Call for Google Summer of Code 2021 project ideas

Call for Google Summer of Code 2021 project ideas

[Stefan Hajnoczi]

Dear QEMU, KVM, and rust-vmm community,
QEMU will apply for Google Summer of Code
(https://summerofcode.withgoogle.com/) again this year.  This internship
program offers paid, 10-week, remote work internships for
contributing to open source.  QEMU can act as an umbrella organization
for KVM kernel and rust-vmm projects too.

Please post project ideas on the QEMU wiki before February 14th:
https://wiki.qemu.org/Google_Summer_of_Code_2021

What's new this year:
 * The number of internship hours has been halved to 175 hours over
   10 weeks. Project ideas must be smaller to fit and students will have
   more flexibility with their working hours.
 * Eligibility has been expanded to include "licensed coding school or
   similar type of program".

Good project ideas are suitable for 175 hours (10 weeks half-day) work by a
competent programmer who is not yet familiar with the codebase.  In
addition, they are:
 * Well-defined - the scope is clear
 * Self-contained - there are few dependencies
 * Uncontroversial - they are acceptable to the community
 * Incremental - they produce deliverables along the way

Feel free to post ideas even if you are unable to mentor the project.
It doesn't hurt to share the idea!

I will review project ideas and keep you up-to-date on QEMU's
acceptance into GSoC.

For more background on QEMU internships, check out this video:
https://www.youtube.com/watch?v=xNVCX7YMUL8

Stefan

Re: Call for Google Summer of Code 2021 project ideas

[John Snow]

On 1/11/21 6:47 AM, Stefan Hajnoczi wrote:
> Dear QEMU, KVM, and rust-vmm community,
> QEMU will apply for Google Summer of Code
> (https://summerofcode.withgoogle.com/) again this year.  This internship
> program offers paid, 10-week, remote work internships for
> contributing to open source.  QEMU can act as an umbrella organization
> for KVM kernel and rust-vmm projects too.
> 
> Please post project ideas on the QEMU wiki before February 14th:
> https://wiki.qemu.org/Google_Summer_of_Code_2021
> 
> What's new this year:
>   * The number of internship hours has been halved to 175 hours over
>     10 weeks. Project ideas must be smaller to fit and students will have
>     more flexibility with their working hours.
>   * Eligibility has been expanded to include "licensed coding school or
>     similar type of program".
> 
> Good project ideas are suitable for 175 hours (10 weeks half-day) work by a
> competent programmer who is not yet familiar with the codebase.  In
> addition, they are:
>   * Well-defined - the scope is clear
>   * Self-contained - there are few dependencies
>   * Uncontroversial - they are acceptable to the community
>   * Incremental - they produce deliverables along the way
> 
> Feel free to post ideas even if you are unable to mentor the project.
> It doesn't hurt to share the idea!
> 

I have one that is probably way too ambitious, but requires a particular 
skillset that might be of good interest to a student that has some 
experience in the area already.

The idea is for a TUI qmp-shell (maybe using urwid?) to create an 
irssi-like REPL interface for QMP. The idea would be to mimic the 
mitmproxy TUI interface (Check it out if you haven't!)

All the ideas below are extremely tentative to give a basic gist of what 
I mean; exact layouts/hotkeys/etc are for the sake of explanation only.

Essentially, I want an interface like this:

-----------------------------------------------------------
| QMP Mode                                                |
|=========================================================|
|                                                         |
| Welcome to the Qemu Machine Protocol shell.             |
| Please type /help or Ctrl+H to see available commands.  |
|                                                         |
|                                                         |
|                                                         |
|---------------------------------------------------------|
| >                                                       |
-----------------------------------------------------------

commands are entered in the bottom and appear in a log window above, 
appearing most-recent last, like irssi works.

As an example, let's say we issue block-dirty-bitmap-add:

--------------------------------------------------------
| > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 |
--------------------------------------------------------

(...syntax up for debate...! We want something easy to parse, but easy 
to type. For structured data, that's Hard.)

we press [Enter] to submit the command, and the history now shows:

-------------------------------------------------
| QMP Mode                                      |
|===============================================|
|                                               |
|                                               |
|                                               |
| [11:28] block-dirty-bitmap-add [Pending ...]  |
|-----------------------------------------------|
| >                                             |
-------------------------------------------------

After a few moments, the command resolves:

----------------------------------------
| QMP Mode                             |
|======================================|
|                                      |
|                                      |
|                                      |
| [11:28] block-dirty-bitmap-add [OK]  |
----------------------------------------
| >                                    |
----------------------------------------

The basic unit of history here is either an RPC call/response pair or an 
asynchronous QMP event.

Clicking on the history pane or pressing Alt+↑ or Alt+↓ (Mimics irssi 
keys for switching buffers, let's say it's user-configurable) to change 
focus to the history pane:

------------------------------------------------------------
| Command History Mode                                     |
|==========================================================|
|                                                          |
|                                                          |
|                                                          |
| [11:28] block-dirty-bitmap-add [OK]                      |
-----------------------------------------------------------|
| Use ↑ or ↓ to select a command, press [Enter] to inspect |
------------------------------------------------------------

Using the arrow keys, you can highlight the history item and then press 
<Enter> to change the log pane to a details pane for that item, showing:

------------------------------------------------------
| Details: block-dirty-bitmap-add                [X] |
|====================================================|
|                                                    |
| [11:28:01] -> {                                    |
|    "execute": "block-dirty-bitmap-add",            |
|    "arguments": {                                  |
|      "node": "ide0hd0",                            |
|      "name": "mybitmap0"                           |
|    }                                               |
| }                                                  |
|                                                    |
| [11:28:02] <- {                                    |
|   "return": {}                                     |
| }                                                  |
|                                                    |
|----------------------------------------------------|
| Press 'Q' or Backspace to close details pane       |
------------------------------------------------------

 From here, maybe you could do a few things, like press a button to view 
the raw QMP I/O for the purposes of copy-pasting elsewhere, or just 
press Q/H/Backspace to go back to the history view.


Interesting abilities this style of shell can provide:

(1) The ability to render asynchronous events from the server in the log 
window without overwriting the user's command buffer. (In contrast to 
qmp-shell today, which cannot show async events until the user presses 
enter again to prompt a check for them.)

(2) The ability to asynchronously reconnect to servers on connection 
failures.

(3) The ability to collapse command:response pairs into one summarized 
line in the log, which can be expanded for more information later with a 
keypress.

(4) (As a future add-on, beyond current scope) The ability to add a 
status pane widget that might show current running jobs, the current 
machine execution status, etc.



As options, users should be able to:

1. Choose the history display mode live at runtime:
   a) Summarized command history (as shown above)
   b) Summarized JSON history
      (one item per message, no grouping, purely linear,
       shows a one-line summary of the message.)
   c) Full JSON history
      (one item per message, JSON messages are rendered in full.)
   d) Pretty JSON history
      (one item per message, JSON messages are pretty-printed.)

   e.g. Ctrl+V could change the "verbosity" of the messages, and the
   history log will be re-rendered to reflect the new setting.

2. Customize hotkeys for switching between command mode and history 
mode; quitting the details view, etc.

3. toggle on/off automatic command execution logging; or manually 
(Ctrl+S?) save history to file. The format might need some light markup 
to indicate directionality, timestamps, and so on.



Other things that might be of interest, but are beyond the scope of a 
GSoC/Outreachy internship:

1. readline-esque hotkey functionality for the command input bar, 
including command history, autocomplete for command names, etc. If we 
aren't using readline itself anymore, these things will have to be 
implemented somehow. There are libraries that support this: 
https://github.com/rr-/urwid_readline

2. Recording execution macros; subsets of commands you wish to replay 
easily.

3. Raw input mode: allow the user to paste a raw JSON command directly 
into the command bar.

4. Configuring our GTK interface to open up qmp-shell inside one of the 
terminal panes. (Kevin Wolf suggested this or something like it once, 
and I rather like the idea.)

5. "Shell Builtins" -- analogous to the same in bash -- additional 
commands and routines that help perform certain actions that might 
require complex, multi-part steps.

6. Augmented commands -- commands that don't require multi-part phases, 
but ones in which the shell might offer enhanced readouts for. An 
example would be integrating Vladimir's block graph rendering script to 
automatically fire off when block-query is issued. (If the user has a 
graphical session running, the libraries are installed, etc.)



See alsos; prior art, &c:

- irssi as an example of continuous, asynchronous history:

https://irssi.org/

- mitmproxy, which heavily inspired this idea with its use of urwid to 
show a history of "packets" which can be interrogated later from within 
the interface

https://mitmproxy.org/

- aioconsole -- an async python REPL loop for interactively writing 
async code in python. It might have good ideas to steal!

https://pypi.org/project/aioconsole/

- urwid: a TUI framework for Python; it is used by mitmproxy to 
implement its interface.

http://urwid.org/

- urwid readline library: implements readline-esque hotkeys for urwid, 
but (maybe) doesn't respect readline config, if any. It is a 
reimplementation.

https://github.com/rr-/urwid_readline





Target skillset for students:
- Intermediate Python
- Some exposure to async programming, coroutines, etc.
- Some experience with GUI programming concepts, in any toolkit or language
- A willingness to use gradually typed python O:-)

I don't expect this project to involve much, if any, C; but familiarity 
with the linux shell ecosystem will be an asset when choosing default 
behaviors, mimicking other tools, etc.

I have already written a fully asyncio QMP library; I would expect this 
new library to be used. It should be fully operational, but might 
require API changes when we begin to use it for a serious endeavor.

> I will review project ideas and keep you up-to-date on QEMU's
> acceptance into GSoC.
> 
> For more background on QEMU internships, check out this video:
> https://www.youtube.com/watch?v=xNVCX7YMUL8
> 
> Stefan
> 

Re: Call for Google Summer of Code 2021 project ideas

[Stefan Hajnoczi]

On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
> I have one that is probably way too ambitious, but requires a particular
> skillset that might be of good interest to a student that has some
> experience in the area already.
>
> The idea is for a TUI qmp-shell (maybe using urwid?) to create an
> irssi-like REPL interface for QMP. The idea would be to mimic the
> mitmproxy TUI interface (Check it out if you haven't!)

Great, I think this project idea lends itself to an incremental
milestones. How far it gets will depend on the intern and we'll be
able to merge useful patches regardless of how far they take it.

Two more ideas:
1. Ability to load libvirt log files for offline viewing. This could
be a major use case for this tool because the raw libvirt logs can be
hard to read today.
2. Ability to watch QMP activity on a running QEMU process, e.g. even
when libvirt is directly connected to the monitor.

Stefan

Re: Call for Google Summer of Code 2021 project ideas

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote:
>> Dear QEMU, KVM, and rust-vmm community,
>> QEMU will apply for Google Summer of Code
>> (https://summerofcode.withgoogle.com/) again this year.  This internship
>> program offers paid, 10-week, remote work internships for
>> contributing to open source.  QEMU can act as an umbrella organization
>> for KVM kernel and rust-vmm projects too.
>> Please post project ideas on the QEMU wiki before February 14th:
>> https://wiki.qemu.org/Google_Summer_of_Code_2021
>> What's new this year:
>>   * The number of internship hours has been halved to 175 hours over
>>     10 weeks. Project ideas must be smaller to fit and students will have
>>     more flexibility with their working hours.
>>   * Eligibility has been expanded to include "licensed coding school or
>>     similar type of program".
>> Good project ideas are suitable for 175 hours (10 weeks half-day)
>> work by a
>> competent programmer who is not yet familiar with the codebase.  In
>> addition, they are:
>>   * Well-defined - the scope is clear
>>   * Self-contained - there are few dependencies
>>   * Uncontroversial - they are acceptable to the community
>>   * Incremental - they produce deliverables along the way
>> Feel free to post ideas even if you are unable to mentor the
>> project.
>> It doesn't hurt to share the idea!
>> 
>
> I have one that is probably way too ambitious, but requires a
> particular skillset that might be of good interest to a student that
> has some experience in the area already.
>
> The idea is for a TUI qmp-shell (maybe using urwid?) to create an
> irssi-like REPL interface for QMP. The idea would be to mimic the 
> mitmproxy TUI interface (Check it out if you haven't!)
>
> All the ideas below are extremely tentative to give a basic gist of
> what I mean; exact layouts/hotkeys/etc are for the sake of explanation
> only.
>
> Essentially, I want an interface like this:
>
> -----------------------------------------------------------
> | QMP Mode                                                |
> |=========================================================|
> |                                                         |
> | Welcome to the Qemu Machine Protocol shell.             |
> | Please type /help or Ctrl+H to see available commands.  |
> |                                                         |
> |                                                         |
> |                                                         |
> |---------------------------------------------------------|
> | >                                                       |
> -----------------------------------------------------------
>
> commands are entered in the bottom and appear in a log window above,
> appearing most-recent last, like irssi works.
>
> As an example, let's say we issue block-dirty-bitmap-add:
>
> --------------------------------------------------------
> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 |
> --------------------------------------------------------
>
> (...syntax up for debate...! We want something easy to parse, but easy
> to type. For structured data, that's Hard.)

"Hard" is a red flag for such a brief internship.  We need to solve or
somehow exclude this design problem before we hand the task to the
student.

[...]

qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[John Snow]

On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
> On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
>> I have one that is probably way too ambitious, but requires a particular
>> skillset that might be of good interest to a student that has some
>> experience in the area already.
>>
>> The idea is for a TUI qmp-shell (maybe using urwid?) to create an
>> irssi-like REPL interface for QMP. The idea would be to mimic the
>> mitmproxy TUI interface (Check it out if you haven't!)
> 
> Great, I think this project idea lends itself to an incremental
> milestones. How far it gets will depend on the intern and we'll be
> able to merge useful patches regardless of how far they take it.
> 

Yeah. I wrote a lot, but I think a lot of the desires and goals can 
actually be split out.

You can start with just the REPL mode; no bells or whistles. Just type 
commands, issue them, and have the log pane populate with the raw JSON 
as a starting point.

That'd be useful enough already. From there, the bells and whistles that 
make it truly shine can be added.

> Two more ideas:
> 1. Ability to load libvirt log files for offline viewing. This could
> be a major use case for this tool because the raw libvirt logs can be
> hard to read today.

Yeah, that would be excellent. (Especially because I have such a hard 
time understanding libvirt-ese; seeing the QMP log in the same tool I 
use to communicate directly with QEMU would be an excellent debugging boon.)

mitmproxy has a similar feature where packet captures can be saved to 
file and loaded again later for analysis. A similar thing here would be 
nice.

I mentioned wanting to be able to save sessions for later viewing in my 
proposal; which likely means developing a QMP log format.

It's unlikely we'll want to use the libvirt log format here, so we'd 
need to develop a reader that can parse libvirt logs; but a writer is 
not likely important.

> 2. Ability to watch QMP activity on a running QEMU process, e.g. even
> when libvirt is directly connected to the monitor.
> 

That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy 
works.

(Actually, mitmproxy could theoretically be taught how to read and 
understand QMP traffic, but that's not something I know how to do or 
would be prepared to mentor.)

Is this possible to do in a post-hoc fashion? Let's say you are using 
production environment QEMU, how do we attach the QMP listener to it? Or 
does this idea require that we start QEMU in a specific fashion with a 
second debug socket that qmp-shell can connect to in order to listen?

... Or do we engineer qmp-shell to open its own socket that libvirt 
connects to ...?

> Stefan
> 

Re: Call for Google Summer of Code 2021 project ideas

[John Snow]

On 1/13/21 4:19 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote:
>>> Dear QEMU, KVM, and rust-vmm community,
>>> QEMU will apply for Google Summer of Code
>>> (https://summerofcode.withgoogle.com/) again this year.  This internship
>>> program offers paid, 10-week, remote work internships for
>>> contributing to open source.  QEMU can act as an umbrella organization
>>> for KVM kernel and rust-vmm projects too.
>>> Please post project ideas on the QEMU wiki before February 14th:
>>> https://wiki.qemu.org/Google_Summer_of_Code_2021
>>> What's new this year:
>>>    * The number of internship hours has been halved to 175 hours over
>>>      10 weeks. Project ideas must be smaller to fit and students will have
>>>      more flexibility with their working hours.
>>>    * Eligibility has been expanded to include "licensed coding school or
>>>      similar type of program".
>>> Good project ideas are suitable for 175 hours (10 weeks half-day)
>>> work by a
>>> competent programmer who is not yet familiar with the codebase.  In
>>> addition, they are:
>>>    * Well-defined - the scope is clear
>>>    * Self-contained - there are few dependencies
>>>    * Uncontroversial - they are acceptable to the community
>>>    * Incremental - they produce deliverables along the way
>>> Feel free to post ideas even if you are unable to mentor the
>>> project.
>>> It doesn't hurt to share the idea!
>>>
>>
>> I have one that is probably way too ambitious, but requires a
>> particular skillset that might be of good interest to a student that
>> has some experience in the area already.
>>
>> The idea is for a TUI qmp-shell (maybe using urwid?) to create an
>> irssi-like REPL interface for QMP. The idea would be to mimic the
>> mitmproxy TUI interface (Check it out if you haven't!)
>>
>> All the ideas below are extremely tentative to give a basic gist of
>> what I mean; exact layouts/hotkeys/etc are for the sake of explanation
>> only.
>>
>> Essentially, I want an interface like this:
>>
>> -----------------------------------------------------------
>> | QMP Mode                                                |
>> |=========================================================|
>> |                                                         |
>> | Welcome to the Qemu Machine Protocol shell.             |
>> | Please type /help or Ctrl+H to see available commands.  |
>> |                                                         |
>> |                                                         |
>> |                                                         |
>> |---------------------------------------------------------|
>> | >                                                       |
>> -----------------------------------------------------------
>>
>> commands are entered in the bottom and appear in a log window above,
>> appearing most-recent last, like irssi works.
>>
>> As an example, let's say we issue block-dirty-bitmap-add:
>>
>> --------------------------------------------------------
>> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 |
>> --------------------------------------------------------
>>
>> (...syntax up for debate...! We want something easy to parse, but easy
>> to type. For structured data, that's Hard.)
> 
> "Hard" is a red flag for such a brief internship.  We need to solve or
> somehow exclude this design problem before we hand the task to the
> student.
> 
> [...]
> 

Yes, there is a reason I capitalized "Hard". :~)

This task CAN be split out, though. For starters, we can use an 
extremely simplified parsing strategy while we work on the REPL and and 
UI elements.

We can accept things of the form:
command arg=value arg2=value

and we can worry about how to input structured data later. I don't think 
it's a blocker to have someone work on the TUI and asynchronous dispatch 
elements. I think even just keeping our current parsing but adding some 
of the features outlined in the proposal would be a big usability win.

--js

Re: Call for Google Summer of Code 2021 project ideas

[Markus Armbruster]

John Snow <jsnow@redhat.com> writes:

> On 1/13/21 4:19 AM, Markus Armbruster wrote:
>> John Snow <jsnow@redhat.com> writes:
>> 
>>> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote:
>>>> Dear QEMU, KVM, and rust-vmm community,
>>>> QEMU will apply for Google Summer of Code
>>>> (https://summerofcode.withgoogle.com/) again this year.  This internship
>>>> program offers paid, 10-week, remote work internships for
>>>> contributing to open source.  QEMU can act as an umbrella organization
>>>> for KVM kernel and rust-vmm projects too.
>>>> Please post project ideas on the QEMU wiki before February 14th:
>>>> https://wiki.qemu.org/Google_Summer_of_Code_2021
>>>> What's new this year:
>>>>    * The number of internship hours has been halved to 175 hours over
>>>>      10 weeks. Project ideas must be smaller to fit and students will have
>>>>      more flexibility with their working hours.
>>>>    * Eligibility has been expanded to include "licensed coding school or
>>>>      similar type of program".
>>>> Good project ideas are suitable for 175 hours (10 weeks half-day)
>>>> work by a
>>>> competent programmer who is not yet familiar with the codebase.  In
>>>> addition, they are:
>>>>    * Well-defined - the scope is clear
>>>>    * Self-contained - there are few dependencies
>>>>    * Uncontroversial - they are acceptable to the community
>>>>    * Incremental - they produce deliverables along the way
>>>> Feel free to post ideas even if you are unable to mentor the
>>>> project.
>>>> It doesn't hurt to share the idea!
>>>>
>>>
>>> I have one that is probably way too ambitious, but requires a
>>> particular skillset that might be of good interest to a student that
>>> has some experience in the area already.
>>>
>>> The idea is for a TUI qmp-shell (maybe using urwid?) to create an
>>> irssi-like REPL interface for QMP. The idea would be to mimic the
>>> mitmproxy TUI interface (Check it out if you haven't!)
>>>
>>> All the ideas below are extremely tentative to give a basic gist of
>>> what I mean; exact layouts/hotkeys/etc are for the sake of explanation
>>> only.
>>>
>>> Essentially, I want an interface like this:
>>>
>>> -----------------------------------------------------------
>>> | QMP Mode                                                |
>>> |=========================================================|
>>> |                                                         |
>>> | Welcome to the Qemu Machine Protocol shell.             |
>>> | Please type /help or Ctrl+H to see available commands.  |
>>> |                                                         |
>>> |                                                         |
>>> |                                                         |
>>> |---------------------------------------------------------|
>>> | >                                                       |
>>> -----------------------------------------------------------
>>>
>>> commands are entered in the bottom and appear in a log window above,
>>> appearing most-recent last, like irssi works.
>>>
>>> As an example, let's say we issue block-dirty-bitmap-add:
>>>
>>> --------------------------------------------------------
>>> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 |
>>> --------------------------------------------------------
>>>
>>> (...syntax up for debate...! We want something easy to parse, but easy
>>> to type. For structured data, that's Hard.)
>> "Hard" is a red flag for such a brief internship.  We need to solve
>> or
>> somehow exclude this design problem before we hand the task to the
>> student.
>> [...]
>> 
>
> Yes, there is a reason I capitalized "Hard". :~)
>
> This task CAN be split out, though. For starters, we can use an
> extremely simplified parsing strategy while we work on the REPL and
> and UI elements.
>
> We can accept things of the form:
> command arg=value arg2=value
>
> and we can worry about how to input structured data later.

Bzzzt!  You just designed a language.  The fact that it is obviously
inadequate for the task does not make this not language design.

Two outcomes:

1. The project fails for other reasons.  The language doesn't matter.

2. The project succeeds, i.e. it becomes useful to people.  Very quickly
these people will demand you evolve your inadequate language into a less
inadequate one, compatibly.  Your evasion of the language design problem
(because hard) has now added accidental constraints, making the design
problem even harder.

The sane way to evade the language design problem is to use the existing
QMP language.

The project then aims to build a tool that adds useful features over
"socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>"
UNIX-CONNECT:/path/to/socket".

If it succeeds, you can still design and implement a "better" language,
and let users choose the one they prefer.  Or you could add features to
help with typing QMP.

>                                                            I don't
> think it's a blocker to have someone work on the TUI and asynchronous
> dispatch elements. I think even just keeping our current parsing but
> adding some of the features outlined in the proposal would be a big
> usability win.

I don't feel this particular itch, but I'm certainly not objecting to
anyone scratching.

Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 3018 bytes --]

On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
> On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
> > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
> > 2. Ability to watch QMP activity on a running QEMU process, e.g. even
> > when libvirt is directly connected to the monitor.
> > 
> 
> That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
> works.
> 
> (Actually, mitmproxy could theoretically be taught how to read and
> understand QMP traffic, but that's not something I know how to do or would
> be prepared to mentor.)
> 
> Is this possible to do in a post-hoc fashion? Let's say you are using
> production environment QEMU, how do we attach the QMP listener to it? Or
> does this idea require that we start QEMU in a specific fashion with a
> second debug socket that qmp-shell can connect to in order to listen?
> 
> ... Or do we engineer qmp-shell to open its own socket that libvirt connects
> to ...?

Here is the QEMU command-line that libvirt uses on my F33 system:

  -chardev socket,id=charmonitor,fd=36,server,nowait
  -mon chardev=charmonitor,id=monitor,mode=control

Goals for this feature:

1. No manual steps required for setup.
2. Ability to start/stop monitoring traffic at runtime without
   restarting QEMU.
3. Available to unprivileged users.

I think the easiest way to achieve this is through a new QEMU monitor
command. Approaches that come to mind:

1. Add a -mon debug-chardev property and a QMP command to set it at
   runtime. The debug-chardev receives both monitor input (commands) and
   output (responses and events). This does not allow MITM, rather it
   mirrors traffic.

2. Add a chardev-get-fd command that fetches the fd from a chardev and
   then use the existing chardev-change command to replace the monitor
   chardev with a chardev connected to qmp-shell. This inserts qmp-shell
   as a proxy between the QMP client and server. qmp-shell can remove
   itself again with another chardev-change command. This approach
   allows MITM. The downside is it assumes the QMP chardev is a file
   descriptor, so it won't work with all types of chardev.

3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
   source chardev, 2. a monitoring sink chardev, and 3. a monitoring
   source chardev. The data flow is origin <-> monitoring sink <->
   monitoring source <-> QMP monitor. qmp-shell creates the monitoring
   sink (for receiving incoming QMP commands) and monitoring source
   chardev (for forwarding QMP commands or MITM commands), and then it
   uses change-chardev to instantiate a chardev-proxy that directs the
   original libvirt chardev through the monitoring sink and source.

   This is the most complex but also completely contained within the
   QEMU chardev layer.

In all these approaches qmp-shell uses virsh qemu-monitor-command or an
equivalent API to start/stop monitoring a running VM without manual
setup steps.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[Daniel P. Berrangé]

On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
> On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
> > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
> > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
> > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even
> > > when libvirt is directly connected to the monitor.
> > > 
> > 
> > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
> > works.
> > 
> > (Actually, mitmproxy could theoretically be taught how to read and
> > understand QMP traffic, but that's not something I know how to do or would
> > be prepared to mentor.)
> > 
> > Is this possible to do in a post-hoc fashion? Let's say you are using
> > production environment QEMU, how do we attach the QMP listener to it? Or
> > does this idea require that we start QEMU in a specific fashion with a
> > second debug socket that qmp-shell can connect to in order to listen?
> > 
> > ... Or do we engineer qmp-shell to open its own socket that libvirt connects
> > to ...?
> 
> Here is the QEMU command-line that libvirt uses on my F33 system:
> 
>   -chardev socket,id=charmonitor,fd=36,server,nowait
>   -mon chardev=charmonitor,id=monitor,mode=control
> 
> Goals for this feature:
> 
> 1. No manual steps required for setup.
> 2. Ability to start/stop monitoring traffic at runtime without
>    restarting QEMU.
> 3. Available to unprivileged users.
> 
> I think the easiest way to achieve this is through a new QEMU monitor
> command. Approaches that come to mind:
> 
> 1. Add a -mon debug-chardev property and a QMP command to set it at
>    runtime. The debug-chardev receives both monitor input (commands) and
>    output (responses and events). This does not allow MITM, rather it
>    mirrors traffic.
> 
> 2. Add a chardev-get-fd command that fetches the fd from a chardev and
>    then use the existing chardev-change command to replace the monitor
>    chardev with a chardev connected to qmp-shell. This inserts qmp-shell
>    as a proxy between the QMP client and server. qmp-shell can remove
>    itself again with another chardev-change command. This approach
>    allows MITM. The downside is it assumes the QMP chardev is a file
>    descriptor, so it won't work with all types of chardev.
> 
> 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
>    source chardev, 2. a monitoring sink chardev, and 3. a monitoring
>    source chardev. The data flow is origin <-> monitoring sink <->
>    monitoring source <-> QMP monitor. qmp-shell creates the monitoring
>    sink (for receiving incoming QMP commands) and monitoring source
>    chardev (for forwarding QMP commands or MITM commands), and then it
>    uses change-chardev to instantiate a chardev-proxy that directs the
>    original libvirt chardev through the monitoring sink and source.
> 
>    This is the most complex but also completely contained within the
>    QEMU chardev layer.
> 
> In all these approaches qmp-shell uses virsh qemu-monitor-command or an
> equivalent API to start/stop monitoring a running VM without manual
> setup steps.

Why go to the trouble of adding more chardevs to a running QEMU that
libvirt has.  qmp-shell can just directly use the libvirt Python API
to invoke  virDomainQemuMonitorCommand to invoke QMP commands, and
the othe API for receiving QMP events.

Essentially it just needs to be split into two layers. The upper
layer works in terms of individual QMP command/replies, and QMP
events. The lower layer provides a transport that is either a
UNIX socket, or is the libvirt QMP passthrough API.

Or alternatively, provide a virt-qmp-shim  command that listens on
a UNIX socket, accepts QMP commands and turns them into calls to
virDomainQemuMonitorCommand, and funnells back the response.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Call for Google Summer of Code 2021 project ideas

[Kevin Wolf]

Am 14.01.2021 um 13:29 hat Markus Armbruster geschrieben:
> John Snow <jsnow@redhat.com> writes:
> 
> > On 1/13/21 4:19 AM, Markus Armbruster wrote:
> >> John Snow <jsnow@redhat.com> writes:
> >> 
> >>> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote:
> >>>> Dear QEMU, KVM, and rust-vmm community,
> >>>> QEMU will apply for Google Summer of Code
> >>>> (https://summerofcode.withgoogle.com/) again this year.  This internship
> >>>> program offers paid, 10-week, remote work internships for
> >>>> contributing to open source.  QEMU can act as an umbrella organization
> >>>> for KVM kernel and rust-vmm projects too.
> >>>> Please post project ideas on the QEMU wiki before February 14th:
> >>>> https://wiki.qemu.org/Google_Summer_of_Code_2021
> >>>> What's new this year:
> >>>>    * The number of internship hours has been halved to 175 hours over
> >>>>      10 weeks. Project ideas must be smaller to fit and students will have
> >>>>      more flexibility with their working hours.
> >>>>    * Eligibility has been expanded to include "licensed coding school or
> >>>>      similar type of program".
> >>>> Good project ideas are suitable for 175 hours (10 weeks half-day)
> >>>> work by a
> >>>> competent programmer who is not yet familiar with the codebase.  In
> >>>> addition, they are:
> >>>>    * Well-defined - the scope is clear
> >>>>    * Self-contained - there are few dependencies
> >>>>    * Uncontroversial - they are acceptable to the community
> >>>>    * Incremental - they produce deliverables along the way
> >>>> Feel free to post ideas even if you are unable to mentor the
> >>>> project.
> >>>> It doesn't hurt to share the idea!
> >>>>
> >>>
> >>> I have one that is probably way too ambitious, but requires a
> >>> particular skillset that might be of good interest to a student that
> >>> has some experience in the area already.
> >>>
> >>> The idea is for a TUI qmp-shell (maybe using urwid?) to create an
> >>> irssi-like REPL interface for QMP. The idea would be to mimic the
> >>> mitmproxy TUI interface (Check it out if you haven't!)
> >>>
> >>> All the ideas below are extremely tentative to give a basic gist of
> >>> what I mean; exact layouts/hotkeys/etc are for the sake of explanation
> >>> only.
> >>>
> >>> Essentially, I want an interface like this:
> >>>
> >>> -----------------------------------------------------------
> >>> | QMP Mode                                                |
> >>> |=========================================================|
> >>> |                                                         |
> >>> | Welcome to the Qemu Machine Protocol shell.             |
> >>> | Please type /help or Ctrl+H to see available commands.  |
> >>> |                                                         |
> >>> |                                                         |
> >>> |                                                         |
> >>> |---------------------------------------------------------|
> >>> | >                                                       |
> >>> -----------------------------------------------------------
> >>>
> >>> commands are entered in the bottom and appear in a log window above,
> >>> appearing most-recent last, like irssi works.
> >>>
> >>> As an example, let's say we issue block-dirty-bitmap-add:
> >>>
> >>> --------------------------------------------------------
> >>> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 |
> >>> --------------------------------------------------------
> >>>
> >>> (...syntax up for debate...! We want something easy to parse, but easy
> >>> to type. For structured data, that's Hard.)
> >> "Hard" is a red flag for such a brief internship.  We need to solve
> >> or
> >> somehow exclude this design problem before we hand the task to the
> >> student.
> >> [...]
> >> 
> >
> > Yes, there is a reason I capitalized "Hard". :~)
> >
> > This task CAN be split out, though. For starters, we can use an
> > extremely simplified parsing strategy while we work on the REPL and
> > and UI elements.
> >
> > We can accept things of the form:
> > command arg=value arg2=value
> >
> > and we can worry about how to input structured data later.
> 
> Bzzzt!  You just designed a language.  The fact that it is obviously
> inadequate for the task does not make this not language design.
> 
> Two outcomes:
> 
> 1. The project fails for other reasons.  The language doesn't matter.
> 
> 2. The project succeeds, i.e. it becomes useful to people.  Very quickly
> these people will demand you evolve your inadequate language into a less
> inadequate one, compatibly.  Your evasion of the language design problem
> (because hard) has now added accidental constraints, making the design
> problem even harder.

We don't even promise compatibility for HMP, which is a part of the
system emulator binary and has existed since I first used QEMU, which
must have been around 0.6 times. If it's possible to simply say "no" to
compatibility demands there, then surely an external shell with a
similar target audience can do the same.

I think it would be useful to have a raw QMP mode (especially so you can
point people to that when they ask for compatibility), but it's probably
not the focus for this project because you can already have raw QMP
today.

Kevin

Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[Kevin Wolf]

Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
> On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
> > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
> > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
> > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
> > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even
> > > > when libvirt is directly connected to the monitor.
> > > > 
> > > 
> > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
> > > works.
> > > 
> > > (Actually, mitmproxy could theoretically be taught how to read and
> > > understand QMP traffic, but that's not something I know how to do or would
> > > be prepared to mentor.)
> > > 
> > > Is this possible to do in a post-hoc fashion? Let's say you are using
> > > production environment QEMU, how do we attach the QMP listener to it? Or
> > > does this idea require that we start QEMU in a specific fashion with a
> > > second debug socket that qmp-shell can connect to in order to listen?
> > > 
> > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects
> > > to ...?
> > 
> > Here is the QEMU command-line that libvirt uses on my F33 system:
> > 
> >   -chardev socket,id=charmonitor,fd=36,server,nowait
> >   -mon chardev=charmonitor,id=monitor,mode=control
> > 
> > Goals for this feature:
> > 
> > 1. No manual steps required for setup.
> > 2. Ability to start/stop monitoring traffic at runtime without
> >    restarting QEMU.
> > 3. Available to unprivileged users.
> > 
> > I think the easiest way to achieve this is through a new QEMU monitor
> > command. Approaches that come to mind:
> > 
> > 1. Add a -mon debug-chardev property and a QMP command to set it at
> >    runtime. The debug-chardev receives both monitor input (commands) and
> >    output (responses and events). This does not allow MITM, rather it
> >    mirrors traffic.
> > 
> > 2. Add a chardev-get-fd command that fetches the fd from a chardev and
> >    then use the existing chardev-change command to replace the monitor
> >    chardev with a chardev connected to qmp-shell. This inserts qmp-shell
> >    as a proxy between the QMP client and server. qmp-shell can remove
> >    itself again with another chardev-change command. This approach
> >    allows MITM. The downside is it assumes the QMP chardev is a file
> >    descriptor, so it won't work with all types of chardev.
> > 
> > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
> >    source chardev, 2. a monitoring sink chardev, and 3. a monitoring
> >    source chardev. The data flow is origin <-> monitoring sink <->
> >    monitoring source <-> QMP monitor. qmp-shell creates the monitoring
> >    sink (for receiving incoming QMP commands) and monitoring source
> >    chardev (for forwarding QMP commands or MITM commands), and then it
> >    uses change-chardev to instantiate a chardev-proxy that directs the
> >    original libvirt chardev through the monitoring sink and source.
> > 
> >    This is the most complex but also completely contained within the
> >    QEMU chardev layer.

I have an idea for the QMP command name: chardev-snapshot-sync!

Finally we get backing file chains for chardevs! :-)

> > In all these approaches qmp-shell uses virsh qemu-monitor-command or an
> > equivalent API to start/stop monitoring a running VM without manual
> > setup steps.
> 
> Why go to the trouble of adding more chardevs to a running QEMU that
> libvirt has.  qmp-shell can just directly use the libvirt Python API
> to invoke  virDomainQemuMonitorCommand to invoke QMP commands, and
> the othe API for receiving QMP events.
> 
> Essentially it just needs to be split into two layers. The upper
> layer works in terms of individual QMP command/replies, and QMP
> events. The lower layer provides a transport that is either a
> UNIX socket, or is the libvirt QMP passthrough API.
> 
> Or alternatively, provide a virt-qmp-shim  command that listens on
> a UNIX socket, accepts QMP commands and turns them into calls to
> virDomainQemuMonitorCommand, and funnells back the response.

I think the idea was to show the QMP traffic that libvirt produces for
other management applications, not for the QMP shell. These APIs
probably don't allow this?

Kevin

Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[Daniel P. Berrangé]

On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
> Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
> > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
> > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
> > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
> > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
> > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even
> > > > > when libvirt is directly connected to the monitor.
> > > > > 
> > > > 
> > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
> > > > works.
> > > > 
> > > > (Actually, mitmproxy could theoretically be taught how to read and
> > > > understand QMP traffic, but that's not something I know how to do or would
> > > > be prepared to mentor.)
> > > > 
> > > > Is this possible to do in a post-hoc fashion? Let's say you are using
> > > > production environment QEMU, how do we attach the QMP listener to it? Or
> > > > does this idea require that we start QEMU in a specific fashion with a
> > > > second debug socket that qmp-shell can connect to in order to listen?
> > > > 
> > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects
> > > > to ...?
> > > 
> > > Here is the QEMU command-line that libvirt uses on my F33 system:
> > > 
> > >   -chardev socket,id=charmonitor,fd=36,server,nowait
> > >   -mon chardev=charmonitor,id=monitor,mode=control
> > > 
> > > Goals for this feature:
> > > 
> > > 1. No manual steps required for setup.
> > > 2. Ability to start/stop monitoring traffic at runtime without
> > >    restarting QEMU.
> > > 3. Available to unprivileged users.
> > > 
> > > I think the easiest way to achieve this is through a new QEMU monitor
> > > command. Approaches that come to mind:
> > > 
> > > 1. Add a -mon debug-chardev property and a QMP command to set it at
> > >    runtime. The debug-chardev receives both monitor input (commands) and
> > >    output (responses and events). This does not allow MITM, rather it
> > >    mirrors traffic.
> > > 
> > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and
> > >    then use the existing chardev-change command to replace the monitor
> > >    chardev with a chardev connected to qmp-shell. This inserts qmp-shell
> > >    as a proxy between the QMP client and server. qmp-shell can remove
> > >    itself again with another chardev-change command. This approach
> > >    allows MITM. The downside is it assumes the QMP chardev is a file
> > >    descriptor, so it won't work with all types of chardev.
> > > 
> > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
> > >    source chardev, 2. a monitoring sink chardev, and 3. a monitoring
> > >    source chardev. The data flow is origin <-> monitoring sink <->
> > >    monitoring source <-> QMP monitor. qmp-shell creates the monitoring
> > >    sink (for receiving incoming QMP commands) and monitoring source
> > >    chardev (for forwarding QMP commands or MITM commands), and then it
> > >    uses change-chardev to instantiate a chardev-proxy that directs the
> > >    original libvirt chardev through the monitoring sink and source.
> > > 
> > >    This is the most complex but also completely contained within the
> > >    QEMU chardev layer.
> 
> I have an idea for the QMP command name: chardev-snapshot-sync!
> 
> Finally we get backing file chains for chardevs! :-)
> 
> > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an
> > > equivalent API to start/stop monitoring a running VM without manual
> > > setup steps.
> > 
> > Why go to the trouble of adding more chardevs to a running QEMU that
> > libvirt has.  qmp-shell can just directly use the libvirt Python API
> > to invoke  virDomainQemuMonitorCommand to invoke QMP commands, and
> > the othe API for receiving QMP events.
> > 
> > Essentially it just needs to be split into two layers. The upper
> > layer works in terms of individual QMP command/replies, and QMP
> > events. The lower layer provides a transport that is either a
> > UNIX socket, or is the libvirt QMP passthrough API.
> > 
> > Or alternatively, provide a virt-qmp-shim  command that listens on
> > a UNIX socket, accepts QMP commands and turns them into calls to
> > virDomainQemuMonitorCommand, and funnells back the response.
> 
> I think the idea was to show the QMP traffic that libvirt produces for
> other management applications, not for the QMP shell. These APIs
> probably don't allow this?

FWIW if you want to monitor what libvirt is sending/receiving we have
a script for that that uses our systemtap probe points:

  https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: Call for Google Summer of Code 2021 project ideas

[John Snow]

On 1/14/21 7:29 AM, Markus Armbruster wrote:
> John Snow <jsnow@redhat.com> writes:
> 
>> On 1/13/21 4:19 AM, Markus Armbruster wrote:
>>> John Snow <jsnow@redhat.com> writes:
>>>
>>>> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote:
>>>>> Dear QEMU, KVM, and rust-vmm community,
>>>>> QEMU will apply for Google Summer of Code
>>>>> (https://summerofcode.withgoogle.com/) again this year.  This internship
>>>>> program offers paid, 10-week, remote work internships for
>>>>> contributing to open source.  QEMU can act as an umbrella organization
>>>>> for KVM kernel and rust-vmm projects too.
>>>>> Please post project ideas on the QEMU wiki before February 14th:
>>>>> https://wiki.qemu.org/Google_Summer_of_Code_2021
>>>>> What's new this year:
>>>>>     * The number of internship hours has been halved to 175 hours over
>>>>>       10 weeks. Project ideas must be smaller to fit and students will have
>>>>>       more flexibility with their working hours.
>>>>>     * Eligibility has been expanded to include "licensed coding school or
>>>>>       similar type of program".
>>>>> Good project ideas are suitable for 175 hours (10 weeks half-day)
>>>>> work by a
>>>>> competent programmer who is not yet familiar with the codebase.  In
>>>>> addition, they are:
>>>>>     * Well-defined - the scope is clear
>>>>>     * Self-contained - there are few dependencies
>>>>>     * Uncontroversial - they are acceptable to the community
>>>>>     * Incremental - they produce deliverables along the way
>>>>> Feel free to post ideas even if you are unable to mentor the
>>>>> project.
>>>>> It doesn't hurt to share the idea!
>>>>>
>>>>
>>>> I have one that is probably way too ambitious, but requires a
>>>> particular skillset that might be of good interest to a student that
>>>> has some experience in the area already.
>>>>
>>>> The idea is for a TUI qmp-shell (maybe using urwid?) to create an
>>>> irssi-like REPL interface for QMP. The idea would be to mimic the
>>>> mitmproxy TUI interface (Check it out if you haven't!)
>>>>
>>>> All the ideas below are extremely tentative to give a basic gist of
>>>> what I mean; exact layouts/hotkeys/etc are for the sake of explanation
>>>> only.
>>>>
>>>> Essentially, I want an interface like this:
>>>>
>>>> -----------------------------------------------------------
>>>> | QMP Mode                                                |
>>>> |=========================================================|
>>>> |                                                         |
>>>> | Welcome to the Qemu Machine Protocol shell.             |
>>>> | Please type /help or Ctrl+H to see available commands.  |
>>>> |                                                         |
>>>> |                                                         |
>>>> |                                                         |
>>>> |---------------------------------------------------------|
>>>> | >                                                       |
>>>> -----------------------------------------------------------
>>>>
>>>> commands are entered in the bottom and appear in a log window above,
>>>> appearing most-recent last, like irssi works.
>>>>
>>>> As an example, let's say we issue block-dirty-bitmap-add:
>>>>
>>>> --------------------------------------------------------
>>>> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 |
>>>> --------------------------------------------------------
>>>>
>>>> (...syntax up for debate...! We want something easy to parse, but easy
>>>> to type. For structured data, that's Hard.)
>>> "Hard" is a red flag for such a brief internship.  We need to solve
>>> or
>>> somehow exclude this design problem before we hand the task to the
>>> student.
>>> [...]
>>>
>>
>> Yes, there is a reason I capitalized "Hard". :~)
>>
>> This task CAN be split out, though. For starters, we can use an
>> extremely simplified parsing strategy while we work on the REPL and
>> and UI elements.
>>
>> We can accept things of the form:
>> command arg=value arg2=value
>>
>> and we can worry about how to input structured data later.
> 
> Bzzzt!  You just designed a language.  The fact that it is obviously
> inadequate for the task does not make this not language design.
> 

Did *I* design a language, or did qmp-shell authors design a language? O:-)

(Oh, I'm one of the authors. Well then.)

> Two outcomes:
> 
> 1. The project fails for other reasons.  The language doesn't matter.
> 
> 2. The project succeeds, i.e. it becomes useful to people.  Very quickly
> these people will demand you evolve your inadequate language into a less
> inadequate one, compatibly.  Your evasion of the language design problem
> (because hard) has now added accidental constraints, making the design
> problem even harder.
> 

There's no reason that an interactive shell has to stay backwards 
compatible. Version 0.1 and 0.2 might offer different input paradigms. 
It's not nice, but it's fine for beta-level stuff.

Still, I appreciate that "Beta! Might change!" is an oft-ignored 
warning. This isn't a library or API though, so I still think it's okay 
to take some liberty here.

> The sane way to evade the language design problem is to use the existing
> QMP language.
> 

I wouldn't mind implementing this for version 0.1 -- just allow 
copy-pasting JSON into the input bar -- it's a feature I wanted anyway.

Some things I would like to see, but suspect are not reasonably possible 
to implement quickly:

1. A shorthand syntax mode similar to qmp-shell that is reasonably 
sufficient for most commands. This involves solving a fairly difficult 
language design problem on how to facilitate the input of structured 
data. The biggest pain point here is nested data; we allow JSON as an 
input here but require that it has no spaces. We do not offer a "native" 
shorthand for the nested data.

We *could* co-opt the dotted key syntax, but that's cumbersome for some 
constructs because it involves a lot of repeated parent keys.

As you say, the language design aspect is hard.


2. a multi-line editor mode that facilitates typing QMP messages that 
offers syntax highlighting, field and type validation, autocomplete, 
etc. I have no idea how to implement this -- it's basically an entire 
bespoke editor.


That's *very* hard stuff, I expect, so I won't go there for starters.

So I see two possible options for "not inventing a language":

1. Raw QMP
2. The existing qmp-shell syntax, warts and all.

I don't see a tremendous problem with doing both; but we can start with 
raw QMP. The existing qmp-shell syntax is at least definitely very easy 
to write a new parser for, even if it's kind of ugly and insufficient. I 
still see value in designing a new TUI with the old syntax.

> The project then aims to build a tool that adds useful features over
> "socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>"
> UNIX-CONNECT:/path/to/socket".
> 
> If it succeeds, you can still design and implement a "better" language,
> and let users choose the one they prefer.  Or you could add features to
> help with typing QMP.
> 
>>                                                             I don't
>> think it's a blocker to have someone work on the TUI and asynchronous
>> dispatch elements. I think even just keeping our current parsing but
>> adding some of the features outlined in the proposal would be a big
>> usability win.
> 
> I don't feel this particular itch, but I'm certainly not objecting to
> anyone scratching.
> 

It's something I'd like to see so that I can walk non-QEMU devs through 
interacting with QEMU at a low level for the purposes of debugging, 
reproducing problems, prototyping features, etc.

I use qmp-shell all the time for debugging things myself, I find it 
easier to use than copy-pasting things directly into socat. I wouldn't 
mind the shell getting a little smarter to help me out -- the ability to 
see async events and reconnect on disconnect would already be a massive 
improvement to *my* quality of life.

So much so that I spent a lot of time in December to write an async qmp 
library O:-)

--js

Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 4716 bytes --]

On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
> Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
> > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
> > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
> > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
> > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
> > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even
> > > > > when libvirt is directly connected to the monitor.
> > > > > 
> > > > 
> > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
> > > > works.
> > > > 
> > > > (Actually, mitmproxy could theoretically be taught how to read and
> > > > understand QMP traffic, but that's not something I know how to do or would
> > > > be prepared to mentor.)
> > > > 
> > > > Is this possible to do in a post-hoc fashion? Let's say you are using
> > > > production environment QEMU, how do we attach the QMP listener to it? Or
> > > > does this idea require that we start QEMU in a specific fashion with a
> > > > second debug socket that qmp-shell can connect to in order to listen?
> > > > 
> > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects
> > > > to ...?
> > > 
> > > Here is the QEMU command-line that libvirt uses on my F33 system:
> > > 
> > >   -chardev socket,id=charmonitor,fd=36,server,nowait
> > >   -mon chardev=charmonitor,id=monitor,mode=control
> > > 
> > > Goals for this feature:
> > > 
> > > 1. No manual steps required for setup.
> > > 2. Ability to start/stop monitoring traffic at runtime without
> > >    restarting QEMU.
> > > 3. Available to unprivileged users.
> > > 
> > > I think the easiest way to achieve this is through a new QEMU monitor
> > > command. Approaches that come to mind:
> > > 
> > > 1. Add a -mon debug-chardev property and a QMP command to set it at
> > >    runtime. The debug-chardev receives both monitor input (commands) and
> > >    output (responses and events). This does not allow MITM, rather it
> > >    mirrors traffic.
> > > 
> > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and
> > >    then use the existing chardev-change command to replace the monitor
> > >    chardev with a chardev connected to qmp-shell. This inserts qmp-shell
> > >    as a proxy between the QMP client and server. qmp-shell can remove
> > >    itself again with another chardev-change command. This approach
> > >    allows MITM. The downside is it assumes the QMP chardev is a file
> > >    descriptor, so it won't work with all types of chardev.
> > > 
> > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
> > >    source chardev, 2. a monitoring sink chardev, and 3. a monitoring
> > >    source chardev. The data flow is origin <-> monitoring sink <->
> > >    monitoring source <-> QMP monitor. qmp-shell creates the monitoring
> > >    sink (for receiving incoming QMP commands) and monitoring source
> > >    chardev (for forwarding QMP commands or MITM commands), and then it
> > >    uses change-chardev to instantiate a chardev-proxy that directs the
> > >    original libvirt chardev through the monitoring sink and source.
> > > 
> > >    This is the most complex but also completely contained within the
> > >    QEMU chardev layer.
> 
> I have an idea for the QMP command name: chardev-snapshot-sync!
> 
> Finally we get backing file chains for chardevs! :-)

Help mom, I'm being made fun of on qemu-devel! :)

> > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an
> > > equivalent API to start/stop monitoring a running VM without manual
> > > setup steps.
> > 
> > Why go to the trouble of adding more chardevs to a running QEMU that
> > libvirt has.  qmp-shell can just directly use the libvirt Python API
> > to invoke  virDomainQemuMonitorCommand to invoke QMP commands, and
> > the othe API for receiving QMP events.
> > 
> > Essentially it just needs to be split into two layers. The upper
> > layer works in terms of individual QMP command/replies, and QMP
> > events. The lower layer provides a transport that is either a
> > UNIX socket, or is the libvirt QMP passthrough API.
> > 
> > Or alternatively, provide a virt-qmp-shim  command that listens on
> > a UNIX socket, accepts QMP commands and turns them into calls to
> > virDomainQemuMonitorCommand, and funnells back the response.
> 
> I think the idea was to show the QMP traffic that libvirt produces for
> other management applications, not for the QMP shell. These APIs
> probably don't allow this?

Yes.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 5163 bytes --]

On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote:
> On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
> > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
> > > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
> > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
> > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
> > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
> > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even
> > > > > > when libvirt is directly connected to the monitor.
> > > > > > 
> > > > > 
> > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
> > > > > works.
> > > > > 
> > > > > (Actually, mitmproxy could theoretically be taught how to read and
> > > > > understand QMP traffic, but that's not something I know how to do or would
> > > > > be prepared to mentor.)
> > > > > 
> > > > > Is this possible to do in a post-hoc fashion? Let's say you are using
> > > > > production environment QEMU, how do we attach the QMP listener to it? Or
> > > > > does this idea require that we start QEMU in a specific fashion with a
> > > > > second debug socket that qmp-shell can connect to in order to listen?
> > > > > 
> > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects
> > > > > to ...?
> > > > 
> > > > Here is the QEMU command-line that libvirt uses on my F33 system:
> > > > 
> > > >   -chardev socket,id=charmonitor,fd=36,server,nowait
> > > >   -mon chardev=charmonitor,id=monitor,mode=control
> > > > 
> > > > Goals for this feature:
> > > > 
> > > > 1. No manual steps required for setup.
> > > > 2. Ability to start/stop monitoring traffic at runtime without
> > > >    restarting QEMU.
> > > > 3. Available to unprivileged users.
> > > > 
> > > > I think the easiest way to achieve this is through a new QEMU monitor
> > > > command. Approaches that come to mind:
> > > > 
> > > > 1. Add a -mon debug-chardev property and a QMP command to set it at
> > > >    runtime. The debug-chardev receives both monitor input (commands) and
> > > >    output (responses and events). This does not allow MITM, rather it
> > > >    mirrors traffic.
> > > > 
> > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and
> > > >    then use the existing chardev-change command to replace the monitor
> > > >    chardev with a chardev connected to qmp-shell. This inserts qmp-shell
> > > >    as a proxy between the QMP client and server. qmp-shell can remove
> > > >    itself again with another chardev-change command. This approach
> > > >    allows MITM. The downside is it assumes the QMP chardev is a file
> > > >    descriptor, so it won't work with all types of chardev.
> > > > 
> > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
> > > >    source chardev, 2. a monitoring sink chardev, and 3. a monitoring
> > > >    source chardev. The data flow is origin <-> monitoring sink <->
> > > >    monitoring source <-> QMP monitor. qmp-shell creates the monitoring
> > > >    sink (for receiving incoming QMP commands) and monitoring source
> > > >    chardev (for forwarding QMP commands or MITM commands), and then it
> > > >    uses change-chardev to instantiate a chardev-proxy that directs the
> > > >    original libvirt chardev through the monitoring sink and source.
> > > > 
> > > >    This is the most complex but also completely contained within the
> > > >    QEMU chardev layer.
> > 
> > I have an idea for the QMP command name: chardev-snapshot-sync!
> > 
> > Finally we get backing file chains for chardevs! :-)
> > 
> > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an
> > > > equivalent API to start/stop monitoring a running VM without manual
> > > > setup steps.
> > > 
> > > Why go to the trouble of adding more chardevs to a running QEMU that
> > > libvirt has.  qmp-shell can just directly use the libvirt Python API
> > > to invoke  virDomainQemuMonitorCommand to invoke QMP commands, and
> > > the othe API for receiving QMP events.
> > > 
> > > Essentially it just needs to be split into two layers. The upper
> > > layer works in terms of individual QMP command/replies, and QMP
> > > events. The lower layer provides a transport that is either a
> > > UNIX socket, or is the libvirt QMP passthrough API.
> > > 
> > > Or alternatively, provide a virt-qmp-shim  command that listens on
> > > a UNIX socket, accepts QMP commands and turns them into calls to
> > > virDomainQemuMonitorCommand, and funnells back the response.
> > 
> > I think the idea was to show the QMP traffic that libvirt produces for
> > other management applications, not for the QMP shell. These APIs
> > probably don't allow this?
> 
> FWIW if you want to monitor what libvirt is sending/receiving we have
> a script for that that uses our systemtap probe points:
> 
>   https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp

Does that require root?

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[Daniel P. Berrangé]

On Thu, Jan 14, 2021 at 04:49:17PM +0000, Stefan Hajnoczi wrote:
> On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote:
> > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
> > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
> > > > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
> > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
> > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
> > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
> > > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even
> > > > > > > when libvirt is directly connected to the monitor.
> > > > > > > 
> > > > > > 
> > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
> > > > > > works.
> > > > > > 
> > > > > > (Actually, mitmproxy could theoretically be taught how to read and
> > > > > > understand QMP traffic, but that's not something I know how to do or would
> > > > > > be prepared to mentor.)
> > > > > > 
> > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using
> > > > > > production environment QEMU, how do we attach the QMP listener to it? Or
> > > > > > does this idea require that we start QEMU in a specific fashion with a
> > > > > > second debug socket that qmp-shell can connect to in order to listen?
> > > > > > 
> > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects
> > > > > > to ...?
> > > > > 
> > > > > Here is the QEMU command-line that libvirt uses on my F33 system:
> > > > > 
> > > > >   -chardev socket,id=charmonitor,fd=36,server,nowait
> > > > >   -mon chardev=charmonitor,id=monitor,mode=control
> > > > > 
> > > > > Goals for this feature:
> > > > > 
> > > > > 1. No manual steps required for setup.
> > > > > 2. Ability to start/stop monitoring traffic at runtime without
> > > > >    restarting QEMU.
> > > > > 3. Available to unprivileged users.
> > > > > 
> > > > > I think the easiest way to achieve this is through a new QEMU monitor
> > > > > command. Approaches that come to mind:
> > > > > 
> > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at
> > > > >    runtime. The debug-chardev receives both monitor input (commands) and
> > > > >    output (responses and events). This does not allow MITM, rather it
> > > > >    mirrors traffic.
> > > > > 
> > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and
> > > > >    then use the existing chardev-change command to replace the monitor
> > > > >    chardev with a chardev connected to qmp-shell. This inserts qmp-shell
> > > > >    as a proxy between the QMP client and server. qmp-shell can remove
> > > > >    itself again with another chardev-change command. This approach
> > > > >    allows MITM. The downside is it assumes the QMP chardev is a file
> > > > >    descriptor, so it won't work with all types of chardev.
> > > > > 
> > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
> > > > >    source chardev, 2. a monitoring sink chardev, and 3. a monitoring
> > > > >    source chardev. The data flow is origin <-> monitoring sink <->
> > > > >    monitoring source <-> QMP monitor. qmp-shell creates the monitoring
> > > > >    sink (for receiving incoming QMP commands) and monitoring source
> > > > >    chardev (for forwarding QMP commands or MITM commands), and then it
> > > > >    uses change-chardev to instantiate a chardev-proxy that directs the
> > > > >    original libvirt chardev through the monitoring sink and source.
> > > > > 
> > > > >    This is the most complex but also completely contained within the
> > > > >    QEMU chardev layer.
> > > 
> > > I have an idea for the QMP command name: chardev-snapshot-sync!
> > > 
> > > Finally we get backing file chains for chardevs! :-)
> > > 
> > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an
> > > > > equivalent API to start/stop monitoring a running VM without manual
> > > > > setup steps.
> > > > 
> > > > Why go to the trouble of adding more chardevs to a running QEMU that
> > > > libvirt has.  qmp-shell can just directly use the libvirt Python API
> > > > to invoke  virDomainQemuMonitorCommand to invoke QMP commands, and
> > > > the othe API for receiving QMP events.
> > > > 
> > > > Essentially it just needs to be split into two layers. The upper
> > > > layer works in terms of individual QMP command/replies, and QMP
> > > > events. The lower layer provides a transport that is either a
> > > > UNIX socket, or is the libvirt QMP passthrough API.
> > > > 
> > > > Or alternatively, provide a virt-qmp-shim  command that listens on
> > > > a UNIX socket, accepts QMP commands and turns them into calls to
> > > > virDomainQemuMonitorCommand, and funnells back the response.
> > > 
> > > I think the idea was to show the QMP traffic that libvirt produces for
> > > other management applications, not for the QMP shell. These APIs
> > > probably don't allow this?
> > 
> > FWIW if you want to monitor what libvirt is sending/receiving we have
> > a script for that that uses our systemtap probe points:
> > 
> >   https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp
> 
> Does that require root?

Yeah, systemtap generally requires root.

The same info is also written to the log files. For example:

  virt-admin daemon-log-filters "2:qemu_monitor_json"
  virt-admin daemon-lop-outputs "2:file:/var/log/libvirt/libvirtd.log"

nb, i'm using level '2' there to avoid enabling  debug logs, only
info level logs which is the level dynamic probes  log at.

Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 6549 bytes --]

On Thu, Jan 14, 2021 at 04:55:30PM +0000, Daniel P. Berrangé wrote:
> On Thu, Jan 14, 2021 at 04:49:17PM +0000, Stefan Hajnoczi wrote:
> > On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote:
> > > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
> > > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
> > > > > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
> > > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
> > > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
> > > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
> > > > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even
> > > > > > > > when libvirt is directly connected to the monitor.
> > > > > > > > 
> > > > > > > 
> > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
> > > > > > > works.
> > > > > > > 
> > > > > > > (Actually, mitmproxy could theoretically be taught how to read and
> > > > > > > understand QMP traffic, but that's not something I know how to do or would
> > > > > > > be prepared to mentor.)
> > > > > > > 
> > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using
> > > > > > > production environment QEMU, how do we attach the QMP listener to it? Or
> > > > > > > does this idea require that we start QEMU in a specific fashion with a
> > > > > > > second debug socket that qmp-shell can connect to in order to listen?
> > > > > > > 
> > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects
> > > > > > > to ...?
> > > > > > 
> > > > > > Here is the QEMU command-line that libvirt uses on my F33 system:
> > > > > > 
> > > > > >   -chardev socket,id=charmonitor,fd=36,server,nowait
> > > > > >   -mon chardev=charmonitor,id=monitor,mode=control
> > > > > > 
> > > > > > Goals for this feature:
> > > > > > 
> > > > > > 1. No manual steps required for setup.
> > > > > > 2. Ability to start/stop monitoring traffic at runtime without
> > > > > >    restarting QEMU.
> > > > > > 3. Available to unprivileged users.
> > > > > > 
> > > > > > I think the easiest way to achieve this is through a new QEMU monitor
> > > > > > command. Approaches that come to mind:
> > > > > > 
> > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at
> > > > > >    runtime. The debug-chardev receives both monitor input (commands) and
> > > > > >    output (responses and events). This does not allow MITM, rather it
> > > > > >    mirrors traffic.
> > > > > > 
> > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and
> > > > > >    then use the existing chardev-change command to replace the monitor
> > > > > >    chardev with a chardev connected to qmp-shell. This inserts qmp-shell
> > > > > >    as a proxy between the QMP client and server. qmp-shell can remove
> > > > > >    itself again with another chardev-change command. This approach
> > > > > >    allows MITM. The downside is it assumes the QMP chardev is a file
> > > > > >    descriptor, so it won't work with all types of chardev.
> > > > > > 
> > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
> > > > > >    source chardev, 2. a monitoring sink chardev, and 3. a monitoring
> > > > > >    source chardev. The data flow is origin <-> monitoring sink <->
> > > > > >    monitoring source <-> QMP monitor. qmp-shell creates the monitoring
> > > > > >    sink (for receiving incoming QMP commands) and monitoring source
> > > > > >    chardev (for forwarding QMP commands or MITM commands), and then it
> > > > > >    uses change-chardev to instantiate a chardev-proxy that directs the
> > > > > >    original libvirt chardev through the monitoring sink and source.
> > > > > > 
> > > > > >    This is the most complex but also completely contained within the
> > > > > >    QEMU chardev layer.
> > > > 
> > > > I have an idea for the QMP command name: chardev-snapshot-sync!
> > > > 
> > > > Finally we get backing file chains for chardevs! :-)
> > > > 
> > > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an
> > > > > > equivalent API to start/stop monitoring a running VM without manual
> > > > > > setup steps.
> > > > > 
> > > > > Why go to the trouble of adding more chardevs to a running QEMU that
> > > > > libvirt has.  qmp-shell can just directly use the libvirt Python API
> > > > > to invoke  virDomainQemuMonitorCommand to invoke QMP commands, and
> > > > > the othe API for receiving QMP events.
> > > > > 
> > > > > Essentially it just needs to be split into two layers. The upper
> > > > > layer works in terms of individual QMP command/replies, and QMP
> > > > > events. The lower layer provides a transport that is either a
> > > > > UNIX socket, or is the libvirt QMP passthrough API.
> > > > > 
> > > > > Or alternatively, provide a virt-qmp-shim  command that listens on
> > > > > a UNIX socket, accepts QMP commands and turns them into calls to
> > > > > virDomainQemuMonitorCommand, and funnells back the response.
> > > > 
> > > > I think the idea was to show the QMP traffic that libvirt produces for
> > > > other management applications, not for the QMP shell. These APIs
> > > > probably don't allow this?
> > > 
> > > FWIW if you want to monitor what libvirt is sending/receiving we have
> > > a script for that that uses our systemtap probe points:
> > > 
> > >   https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp
> > 
> > Does that require root?
> 
> Yeah, systemtap generally requires root.
> 
> The same info is also written to the log files. For example:
> 
>   virt-admin daemon-log-filters "2:qemu_monitor_json"
>   virt-admin daemon-lop-outputs "2:file:/var/log/libvirt/libvirtd.log"
> 
> nb, i'm using level '2' there to avoid enabling  debug logs, only
> info level logs which is the level dynamic probes  log at.

On my F33 system /var/log/libvirt is owned by root:root and rwx------,
so I guess it would be necessary to reconfigure log output so that
unprivileged users can access it.

If it can be used in conjunction with virDomainQemuMonitorCommand(),
then that eliminates the need to introduce new chardev functionality in
QEMU.

Parsing libvirt logs was one of the things I suggested, though. I think
it would be a nice feature for troubleshooting QMP conversations.

Stefan

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[Daniel P. Berrangé]

On Thu, Jan 14, 2021 at 05:14:28PM +0000, Stefan Hajnoczi wrote:
> On Thu, Jan 14, 2021 at 04:55:30PM +0000, Daniel P. Berrangé wrote:
> > On Thu, Jan 14, 2021 at 04:49:17PM +0000, Stefan Hajnoczi wrote:
> > > On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote:
> > > > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote:
> > > > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben:
> > > > > > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote:
> > > > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
> > > > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
> > > > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
> > > > > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even
> > > > > > > > > when libvirt is directly connected to the monitor.
> > > > > > > > > 
> > > > > > > > 
> > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
> > > > > > > > works.
> > > > > > > > 
> > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and
> > > > > > > > understand QMP traffic, but that's not something I know how to do or would
> > > > > > > > be prepared to mentor.)
> > > > > > > > 
> > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using
> > > > > > > > production environment QEMU, how do we attach the QMP listener to it? Or
> > > > > > > > does this idea require that we start QEMU in a specific fashion with a
> > > > > > > > second debug socket that qmp-shell can connect to in order to listen?
> > > > > > > > 
> > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects
> > > > > > > > to ...?
> > > > > > > 
> > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system:
> > > > > > > 
> > > > > > >   -chardev socket,id=charmonitor,fd=36,server,nowait
> > > > > > >   -mon chardev=charmonitor,id=monitor,mode=control
> > > > > > > 
> > > > > > > Goals for this feature:
> > > > > > > 
> > > > > > > 1. No manual steps required for setup.
> > > > > > > 2. Ability to start/stop monitoring traffic at runtime without
> > > > > > >    restarting QEMU.
> > > > > > > 3. Available to unprivileged users.
> > > > > > > 
> > > > > > > I think the easiest way to achieve this is through a new QEMU monitor
> > > > > > > command. Approaches that come to mind:
> > > > > > > 
> > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at
> > > > > > >    runtime. The debug-chardev receives both monitor input (commands) and
> > > > > > >    output (responses and events). This does not allow MITM, rather it
> > > > > > >    mirrors traffic.
> > > > > > > 
> > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and
> > > > > > >    then use the existing chardev-change command to replace the monitor
> > > > > > >    chardev with a chardev connected to qmp-shell. This inserts qmp-shell
> > > > > > >    as a proxy between the QMP client and server. qmp-shell can remove
> > > > > > >    itself again with another chardev-change command. This approach
> > > > > > >    allows MITM. The downside is it assumes the QMP chardev is a file
> > > > > > >    descriptor, so it won't work with all types of chardev.
> > > > > > > 
> > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
> > > > > > >    source chardev, 2. a monitoring sink chardev, and 3. a monitoring
> > > > > > >    source chardev. The data flow is origin <-> monitoring sink <->
> > > > > > >    monitoring source <-> QMP monitor. qmp-shell creates the monitoring
> > > > > > >    sink (for receiving incoming QMP commands) and monitoring source
> > > > > > >    chardev (for forwarding QMP commands or MITM commands), and then it
> > > > > > >    uses change-chardev to instantiate a chardev-proxy that directs the
> > > > > > >    original libvirt chardev through the monitoring sink and source.
> > > > > > > 
> > > > > > >    This is the most complex but also completely contained within the
> > > > > > >    QEMU chardev layer.
> > > > > 
> > > > > I have an idea for the QMP command name: chardev-snapshot-sync!
> > > > > 
> > > > > Finally we get backing file chains for chardevs! :-)
> > > > > 
> > > > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an
> > > > > > > equivalent API to start/stop monitoring a running VM without manual
> > > > > > > setup steps.
> > > > > > 
> > > > > > Why go to the trouble of adding more chardevs to a running QEMU that
> > > > > > libvirt has.  qmp-shell can just directly use the libvirt Python API
> > > > > > to invoke  virDomainQemuMonitorCommand to invoke QMP commands, and
> > > > > > the othe API for receiving QMP events.
> > > > > > 
> > > > > > Essentially it just needs to be split into two layers. The upper
> > > > > > layer works in terms of individual QMP command/replies, and QMP
> > > > > > events. The lower layer provides a transport that is either a
> > > > > > UNIX socket, or is the libvirt QMP passthrough API.
> > > > > > 
> > > > > > Or alternatively, provide a virt-qmp-shim  command that listens on
> > > > > > a UNIX socket, accepts QMP commands and turns them into calls to
> > > > > > virDomainQemuMonitorCommand, and funnells back the response.
> > > > > 
> > > > > I think the idea was to show the QMP traffic that libvirt produces for
> > > > > other management applications, not for the QMP shell. These APIs
> > > > > probably don't allow this?
> > > > 
> > > > FWIW if you want to monitor what libvirt is sending/receiving we have
> > > > a script for that that uses our systemtap probe points:
> > > > 
> > > >   https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp
> > > 
> > > Does that require root?
> > 
> > Yeah, systemtap generally requires root.
> > 
> > The same info is also written to the log files. For example:
> > 
> >   virt-admin daemon-log-filters "2:qemu_monitor_json"
> >   virt-admin daemon-lop-outputs "2:file:/var/log/libvirt/libvirtd.log"
> > 
> > nb, i'm using level '2' there to avoid enabling  debug logs, only
> > info level logs which is the level dynamic probes  log at.
> 
> On my F33 system /var/log/libvirt is owned by root:root and rwx------,
> so I guess it would be necessary to reconfigure log output so that
> unprivileged users can access it.

If you're connected to the privileged libvirtd then you have
effectively got root privileges, so that shouldn't be an issue.

If you're connecting to the unprivileged libvirtd, then you would
tell it to use $HOME/.cache/libvirt/libvirtd.log instead.

> If it can be used in conjunction with virDomainQemuMonitorCommand(),
> then that eliminates the need to introduce new chardev functionality in
> QEMU.
> 
> Parsing libvirt logs was one of the things I suggested, though. I think
> it would be a nice feature for troubleshooting QMP conversations.


Regards,
Daniel
-- 
|: https://berrange.com      -o-    https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org         -o-            https://fstop138.berrange.com :|
|: https://entangle-photo.org    -o-    https://www.instagram.com/dberrange :|

Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas)

[John Snow]

On 1/14/21 8:52 AM, Stefan Hajnoczi wrote:
> On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote:
>> On 1/13/21 3:53 AM, Stefan Hajnoczi wrote:
>>> On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote:
>>> 2. Ability to watch QMP activity on a running QEMU process, e.g. even
>>> when libvirt is directly connected to the monitor.
>>>
>>
>> That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy
>> works.
>>
>> (Actually, mitmproxy could theoretically be taught how to read and
>> understand QMP traffic, but that's not something I know how to do or would
>> be prepared to mentor.)
>>
>> Is this possible to do in a post-hoc fashion? Let's say you are using
>> production environment QEMU, how do we attach the QMP listener to it? Or
>> does this idea require that we start QEMU in a specific fashion with a
>> second debug socket that qmp-shell can connect to in order to listen?
>>
>> ... Or do we engineer qmp-shell to open its own socket that libvirt connects
>> to ...?
> 
> Here is the QEMU command-line that libvirt uses on my F33 system:
> 
>    -chardev socket,id=charmonitor,fd=36,server,nowait
>    -mon chardev=charmonitor,id=monitor,mode=control
> 
> Goals for this feature:
> 
> 1. No manual steps required for setup.
> 2. Ability to start/stop monitoring traffic at runtime without
>     restarting QEMU.
> 3. Available to unprivileged users.
> 

Excellent goals, and I agree completely.

> I think the easiest way to achieve this is through a new QEMU monitor
> command. Approaches that come to mind:
> 
> 1. Add a -mon debug-chardev property and a QMP command to set it at
>     runtime. The debug-chardev receives both monitor input (commands) and
>     output (responses and events). This does not allow MITM, rather it
>     mirrors traffic.
> 

So you have a socket that relays I/O. I wonder if it needs to modify the 
stream format to some extent to annotate directionality?

For now, directionality can be inferred, but maybe that's brittle.
(greeting messages, events and return statements are from the server; 
negotiation and execute statements are from the client.)

Maybe if we used a hypothetical qmp-shell log format, we could add 
timestamps here instead of relying on the client to produce them. This 
might be interesting for analyzing race conditions and measuring 
response delays as experienced by the server.

{"message": original_json_message_here, "direction": "in", "timestamp": 
1610627721}

(Downside: JSON is still not a streaming message format, but I guess 
it's one we already use all over the place anyway.)

> 2. Add a chardev-get-fd command that fetches the fd from a chardev and
>     then use the existing chardev-change command to replace the monitor
>     chardev with a chardev connected to qmp-shell. This inserts qmp-shell
>     as a proxy between the QMP client and server. qmp-shell can remove
>     itself again with another chardev-change command. This approach
>     allows MITM. The downside is it assumes the QMP chardev is a file
>     descriptor, so it won't work with all types of chardev.
> 

It seems a little more prone to failure if the insertion/removal fails, 
and has some downsides about which configurations it can inject into.

> 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin
>     source chardev, 2. a monitoring sink chardev, and 3. a monitoring
>     source chardev. The data flow is origin <-> monitoring sink <->
>     monitoring source <-> QMP monitor. qmp-shell creates the monitoring
>     sink (for receiving incoming QMP commands) and monitoring source
>     chardev (for forwarding QMP commands or MITM commands), and then it
>     uses change-chardev to instantiate a chardev-proxy that directs the
>     original libvirt chardev through the monitoring sink and source.
> 

I'm not sure I understand the topology here, exactly. I could stand to 
be a little more familiar with how chardevs are modeled in QEMU ...

>     This is the most complex but also completely contained within the
>     QEMU chardev layer.
> 
> In all these approaches qmp-shell uses virsh qemu-monitor-command or an
> equivalent API to start/stop monitoring a running VM without manual
> setup steps.
> 

Gotcha. I think I am leaning towards the first suggestion, but maybe the 
third one that I don't quite grasp yet is good too.

> Stefan
> 

Re: Call for Google Summer of Code 2021 project ideas

[Kashyap Chamarthy]

On Thu, Jan 14, 2021 at 11:36:23AM -0500, John Snow wrote:
> On 1/14/21 7:29 AM, Markus Armbruster wrote:

[...]

> So I see two possible options for "not inventing a language":
> 
> 1. Raw QMP
> 2. The existing qmp-shell syntax, warts and all.
> 
> I don't see a tremendous problem with doing both; but we can start with raw
> QMP. The existing qmp-shell syntax is at least definitely very easy to write
> a new parser for, even if it's kind of ugly and insufficient. I still see
> value in designing a new TUI with the old syntax.
> 
> > The project then aims to build a tool that adds useful features over
> > "socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>"
> > UNIX-CONNECT:/path/to/socket".
> > 
> > If it succeeds, you can still design and implement a "better" language,
> > and let users choose the one they prefer.  Or you could add features to
> > help with typing QMP.
> > 
> > >                                                             I don't
> > > think it's a blocker to have someone work on the TUI and asynchronous
> > > dispatch elements. I think even just keeping our current parsing but
> > > adding some of the features outlined in the proposal would be a big
> > > usability win.
> > 
> > I don't feel this particular itch, but I'm certainly not objecting to
> > anyone scratching.
> > 
> 
> It's something I'd like to see so that I can walk non-QEMU devs through
> interacting with QEMU at a low level for the purposes of debugging,
> reproducing problems, prototyping features, etc.
>
> I use qmp-shell all the time for debugging things myself, I find it easier
> to use than copy-pasting things directly into socat. I wouldn't mind the
> shell getting a little smarter to help me out -- the ability to see async
> events and reconnect on disconnect would already be a massive improvement to
> *my* quality of life.

As an infrequent user of `qmp-shell`, the async events stuff is really
beneficial to me too.  And, I'm happy to play the test guinea pig to
give the patchs a spin.  (I'm somewhat behind on the goings-on in this
area, very slowly catching up.)

> So much so that I spent a lot of time in December to write an async qmp
> library O:-)

Nice, I recall that you planned to use the 'asyncio' primitives from
Python 3.6.

-- 
/kashyap

Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas

[Florescu, Andreea]

[-- Attachment #1: Type: text/plain, Size: 7094 bytes --]

Hey Stefan,


Thanks for taking care of organizing GSoC, and for allowing rust-vmm to also participate under the QEMU umbrella!

I am a bit unsure of how can we propose projects related to rust-vmm.

We did a bit of brainstorming in our team, and we came up with 3 project ideas.

I'll just paste them below, but please let me know if we were supposed to propose them some other way.


=== Implement the Virtio Console device in Rust ===

'''Summary:''' Implement the basic emulation for the Virtio Console device in Rust

Implement the basic functionality (excluding the optional features:
VIRTIO_CONSOLE_F_SIZE, VIRTIO_CONSOLE_F_MULTIPORT, or VIRTIO_CONSOLE_F_EMERG_WRITE)
of the Virtio Console Device, using the Virtio building blocks (queue implementations,
VirtioDevice traits) defined in rust-vmm/vm-virtio. The virtio console device uses
one virtio queue for transmitting data, and one virtio queue for receiving data.
The implementation can be extended to also support a subset of the previously
mentioned optional features.

'''Links:'''
* About rust-vmm: https://github.com/rust-vmm/community
* rust-vmm/vm-virtio: https://github.com/rust-vmm/vm-virtio
* virtio-console spec: https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-2550003

'''Details:'''
* Skill level: intermediate
* Language: Rust
* Mentor: iul@amazon.com
* Suggested by: fandree@amazon.com<mailto:fandree@amazon.com>


=== Mocking framework for Virtio Queues ===

'''Summary:''' Implement a mocking framework for virtio queues

Paravirtualized devices (such as those defined by the Virtio standard) are used
to provide high performance device emulation. Virtio drivers from a guest VM
communicate with the device model using an efficient mechanism based on queues
stored in a shared memory area that operate based on a protocol and message format
defined by the standard. Various implementations of devices and other
virtualization building blocks require mocking the contents that a driver would
place into a Virtio queue for validation, testing, and evaluation purposes.

This project aims to lay the foundations of a reusable framework for mocking the
driver side of Virtio queue operation, that can be consumed by rust-vmm crates and
other projects. At the basic level, this means providing a flexible and easy to
use interface for users to set up the underlying memory areas and populate contents
(as the driver would do) for the basic split queue format in a generic manner. This
can further be extended for the packed format and with device-specific mocking
capabilities.

'''Links:'''
* About rust-vmm: https://github.com/rust-vmm/community
* Virtio queue spec: https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-230005
Issue in rust-vmm about reusing the mocking logic: rust-vmm/vm-virtio: https://github.com/rust-vmm/vm-virtio

'''Details:'''
* Skill level: intermediate
* Language: Rust
* Mentor: aagch@amazon.com
* Suggested by: aagch@amazon.com


=== Local running rust-vmm-ci ===

'''Summary:''' Run the rust-vmm-ci locally

The rust-vmm-ci provides automation for uniformely running the tests on
all rust-vmm repositories. It is built on top of Buildkite, and only allows
running the tests in the Buildkite context. To run the same tests as in the CI
locally, users need to manually copy the Buildkite pipeline steps.

The scope of this project is to make it possible for the same tests to easily run
locally. This project makes it easier to contribute to all rust-vmm repositories.

In order for that to be possible, the following steps are required:
- the Buildlkite pipeline is autogenerated from code instead of being a static
list of tests to run. This also allows us to uniformely use the same container
version for running all the tests (instead of manually modifying each step in
the pipeline)
- the code for autogenerating the Buildkite pipeline is reused for generating
a Python script which can be run locally


'''Links:'''
* rust-vmm-ci: https://github.com/rust-vmm/rust-vmm-ci
* Buildkite pipeline that currently runs the tests: https://github.com/rust-vmm/rust-vmm-ci/blob/master/.buildkite/pipeline.yml
* About rust-vmm: https://github.com/rust-vmm/community
* Buildkite documentation: https://buildkite.com/docs/tutorials/getting-started

'''Details:'''
* Skill level: intermediate
* Language: Python
* Mentor: fandree@amazon.com
* Suggested by: fandree@amazon.com


?Thanks again!

Andreea

________________________________
From: Stefan Hajnoczi <stefanha@gmail.com>
Sent: Monday, January 11, 2021 1:47 PM
To: qemu-devel; kvm; rust-vmm@lists.opendev.org; Alex Bennée; Alexander Graf; Alberto Garcia; David Hildenbrand; Eduardo Habkost; Igor Mammedov; John Snow; Julia Suvorova; Gerd Hoffmann; Kevin Wolf; Laurent Vivier; Marc-André Lureau; Aleksandar Markovic; Sergio Lopez; Stefano Garzarella; Paolo Bonzini; Philippe Mathieu-Daudé
Subject: [EXTERNAL] [Rust-VMM] Call for Google Summer of Code 2021 project ideas

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.



Dear QEMU, KVM, and rust-vmm community,
QEMU will apply for Google Summer of Code
(https://summerofcode.withgoogle.com/) again this year.  This internship
program offers paid, 10-week, remote work internships for
contributing to open source.  QEMU can act as an umbrella organization
for KVM kernel and rust-vmm projects too.

Please post project ideas on the QEMU wiki before February 14th:
https://wiki.qemu.org/Google_Summer_of_Code_2021

What's new this year:
 * The number of internship hours has been halved to 175 hours over
   10 weeks. Project ideas must be smaller to fit and students will have
   more flexibility with their working hours.
 * Eligibility has been expanded to include "licensed coding school or
   similar type of program".

Good project ideas are suitable for 175 hours (10 weeks half-day) work by a
competent programmer who is not yet familiar with the codebase.  In
addition, they are:
 * Well-defined - the scope is clear
 * Self-contained - there are few dependencies
 * Uncontroversial - they are acceptable to the community
 * Incremental - they produce deliverables along the way

Feel free to post ideas even if you are unable to mentor the project.
It doesn't hurt to share the idea!

I will review project ideas and keep you up-to-date on QEMU's
acceptance into GSoC.

For more background on QEMU internships, check out this video:
https://www.youtube.com/watch?v=xNVCX7YMUL8

Stefan

_______________________________________________
Rust-vmm mailing list
Rust-vmm@lists.opendev.org
http://lists.opendev.org/cgi-bin/mailman/listinfo/rust-vmm



Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[-- Attachment #2: Type: text/html, Size: 9566 bytes --]

Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas

[Sergio Lopez]

[-- Attachment #1: Type: text/plain, Size: 7770 bytes --]

On Fri, Feb 12, 2021 at 01:22:43PM +0000, Florescu, Andreea wrote:
> Hey Stefan,
> 
> 
> Thanks for taking care of organizing GSoC, and for allowing rust-vmm to also participate under the QEMU umbrella!
> 
> I am a bit unsure of how can we propose projects related to rust-vmm.
> 
> We did a bit of brainstorming in our team, and we came up with 3 project ideas.
> 
> I'll just paste them below, but please let me know if we were supposed to propose them some other way.
> 
> 
> === Implement the Virtio Console device in Rust ===
> 
> '''Summary:''' Implement the basic emulation for the Virtio Console device in Rust
> 
> Implement the basic functionality (excluding the optional features:
> VIRTIO_CONSOLE_F_SIZE, VIRTIO_CONSOLE_F_MULTIPORT, or VIRTIO_CONSOLE_F_EMERG_WRITE)
> of the Virtio Console Device, using the Virtio building blocks (queue implementations,
> VirtioDevice traits) defined in rust-vmm/vm-virtio. The virtio console device uses
> one virtio queue for transmitting data, and one virtio queue for receiving data.
> The implementation can be extended to also support a subset of the previously
> mentioned optional features.

FWIW, libkrun already has support for virtio-console with the basic
functionality and VIRTIO_CONSOLE_F_SIZE, and this code could be easily
borrowed for implementing it in some rust-vmm crate:

https://github.com/containers/libkrun/blob/main/src/devices/src/virtio/console

Sergio.

> '''Links:'''
> * About rust-vmm: https://github.com/rust-vmm/community
> * rust-vmm/vm-virtio: https://github.com/rust-vmm/vm-virtio
> * virtio-console spec: https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-2550003
> 
> '''Details:'''
> * Skill level: intermediate
> * Language: Rust
> * Mentor: iul@amazon.com
> * Suggested by: fandree@amazon.com<mailto:fandree@amazon.com>
> 
> 
> === Mocking framework for Virtio Queues ===
> 
> '''Summary:''' Implement a mocking framework for virtio queues
> 
> Paravirtualized devices (such as those defined by the Virtio standard) are used
> to provide high performance device emulation. Virtio drivers from a guest VM
> communicate with the device model using an efficient mechanism based on queues
> stored in a shared memory area that operate based on a protocol and message format
> defined by the standard. Various implementations of devices and other
> virtualization building blocks require mocking the contents that a driver would
> place into a Virtio queue for validation, testing, and evaluation purposes.
> 
> This project aims to lay the foundations of a reusable framework for mocking the
> driver side of Virtio queue operation, that can be consumed by rust-vmm crates and
> other projects. At the basic level, this means providing a flexible and easy to
> use interface for users to set up the underlying memory areas and populate contents
> (as the driver would do) for the basic split queue format in a generic manner. This
> can further be extended for the packed format and with device-specific mocking
> capabilities.
> 
> '''Links:'''
> * About rust-vmm: https://github.com/rust-vmm/community
> * Virtio queue spec: https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-230005
> Issue in rust-vmm about reusing the mocking logic: rust-vmm/vm-virtio: https://github.com/rust-vmm/vm-virtio
> 
> '''Details:'''
> * Skill level: intermediate
> * Language: Rust
> * Mentor: aagch@amazon.com
> * Suggested by: aagch@amazon.com
> 
> 
> === Local running rust-vmm-ci ===
> 
> '''Summary:''' Run the rust-vmm-ci locally
> 
> The rust-vmm-ci provides automation for uniformely running the tests on
> all rust-vmm repositories. It is built on top of Buildkite, and only allows
> running the tests in the Buildkite context. To run the same tests as in the CI
> locally, users need to manually copy the Buildkite pipeline steps.
> 
> The scope of this project is to make it possible for the same tests to easily run
> locally. This project makes it easier to contribute to all rust-vmm repositories.
> 
> In order for that to be possible, the following steps are required:
> - the Buildlkite pipeline is autogenerated from code instead of being a static
> list of tests to run. This also allows us to uniformely use the same container
> version for running all the tests (instead of manually modifying each step in
> the pipeline)
> - the code for autogenerating the Buildkite pipeline is reused for generating
> a Python script which can be run locally
> 
> 
> '''Links:'''
> * rust-vmm-ci: https://github.com/rust-vmm/rust-vmm-ci
> * Buildkite pipeline that currently runs the tests: https://github.com/rust-vmm/rust-vmm-ci/blob/master/.buildkite/pipeline.yml
> * About rust-vmm: https://github.com/rust-vmm/community
> * Buildkite documentation: https://buildkite.com/docs/tutorials/getting-started
> 
> '''Details:'''
> * Skill level: intermediate
> * Language: Python
> * Mentor: fandree@amazon.com
> * Suggested by: fandree@amazon.com
> 
> 
> ?Thanks again!
> 
> Andreea
> 
> ________________________________
> From: Stefan Hajnoczi <stefanha@gmail.com>
> Sent: Monday, January 11, 2021 1:47 PM
> To: qemu-devel; kvm; rust-vmm@lists.opendev.org; Alex Bennée; Alexander Graf; Alberto Garcia; David Hildenbrand; Eduardo Habkost; Igor Mammedov; John Snow; Julia Suvorova; Gerd Hoffmann; Kevin Wolf; Laurent Vivier; Marc-André Lureau; Aleksandar Markovic; Sergio Lopez; Stefano Garzarella; Paolo Bonzini; Philippe Mathieu-Daudé
> Subject: [EXTERNAL] [Rust-VMM] Call for Google Summer of Code 2021 project ideas
> 
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> 
> 
> 
> Dear QEMU, KVM, and rust-vmm community,
> QEMU will apply for Google Summer of Code
> (https://summerofcode.withgoogle.com/) again this year.  This internship
> program offers paid, 10-week, remote work internships for
> contributing to open source.  QEMU can act as an umbrella organization
> for KVM kernel and rust-vmm projects too.
> 
> Please post project ideas on the QEMU wiki before February 14th:
> https://wiki.qemu.org/Google_Summer_of_Code_2021
> 
> What's new this year:
>  * The number of internship hours has been halved to 175 hours over
>    10 weeks. Project ideas must be smaller to fit and students will have
>    more flexibility with their working hours.
>  * Eligibility has been expanded to include "licensed coding school or
>    similar type of program".
> 
> Good project ideas are suitable for 175 hours (10 weeks half-day) work by a
> competent programmer who is not yet familiar with the codebase.  In
> addition, they are:
>  * Well-defined - the scope is clear
>  * Self-contained - there are few dependencies
>  * Uncontroversial - they are acceptable to the community
>  * Incremental - they produce deliverables along the way
> 
> Feel free to post ideas even if you are unable to mentor the project.
> It doesn't hurt to share the idea!
> 
> I will review project ideas and keep you up-to-date on QEMU's
> acceptance into GSoC.
> 
> For more background on QEMU internships, check out this video:
> https://www.youtube.com/watch?v=xNVCX7YMUL8
> 
> Stefan
> 
> _______________________________________________
> Rust-vmm mailing list
> Rust-vmm@lists.opendev.org
> http://lists.opendev.org/cgi-bin/mailman/listinfo/rust-vmm
> 
> 
> 
> Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Call for Google Summer of Code 2021 project ideas

[Paolo Bonzini]

On 14/01/21 17:36, John Snow wrote:
> 
>> The sane way to evade the language design problem is to use the existing
>> QMP language.
> 
> I wouldn't mind implementing this for version 0.1 -- just allow 
> copy-pasting JSON into the input bar -- it's a feature I wanted anyway.

I think the only way out of language design is to instead design a TUI 
for inputting JSON.  For example:

* after typing the ' for a key you can autocomplete the next field, 
using the TAB key similar to vi

* after typing the : the TUI tells you the field type

* after typing the ' for an enum, the TAB brings up a menu to pick an enum

* after typing the last character in a key or value you automatically 
get a suggestion for what to type next (comma and next apostrophe after 
a value, colon and possible apostrophe/bracket/brace for a key)

One idea that has worked for me in the past was to write a mockup that 
shows what things are going to look like, with fake user interaction. 
You would have something like

    // {
    keypress("{")
    show_suggestion("'")
    // '
    keypress("'")
    start_autocomplete_choices(["execute", "arguments"])
    // TAB
    await asyncio.sleep(1)
    do_autocomplete()
    // '
    await asyncio.sleep(1)
    keypress("'")
    // string argument, propose ' for the value automatically
    show_suggestion(": '")
    // TAB
    await asyncio.sleep(1)
    do_autocomplete()
    start_autocomplete_choices("query-status", "query-kvm") # many more

etc.

Then you plug in an incremental lexer, so that you can e.g. replace

    show_autocomplete(": '")

with

    lex_state(Lexer.AFTER_KEY); // this would come from the lexer
    show_autocomplete("'")      // this would come from the schema

And then again plug the incremental visitor to autocomplete on the 
schema types.

Another advantage of this approach is that you also have a natural API 
for the mocks, and thus it becomes easier to write testcases.

Paolo

Re: Call for Google Summer of Code 2021 project ideas

[John Snow]

On 2/15/21 6:05 AM, Paolo Bonzini wrote:
> On 14/01/21 17:36, John Snow wrote:
>>
>>> The sane way to evade the language design problem is to use the existing
>>> QMP language.
>>
>> I wouldn't mind implementing this for version 0.1 -- just allow 
>> copy-pasting JSON into the input bar -- it's a feature I wanted anyway.
> 
> I think the only way out of language design is to instead design a TUI 
> for inputting JSON.  For example:
>  > * after typing the ' for a key you can autocomplete the next field,
> using the TAB key similar to vi
> 
> * after typing the : the TUI tells you the field type
> 
> * after typing the ' for an enum, the TAB brings up a menu to pick an enum
> 
> * after typing the last character in a key or value you automatically 
> get a suggestion for what to type next (comma and next apostrophe after 
> a value, colon and possible apostrophe/bracket/brace for a key)
> 
> One idea that has worked for me in the past was to write a mockup that 
> shows what things are going to look like, with fake user interaction. 
> You would have something like
> 
>     // {
>     keypress("{")
>     show_suggestion("'")
>     // '
>     keypress("'")
>     start_autocomplete_choices(["execute", "arguments"])
>     // TAB
>     await asyncio.sleep(1)
>     do_autocomplete()
>     // '
>     await asyncio.sleep(1)
>     keypress("'")
>     // string argument, propose ' for the value automatically
>     show_suggestion(": '")
>     // TAB
>     await asyncio.sleep(1)
>     do_autocomplete()
>     start_autocomplete_choices("query-status", "query-kvm") # many more
> 
> etc.
> 
> Then you plug in an incremental lexer, so that you can e.g. replace
> 
>     show_autocomplete(": '")
> 
> with
> 
>     lex_state(Lexer.AFTER_KEY); // this would come from the lexer
>     show_autocomplete("'")      // this would come from the schema
> 
> And then again plug the incremental visitor to autocomplete on the 
> schema types.
> 
> Another advantage of this approach is that you also have a natural API 
> for the mocks, and thus it becomes easier to write testcases.
> 
> Paolo
> 

Yes, I follow.

I quite like the VSCode auto-suggest behavior for writing JSON when you 
have a schema, but that's baked into their borg-cube, I assume.

I wonder if there's any libraries or more re-usable pieces out there 
that would allow us to write a nice interactive TUI for complex 
structures in Python, or if we'd have to design and build it ground-up.

Changes how willing I am to embark upon the quest.

--js

Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas

[Stefan Hajnoczi]

Thanks, I have published the rust-vmm project ideas on the wiki:
https://wiki.qemu.org/Google_Summer_of_Code_2021

Please see Sergio's reply about virtio-consoile is libkrun. Maybe it
affects the project idea?

Stefan

Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas

[Andreea Florescu]

[-- Attachment #1: Type: text/plain, Size: 1591 bytes --]



On 2/17/21 1:20 PM, Stefan Hajnoczi wrote:
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
>
>
>
> Thanks, I have published the rust-vmm project ideas on the wiki:
> https://wiki.qemu.org/Google_Summer_of_Code_2021
Thanks, Stefan!
>
> Please see Sergio's reply about virtio-consoile is libkrun. Maybe it
> affects the project idea?
I synced offline with Sergio. It seems I've misread his comment.
The code is already available in libkrun, and to port it to rust-vmm
will likely take just a couple of days. We explored the option of also
requesting to implement VIRTIO_CONSOLE_F_MULTIPORT to make it an
appropriate GSoC project, but we decided this is not a good idea since
it doesn't look like that feature is useful for the projects consuming
rust-vmm. It also adds complexity, and we would need to maintain that as
well.

Since it would still be nice to have virtio-console in rust-vmm, I'll
just open an issue in vm-virtio and label it with "help wanted" so
people can pick it up.
Can we remove the virtio-console project from the list of GSoC ideas?

Also, iul@amazon.com will like to help with mentoring the GSoC projects.
Can he be added as a co-mentor of: "Mocking framework for Virtio Queues"?

Sorry for the ping-pong, and thanks again for everything!

Andreea




Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005.

[-- Attachment #2: 0x9B51563C1FA36782.asc --]
[-- Type: application/pgp-keys, Size: 2501 bytes --]

Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas

[Stefan Hajnoczi]

On Thu, Feb 18, 2021 at 11:50 AM Andreea Florescu <fandree@amazon.com> wrote:
> On 2/17/21 1:20 PM, Stefan Hajnoczi wrote:
> > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe.
> >
> >
> >
> > Thanks, I have published the rust-vmm project ideas on the wiki:
> > https://wiki.qemu.org/Google_Summer_of_Code_2021
> Thanks, Stefan!
> >
> > Please see Sergio's reply about virtio-consoile is libkrun. Maybe it
> > affects the project idea?
> I synced offline with Sergio. It seems I've misread his comment.
> The code is already available in libkrun, and to port it to rust-vmm
> will likely take just a couple of days. We explored the option of also
> requesting to implement VIRTIO_CONSOLE_F_MULTIPORT to make it an
> appropriate GSoC project, but we decided this is not a good idea since
> it doesn't look like that feature is useful for the projects consuming
> rust-vmm. It also adds complexity, and we would need to maintain that as
> well.
>
> Since it would still be nice to have virtio-console in rust-vmm, I'll
> just open an issue in vm-virtio and label it with "help wanted" so
> people can pick it up.
> Can we remove the virtio-console project from the list of GSoC ideas?

Done.

> Also, iul@amazon.com will like to help with mentoring the GSoC projects.
> Can he be added as a co-mentor of: "Mocking framework for Virtio Queues"?

Done.

Thanks for getting the rust-vmm mentors together! I'm looking forward
to more Rust virtualization projects :).

Stefan