* Call for Google Summer of Code 2021 project ideas @ 2021-01-11 11:47 Stefan Hajnoczi 2021-01-12 21:10 ` John Snow 2021-02-12 13:22 ` [Rust-VMM] " Florescu, Andreea 0 siblings, 2 replies; 27+ messages in thread From: Stefan Hajnoczi @ 2021-01-11 11:47 UTC (permalink / raw) To: qemu-devel, kvm, rust-vmm, Alex Bennée, Alexander Graf, Alberto Garcia, David Hildenbrand, Eduardo Habkost, Igor Mammedov, John Snow, Julia Suvorova, Gerd Hoffmann, Kevin Wolf, Laurent Vivier, Marc-André Lureau, Aleksandar Markovic, Sergio Lopez, Stefano Garzarella, Paolo Bonzini, Philippe Mathieu-Daudé Dear QEMU, KVM, and rust-vmm community, QEMU will apply for Google Summer of Code (https://summerofcode.withgoogle.com/) again this year. This internship program offers paid, 10-week, remote work internships for contributing to open source. QEMU can act as an umbrella organization for KVM kernel and rust-vmm projects too. Please post project ideas on the QEMU wiki before February 14th: https://wiki.qemu.org/Google_Summer_of_Code_2021 What's new this year: * The number of internship hours has been halved to 175 hours over 10 weeks. Project ideas must be smaller to fit and students will have more flexibility with their working hours. * Eligibility has been expanded to include "licensed coding school or similar type of program". Good project ideas are suitable for 175 hours (10 weeks half-day) work by a competent programmer who is not yet familiar with the codebase. In addition, they are: * Well-defined - the scope is clear * Self-contained - there are few dependencies * Uncontroversial - they are acceptable to the community * Incremental - they produce deliverables along the way Feel free to post ideas even if you are unable to mentor the project. It doesn't hurt to share the idea! I will review project ideas and keep you up-to-date on QEMU's acceptance into GSoC. For more background on QEMU internships, check out this video: https://www.youtube.com/watch?v=xNVCX7YMUL8 Stefan ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Call for Google Summer of Code 2021 project ideas 2021-01-11 11:47 Call for Google Summer of Code 2021 project ideas Stefan Hajnoczi @ 2021-01-12 21:10 ` John Snow 2021-01-13 8:53 ` Stefan Hajnoczi 2021-01-13 9:19 ` Call for Google Summer of Code 2021 project ideas Markus Armbruster 2021-02-12 13:22 ` [Rust-VMM] " Florescu, Andreea 1 sibling, 2 replies; 27+ messages in thread From: John Snow @ 2021-01-12 21:10 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: QEMU Developers, Stefan Hajnoczi, Eduardo Habkost On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: > Dear QEMU, KVM, and rust-vmm community, > QEMU will apply for Google Summer of Code > (https://summerofcode.withgoogle.com/) again this year. This internship > program offers paid, 10-week, remote work internships for > contributing to open source. QEMU can act as an umbrella organization > for KVM kernel and rust-vmm projects too. > > Please post project ideas on the QEMU wiki before February 14th: > https://wiki.qemu.org/Google_Summer_of_Code_2021 > > What's new this year: > * The number of internship hours has been halved to 175 hours over > 10 weeks. Project ideas must be smaller to fit and students will have > more flexibility with their working hours. > * Eligibility has been expanded to include "licensed coding school or > similar type of program". > > Good project ideas are suitable for 175 hours (10 weeks half-day) work by a > competent programmer who is not yet familiar with the codebase. In > addition, they are: > * Well-defined - the scope is clear > * Self-contained - there are few dependencies > * Uncontroversial - they are acceptable to the community > * Incremental - they produce deliverables along the way > > Feel free to post ideas even if you are unable to mentor the project. > It doesn't hurt to share the idea! > I have one that is probably way too ambitious, but requires a particular skillset that might be of good interest to a student that has some experience in the area already. The idea is for a TUI qmp-shell (maybe using urwid?) to create an irssi-like REPL interface for QMP. The idea would be to mimic the mitmproxy TUI interface (Check it out if you haven't!) All the ideas below are extremely tentative to give a basic gist of what I mean; exact layouts/hotkeys/etc are for the sake of explanation only. Essentially, I want an interface like this: ----------------------------------------------------------- | QMP Mode | |=========================================================| | | | Welcome to the Qemu Machine Protocol shell. | | Please type /help or Ctrl+H to see available commands. | | | | | | | |---------------------------------------------------------| | > | ----------------------------------------------------------- commands are entered in the bottom and appear in a log window above, appearing most-recent last, like irssi works. As an example, let's say we issue block-dirty-bitmap-add: -------------------------------------------------------- | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | -------------------------------------------------------- (...syntax up for debate...! We want something easy to parse, but easy to type. For structured data, that's Hard.) we press [Enter] to submit the command, and the history now shows: ------------------------------------------------- | QMP Mode | |===============================================| | | | | | | | [11:28] block-dirty-bitmap-add [Pending ...] | |-----------------------------------------------| | > | ------------------------------------------------- After a few moments, the command resolves: ---------------------------------------- | QMP Mode | |======================================| | | | | | | | [11:28] block-dirty-bitmap-add [OK] | ---------------------------------------- | > | ---------------------------------------- The basic unit of history here is either an RPC call/response pair or an asynchronous QMP event. Clicking on the history pane or pressing Alt+↑ or Alt+↓ (Mimics irssi keys for switching buffers, let's say it's user-configurable) to change focus to the history pane: ------------------------------------------------------------ | Command History Mode | |==========================================================| | | | | | | | [11:28] block-dirty-bitmap-add [OK] | -----------------------------------------------------------| | Use ↑ or ↓ to select a command, press [Enter] to inspect | ------------------------------------------------------------ Using the arrow keys, you can highlight the history item and then press <Enter> to change the log pane to a details pane for that item, showing: ------------------------------------------------------ | Details: block-dirty-bitmap-add [X] | |====================================================| | | | [11:28:01] -> { | | "execute": "block-dirty-bitmap-add", | | "arguments": { | | "node": "ide0hd0", | | "name": "mybitmap0" | | } | | } | | | | [11:28:02] <- { | | "return": {} | | } | | | |----------------------------------------------------| | Press 'Q' or Backspace to close details pane | ------------------------------------------------------ From here, maybe you could do a few things, like press a button to view the raw QMP I/O for the purposes of copy-pasting elsewhere, or just press Q/H/Backspace to go back to the history view. Interesting abilities this style of shell can provide: (1) The ability to render asynchronous events from the server in the log window without overwriting the user's command buffer. (In contrast to qmp-shell today, which cannot show async events until the user presses enter again to prompt a check for them.) (2) The ability to asynchronously reconnect to servers on connection failures. (3) The ability to collapse command:response pairs into one summarized line in the log, which can be expanded for more information later with a keypress. (4) (As a future add-on, beyond current scope) The ability to add a status pane widget that might show current running jobs, the current machine execution status, etc. As options, users should be able to: 1. Choose the history display mode live at runtime: a) Summarized command history (as shown above) b) Summarized JSON history (one item per message, no grouping, purely linear, shows a one-line summary of the message.) c) Full JSON history (one item per message, JSON messages are rendered in full.) d) Pretty JSON history (one item per message, JSON messages are pretty-printed.) e.g. Ctrl+V could change the "verbosity" of the messages, and the history log will be re-rendered to reflect the new setting. 2. Customize hotkeys for switching between command mode and history mode; quitting the details view, etc. 3. toggle on/off automatic command execution logging; or manually (Ctrl+S?) save history to file. The format might need some light markup to indicate directionality, timestamps, and so on. Other things that might be of interest, but are beyond the scope of a GSoC/Outreachy internship: 1. readline-esque hotkey functionality for the command input bar, including command history, autocomplete for command names, etc. If we aren't using readline itself anymore, these things will have to be implemented somehow. There are libraries that support this: https://github.com/rr-/urwid_readline 2. Recording execution macros; subsets of commands you wish to replay easily. 3. Raw input mode: allow the user to paste a raw JSON command directly into the command bar. 4. Configuring our GTK interface to open up qmp-shell inside one of the terminal panes. (Kevin Wolf suggested this or something like it once, and I rather like the idea.) 5. "Shell Builtins" -- analogous to the same in bash -- additional commands and routines that help perform certain actions that might require complex, multi-part steps. 6. Augmented commands -- commands that don't require multi-part phases, but ones in which the shell might offer enhanced readouts for. An example would be integrating Vladimir's block graph rendering script to automatically fire off when block-query is issued. (If the user has a graphical session running, the libraries are installed, etc.) See alsos; prior art, &c: - irssi as an example of continuous, asynchronous history: https://irssi.org/ - mitmproxy, which heavily inspired this idea with its use of urwid to show a history of "packets" which can be interrogated later from within the interface https://mitmproxy.org/ - aioconsole -- an async python REPL loop for interactively writing async code in python. It might have good ideas to steal! https://pypi.org/project/aioconsole/ - urwid: a TUI framework for Python; it is used by mitmproxy to implement its interface. http://urwid.org/ - urwid readline library: implements readline-esque hotkeys for urwid, but (maybe) doesn't respect readline config, if any. It is a reimplementation. https://github.com/rr-/urwid_readline Target skillset for students: - Intermediate Python - Some exposure to async programming, coroutines, etc. - Some experience with GUI programming concepts, in any toolkit or language - A willingness to use gradually typed python O:-) I don't expect this project to involve much, if any, C; but familiarity with the linux shell ecosystem will be an asset when choosing default behaviors, mimicking other tools, etc. I have already written a fully asyncio QMP library; I would expect this new library to be used. It should be fully operational, but might require API changes when we begin to use it for a serious endeavor. > I will review project ideas and keep you up-to-date on QEMU's > acceptance into GSoC. > > For more background on QEMU internships, check out this video: > https://www.youtube.com/watch?v=xNVCX7YMUL8 > > Stefan > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Call for Google Summer of Code 2021 project ideas 2021-01-12 21:10 ` John Snow @ 2021-01-13 8:53 ` Stefan Hajnoczi 2021-01-13 18:59 ` qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) John Snow 2021-01-13 9:19 ` Call for Google Summer of Code 2021 project ideas Markus Armbruster 1 sibling, 1 reply; 27+ messages in thread From: Stefan Hajnoczi @ 2021-01-13 8:53 UTC (permalink / raw) To: John Snow; +Cc: QEMU Developers, Stefan Hajnoczi, Eduardo Habkost On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > I have one that is probably way too ambitious, but requires a particular > skillset that might be of good interest to a student that has some > experience in the area already. > > The idea is for a TUI qmp-shell (maybe using urwid?) to create an > irssi-like REPL interface for QMP. The idea would be to mimic the > mitmproxy TUI interface (Check it out if you haven't!) Great, I think this project idea lends itself to an incremental milestones. How far it gets will depend on the intern and we'll be able to merge useful patches regardless of how far they take it. Two more ideas: 1. Ability to load libvirt log files for offline viewing. This could be a major use case for this tool because the raw libvirt logs can be hard to read today. 2. Ability to watch QMP activity on a running QEMU process, e.g. even when libvirt is directly connected to the monitor. Stefan ^ permalink raw reply [flat|nested] 27+ messages in thread
* qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-13 8:53 ` Stefan Hajnoczi @ 2021-01-13 18:59 ` John Snow 2021-01-14 13:52 ` Stefan Hajnoczi 0 siblings, 1 reply; 27+ messages in thread From: John Snow @ 2021-01-13 18:59 UTC (permalink / raw) To: Stefan Hajnoczi; +Cc: QEMU Developers, Stefan Hajnoczi, Eduardo Habkost On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: >> I have one that is probably way too ambitious, but requires a particular >> skillset that might be of good interest to a student that has some >> experience in the area already. >> >> The idea is for a TUI qmp-shell (maybe using urwid?) to create an >> irssi-like REPL interface for QMP. The idea would be to mimic the >> mitmproxy TUI interface (Check it out if you haven't!) > > Great, I think this project idea lends itself to an incremental > milestones. How far it gets will depend on the intern and we'll be > able to merge useful patches regardless of how far they take it. > Yeah. I wrote a lot, but I think a lot of the desires and goals can actually be split out. You can start with just the REPL mode; no bells or whistles. Just type commands, issue them, and have the log pane populate with the raw JSON as a starting point. That'd be useful enough already. From there, the bells and whistles that make it truly shine can be added. > Two more ideas: > 1. Ability to load libvirt log files for offline viewing. This could > be a major use case for this tool because the raw libvirt logs can be > hard to read today. Yeah, that would be excellent. (Especially because I have such a hard time understanding libvirt-ese; seeing the QMP log in the same tool I use to communicate directly with QEMU would be an excellent debugging boon.) mitmproxy has a similar feature where packet captures can be saved to file and loaded again later for analysis. A similar thing here would be nice. I mentioned wanting to be able to save sessions for later viewing in my proposal; which likely means developing a QMP log format. It's unlikely we'll want to use the libvirt log format here, so we'd need to develop a reader that can parse libvirt logs; but a writer is not likely important. > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > when libvirt is directly connected to the monitor. > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy works. (Actually, mitmproxy could theoretically be taught how to read and understand QMP traffic, but that's not something I know how to do or would be prepared to mentor.) Is this possible to do in a post-hoc fashion? Let's say you are using production environment QEMU, how do we attach the QMP listener to it? Or does this idea require that we start QEMU in a specific fashion with a second debug socket that qmp-shell can connect to in order to listen? ... Or do we engineer qmp-shell to open its own socket that libvirt connects to ...? > Stefan > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-13 18:59 ` qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) John Snow @ 2021-01-14 13:52 ` Stefan Hajnoczi 2021-01-14 13:59 ` Daniel P. Berrangé 2021-01-14 17:48 ` John Snow 0 siblings, 2 replies; 27+ messages in thread From: Stefan Hajnoczi @ 2021-01-14 13:52 UTC (permalink / raw) To: John Snow; +Cc: libvir-list, Stefan Hajnoczi, QEMU Developers, Eduardo Habkost [-- Attachment #1: Type: text/plain, Size: 3018 bytes --] On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > when libvirt is directly connected to the monitor. > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > works. > > (Actually, mitmproxy could theoretically be taught how to read and > understand QMP traffic, but that's not something I know how to do or would > be prepared to mentor.) > > Is this possible to do in a post-hoc fashion? Let's say you are using > production environment QEMU, how do we attach the QMP listener to it? Or > does this idea require that we start QEMU in a specific fashion with a > second debug socket that qmp-shell can connect to in order to listen? > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > to ...? Here is the QEMU command-line that libvirt uses on my F33 system: -chardev socket,id=charmonitor,fd=36,server,nowait -mon chardev=charmonitor,id=monitor,mode=control Goals for this feature: 1. No manual steps required for setup. 2. Ability to start/stop monitoring traffic at runtime without restarting QEMU. 3. Available to unprivileged users. I think the easiest way to achieve this is through a new QEMU monitor command. Approaches that come to mind: 1. Add a -mon debug-chardev property and a QMP command to set it at runtime. The debug-chardev receives both monitor input (commands) and output (responses and events). This does not allow MITM, rather it mirrors traffic. 2. Add a chardev-get-fd command that fetches the fd from a chardev and then use the existing chardev-change command to replace the monitor chardev with a chardev connected to qmp-shell. This inserts qmp-shell as a proxy between the QMP client and server. qmp-shell can remove itself again with another chardev-change command. This approach allows MITM. The downside is it assumes the QMP chardev is a file descriptor, so it won't work with all types of chardev. 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin source chardev, 2. a monitoring sink chardev, and 3. a monitoring source chardev. The data flow is origin <-> monitoring sink <-> monitoring source <-> QMP monitor. qmp-shell creates the monitoring sink (for receiving incoming QMP commands) and monitoring source chardev (for forwarding QMP commands or MITM commands), and then it uses change-chardev to instantiate a chardev-proxy that directs the original libvirt chardev through the monitoring sink and source. This is the most complex but also completely contained within the QEMU chardev layer. In all these approaches qmp-shell uses virsh qemu-monitor-command or an equivalent API to start/stop monitoring a running VM without manual setup steps. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-14 13:52 ` Stefan Hajnoczi @ 2021-01-14 13:59 ` Daniel P. Berrangé 2021-01-14 15:02 ` Kevin Wolf 2021-01-14 17:48 ` John Snow 1 sibling, 1 reply; 27+ messages in thread From: Daniel P. Berrangé @ 2021-01-14 13:59 UTC (permalink / raw) To: Stefan Hajnoczi Cc: libvir-list, Stefan Hajnoczi, John Snow, QEMU Developers, Eduardo Habkost On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote: > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > when libvirt is directly connected to the monitor. > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > works. > > > > (Actually, mitmproxy could theoretically be taught how to read and > > understand QMP traffic, but that's not something I know how to do or would > > be prepared to mentor.) > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > production environment QEMU, how do we attach the QMP listener to it? Or > > does this idea require that we start QEMU in a specific fashion with a > > second debug socket that qmp-shell can connect to in order to listen? > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > > to ...? > > Here is the QEMU command-line that libvirt uses on my F33 system: > > -chardev socket,id=charmonitor,fd=36,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control > > Goals for this feature: > > 1. No manual steps required for setup. > 2. Ability to start/stop monitoring traffic at runtime without > restarting QEMU. > 3. Available to unprivileged users. > > I think the easiest way to achieve this is through a new QEMU monitor > command. Approaches that come to mind: > > 1. Add a -mon debug-chardev property and a QMP command to set it at > runtime. The debug-chardev receives both monitor input (commands) and > output (responses and events). This does not allow MITM, rather it > mirrors traffic. > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > then use the existing chardev-change command to replace the monitor > chardev with a chardev connected to qmp-shell. This inserts qmp-shell > as a proxy between the QMP client and server. qmp-shell can remove > itself again with another chardev-change command. This approach > allows MITM. The downside is it assumes the QMP chardev is a file > descriptor, so it won't work with all types of chardev. > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > source chardev, 2. a monitoring sink chardev, and 3. a monitoring > source chardev. The data flow is origin <-> monitoring sink <-> > monitoring source <-> QMP monitor. qmp-shell creates the monitoring > sink (for receiving incoming QMP commands) and monitoring source > chardev (for forwarding QMP commands or MITM commands), and then it > uses change-chardev to instantiate a chardev-proxy that directs the > original libvirt chardev through the monitoring sink and source. > > This is the most complex but also completely contained within the > QEMU chardev layer. > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > equivalent API to start/stop monitoring a running VM without manual > setup steps. Why go to the trouble of adding more chardevs to a running QEMU that libvirt has. qmp-shell can just directly use the libvirt Python API to invoke virDomainQemuMonitorCommand to invoke QMP commands, and the othe API for receiving QMP events. Essentially it just needs to be split into two layers. The upper layer works in terms of individual QMP command/replies, and QMP events. The lower layer provides a transport that is either a UNIX socket, or is the libvirt QMP passthrough API. Or alternatively, provide a virt-qmp-shim command that listens on a UNIX socket, accepts QMP commands and turns them into calls to virDomainQemuMonitorCommand, and funnells back the response. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-14 13:59 ` Daniel P. Berrangé @ 2021-01-14 15:02 ` Kevin Wolf 2021-01-14 15:22 ` Daniel P. Berrangé 2021-01-14 16:48 ` Stefan Hajnoczi 0 siblings, 2 replies; 27+ messages in thread From: Kevin Wolf @ 2021-01-14 15:02 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Eduardo Habkost, libvir-list, Stefan Hajnoczi, QEMU Developers, Stefan Hajnoczi, John Snow Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote: > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > > works. > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > understand QMP traffic, but that's not something I know how to do or would > > > be prepared to mentor.) > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > production environment QEMU, how do we attach the QMP listener to it? Or > > > does this idea require that we start QEMU in a specific fashion with a > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > > > to ...? > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > -mon chardev=charmonitor,id=monitor,mode=control > > > > Goals for this feature: > > > > 1. No manual steps required for setup. > > 2. Ability to start/stop monitoring traffic at runtime without > > restarting QEMU. > > 3. Available to unprivileged users. > > > > I think the easiest way to achieve this is through a new QEMU monitor > > command. Approaches that come to mind: > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > runtime. The debug-chardev receives both monitor input (commands) and > > output (responses and events). This does not allow MITM, rather it > > mirrors traffic. > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > then use the existing chardev-change command to replace the monitor > > chardev with a chardev connected to qmp-shell. This inserts qmp-shell > > as a proxy between the QMP client and server. qmp-shell can remove > > itself again with another chardev-change command. This approach > > allows MITM. The downside is it assumes the QMP chardev is a file > > descriptor, so it won't work with all types of chardev. > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > > source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > source chardev. The data flow is origin <-> monitoring sink <-> > > monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > sink (for receiving incoming QMP commands) and monitoring source > > chardev (for forwarding QMP commands or MITM commands), and then it > > uses change-chardev to instantiate a chardev-proxy that directs the > > original libvirt chardev through the monitoring sink and source. > > > > This is the most complex but also completely contained within the > > QEMU chardev layer. I have an idea for the QMP command name: chardev-snapshot-sync! Finally we get backing file chains for chardevs! :-) > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > equivalent API to start/stop monitoring a running VM without manual > > setup steps. > > Why go to the trouble of adding more chardevs to a running QEMU that > libvirt has. qmp-shell can just directly use the libvirt Python API > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > the othe API for receiving QMP events. > > Essentially it just needs to be split into two layers. The upper > layer works in terms of individual QMP command/replies, and QMP > events. The lower layer provides a transport that is either a > UNIX socket, or is the libvirt QMP passthrough API. > > Or alternatively, provide a virt-qmp-shim command that listens on > a UNIX socket, accepts QMP commands and turns them into calls to > virDomainQemuMonitorCommand, and funnells back the response. I think the idea was to show the QMP traffic that libvirt produces for other management applications, not for the QMP shell. These APIs probably don't allow this? Kevin ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-14 15:02 ` Kevin Wolf @ 2021-01-14 15:22 ` Daniel P. Berrangé 2021-01-14 16:49 ` Stefan Hajnoczi 2021-01-14 16:48 ` Stefan Hajnoczi 1 sibling, 1 reply; 27+ messages in thread From: Daniel P. Berrangé @ 2021-01-14 15:22 UTC (permalink / raw) To: Kevin Wolf Cc: Eduardo Habkost, libvir-list, Stefan Hajnoczi, QEMU Developers, Stefan Hajnoczi, John Snow On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote: > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > > > works. > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > understand QMP traffic, but that's not something I know how to do or would > > > > be prepared to mentor.) > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > > production environment QEMU, how do we attach the QMP listener to it? Or > > > > does this idea require that we start QEMU in a specific fashion with a > > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > > > > to ...? > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > Goals for this feature: > > > > > > 1. No manual steps required for setup. > > > 2. Ability to start/stop monitoring traffic at runtime without > > > restarting QEMU. > > > 3. Available to unprivileged users. > > > > > > I think the easiest way to achieve this is through a new QEMU monitor > > > command. Approaches that come to mind: > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > > runtime. The debug-chardev receives both monitor input (commands) and > > > output (responses and events). This does not allow MITM, rather it > > > mirrors traffic. > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > > then use the existing chardev-change command to replace the monitor > > > chardev with a chardev connected to qmp-shell. This inserts qmp-shell > > > as a proxy between the QMP client and server. qmp-shell can remove > > > itself again with another chardev-change command. This approach > > > allows MITM. The downside is it assumes the QMP chardev is a file > > > descriptor, so it won't work with all types of chardev. > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > > > source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > > source chardev. The data flow is origin <-> monitoring sink <-> > > > monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > > sink (for receiving incoming QMP commands) and monitoring source > > > chardev (for forwarding QMP commands or MITM commands), and then it > > > uses change-chardev to instantiate a chardev-proxy that directs the > > > original libvirt chardev through the monitoring sink and source. > > > > > > This is the most complex but also completely contained within the > > > QEMU chardev layer. > > I have an idea for the QMP command name: chardev-snapshot-sync! > > Finally we get backing file chains for chardevs! :-) > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > > equivalent API to start/stop monitoring a running VM without manual > > > setup steps. > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > libvirt has. qmp-shell can just directly use the libvirt Python API > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > the othe API for receiving QMP events. > > > > Essentially it just needs to be split into two layers. The upper > > layer works in terms of individual QMP command/replies, and QMP > > events. The lower layer provides a transport that is either a > > UNIX socket, or is the libvirt QMP passthrough API. > > > > Or alternatively, provide a virt-qmp-shim command that listens on > > a UNIX socket, accepts QMP commands and turns them into calls to > > virDomainQemuMonitorCommand, and funnells back the response. > > I think the idea was to show the QMP traffic that libvirt produces for > other management applications, not for the QMP shell. These APIs > probably don't allow this? FWIW if you want to monitor what libvirt is sending/receiving we have a script for that that uses our systemtap probe points: https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-14 15:22 ` Daniel P. Berrangé @ 2021-01-14 16:49 ` Stefan Hajnoczi 2021-01-14 16:55 ` Daniel P. Berrangé 0 siblings, 1 reply; 27+ messages in thread From: Stefan Hajnoczi @ 2021-01-14 16:49 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Kevin Wolf, Eduardo Habkost, libvir-list, Stefan Hajnoczi, QEMU Developers, John Snow [-- Attachment #1: Type: text/plain, Size: 5163 bytes --] On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote: > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote: > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > > > > works. > > > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > > understand QMP traffic, but that's not something I know how to do or would > > > > > be prepared to mentor.) > > > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > > > production environment QEMU, how do we attach the QMP listener to it? Or > > > > > does this idea require that we start QEMU in a specific fashion with a > > > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > > > > > to ...? > > > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > > > Goals for this feature: > > > > > > > > 1. No manual steps required for setup. > > > > 2. Ability to start/stop monitoring traffic at runtime without > > > > restarting QEMU. > > > > 3. Available to unprivileged users. > > > > > > > > I think the easiest way to achieve this is through a new QEMU monitor > > > > command. Approaches that come to mind: > > > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > > > runtime. The debug-chardev receives both monitor input (commands) and > > > > output (responses and events). This does not allow MITM, rather it > > > > mirrors traffic. > > > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > > > then use the existing chardev-change command to replace the monitor > > > > chardev with a chardev connected to qmp-shell. This inserts qmp-shell > > > > as a proxy between the QMP client and server. qmp-shell can remove > > > > itself again with another chardev-change command. This approach > > > > allows MITM. The downside is it assumes the QMP chardev is a file > > > > descriptor, so it won't work with all types of chardev. > > > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > > > > source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > > > source chardev. The data flow is origin <-> monitoring sink <-> > > > > monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > > > sink (for receiving incoming QMP commands) and monitoring source > > > > chardev (for forwarding QMP commands or MITM commands), and then it > > > > uses change-chardev to instantiate a chardev-proxy that directs the > > > > original libvirt chardev through the monitoring sink and source. > > > > > > > > This is the most complex but also completely contained within the > > > > QEMU chardev layer. > > > > I have an idea for the QMP command name: chardev-snapshot-sync! > > > > Finally we get backing file chains for chardevs! :-) > > > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > > > equivalent API to start/stop monitoring a running VM without manual > > > > setup steps. > > > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > > libvirt has. qmp-shell can just directly use the libvirt Python API > > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > > the othe API for receiving QMP events. > > > > > > Essentially it just needs to be split into two layers. The upper > > > layer works in terms of individual QMP command/replies, and QMP > > > events. The lower layer provides a transport that is either a > > > UNIX socket, or is the libvirt QMP passthrough API. > > > > > > Or alternatively, provide a virt-qmp-shim command that listens on > > > a UNIX socket, accepts QMP commands and turns them into calls to > > > virDomainQemuMonitorCommand, and funnells back the response. > > > > I think the idea was to show the QMP traffic that libvirt produces for > > other management applications, not for the QMP shell. These APIs > > probably don't allow this? > > FWIW if you want to monitor what libvirt is sending/receiving we have > a script for that that uses our systemtap probe points: > > https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp Does that require root? Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-14 16:49 ` Stefan Hajnoczi @ 2021-01-14 16:55 ` Daniel P. Berrangé 2021-01-14 17:14 ` Stefan Hajnoczi 0 siblings, 1 reply; 27+ messages in thread From: Daniel P. Berrangé @ 2021-01-14 16:55 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Kevin Wolf, Eduardo Habkost, libvir-list, Stefan Hajnoczi, QEMU Developers, John Snow On Thu, Jan 14, 2021 at 04:49:17PM +0000, Stefan Hajnoczi wrote: > On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote: > > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > > > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote: > > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > > > > > works. > > > > > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > > > understand QMP traffic, but that's not something I know how to do or would > > > > > > be prepared to mentor.) > > > > > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > > > > production environment QEMU, how do we attach the QMP listener to it? Or > > > > > > does this idea require that we start QEMU in a specific fashion with a > > > > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > > > > > > to ...? > > > > > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > > > > > Goals for this feature: > > > > > > > > > > 1. No manual steps required for setup. > > > > > 2. Ability to start/stop monitoring traffic at runtime without > > > > > restarting QEMU. > > > > > 3. Available to unprivileged users. > > > > > > > > > > I think the easiest way to achieve this is through a new QEMU monitor > > > > > command. Approaches that come to mind: > > > > > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > > > > runtime. The debug-chardev receives both monitor input (commands) and > > > > > output (responses and events). This does not allow MITM, rather it > > > > > mirrors traffic. > > > > > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > > > > then use the existing chardev-change command to replace the monitor > > > > > chardev with a chardev connected to qmp-shell. This inserts qmp-shell > > > > > as a proxy between the QMP client and server. qmp-shell can remove > > > > > itself again with another chardev-change command. This approach > > > > > allows MITM. The downside is it assumes the QMP chardev is a file > > > > > descriptor, so it won't work with all types of chardev. > > > > > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > > > > > source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > > > > source chardev. The data flow is origin <-> monitoring sink <-> > > > > > monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > > > > sink (for receiving incoming QMP commands) and monitoring source > > > > > chardev (for forwarding QMP commands or MITM commands), and then it > > > > > uses change-chardev to instantiate a chardev-proxy that directs the > > > > > original libvirt chardev through the monitoring sink and source. > > > > > > > > > > This is the most complex but also completely contained within the > > > > > QEMU chardev layer. > > > > > > I have an idea for the QMP command name: chardev-snapshot-sync! > > > > > > Finally we get backing file chains for chardevs! :-) > > > > > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > > > > equivalent API to start/stop monitoring a running VM without manual > > > > > setup steps. > > > > > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > > > libvirt has. qmp-shell can just directly use the libvirt Python API > > > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > > > the othe API for receiving QMP events. > > > > > > > > Essentially it just needs to be split into two layers. The upper > > > > layer works in terms of individual QMP command/replies, and QMP > > > > events. The lower layer provides a transport that is either a > > > > UNIX socket, or is the libvirt QMP passthrough API. > > > > > > > > Or alternatively, provide a virt-qmp-shim command that listens on > > > > a UNIX socket, accepts QMP commands and turns them into calls to > > > > virDomainQemuMonitorCommand, and funnells back the response. > > > > > > I think the idea was to show the QMP traffic that libvirt produces for > > > other management applications, not for the QMP shell. These APIs > > > probably don't allow this? > > > > FWIW if you want to monitor what libvirt is sending/receiving we have > > a script for that that uses our systemtap probe points: > > > > https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp > > Does that require root? Yeah, systemtap generally requires root. The same info is also written to the log files. For example: virt-admin daemon-log-filters "2:qemu_monitor_json" virt-admin daemon-lop-outputs "2:file:/var/log/libvirt/libvirtd.log" nb, i'm using level '2' there to avoid enabling debug logs, only info level logs which is the level dynamic probes log at. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-14 16:55 ` Daniel P. Berrangé @ 2021-01-14 17:14 ` Stefan Hajnoczi 2021-01-14 17:24 ` Daniel P. Berrangé 0 siblings, 1 reply; 27+ messages in thread From: Stefan Hajnoczi @ 2021-01-14 17:14 UTC (permalink / raw) To: Daniel P. Berrangé Cc: Kevin Wolf, Eduardo Habkost, libvir-list, Stefan Hajnoczi, QEMU Developers, John Snow [-- Attachment #1: Type: text/plain, Size: 6549 bytes --] On Thu, Jan 14, 2021 at 04:55:30PM +0000, Daniel P. Berrangé wrote: > On Thu, Jan 14, 2021 at 04:49:17PM +0000, Stefan Hajnoczi wrote: > > On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote: > > > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > > > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > > > > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote: > > > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > > > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > > > > > > works. > > > > > > > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > > > > understand QMP traffic, but that's not something I know how to do or would > > > > > > > be prepared to mentor.) > > > > > > > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > > > > > production environment QEMU, how do we attach the QMP listener to it? Or > > > > > > > does this idea require that we start QEMU in a specific fashion with a > > > > > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > > > > > > > to ...? > > > > > > > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > > > > > > > Goals for this feature: > > > > > > > > > > > > 1. No manual steps required for setup. > > > > > > 2. Ability to start/stop monitoring traffic at runtime without > > > > > > restarting QEMU. > > > > > > 3. Available to unprivileged users. > > > > > > > > > > > > I think the easiest way to achieve this is through a new QEMU monitor > > > > > > command. Approaches that come to mind: > > > > > > > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > > > > > runtime. The debug-chardev receives both monitor input (commands) and > > > > > > output (responses and events). This does not allow MITM, rather it > > > > > > mirrors traffic. > > > > > > > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > > > > > then use the existing chardev-change command to replace the monitor > > > > > > chardev with a chardev connected to qmp-shell. This inserts qmp-shell > > > > > > as a proxy between the QMP client and server. qmp-shell can remove > > > > > > itself again with another chardev-change command. This approach > > > > > > allows MITM. The downside is it assumes the QMP chardev is a file > > > > > > descriptor, so it won't work with all types of chardev. > > > > > > > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > > > > > > source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > > > > > source chardev. The data flow is origin <-> monitoring sink <-> > > > > > > monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > > > > > sink (for receiving incoming QMP commands) and monitoring source > > > > > > chardev (for forwarding QMP commands or MITM commands), and then it > > > > > > uses change-chardev to instantiate a chardev-proxy that directs the > > > > > > original libvirt chardev through the monitoring sink and source. > > > > > > > > > > > > This is the most complex but also completely contained within the > > > > > > QEMU chardev layer. > > > > > > > > I have an idea for the QMP command name: chardev-snapshot-sync! > > > > > > > > Finally we get backing file chains for chardevs! :-) > > > > > > > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > > > > > equivalent API to start/stop monitoring a running VM without manual > > > > > > setup steps. > > > > > > > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > > > > libvirt has. qmp-shell can just directly use the libvirt Python API > > > > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > > > > the othe API for receiving QMP events. > > > > > > > > > > Essentially it just needs to be split into two layers. The upper > > > > > layer works in terms of individual QMP command/replies, and QMP > > > > > events. The lower layer provides a transport that is either a > > > > > UNIX socket, or is the libvirt QMP passthrough API. > > > > > > > > > > Or alternatively, provide a virt-qmp-shim command that listens on > > > > > a UNIX socket, accepts QMP commands and turns them into calls to > > > > > virDomainQemuMonitorCommand, and funnells back the response. > > > > > > > > I think the idea was to show the QMP traffic that libvirt produces for > > > > other management applications, not for the QMP shell. These APIs > > > > probably don't allow this? > > > > > > FWIW if you want to monitor what libvirt is sending/receiving we have > > > a script for that that uses our systemtap probe points: > > > > > > https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp > > > > Does that require root? > > Yeah, systemtap generally requires root. > > The same info is also written to the log files. For example: > > virt-admin daemon-log-filters "2:qemu_monitor_json" > virt-admin daemon-lop-outputs "2:file:/var/log/libvirt/libvirtd.log" > > nb, i'm using level '2' there to avoid enabling debug logs, only > info level logs which is the level dynamic probes log at. On my F33 system /var/log/libvirt is owned by root:root and rwx------, so I guess it would be necessary to reconfigure log output so that unprivileged users can access it. If it can be used in conjunction with virDomainQemuMonitorCommand(), then that eliminates the need to introduce new chardev functionality in QEMU. Parsing libvirt logs was one of the things I suggested, though. I think it would be a nice feature for troubleshooting QMP conversations. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-14 17:14 ` Stefan Hajnoczi @ 2021-01-14 17:24 ` Daniel P. Berrangé 0 siblings, 0 replies; 27+ messages in thread From: Daniel P. Berrangé @ 2021-01-14 17:24 UTC (permalink / raw) To: Stefan Hajnoczi Cc: Kevin Wolf, Eduardo Habkost, libvir-list, Stefan Hajnoczi, QEMU Developers, John Snow On Thu, Jan 14, 2021 at 05:14:28PM +0000, Stefan Hajnoczi wrote: > On Thu, Jan 14, 2021 at 04:55:30PM +0000, Daniel P. Berrangé wrote: > > On Thu, Jan 14, 2021 at 04:49:17PM +0000, Stefan Hajnoczi wrote: > > > On Thu, Jan 14, 2021 at 03:22:41PM +0000, Daniel P. Berrangé wrote: > > > > On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > > > > > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > > > > > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote: > > > > > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > > > > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > > > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > > > > > > > works. > > > > > > > > > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > > > > > understand QMP traffic, but that's not something I know how to do or would > > > > > > > > be prepared to mentor.) > > > > > > > > > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > > > > > > production environment QEMU, how do we attach the QMP listener to it? Or > > > > > > > > does this idea require that we start QEMU in a specific fashion with a > > > > > > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > > > > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > > > > > > > > to ...? > > > > > > > > > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > > > > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > > > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > > > > > > > > > Goals for this feature: > > > > > > > > > > > > > > 1. No manual steps required for setup. > > > > > > > 2. Ability to start/stop monitoring traffic at runtime without > > > > > > > restarting QEMU. > > > > > > > 3. Available to unprivileged users. > > > > > > > > > > > > > > I think the easiest way to achieve this is through a new QEMU monitor > > > > > > > command. Approaches that come to mind: > > > > > > > > > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > > > > > > runtime. The debug-chardev receives both monitor input (commands) and > > > > > > > output (responses and events). This does not allow MITM, rather it > > > > > > > mirrors traffic. > > > > > > > > > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > > > > > > then use the existing chardev-change command to replace the monitor > > > > > > > chardev with a chardev connected to qmp-shell. This inserts qmp-shell > > > > > > > as a proxy between the QMP client and server. qmp-shell can remove > > > > > > > itself again with another chardev-change command. This approach > > > > > > > allows MITM. The downside is it assumes the QMP chardev is a file > > > > > > > descriptor, so it won't work with all types of chardev. > > > > > > > > > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > > > > > > > source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > > > > > > source chardev. The data flow is origin <-> monitoring sink <-> > > > > > > > monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > > > > > > sink (for receiving incoming QMP commands) and monitoring source > > > > > > > chardev (for forwarding QMP commands or MITM commands), and then it > > > > > > > uses change-chardev to instantiate a chardev-proxy that directs the > > > > > > > original libvirt chardev through the monitoring sink and source. > > > > > > > > > > > > > > This is the most complex but also completely contained within the > > > > > > > QEMU chardev layer. > > > > > > > > > > I have an idea for the QMP command name: chardev-snapshot-sync! > > > > > > > > > > Finally we get backing file chains for chardevs! :-) > > > > > > > > > > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > > > > > > equivalent API to start/stop monitoring a running VM without manual > > > > > > > setup steps. > > > > > > > > > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > > > > > libvirt has. qmp-shell can just directly use the libvirt Python API > > > > > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > > > > > the othe API for receiving QMP events. > > > > > > > > > > > > Essentially it just needs to be split into two layers. The upper > > > > > > layer works in terms of individual QMP command/replies, and QMP > > > > > > events. The lower layer provides a transport that is either a > > > > > > UNIX socket, or is the libvirt QMP passthrough API. > > > > > > > > > > > > Or alternatively, provide a virt-qmp-shim command that listens on > > > > > > a UNIX socket, accepts QMP commands and turns them into calls to > > > > > > virDomainQemuMonitorCommand, and funnells back the response. > > > > > > > > > > I think the idea was to show the QMP traffic that libvirt produces for > > > > > other management applications, not for the QMP shell. These APIs > > > > > probably don't allow this? > > > > > > > > FWIW if you want to monitor what libvirt is sending/receiving we have > > > > a script for that that uses our systemtap probe points: > > > > > > > > https://gitlab.com/libvirt/libvirt/-/blob/master/examples/systemtap/qemu-monitor.stp > > > > > > Does that require root? > > > > Yeah, systemtap generally requires root. > > > > The same info is also written to the log files. For example: > > > > virt-admin daemon-log-filters "2:qemu_monitor_json" > > virt-admin daemon-lop-outputs "2:file:/var/log/libvirt/libvirtd.log" > > > > nb, i'm using level '2' there to avoid enabling debug logs, only > > info level logs which is the level dynamic probes log at. > > On my F33 system /var/log/libvirt is owned by root:root and rwx------, > so I guess it would be necessary to reconfigure log output so that > unprivileged users can access it. If you're connected to the privileged libvirtd then you have effectively got root privileges, so that shouldn't be an issue. If you're connecting to the unprivileged libvirtd, then you would tell it to use $HOME/.cache/libvirt/libvirtd.log instead. > If it can be used in conjunction with virDomainQemuMonitorCommand(), > then that eliminates the need to introduce new chardev functionality in > QEMU. > > Parsing libvirt logs was one of the things I suggested, though. I think > it would be a nice feature for troubleshooting QMP conversations. Regards, Daniel -- |: https://berrange.com -o- https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o- https://fstop138.berrange.com :| |: https://entangle-photo.org -o- https://www.instagram.com/dberrange :| ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-14 15:02 ` Kevin Wolf 2021-01-14 15:22 ` Daniel P. Berrangé @ 2021-01-14 16:48 ` Stefan Hajnoczi 1 sibling, 0 replies; 27+ messages in thread From: Stefan Hajnoczi @ 2021-01-14 16:48 UTC (permalink / raw) To: Kevin Wolf Cc: Daniel P. Berrangé, Eduardo Habkost, libvir-list, Stefan Hajnoczi, QEMU Developers, John Snow [-- Attachment #1: Type: text/plain, Size: 4716 bytes --] On Thu, Jan 14, 2021 at 04:02:56PM +0100, Kevin Wolf wrote: > Am 14.01.2021 um 14:59 hat Daniel P. Berrangé geschrieben: > > On Thu, Jan 14, 2021 at 01:52:34PM +0000, Stefan Hajnoczi wrote: > > > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: > > > > On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: > > > > > On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: > > > > > 2. Ability to watch QMP activity on a running QEMU process, e.g. even > > > > > when libvirt is directly connected to the monitor. > > > > > > > > > > > > > That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy > > > > works. > > > > > > > > (Actually, mitmproxy could theoretically be taught how to read and > > > > understand QMP traffic, but that's not something I know how to do or would > > > > be prepared to mentor.) > > > > > > > > Is this possible to do in a post-hoc fashion? Let's say you are using > > > > production environment QEMU, how do we attach the QMP listener to it? Or > > > > does this idea require that we start QEMU in a specific fashion with a > > > > second debug socket that qmp-shell can connect to in order to listen? > > > > > > > > ... Or do we engineer qmp-shell to open its own socket that libvirt connects > > > > to ...? > > > > > > Here is the QEMU command-line that libvirt uses on my F33 system: > > > > > > -chardev socket,id=charmonitor,fd=36,server,nowait > > > -mon chardev=charmonitor,id=monitor,mode=control > > > > > > Goals for this feature: > > > > > > 1. No manual steps required for setup. > > > 2. Ability to start/stop monitoring traffic at runtime without > > > restarting QEMU. > > > 3. Available to unprivileged users. > > > > > > I think the easiest way to achieve this is through a new QEMU monitor > > > command. Approaches that come to mind: > > > > > > 1. Add a -mon debug-chardev property and a QMP command to set it at > > > runtime. The debug-chardev receives both monitor input (commands) and > > > output (responses and events). This does not allow MITM, rather it > > > mirrors traffic. > > > > > > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > > > then use the existing chardev-change command to replace the monitor > > > chardev with a chardev connected to qmp-shell. This inserts qmp-shell > > > as a proxy between the QMP client and server. qmp-shell can remove > > > itself again with another chardev-change command. This approach > > > allows MITM. The downside is it assumes the QMP chardev is a file > > > descriptor, so it won't work with all types of chardev. > > > > > > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > > > source chardev, 2. a monitoring sink chardev, and 3. a monitoring > > > source chardev. The data flow is origin <-> monitoring sink <-> > > > monitoring source <-> QMP monitor. qmp-shell creates the monitoring > > > sink (for receiving incoming QMP commands) and monitoring source > > > chardev (for forwarding QMP commands or MITM commands), and then it > > > uses change-chardev to instantiate a chardev-proxy that directs the > > > original libvirt chardev through the monitoring sink and source. > > > > > > This is the most complex but also completely contained within the > > > QEMU chardev layer. > > I have an idea for the QMP command name: chardev-snapshot-sync! > > Finally we get backing file chains for chardevs! :-) Help mom, I'm being made fun of on qemu-devel! :) > > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > > > equivalent API to start/stop monitoring a running VM without manual > > > setup steps. > > > > Why go to the trouble of adding more chardevs to a running QEMU that > > libvirt has. qmp-shell can just directly use the libvirt Python API > > to invoke virDomainQemuMonitorCommand to invoke QMP commands, and > > the othe API for receiving QMP events. > > > > Essentially it just needs to be split into two layers. The upper > > layer works in terms of individual QMP command/replies, and QMP > > events. The lower layer provides a transport that is either a > > UNIX socket, or is the libvirt QMP passthrough API. > > > > Or alternatively, provide a virt-qmp-shim command that listens on > > a UNIX socket, accepts QMP commands and turns them into calls to > > virDomainQemuMonitorCommand, and funnells back the response. > > I think the idea was to show the QMP traffic that libvirt produces for > other management applications, not for the QMP shell. These APIs > probably don't allow this? Yes. Stefan [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 488 bytes --] ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) 2021-01-14 13:52 ` Stefan Hajnoczi 2021-01-14 13:59 ` Daniel P. Berrangé @ 2021-01-14 17:48 ` John Snow 1 sibling, 0 replies; 27+ messages in thread From: John Snow @ 2021-01-14 17:48 UTC (permalink / raw) To: Stefan Hajnoczi Cc: libvir-list, Stefan Hajnoczi, QEMU Developers, Eduardo Habkost On 1/14/21 8:52 AM, Stefan Hajnoczi wrote: > On Wed, Jan 13, 2021 at 01:59:43PM -0500, John Snow wrote: >> On 1/13/21 3:53 AM, Stefan Hajnoczi wrote: >>> On Tue, Jan 12, 2021 at 9:10 PM John Snow <jsnow@redhat.com> wrote: >>> 2. Ability to watch QMP activity on a running QEMU process, e.g. even >>> when libvirt is directly connected to the monitor. >>> >> >> That *WOULD* be extremely cool, and moves a lot closer to how mitmproxy >> works. >> >> (Actually, mitmproxy could theoretically be taught how to read and >> understand QMP traffic, but that's not something I know how to do or would >> be prepared to mentor.) >> >> Is this possible to do in a post-hoc fashion? Let's say you are using >> production environment QEMU, how do we attach the QMP listener to it? Or >> does this idea require that we start QEMU in a specific fashion with a >> second debug socket that qmp-shell can connect to in order to listen? >> >> ... Or do we engineer qmp-shell to open its own socket that libvirt connects >> to ...? > > Here is the QEMU command-line that libvirt uses on my F33 system: > > -chardev socket,id=charmonitor,fd=36,server,nowait > -mon chardev=charmonitor,id=monitor,mode=control > > Goals for this feature: > > 1. No manual steps required for setup. > 2. Ability to start/stop monitoring traffic at runtime without > restarting QEMU. > 3. Available to unprivileged users. > Excellent goals, and I agree completely. > I think the easiest way to achieve this is through a new QEMU monitor > command. Approaches that come to mind: > > 1. Add a -mon debug-chardev property and a QMP command to set it at > runtime. The debug-chardev receives both monitor input (commands) and > output (responses and events). This does not allow MITM, rather it > mirrors traffic. > So you have a socket that relays I/O. I wonder if it needs to modify the stream format to some extent to annotate directionality? For now, directionality can be inferred, but maybe that's brittle. (greeting messages, events and return statements are from the server; negotiation and execute statements are from the client.) Maybe if we used a hypothetical qmp-shell log format, we could add timestamps here instead of relying on the client to produce them. This might be interesting for analyzing race conditions and measuring response delays as experienced by the server. {"message": original_json_message_here, "direction": "in", "timestamp": 1610627721} (Downside: JSON is still not a streaming message format, but I guess it's one we already use all over the place anyway.) > 2. Add a chardev-get-fd command that fetches the fd from a chardev and > then use the existing chardev-change command to replace the monitor > chardev with a chardev connected to qmp-shell. This inserts qmp-shell > as a proxy between the QMP client and server. qmp-shell can remove > itself again with another chardev-change command. This approach > allows MITM. The downside is it assumes the QMP chardev is a file > descriptor, so it won't work with all types of chardev. > It seems a little more prone to failure if the insertion/removal fails, and has some downsides about which configurations it can inject into. > 3. Add a new chardev-proxy type that aggregates 3 chardevs: 1. an origin > source chardev, 2. a monitoring sink chardev, and 3. a monitoring > source chardev. The data flow is origin <-> monitoring sink <-> > monitoring source <-> QMP monitor. qmp-shell creates the monitoring > sink (for receiving incoming QMP commands) and monitoring source > chardev (for forwarding QMP commands or MITM commands), and then it > uses change-chardev to instantiate a chardev-proxy that directs the > original libvirt chardev through the monitoring sink and source. > I'm not sure I understand the topology here, exactly. I could stand to be a little more familiar with how chardevs are modeled in QEMU ... > This is the most complex but also completely contained within the > QEMU chardev layer. > > In all these approaches qmp-shell uses virsh qemu-monitor-command or an > equivalent API to start/stop monitoring a running VM without manual > setup steps. > Gotcha. I think I am leaning towards the first suggestion, but maybe the third one that I don't quite grasp yet is good too. > Stefan > ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Call for Google Summer of Code 2021 project ideas 2021-01-12 21:10 ` John Snow 2021-01-13 8:53 ` Stefan Hajnoczi @ 2021-01-13 9:19 ` Markus Armbruster 2021-01-13 19:05 ` John Snow 1 sibling, 1 reply; 27+ messages in thread From: Markus Armbruster @ 2021-01-13 9:19 UTC (permalink / raw) To: John Snow Cc: Stefan Hajnoczi, QEMU Developers, Stefan Hajnoczi, Eduardo Habkost John Snow <jsnow@redhat.com> writes: > On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: >> Dear QEMU, KVM, and rust-vmm community, >> QEMU will apply for Google Summer of Code >> (https://summerofcode.withgoogle.com/) again this year. This internship >> program offers paid, 10-week, remote work internships for >> contributing to open source. QEMU can act as an umbrella organization >> for KVM kernel and rust-vmm projects too. >> Please post project ideas on the QEMU wiki before February 14th: >> https://wiki.qemu.org/Google_Summer_of_Code_2021 >> What's new this year: >> * The number of internship hours has been halved to 175 hours over >> 10 weeks. Project ideas must be smaller to fit and students will have >> more flexibility with their working hours. >> * Eligibility has been expanded to include "licensed coding school or >> similar type of program". >> Good project ideas are suitable for 175 hours (10 weeks half-day) >> work by a >> competent programmer who is not yet familiar with the codebase. In >> addition, they are: >> * Well-defined - the scope is clear >> * Self-contained - there are few dependencies >> * Uncontroversial - they are acceptable to the community >> * Incremental - they produce deliverables along the way >> Feel free to post ideas even if you are unable to mentor the >> project. >> It doesn't hurt to share the idea! >> > > I have one that is probably way too ambitious, but requires a > particular skillset that might be of good interest to a student that > has some experience in the area already. > > The idea is for a TUI qmp-shell (maybe using urwid?) to create an > irssi-like REPL interface for QMP. The idea would be to mimic the > mitmproxy TUI interface (Check it out if you haven't!) > > All the ideas below are extremely tentative to give a basic gist of > what I mean; exact layouts/hotkeys/etc are for the sake of explanation > only. > > Essentially, I want an interface like this: > > ----------------------------------------------------------- > | QMP Mode | > |=========================================================| > | | > | Welcome to the Qemu Machine Protocol shell. | > | Please type /help or Ctrl+H to see available commands. | > | | > | | > | | > |---------------------------------------------------------| > | > | > ----------------------------------------------------------- > > commands are entered in the bottom and appear in a log window above, > appearing most-recent last, like irssi works. > > As an example, let's say we issue block-dirty-bitmap-add: > > -------------------------------------------------------- > | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | > -------------------------------------------------------- > > (...syntax up for debate...! We want something easy to parse, but easy > to type. For structured data, that's Hard.) "Hard" is a red flag for such a brief internship. We need to solve or somehow exclude this design problem before we hand the task to the student. [...] ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Call for Google Summer of Code 2021 project ideas 2021-01-13 9:19 ` Call for Google Summer of Code 2021 project ideas Markus Armbruster @ 2021-01-13 19:05 ` John Snow 2021-01-14 12:29 ` Markus Armbruster 0 siblings, 1 reply; 27+ messages in thread From: John Snow @ 2021-01-13 19:05 UTC (permalink / raw) To: Markus Armbruster Cc: Stefan Hajnoczi, QEMU Developers, Stefan Hajnoczi, Eduardo Habkost On 1/13/21 4:19 AM, Markus Armbruster wrote: > John Snow <jsnow@redhat.com> writes: > >> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: >>> Dear QEMU, KVM, and rust-vmm community, >>> QEMU will apply for Google Summer of Code >>> (https://summerofcode.withgoogle.com/) again this year. This internship >>> program offers paid, 10-week, remote work internships for >>> contributing to open source. QEMU can act as an umbrella organization >>> for KVM kernel and rust-vmm projects too. >>> Please post project ideas on the QEMU wiki before February 14th: >>> https://wiki.qemu.org/Google_Summer_of_Code_2021 >>> What's new this year: >>> * The number of internship hours has been halved to 175 hours over >>> 10 weeks. Project ideas must be smaller to fit and students will have >>> more flexibility with their working hours. >>> * Eligibility has been expanded to include "licensed coding school or >>> similar type of program". >>> Good project ideas are suitable for 175 hours (10 weeks half-day) >>> work by a >>> competent programmer who is not yet familiar with the codebase. In >>> addition, they are: >>> * Well-defined - the scope is clear >>> * Self-contained - there are few dependencies >>> * Uncontroversial - they are acceptable to the community >>> * Incremental - they produce deliverables along the way >>> Feel free to post ideas even if you are unable to mentor the >>> project. >>> It doesn't hurt to share the idea! >>> >> >> I have one that is probably way too ambitious, but requires a >> particular skillset that might be of good interest to a student that >> has some experience in the area already. >> >> The idea is for a TUI qmp-shell (maybe using urwid?) to create an >> irssi-like REPL interface for QMP. The idea would be to mimic the >> mitmproxy TUI interface (Check it out if you haven't!) >> >> All the ideas below are extremely tentative to give a basic gist of >> what I mean; exact layouts/hotkeys/etc are for the sake of explanation >> only. >> >> Essentially, I want an interface like this: >> >> ----------------------------------------------------------- >> | QMP Mode | >> |=========================================================| >> | | >> | Welcome to the Qemu Machine Protocol shell. | >> | Please type /help or Ctrl+H to see available commands. | >> | | >> | | >> | | >> |---------------------------------------------------------| >> | > | >> ----------------------------------------------------------- >> >> commands are entered in the bottom and appear in a log window above, >> appearing most-recent last, like irssi works. >> >> As an example, let's say we issue block-dirty-bitmap-add: >> >> -------------------------------------------------------- >> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | >> -------------------------------------------------------- >> >> (...syntax up for debate...! We want something easy to parse, but easy >> to type. For structured data, that's Hard.) > > "Hard" is a red flag for such a brief internship. We need to solve or > somehow exclude this design problem before we hand the task to the > student. > > [...] > Yes, there is a reason I capitalized "Hard". :~) This task CAN be split out, though. For starters, we can use an extremely simplified parsing strategy while we work on the REPL and and UI elements. We can accept things of the form: command arg=value arg2=value and we can worry about how to input structured data later. I don't think it's a blocker to have someone work on the TUI and asynchronous dispatch elements. I think even just keeping our current parsing but adding some of the features outlined in the proposal would be a big usability win. --js ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Call for Google Summer of Code 2021 project ideas 2021-01-13 19:05 ` John Snow @ 2021-01-14 12:29 ` Markus Armbruster 2021-01-14 14:57 ` Kevin Wolf 2021-01-14 16:36 ` John Snow 0 siblings, 2 replies; 27+ messages in thread From: Markus Armbruster @ 2021-01-14 12:29 UTC (permalink / raw) To: John Snow Cc: Stefan Hajnoczi, QEMU Developers, Stefan Hajnoczi, Eduardo Habkost John Snow <jsnow@redhat.com> writes: > On 1/13/21 4:19 AM, Markus Armbruster wrote: >> John Snow <jsnow@redhat.com> writes: >> >>> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: >>>> Dear QEMU, KVM, and rust-vmm community, >>>> QEMU will apply for Google Summer of Code >>>> (https://summerofcode.withgoogle.com/) again this year. This internship >>>> program offers paid, 10-week, remote work internships for >>>> contributing to open source. QEMU can act as an umbrella organization >>>> for KVM kernel and rust-vmm projects too. >>>> Please post project ideas on the QEMU wiki before February 14th: >>>> https://wiki.qemu.org/Google_Summer_of_Code_2021 >>>> What's new this year: >>>> * The number of internship hours has been halved to 175 hours over >>>> 10 weeks. Project ideas must be smaller to fit and students will have >>>> more flexibility with their working hours. >>>> * Eligibility has been expanded to include "licensed coding school or >>>> similar type of program". >>>> Good project ideas are suitable for 175 hours (10 weeks half-day) >>>> work by a >>>> competent programmer who is not yet familiar with the codebase. In >>>> addition, they are: >>>> * Well-defined - the scope is clear >>>> * Self-contained - there are few dependencies >>>> * Uncontroversial - they are acceptable to the community >>>> * Incremental - they produce deliverables along the way >>>> Feel free to post ideas even if you are unable to mentor the >>>> project. >>>> It doesn't hurt to share the idea! >>>> >>> >>> I have one that is probably way too ambitious, but requires a >>> particular skillset that might be of good interest to a student that >>> has some experience in the area already. >>> >>> The idea is for a TUI qmp-shell (maybe using urwid?) to create an >>> irssi-like REPL interface for QMP. The idea would be to mimic the >>> mitmproxy TUI interface (Check it out if you haven't!) >>> >>> All the ideas below are extremely tentative to give a basic gist of >>> what I mean; exact layouts/hotkeys/etc are for the sake of explanation >>> only. >>> >>> Essentially, I want an interface like this: >>> >>> ----------------------------------------------------------- >>> | QMP Mode | >>> |=========================================================| >>> | | >>> | Welcome to the Qemu Machine Protocol shell. | >>> | Please type /help or Ctrl+H to see available commands. | >>> | | >>> | | >>> | | >>> |---------------------------------------------------------| >>> | > | >>> ----------------------------------------------------------- >>> >>> commands are entered in the bottom and appear in a log window above, >>> appearing most-recent last, like irssi works. >>> >>> As an example, let's say we issue block-dirty-bitmap-add: >>> >>> -------------------------------------------------------- >>> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | >>> -------------------------------------------------------- >>> >>> (...syntax up for debate...! We want something easy to parse, but easy >>> to type. For structured data, that's Hard.) >> "Hard" is a red flag for such a brief internship. We need to solve >> or >> somehow exclude this design problem before we hand the task to the >> student. >> [...] >> > > Yes, there is a reason I capitalized "Hard". :~) > > This task CAN be split out, though. For starters, we can use an > extremely simplified parsing strategy while we work on the REPL and > and UI elements. > > We can accept things of the form: > command arg=value arg2=value > > and we can worry about how to input structured data later. Bzzzt! You just designed a language. The fact that it is obviously inadequate for the task does not make this not language design. Two outcomes: 1. The project fails for other reasons. The language doesn't matter. 2. The project succeeds, i.e. it becomes useful to people. Very quickly these people will demand you evolve your inadequate language into a less inadequate one, compatibly. Your evasion of the language design problem (because hard) has now added accidental constraints, making the design problem even harder. The sane way to evade the language design problem is to use the existing QMP language. The project then aims to build a tool that adds useful features over "socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>" UNIX-CONNECT:/path/to/socket". If it succeeds, you can still design and implement a "better" language, and let users choose the one they prefer. Or you could add features to help with typing QMP. > I don't > think it's a blocker to have someone work on the TUI and asynchronous > dispatch elements. I think even just keeping our current parsing but > adding some of the features outlined in the proposal would be a big > usability win. I don't feel this particular itch, but I'm certainly not objecting to anyone scratching. ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Call for Google Summer of Code 2021 project ideas 2021-01-14 12:29 ` Markus Armbruster @ 2021-01-14 14:57 ` Kevin Wolf 2021-01-14 16:36 ` John Snow 1 sibling, 0 replies; 27+ messages in thread From: Kevin Wolf @ 2021-01-14 14:57 UTC (permalink / raw) To: Markus Armbruster Cc: Stefan Hajnoczi, John Snow, QEMU Developers, Stefan Hajnoczi, Eduardo Habkost Am 14.01.2021 um 13:29 hat Markus Armbruster geschrieben: > John Snow <jsnow@redhat.com> writes: > > > On 1/13/21 4:19 AM, Markus Armbruster wrote: > >> John Snow <jsnow@redhat.com> writes: > >> > >>> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: > >>>> Dear QEMU, KVM, and rust-vmm community, > >>>> QEMU will apply for Google Summer of Code > >>>> (https://summerofcode.withgoogle.com/) again this year. This internship > >>>> program offers paid, 10-week, remote work internships for > >>>> contributing to open source. QEMU can act as an umbrella organization > >>>> for KVM kernel and rust-vmm projects too. > >>>> Please post project ideas on the QEMU wiki before February 14th: > >>>> https://wiki.qemu.org/Google_Summer_of_Code_2021 > >>>> What's new this year: > >>>> * The number of internship hours has been halved to 175 hours over > >>>> 10 weeks. Project ideas must be smaller to fit and students will have > >>>> more flexibility with their working hours. > >>>> * Eligibility has been expanded to include "licensed coding school or > >>>> similar type of program". > >>>> Good project ideas are suitable for 175 hours (10 weeks half-day) > >>>> work by a > >>>> competent programmer who is not yet familiar with the codebase. In > >>>> addition, they are: > >>>> * Well-defined - the scope is clear > >>>> * Self-contained - there are few dependencies > >>>> * Uncontroversial - they are acceptable to the community > >>>> * Incremental - they produce deliverables along the way > >>>> Feel free to post ideas even if you are unable to mentor the > >>>> project. > >>>> It doesn't hurt to share the idea! > >>>> > >>> > >>> I have one that is probably way too ambitious, but requires a > >>> particular skillset that might be of good interest to a student that > >>> has some experience in the area already. > >>> > >>> The idea is for a TUI qmp-shell (maybe using urwid?) to create an > >>> irssi-like REPL interface for QMP. The idea would be to mimic the > >>> mitmproxy TUI interface (Check it out if you haven't!) > >>> > >>> All the ideas below are extremely tentative to give a basic gist of > >>> what I mean; exact layouts/hotkeys/etc are for the sake of explanation > >>> only. > >>> > >>> Essentially, I want an interface like this: > >>> > >>> ----------------------------------------------------------- > >>> | QMP Mode | > >>> |=========================================================| > >>> | | > >>> | Welcome to the Qemu Machine Protocol shell. | > >>> | Please type /help or Ctrl+H to see available commands. | > >>> | | > >>> | | > >>> | | > >>> |---------------------------------------------------------| > >>> | > | > >>> ----------------------------------------------------------- > >>> > >>> commands are entered in the bottom and appear in a log window above, > >>> appearing most-recent last, like irssi works. > >>> > >>> As an example, let's say we issue block-dirty-bitmap-add: > >>> > >>> -------------------------------------------------------- > >>> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | > >>> -------------------------------------------------------- > >>> > >>> (...syntax up for debate...! We want something easy to parse, but easy > >>> to type. For structured data, that's Hard.) > >> "Hard" is a red flag for such a brief internship. We need to solve > >> or > >> somehow exclude this design problem before we hand the task to the > >> student. > >> [...] > >> > > > > Yes, there is a reason I capitalized "Hard". :~) > > > > This task CAN be split out, though. For starters, we can use an > > extremely simplified parsing strategy while we work on the REPL and > > and UI elements. > > > > We can accept things of the form: > > command arg=value arg2=value > > > > and we can worry about how to input structured data later. > > Bzzzt! You just designed a language. The fact that it is obviously > inadequate for the task does not make this not language design. > > Two outcomes: > > 1. The project fails for other reasons. The language doesn't matter. > > 2. The project succeeds, i.e. it becomes useful to people. Very quickly > these people will demand you evolve your inadequate language into a less > inadequate one, compatibly. Your evasion of the language design problem > (because hard) has now added accidental constraints, making the design > problem even harder. We don't even promise compatibility for HMP, which is a part of the system emulator binary and has existed since I first used QEMU, which must have been around 0.6 times. If it's possible to simply say "no" to compatibility demands there, then surely an external shell with a similar target audience can do the same. I think it would be useful to have a raw QMP mode (especially so you can point people to that when they ask for compatibility), but it's probably not the focus for this project because you can already have raw QMP today. Kevin ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Call for Google Summer of Code 2021 project ideas 2021-01-14 12:29 ` Markus Armbruster 2021-01-14 14:57 ` Kevin Wolf @ 2021-01-14 16:36 ` John Snow 2021-01-15 16:31 ` Kashyap Chamarthy 2021-02-15 11:05 ` Paolo Bonzini 1 sibling, 2 replies; 27+ messages in thread From: John Snow @ 2021-01-14 16:36 UTC (permalink / raw) To: Markus Armbruster Cc: Stefan Hajnoczi, QEMU Developers, Stefan Hajnoczi, Eduardo Habkost On 1/14/21 7:29 AM, Markus Armbruster wrote: > John Snow <jsnow@redhat.com> writes: > >> On 1/13/21 4:19 AM, Markus Armbruster wrote: >>> John Snow <jsnow@redhat.com> writes: >>> >>>> On 1/11/21 6:47 AM, Stefan Hajnoczi wrote: >>>>> Dear QEMU, KVM, and rust-vmm community, >>>>> QEMU will apply for Google Summer of Code >>>>> (https://summerofcode.withgoogle.com/) again this year. This internship >>>>> program offers paid, 10-week, remote work internships for >>>>> contributing to open source. QEMU can act as an umbrella organization >>>>> for KVM kernel and rust-vmm projects too. >>>>> Please post project ideas on the QEMU wiki before February 14th: >>>>> https://wiki.qemu.org/Google_Summer_of_Code_2021 >>>>> What's new this year: >>>>> * The number of internship hours has been halved to 175 hours over >>>>> 10 weeks. Project ideas must be smaller to fit and students will have >>>>> more flexibility with their working hours. >>>>> * Eligibility has been expanded to include "licensed coding school or >>>>> similar type of program". >>>>> Good project ideas are suitable for 175 hours (10 weeks half-day) >>>>> work by a >>>>> competent programmer who is not yet familiar with the codebase. In >>>>> addition, they are: >>>>> * Well-defined - the scope is clear >>>>> * Self-contained - there are few dependencies >>>>> * Uncontroversial - they are acceptable to the community >>>>> * Incremental - they produce deliverables along the way >>>>> Feel free to post ideas even if you are unable to mentor the >>>>> project. >>>>> It doesn't hurt to share the idea! >>>>> >>>> >>>> I have one that is probably way too ambitious, but requires a >>>> particular skillset that might be of good interest to a student that >>>> has some experience in the area already. >>>> >>>> The idea is for a TUI qmp-shell (maybe using urwid?) to create an >>>> irssi-like REPL interface for QMP. The idea would be to mimic the >>>> mitmproxy TUI interface (Check it out if you haven't!) >>>> >>>> All the ideas below are extremely tentative to give a basic gist of >>>> what I mean; exact layouts/hotkeys/etc are for the sake of explanation >>>> only. >>>> >>>> Essentially, I want an interface like this: >>>> >>>> ----------------------------------------------------------- >>>> | QMP Mode | >>>> |=========================================================| >>>> | | >>>> | Welcome to the Qemu Machine Protocol shell. | >>>> | Please type /help or Ctrl+H to see available commands. | >>>> | | >>>> | | >>>> | | >>>> |---------------------------------------------------------| >>>> | > | >>>> ----------------------------------------------------------- >>>> >>>> commands are entered in the bottom and appear in a log window above, >>>> appearing most-recent last, like irssi works. >>>> >>>> As an example, let's say we issue block-dirty-bitmap-add: >>>> >>>> -------------------------------------------------------- >>>> | > block-dirty-bitmap-add node=ide0hd0 name=mybitmap0 | >>>> -------------------------------------------------------- >>>> >>>> (...syntax up for debate...! We want something easy to parse, but easy >>>> to type. For structured data, that's Hard.) >>> "Hard" is a red flag for such a brief internship. We need to solve >>> or >>> somehow exclude this design problem before we hand the task to the >>> student. >>> [...] >>> >> >> Yes, there is a reason I capitalized "Hard". :~) >> >> This task CAN be split out, though. For starters, we can use an >> extremely simplified parsing strategy while we work on the REPL and >> and UI elements. >> >> We can accept things of the form: >> command arg=value arg2=value >> >> and we can worry about how to input structured data later. > > Bzzzt! You just designed a language. The fact that it is obviously > inadequate for the task does not make this not language design. > Did *I* design a language, or did qmp-shell authors design a language? O:-) (Oh, I'm one of the authors. Well then.) > Two outcomes: > > 1. The project fails for other reasons. The language doesn't matter. > > 2. The project succeeds, i.e. it becomes useful to people. Very quickly > these people will demand you evolve your inadequate language into a less > inadequate one, compatibly. Your evasion of the language design problem > (because hard) has now added accidental constraints, making the design > problem even harder. > There's no reason that an interactive shell has to stay backwards compatible. Version 0.1 and 0.2 might offer different input paradigms. It's not nice, but it's fine for beta-level stuff. Still, I appreciate that "Beta! Might change!" is an oft-ignored warning. This isn't a library or API though, so I still think it's okay to take some liberty here. > The sane way to evade the language design problem is to use the existing > QMP language. > I wouldn't mind implementing this for version 0.1 -- just allow copy-pasting JSON into the input bar -- it's a feature I wanted anyway. Some things I would like to see, but suspect are not reasonably possible to implement quickly: 1. A shorthand syntax mode similar to qmp-shell that is reasonably sufficient for most commands. This involves solving a fairly difficult language design problem on how to facilitate the input of structured data. The biggest pain point here is nested data; we allow JSON as an input here but require that it has no spaces. We do not offer a "native" shorthand for the nested data. We *could* co-opt the dotted key syntax, but that's cumbersome for some constructs because it involves a lot of repeated parent keys. As you say, the language design aspect is hard. 2. a multi-line editor mode that facilitates typing QMP messages that offers syntax highlighting, field and type validation, autocomplete, etc. I have no idea how to implement this -- it's basically an entire bespoke editor. That's *very* hard stuff, I expect, so I won't go there for starters. So I see two possible options for "not inventing a language": 1. Raw QMP 2. The existing qmp-shell syntax, warts and all. I don't see a tremendous problem with doing both; but we can start with raw QMP. The existing qmp-shell syntax is at least definitely very easy to write a new parser for, even if it's kind of ugly and insufficient. I still see value in designing a new TUI with the old syntax. > The project then aims to build a tool that adds useful features over > "socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>" > UNIX-CONNECT:/path/to/socket". > > If it succeeds, you can still design and implement a "better" language, > and let users choose the one they prefer. Or you could add features to > help with typing QMP. > >> I don't >> think it's a blocker to have someone work on the TUI and asynchronous >> dispatch elements. I think even just keeping our current parsing but >> adding some of the features outlined in the proposal would be a big >> usability win. > > I don't feel this particular itch, but I'm certainly not objecting to > anyone scratching. > It's something I'd like to see so that I can walk non-QEMU devs through interacting with QEMU at a low level for the purposes of debugging, reproducing problems, prototyping features, etc. I use qmp-shell all the time for debugging things myself, I find it easier to use than copy-pasting things directly into socat. I wouldn't mind the shell getting a little smarter to help me out -- the ability to see async events and reconnect on disconnect would already be a massive improvement to *my* quality of life. So much so that I spent a lot of time in December to write an async qmp library O:-) --js ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Call for Google Summer of Code 2021 project ideas 2021-01-14 16:36 ` John Snow @ 2021-01-15 16:31 ` Kashyap Chamarthy 2021-02-15 11:05 ` Paolo Bonzini 1 sibling, 0 replies; 27+ messages in thread From: Kashyap Chamarthy @ 2021-01-15 16:31 UTC (permalink / raw) To: John Snow Cc: Stefan Hajnoczi, Eduardo Habkost, Markus Armbruster, Stefan Hajnoczi, QEMU Developers On Thu, Jan 14, 2021 at 11:36:23AM -0500, John Snow wrote: > On 1/14/21 7:29 AM, Markus Armbruster wrote: [...] > So I see two possible options for "not inventing a language": > > 1. Raw QMP > 2. The existing qmp-shell syntax, warts and all. > > I don't see a tremendous problem with doing both; but we can start with raw > QMP. The existing qmp-shell syntax is at least definitely very easy to write > a new parser for, even if it's kind of ugly and insufficient. I still see > value in designing a new TUI with the old syntax. > > > The project then aims to build a tool that adds useful features over > > "socat "READLINE,history=$HOME/.qmp_history,prompt=QMP>" > > UNIX-CONNECT:/path/to/socket". > > > > If it succeeds, you can still design and implement a "better" language, > > and let users choose the one they prefer. Or you could add features to > > help with typing QMP. > > > > > I don't > > > think it's a blocker to have someone work on the TUI and asynchronous > > > dispatch elements. I think even just keeping our current parsing but > > > adding some of the features outlined in the proposal would be a big > > > usability win. > > > > I don't feel this particular itch, but I'm certainly not objecting to > > anyone scratching. > > > > It's something I'd like to see so that I can walk non-QEMU devs through > interacting with QEMU at a low level for the purposes of debugging, > reproducing problems, prototyping features, etc. > > I use qmp-shell all the time for debugging things myself, I find it easier > to use than copy-pasting things directly into socat. I wouldn't mind the > shell getting a little smarter to help me out -- the ability to see async > events and reconnect on disconnect would already be a massive improvement to > *my* quality of life. As an infrequent user of `qmp-shell`, the async events stuff is really beneficial to me too. And, I'm happy to play the test guinea pig to give the patchs a spin. (I'm somewhat behind on the goings-on in this area, very slowly catching up.) > So much so that I spent a lot of time in December to write an async qmp > library O:-) Nice, I recall that you planned to use the 'asyncio' primitives from Python 3.6. -- /kashyap ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Call for Google Summer of Code 2021 project ideas 2021-01-14 16:36 ` John Snow 2021-01-15 16:31 ` Kashyap Chamarthy @ 2021-02-15 11:05 ` Paolo Bonzini 2021-02-15 21:47 ` John Snow 1 sibling, 1 reply; 27+ messages in thread From: Paolo Bonzini @ 2021-02-15 11:05 UTC (permalink / raw) To: John Snow, Stefan Hajnoczi, Markus Armbruster, qemu-devel On 14/01/21 17:36, John Snow wrote: > >> The sane way to evade the language design problem is to use the existing >> QMP language. > > I wouldn't mind implementing this for version 0.1 -- just allow > copy-pasting JSON into the input bar -- it's a feature I wanted anyway. I think the only way out of language design is to instead design a TUI for inputting JSON. For example: * after typing the ' for a key you can autocomplete the next field, using the TAB key similar to vi * after typing the : the TUI tells you the field type * after typing the ' for an enum, the TAB brings up a menu to pick an enum * after typing the last character in a key or value you automatically get a suggestion for what to type next (comma and next apostrophe after a value, colon and possible apostrophe/bracket/brace for a key) One idea that has worked for me in the past was to write a mockup that shows what things are going to look like, with fake user interaction. You would have something like // { keypress("{") show_suggestion("'") // ' keypress("'") start_autocomplete_choices(["execute", "arguments"]) // TAB await asyncio.sleep(1) do_autocomplete() // ' await asyncio.sleep(1) keypress("'") // string argument, propose ' for the value automatically show_suggestion(": '") // TAB await asyncio.sleep(1) do_autocomplete() start_autocomplete_choices("query-status", "query-kvm") # many more etc. Then you plug in an incremental lexer, so that you can e.g. replace show_autocomplete(": '") with lex_state(Lexer.AFTER_KEY); // this would come from the lexer show_autocomplete("'") // this would come from the schema And then again plug the incremental visitor to autocomplete on the schema types. Another advantage of this approach is that you also have a natural API for the mocks, and thus it becomes easier to write testcases. Paolo ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: Call for Google Summer of Code 2021 project ideas 2021-02-15 11:05 ` Paolo Bonzini @ 2021-02-15 21:47 ` John Snow 0 siblings, 0 replies; 27+ messages in thread From: John Snow @ 2021-02-15 21:47 UTC (permalink / raw) To: Paolo Bonzini, Stefan Hajnoczi, Markus Armbruster, qemu-devel On 2/15/21 6:05 AM, Paolo Bonzini wrote: > On 14/01/21 17:36, John Snow wrote: >> >>> The sane way to evade the language design problem is to use the existing >>> QMP language. >> >> I wouldn't mind implementing this for version 0.1 -- just allow >> copy-pasting JSON into the input bar -- it's a feature I wanted anyway. > > I think the only way out of language design is to instead design a TUI > for inputting JSON. For example: > > * after typing the ' for a key you can autocomplete the next field, > using the TAB key similar to vi > > * after typing the : the TUI tells you the field type > > * after typing the ' for an enum, the TAB brings up a menu to pick an enum > > * after typing the last character in a key or value you automatically > get a suggestion for what to type next (comma and next apostrophe after > a value, colon and possible apostrophe/bracket/brace for a key) > > One idea that has worked for me in the past was to write a mockup that > shows what things are going to look like, with fake user interaction. > You would have something like > > // { > keypress("{") > show_suggestion("'") > // ' > keypress("'") > start_autocomplete_choices(["execute", "arguments"]) > // TAB > await asyncio.sleep(1) > do_autocomplete() > // ' > await asyncio.sleep(1) > keypress("'") > // string argument, propose ' for the value automatically > show_suggestion(": '") > // TAB > await asyncio.sleep(1) > do_autocomplete() > start_autocomplete_choices("query-status", "query-kvm") # many more > > etc. > > Then you plug in an incremental lexer, so that you can e.g. replace > > show_autocomplete(": '") > > with > > lex_state(Lexer.AFTER_KEY); // this would come from the lexer > show_autocomplete("'") // this would come from the schema > > And then again plug the incremental visitor to autocomplete on the > schema types. > > Another advantage of this approach is that you also have a natural API > for the mocks, and thus it becomes easier to write testcases. > > Paolo > Yes, I follow. I quite like the VSCode auto-suggest behavior for writing JSON when you have a schema, but that's baked into their borg-cube, I assume. I wonder if there's any libraries or more re-usable pieces out there that would allow us to write a nice interactive TUI for complex structures in Python, or if we'd have to design and build it ground-up. Changes how willing I am to embark upon the quest. --js ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas 2021-01-11 11:47 Call for Google Summer of Code 2021 project ideas Stefan Hajnoczi 2021-01-12 21:10 ` John Snow @ 2021-02-12 13:22 ` Florescu, Andreea 2021-02-12 13:51 ` Sergio Lopez 2021-02-17 11:20 ` Stefan Hajnoczi 1 sibling, 2 replies; 27+ messages in thread From: Florescu, Andreea @ 2021-02-12 13:22 UTC (permalink / raw) To: Stefan Hajnoczi, qemu-devel, kvm, rust-vmm, Alex Bennée, Alexander Graf, Alberto Garcia, David Hildenbrand, Eduardo Habkost, Igor Mammedov, John Snow, Julia Suvorova, Gerd Hoffmann, Kevin Wolf, Laurent Vivier, Marc-André Lureau, Aleksandar Markovic, Sergio Lopez, Stefano Garzarella, Paolo Bonzini, Philippe Mathieu-Daudé [-- Attachment #1: Type: text/plain, Size: 7094 bytes --] Hey Stefan, Thanks for taking care of organizing GSoC, and for allowing rust-vmm to also participate under the QEMU umbrella! I am a bit unsure of how can we propose projects related to rust-vmm. We did a bit of brainstorming in our team, and we came up with 3 project ideas. I'll just paste them below, but please let me know if we were supposed to propose them some other way. === Implement the Virtio Console device in Rust === '''Summary:''' Implement the basic emulation for the Virtio Console device in Rust Implement the basic functionality (excluding the optional features: VIRTIO_CONSOLE_F_SIZE, VIRTIO_CONSOLE_F_MULTIPORT, or VIRTIO_CONSOLE_F_EMERG_WRITE) of the Virtio Console Device, using the Virtio building blocks (queue implementations, VirtioDevice traits) defined in rust-vmm/vm-virtio. The virtio console device uses one virtio queue for transmitting data, and one virtio queue for receiving data. The implementation can be extended to also support a subset of the previously mentioned optional features. '''Links:''' * About rust-vmm: https://github.com/rust-vmm/community * rust-vmm/vm-virtio: https://github.com/rust-vmm/vm-virtio * virtio-console spec: https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-2550003 '''Details:''' * Skill level: intermediate * Language: Rust * Mentor: iul@amazon.com * Suggested by: fandree@amazon.com<mailto:fandree@amazon.com> === Mocking framework for Virtio Queues === '''Summary:''' Implement a mocking framework for virtio queues Paravirtualized devices (such as those defined by the Virtio standard) are used to provide high performance device emulation. Virtio drivers from a guest VM communicate with the device model using an efficient mechanism based on queues stored in a shared memory area that operate based on a protocol and message format defined by the standard. Various implementations of devices and other virtualization building blocks require mocking the contents that a driver would place into a Virtio queue for validation, testing, and evaluation purposes. This project aims to lay the foundations of a reusable framework for mocking the driver side of Virtio queue operation, that can be consumed by rust-vmm crates and other projects. At the basic level, this means providing a flexible and easy to use interface for users to set up the underlying memory areas and populate contents (as the driver would do) for the basic split queue format in a generic manner. This can further be extended for the packed format and with device-specific mocking capabilities. '''Links:''' * About rust-vmm: https://github.com/rust-vmm/community * Virtio queue spec: https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-230005 Issue in rust-vmm about reusing the mocking logic: rust-vmm/vm-virtio: https://github.com/rust-vmm/vm-virtio '''Details:''' * Skill level: intermediate * Language: Rust * Mentor: aagch@amazon.com * Suggested by: aagch@amazon.com === Local running rust-vmm-ci === '''Summary:''' Run the rust-vmm-ci locally The rust-vmm-ci provides automation for uniformely running the tests on all rust-vmm repositories. It is built on top of Buildkite, and only allows running the tests in the Buildkite context. To run the same tests as in the CI locally, users need to manually copy the Buildkite pipeline steps. The scope of this project is to make it possible for the same tests to easily run locally. This project makes it easier to contribute to all rust-vmm repositories. In order for that to be possible, the following steps are required: - the Buildlkite pipeline is autogenerated from code instead of being a static list of tests to run. This also allows us to uniformely use the same container version for running all the tests (instead of manually modifying each step in the pipeline) - the code for autogenerating the Buildkite pipeline is reused for generating a Python script which can be run locally '''Links:''' * rust-vmm-ci: https://github.com/rust-vmm/rust-vmm-ci * Buildkite pipeline that currently runs the tests: https://github.com/rust-vmm/rust-vmm-ci/blob/master/.buildkite/pipeline.yml * About rust-vmm: https://github.com/rust-vmm/community * Buildkite documentation: https://buildkite.com/docs/tutorials/getting-started '''Details:''' * Skill level: intermediate * Language: Python * Mentor: fandree@amazon.com * Suggested by: fandree@amazon.com ?Thanks again! Andreea ________________________________ From: Stefan Hajnoczi <stefanha@gmail.com> Sent: Monday, January 11, 2021 1:47 PM To: qemu-devel; kvm; rust-vmm@lists.opendev.org; Alex Bennée; Alexander Graf; Alberto Garcia; David Hildenbrand; Eduardo Habkost; Igor Mammedov; John Snow; Julia Suvorova; Gerd Hoffmann; Kevin Wolf; Laurent Vivier; Marc-André Lureau; Aleksandar Markovic; Sergio Lopez; Stefano Garzarella; Paolo Bonzini; Philippe Mathieu-Daudé Subject: [EXTERNAL] [Rust-VMM] Call for Google Summer of Code 2021 project ideas CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Dear QEMU, KVM, and rust-vmm community, QEMU will apply for Google Summer of Code (https://summerofcode.withgoogle.com/) again this year. This internship program offers paid, 10-week, remote work internships for contributing to open source. QEMU can act as an umbrella organization for KVM kernel and rust-vmm projects too. Please post project ideas on the QEMU wiki before February 14th: https://wiki.qemu.org/Google_Summer_of_Code_2021 What's new this year: * The number of internship hours has been halved to 175 hours over 10 weeks. Project ideas must be smaller to fit and students will have more flexibility with their working hours. * Eligibility has been expanded to include "licensed coding school or similar type of program". Good project ideas are suitable for 175 hours (10 weeks half-day) work by a competent programmer who is not yet familiar with the codebase. In addition, they are: * Well-defined - the scope is clear * Self-contained - there are few dependencies * Uncontroversial - they are acceptable to the community * Incremental - they produce deliverables along the way Feel free to post ideas even if you are unable to mentor the project. It doesn't hurt to share the idea! I will review project ideas and keep you up-to-date on QEMU's acceptance into GSoC. For more background on QEMU internships, check out this video: https://www.youtube.com/watch?v=xNVCX7YMUL8 Stefan _______________________________________________ Rust-vmm mailing list Rust-vmm@lists.opendev.org http://lists.opendev.org/cgi-bin/mailman/listinfo/rust-vmm Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005. [-- Attachment #2: Type: text/html, Size: 9566 bytes --] ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas 2021-02-12 13:22 ` [Rust-VMM] " Florescu, Andreea @ 2021-02-12 13:51 ` Sergio Lopez 2021-02-17 11:20 ` Stefan Hajnoczi 1 sibling, 0 replies; 27+ messages in thread From: Sergio Lopez @ 2021-02-12 13:51 UTC (permalink / raw) To: Florescu, Andreea Cc: Kevin Wolf, Laurent Vivier, Alberto Garcia, Eduardo Habkost, kvm, David Hildenbrand, Stefan Hajnoczi, Philippe Mathieu-Daudé, Julia Suvorova, qemu-devel, Paolo Bonzini, John Snow, Marc-André Lureau, Alexander Graf, Gerd Hoffmann, Aleksandar Markovic, rust-vmm, Igor Mammedov, Alex Bennée, Stefano Garzarella [-- Attachment #1: Type: text/plain, Size: 7770 bytes --] On Fri, Feb 12, 2021 at 01:22:43PM +0000, Florescu, Andreea wrote: > Hey Stefan, > > > Thanks for taking care of organizing GSoC, and for allowing rust-vmm to also participate under the QEMU umbrella! > > I am a bit unsure of how can we propose projects related to rust-vmm. > > We did a bit of brainstorming in our team, and we came up with 3 project ideas. > > I'll just paste them below, but please let me know if we were supposed to propose them some other way. > > > === Implement the Virtio Console device in Rust === > > '''Summary:''' Implement the basic emulation for the Virtio Console device in Rust > > Implement the basic functionality (excluding the optional features: > VIRTIO_CONSOLE_F_SIZE, VIRTIO_CONSOLE_F_MULTIPORT, or VIRTIO_CONSOLE_F_EMERG_WRITE) > of the Virtio Console Device, using the Virtio building blocks (queue implementations, > VirtioDevice traits) defined in rust-vmm/vm-virtio. The virtio console device uses > one virtio queue for transmitting data, and one virtio queue for receiving data. > The implementation can be extended to also support a subset of the previously > mentioned optional features. FWIW, libkrun already has support for virtio-console with the basic functionality and VIRTIO_CONSOLE_F_SIZE, and this code could be easily borrowed for implementing it in some rust-vmm crate: https://github.com/containers/libkrun/blob/main/src/devices/src/virtio/console Sergio. > '''Links:''' > * About rust-vmm: https://github.com/rust-vmm/community > * rust-vmm/vm-virtio: https://github.com/rust-vmm/vm-virtio > * virtio-console spec: https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-2550003 > > '''Details:''' > * Skill level: intermediate > * Language: Rust > * Mentor: iul@amazon.com > * Suggested by: fandree@amazon.com<mailto:fandree@amazon.com> > > > === Mocking framework for Virtio Queues === > > '''Summary:''' Implement a mocking framework for virtio queues > > Paravirtualized devices (such as those defined by the Virtio standard) are used > to provide high performance device emulation. Virtio drivers from a guest VM > communicate with the device model using an efficient mechanism based on queues > stored in a shared memory area that operate based on a protocol and message format > defined by the standard. Various implementations of devices and other > virtualization building blocks require mocking the contents that a driver would > place into a Virtio queue for validation, testing, and evaluation purposes. > > This project aims to lay the foundations of a reusable framework for mocking the > driver side of Virtio queue operation, that can be consumed by rust-vmm crates and > other projects. At the basic level, this means providing a flexible and easy to > use interface for users to set up the underlying memory areas and populate contents > (as the driver would do) for the basic split queue format in a generic manner. This > can further be extended for the packed format and with device-specific mocking > capabilities. > > '''Links:''' > * About rust-vmm: https://github.com/rust-vmm/community > * Virtio queue spec: https://docs.oasis-open.org/virtio/virtio/v1.1/csprd01/virtio-v1.1-csprd01.html#x1-230005 > Issue in rust-vmm about reusing the mocking logic: rust-vmm/vm-virtio: https://github.com/rust-vmm/vm-virtio > > '''Details:''' > * Skill level: intermediate > * Language: Rust > * Mentor: aagch@amazon.com > * Suggested by: aagch@amazon.com > > > === Local running rust-vmm-ci === > > '''Summary:''' Run the rust-vmm-ci locally > > The rust-vmm-ci provides automation for uniformely running the tests on > all rust-vmm repositories. It is built on top of Buildkite, and only allows > running the tests in the Buildkite context. To run the same tests as in the CI > locally, users need to manually copy the Buildkite pipeline steps. > > The scope of this project is to make it possible for the same tests to easily run > locally. This project makes it easier to contribute to all rust-vmm repositories. > > In order for that to be possible, the following steps are required: > - the Buildlkite pipeline is autogenerated from code instead of being a static > list of tests to run. This also allows us to uniformely use the same container > version for running all the tests (instead of manually modifying each step in > the pipeline) > - the code for autogenerating the Buildkite pipeline is reused for generating > a Python script which can be run locally > > > '''Links:''' > * rust-vmm-ci: https://github.com/rust-vmm/rust-vmm-ci > * Buildkite pipeline that currently runs the tests: https://github.com/rust-vmm/rust-vmm-ci/blob/master/.buildkite/pipeline.yml > * About rust-vmm: https://github.com/rust-vmm/community > * Buildkite documentation: https://buildkite.com/docs/tutorials/getting-started > > '''Details:''' > * Skill level: intermediate > * Language: Python > * Mentor: fandree@amazon.com > * Suggested by: fandree@amazon.com > > > ?Thanks again! > > Andreea > > ________________________________ > From: Stefan Hajnoczi <stefanha@gmail.com> > Sent: Monday, January 11, 2021 1:47 PM > To: qemu-devel; kvm; rust-vmm@lists.opendev.org; Alex Bennée; Alexander Graf; Alberto Garcia; David Hildenbrand; Eduardo Habkost; Igor Mammedov; John Snow; Julia Suvorova; Gerd Hoffmann; Kevin Wolf; Laurent Vivier; Marc-André Lureau; Aleksandar Markovic; Sergio Lopez; Stefano Garzarella; Paolo Bonzini; Philippe Mathieu-Daudé > Subject: [EXTERNAL] [Rust-VMM] Call for Google Summer of Code 2021 project ideas > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > Dear QEMU, KVM, and rust-vmm community, > QEMU will apply for Google Summer of Code > (https://summerofcode.withgoogle.com/) again this year. This internship > program offers paid, 10-week, remote work internships for > contributing to open source. QEMU can act as an umbrella organization > for KVM kernel and rust-vmm projects too. > > Please post project ideas on the QEMU wiki before February 14th: > https://wiki.qemu.org/Google_Summer_of_Code_2021 > > What's new this year: > * The number of internship hours has been halved to 175 hours over > 10 weeks. Project ideas must be smaller to fit and students will have > more flexibility with their working hours. > * Eligibility has been expanded to include "licensed coding school or > similar type of program". > > Good project ideas are suitable for 175 hours (10 weeks half-day) work by a > competent programmer who is not yet familiar with the codebase. In > addition, they are: > * Well-defined - the scope is clear > * Self-contained - there are few dependencies > * Uncontroversial - they are acceptable to the community > * Incremental - they produce deliverables along the way > > Feel free to post ideas even if you are unable to mentor the project. > It doesn't hurt to share the idea! > > I will review project ideas and keep you up-to-date on QEMU's > acceptance into GSoC. > > For more background on QEMU internships, check out this video: > https://www.youtube.com/watch?v=xNVCX7YMUL8 > > Stefan > > _______________________________________________ > Rust-vmm mailing list > Rust-vmm@lists.opendev.org > http://lists.opendev.org/cgi-bin/mailman/listinfo/rust-vmm > > > > Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005. [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas 2021-02-12 13:22 ` [Rust-VMM] " Florescu, Andreea 2021-02-12 13:51 ` Sergio Lopez @ 2021-02-17 11:20 ` Stefan Hajnoczi 2021-02-18 11:49 ` Andreea Florescu 1 sibling, 1 reply; 27+ messages in thread From: Stefan Hajnoczi @ 2021-02-17 11:20 UTC (permalink / raw) To: Florescu, Andreea Cc: Kevin Wolf, Laurent Vivier, Alberto Garcia, Eduardo Habkost, kvm, David Hildenbrand, Sergio Lopez, John Snow, Philippe Mathieu-Daudé, Julia Suvorova, qemu-devel, Paolo Bonzini, Aleksandar Markovic, Marc-André Lureau, Alexander Graf, Gerd Hoffmann, rust-vmm, Igor Mammedov, Alex Bennée, Stefano Garzarella Thanks, I have published the rust-vmm project ideas on the wiki: https://wiki.qemu.org/Google_Summer_of_Code_2021 Please see Sergio's reply about virtio-consoile is libkrun. Maybe it affects the project idea? Stefan ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas 2021-02-17 11:20 ` Stefan Hajnoczi @ 2021-02-18 11:49 ` Andreea Florescu 2021-02-18 17:43 ` Stefan Hajnoczi 0 siblings, 1 reply; 27+ messages in thread From: Andreea Florescu @ 2021-02-18 11:49 UTC (permalink / raw) To: Stefan Hajnoczi Cc: qemu-devel, kvm, rust-vmm, Alex Bennée, Alexander Graf, Alberto Garcia, David Hildenbrand, Eduardo Habkost, Igor Mammedov, John Snow, Julia Suvorova, Gerd Hoffmann, Kevin Wolf, Laurent Vivier, Marc-André Lureau, Aleksandar Markovic, Sergio Lopez, Stefano Garzarella, Paolo Bonzini, Philippe Mathieu-Daudé [-- Attachment #1: Type: text/plain, Size: 1591 bytes --] On 2/17/21 1:20 PM, Stefan Hajnoczi wrote: > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > Thanks, I have published the rust-vmm project ideas on the wiki: > https://wiki.qemu.org/Google_Summer_of_Code_2021 Thanks, Stefan! > > Please see Sergio's reply about virtio-consoile is libkrun. Maybe it > affects the project idea? I synced offline with Sergio. It seems I've misread his comment. The code is already available in libkrun, and to port it to rust-vmm will likely take just a couple of days. We explored the option of also requesting to implement VIRTIO_CONSOLE_F_MULTIPORT to make it an appropriate GSoC project, but we decided this is not a good idea since it doesn't look like that feature is useful for the projects consuming rust-vmm. It also adds complexity, and we would need to maintain that as well. Since it would still be nice to have virtio-console in rust-vmm, I'll just open an issue in vm-virtio and label it with "help wanted" so people can pick it up. Can we remove the virtio-console project from the list of GSoC ideas? Also, iul@amazon.com will like to help with mentoring the GSoC projects. Can he be added as a co-mentor of: "Mocking framework for Virtio Queues"? Sorry for the ping-pong, and thanks again for everything! Andreea Amazon Development Center (Romania) S.R.L. registered office: 27A Sf. Lazar Street, UBC5, floor 2, Iasi, Iasi County, 700045, Romania. Registered in Romania. Registration number J22/2621/2005. [-- Attachment #2: 0x9B51563C1FA36782.asc --] [-- Type: application/pgp-keys, Size: 2501 bytes --] ^ permalink raw reply [flat|nested] 27+ messages in thread
* Re: [Rust-VMM] Call for Google Summer of Code 2021 project ideas 2021-02-18 11:49 ` Andreea Florescu @ 2021-02-18 17:43 ` Stefan Hajnoczi 0 siblings, 0 replies; 27+ messages in thread From: Stefan Hajnoczi @ 2021-02-18 17:43 UTC (permalink / raw) To: Andreea Florescu Cc: Kevin Wolf, Laurent Vivier, Alberto Garcia, Eduardo Habkost, kvm, David Hildenbrand, Sergio Lopez, John Snow, Philippe Mathieu-Daudé, Julia Suvorova, qemu-devel, Paolo Bonzini, Aleksandar Markovic, Marc-André Lureau, Alexander Graf, Gerd Hoffmann, rust-vmm, Igor Mammedov, Alex Bennée, Stefano Garzarella On Thu, Feb 18, 2021 at 11:50 AM Andreea Florescu <fandree@amazon.com> wrote: > On 2/17/21 1:20 PM, Stefan Hajnoczi wrote: > > CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > > > > > > > > Thanks, I have published the rust-vmm project ideas on the wiki: > > https://wiki.qemu.org/Google_Summer_of_Code_2021 > Thanks, Stefan! > > > > Please see Sergio's reply about virtio-consoile is libkrun. Maybe it > > affects the project idea? > I synced offline with Sergio. It seems I've misread his comment. > The code is already available in libkrun, and to port it to rust-vmm > will likely take just a couple of days. We explored the option of also > requesting to implement VIRTIO_CONSOLE_F_MULTIPORT to make it an > appropriate GSoC project, but we decided this is not a good idea since > it doesn't look like that feature is useful for the projects consuming > rust-vmm. It also adds complexity, and we would need to maintain that as > well. > > Since it would still be nice to have virtio-console in rust-vmm, I'll > just open an issue in vm-virtio and label it with "help wanted" so > people can pick it up. > Can we remove the virtio-console project from the list of GSoC ideas? Done. > Also, iul@amazon.com will like to help with mentoring the GSoC projects. > Can he be added as a co-mentor of: "Mocking framework for Virtio Queues"? Done. Thanks for getting the rust-vmm mentors together! I'm looking forward to more Rust virtualization projects :). Stefan ^ permalink raw reply [flat|nested] 27+ messages in thread
end of thread, other threads:[~2021-02-18 17:57 UTC | newest] Thread overview: 27+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-11 11:47 Call for Google Summer of Code 2021 project ideas Stefan Hajnoczi 2021-01-12 21:10 ` John Snow 2021-01-13 8:53 ` Stefan Hajnoczi 2021-01-13 18:59 ` qmp-shell TUI (was: Re: Call for Google Summer of Code 2021 project ideas) John Snow 2021-01-14 13:52 ` Stefan Hajnoczi 2021-01-14 13:59 ` Daniel P. Berrangé 2021-01-14 15:02 ` Kevin Wolf 2021-01-14 15:22 ` Daniel P. Berrangé 2021-01-14 16:49 ` Stefan Hajnoczi 2021-01-14 16:55 ` Daniel P. Berrangé 2021-01-14 17:14 ` Stefan Hajnoczi 2021-01-14 17:24 ` Daniel P. Berrangé 2021-01-14 16:48 ` Stefan Hajnoczi 2021-01-14 17:48 ` John Snow 2021-01-13 9:19 ` Call for Google Summer of Code 2021 project ideas Markus Armbruster 2021-01-13 19:05 ` John Snow 2021-01-14 12:29 ` Markus Armbruster 2021-01-14 14:57 ` Kevin Wolf 2021-01-14 16:36 ` John Snow 2021-01-15 16:31 ` Kashyap Chamarthy 2021-02-15 11:05 ` Paolo Bonzini 2021-02-15 21:47 ` John Snow 2021-02-12 13:22 ` [Rust-VMM] " Florescu, Andreea 2021-02-12 13:51 ` Sergio Lopez 2021-02-17 11:20 ` Stefan Hajnoczi 2021-02-18 11:49 ` Andreea Florescu 2021-02-18 17:43 ` Stefan Hajnoczi
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).