Interested in contributing to QEMU

Interested in contributing to QEMU

[Niteesh G. S.]

[-- Attachment #1: Type: text/plain, Size: 2057 bytes --]

Hello all,

I am Niteesh, a junior student(3rd year) pursuing Electronics and
Communication
engineering. I was also a GSoC student for RTEMS last year. My main area of
interest is low-level development (OS, Emulators, Hardware design, etc).

I wanted to start contributing from last year itself but was occupied with
academic
work. I have started working on small patches. My ultimate goal is to learn
about
how QEMU works, contribute and learn as much as possible.

I tried going through the Arduino emulation code. I was able to understand
it from
a high level but couldn't understand underlying details. I went through few
blog
posts related to QEMU internals but they didn't help much. I plan to step
through
the code but the sheer size of the codebase is scary(Tips regarding
debugging are
very much welcomed). AFAIK the source code is mostly the documentation for
QEMU. If someone knows any docs or articles that will help a beginner get
started
it would be great.

I would also like to take part in GSoC this year. I find the below two
projects interesting
1)
https://wiki.qemu.org/Google_Summer_of_Code_2020#QEMU_emulated_Arduino_board_visualizer
This one is from last year AFAIK no one has worked on it. If so I would like
to work on it. I have CC'ed the mentors of this project to share some more
details
regarding it. Have you guys decided on the netlist parser lib, UI lib? Is
there something
that I could work on or read to get myself familiarized with the JSON event
IO stuff?

2)
https://wiki.qemu.org/Google_Summer_of_Code_2021#Interactive.2C_asynchronous_QEMU_Machine_Protocol_.28QMP.29_text_user_interface_.28TUI.29
This is something that I don't know much about. I have a basic idea about
what
QMP is but I never used it. The docs say that the Async QMP library is a
work
in progress. If someone can hook me up with some small tasks in this
library it would
be really helpful in improving my understanding.

I would like to work on these projects even outside of GSoC if someone is
ready to
mentor in their free time :).

Thanks
Niteesh.

[-- Attachment #2: Type: text/html, Size: 4553 bytes --]

Re: Interested in contributing to QEMU

[John Snow]

On 2/7/21 12:01 PM, Niteesh G. S. wrote:
> Hello all,
> 
> I am Niteesh, a junior student(3rd year) pursuing Electronics and 
> Communication
> engineering. I was also a GSoC student for RTEMS last year. My main area of
> interest is low-level development (OS, Emulators, Hardware design, etc).
> 

QEMU certainly is the right place for low-level!

> I wanted to start contributing from last year itself but was occupied 
> with academic
> work. I have started working on small patches. My ultimate goal is to 
> learn about
> how QEMU works, contribute and learn as much as possible.
> 
> I tried going through the Arduino emulation code. I was able to 
> understand it from
> a high level but couldn't understand underlying details. I went through 
> few blog
> posts related to QEMU internals but they didn't help much. I plan to 
> step through
> the code but the sheer size of the codebase is scary(Tips regarding 
> debugging are
> very much welcomed). AFAIK the source code is mostly the documentation for
> QEMU. If someone knows any docs or articles that will help a beginner 
> get started
> it would be great.
> 

Depending on what you'd like to debug, the debugging tips are going to 
be different. I am not sure of the quality of our Arduino code as I have 
not touched it personally.

Anything concrete you'd like to ask about how QEMU works? You can always 
stop by the IRC channel to ask some questions if you're afraid of 
cluttering up the email list.

See the end of this mail for instructions on how to join, if you need them.

> I would also like to take part in GSoC this year. I find the below two 
> projects interesting
> 1) 
> https://wiki.qemu.org/Google_Summer_of_Code_2020#QEMU_emulated_Arduino_board_visualizer 
> <https://wiki.qemu.org/Google_Summer_of_Code_2020#QEMU_emulated_Arduino_board_visualizer>
> This one is from last year AFAIK no one has worked on it. If so I would like
> to work on it. I have CC'ed the mentors of this project to share some 
> more details
> regarding it. Have you guys decided on the netlist parser lib, UI lib? 
> Is there something
> that I could work on or read to get myself familiarized with the JSON 
> event IO stuff?
> 

I don't think we have the events this project described in the latest 
development branch of QEMU, but maybe Phil has something in development 
somewhere. I'll let him answer you, but, it's likely they intend to use 
QMP here, which is relevant below, too:

> 2) 
> https://wiki.qemu.org/Google_Summer_of_Code_2021#Interactive.2C_asynchronous_QEMU_Machine_Protocol_.28QMP.29_text_user_interface_.28TUI.29 
> <https://wiki.qemu.org/Google_Summer_of_Code_2021#Interactive.2C_asynchronous_QEMU_Machine_Protocol_.28QMP.29_text_user_interface_.28TUI.29>
> This is something that I don't know much about. I have a basic idea 
> about what
> QMP is but I never used it. The docs say that the Async QMP library is a 
> work
> in progress. If someone can hook me up with some small tasks in this 
> library it would
> be really helpful in improving my understanding.
> 

1) About QMP and QAPI:

QMP is the JSON-like runtime protocol that QEMU supports. These two 
documents should get you started on understanding what QMP is and how it 
works:

https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/qmp-intro.txt

https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/qmp-spec.txt

There's a python script (scripts/qmp/qmp-shell) that can be used to 
issue an unsupported* short-hand syntax that translates into "real" QMP. 
Or, as qmp-intro.txt says, you can use telnet or socat to copy/paste 
JSON straight into the socket.

The commands that QMP accepts are defined by QAPI; those are defined in 
./qapi/*.json -- a component called the QAPI generator digests this 
information and generates the runtime server code that handles parsing 
and dispatch.


2) About QMP Events

Normally, after the initial handshake, QMP is a call-and-response 
protocol. The client sends an RPC request, the server executes the 
command and sends the response.

QMP also supports asynchronous events, though. At any point, QMP can 
send an unprompted status message that informs the client of some state 
change within QEMU. This is seen most often for changes in the system 
emulator runstate, i.e. if QEMU is paused or resumed, etc.

(It's also used for reporting block device errors, long-running task 
status updates like backup, etc.)

It's likely that Phil wants to use this functionality to send 
information about GPIO state changes such that a client can render a 
meaningful visualization.


3) About AQMP

QEMU today has a QMP library written in Python at ./python/qemu/qmp.py. 
This is a synchronous library that blocks execution until the command is 
done executing on the server.

We have augmented it over the years to support caching events we 
received while waiting for execution to finish (for later retrieval), 
but it requires the caller to go back and check those cached events. It 
does not offer event handling callbacks. It is a very low-level library 
that is prone to race conditions depending on how the test using it is 
written.

qmp-shell uses the old qmp library: it does not show you incoming events 
as they happen. You have to press "enter" with an empty buffer to coax 
the shell to check for new events and print them for you, which can be a 
little annoying if you want live updates.

I started writing AQMP using Python's asyncio/await keywords to create a 
more modern, flexible QMP library to replace it. One of the hopes I have 
for this library is that it will handle asynchronous events much more 
nicely. It's my hope that qmp-shell can be upgraded to use this newer 
library instead.

The AQMP library is about half finished: It supports all of the basic 
functionality of the protocol, but needs work on the callback API for 
dispatching event responses. It also likely needs a lot of testing and 
polish that's likely to become obvious as anyone tries to integrate it 
into a real program like a theoretical qmp-shell-2.0.


4) Understanding my project

If you're still interested in my project, I'd recommend trying out 
qmp-shell against a running QEMU instance and issuing a few basic, 
boring commands ("query-status" is a good candidate) and seeing how that 
works.

Then, I'd take a look at some of the other projects I mentioned 
(mitmproxy, irssi) to get a sense of what the work is here. This is 
largely a UI/API programming task, and there's real work to do on the 
AQMP library, but it's probably closer to the surface than the deep 
technical internals of QEMU.

It might be a good introductory project that helps you get a better 
overview of the internals of QEMU if you're interested in more 
hardware-related aspects, but it still requires you have at least some 
interested in UI programming and API design.

Phil's project might involve hardware specifics quite a bit more than 
mine, while still teaching you some overview of QMP as a necessity of 
the project.

If you remain interested after the above, I can point you towards some 
more concrete tasks that need doing for you to get a fairly concrete 
sense of what the project entails.

> I would like to work on these projects even outside of GSoC if someone 
> is ready to
> mentor in their free time :).
> 

Feel free to join #qemu-gsoc on irc.oftc.net. If you've not joined an 
IRC channel before, it's kind of like a prehistoric slack channel.

Linux GUI: xchat, hexchat
Linux TUI: irssi, WeeChat

OSX GUI: LimeChat, Colloquy (I've never used either)
OSX TUI: irssi and weechat should be available via ports (Not tried.)

Windows GUI: mIRC, XChat


I'm jsnow on OFTC. You can use my nickname at the start of a message 
("jsnow: Hello, this is Niteesh from the mailing list") and it will show 
me a notification -- but the hours I am paying attention to IRC are 
around 10AM - 7PM EST. (15:00 - 00:00 GMT)

I can be around later by request (00:00 - 05:00 GMT) if you give me some 
advance notice.

> Thanks
> Niteesh.
> 
> 

Re: Interested in contributing to QEMU

[Niteesh G. S.]

[-- Attachment #1: Type: text/plain, Size: 10009 bytes --]

Hello John,
On Mon, Feb 8, 2021 at 8:44 PM John Snow <jsnow@redhat.com> wrote:

> On 2/7/21 12:01 PM, Niteesh G. S. wrote:
> > Hello all,
> >
> > I am Niteesh, a junior student(3rd year) pursuing Electronics and
> > Communication
> > engineering. I was also a GSoC student for RTEMS last year. My main area
> of
> > interest is low-level development (OS, Emulators, Hardware design, etc).
> >
>
> QEMU certainly is the right place for low-level!
>
> > I wanted to start contributing from last year itself but was occupied
> > with academic
> > work. I have started working on small patches. My ultimate goal is to
> > learn about
> > how QEMU works, contribute and learn as much as possible.
> >
> > I tried going through the Arduino emulation code. I was able to
> > understand it from
> > a high level but couldn't understand underlying details. I went through
> > few blog
> > posts related to QEMU internals but they didn't help much. I plan to
> > step through
> > the code but the sheer size of the codebase is scary(Tips regarding
> > debugging are
> > very much welcomed). AFAIK the source code is mostly the documentation
> for
> > QEMU. If someone knows any docs or articles that will help a beginner
> > get started
> > it would be great.
> >
>
> Depending on what you'd like to debug, the debugging tips are going to
> be different. I am not sure of the quality of our Arduino code as I have
> not touched it personally.
>
> Anything concrete you'd like to ask about how QEMU works? You can always
> stop by the IRC channel to ask some questions if you're afraid of
> cluttering up the email list.


Which one in your opinion will yield a faster response, Mailing list or IRC?
My problem with IRC is, I don't have an IRC bouncer set up so I miss
some conversations. Which one is simpler to use? There are many options
available. Is there something that will not require me to host a server?

See the end of this mail for instructions on how to join, if you need them.


I am already part of the QEMU IRC channel. But thanks for these
instructions.

>
> > I would also like to take part in GSoC this year. I find the below two
> > projects interesting
> > 1)
> >
> https://wiki.qemu.org/Google_Summer_of_Code_2020#QEMU_emulated_Arduino_board_visualizer
> > <
> https://wiki.qemu.org/Google_Summer_of_Code_2020#QEMU_emulated_Arduino_board_visualizer
> >
> > This one is from last year AFAIK no one has worked on it. If so I would
> like
> > to work on it. I have CC'ed the mentors of this project to share some
> > more details
> > regarding it. Have you guys decided on the netlist parser lib, UI lib?
> > Is there something
> > that I could work on or read to get myself familiarized with the JSON
> > event IO stuff?
> >
>
> I don't think we have the events this project described in the latest
> development branch of QEMU, but maybe Phil has something in development
> somewhere. I'll let him answer you, but, it's likely they intend to use
> QMP here, which is relevant below, too:
>

OK and thanks.

>
> > 2)
> >
> https://wiki.qemu.org/Google_Summer_of_Code_2021#Interactive.2C_asynchronous_QEMU_Machine_Protocol_.28QMP.29_text_user_interface_.28TUI.29
> > <
> https://wiki.qemu.org/Google_Summer_of_Code_2021#Interactive.2C_asynchronous_QEMU_Machine_Protocol_.28QMP.29_text_user_interface_.28TUI.29
> >
> > This is something that I don't know much about. I have a basic idea
> > about what
> > QMP is but I never used it. The docs say that the Async QMP library is a
> > work
> > in progress. If someone can hook me up with some small tasks in this
> > library it would
> > be really helpful in improving my understanding.
> >
>
> 1) About QMP and QAPI:
>
> QMP is the JSON-like runtime protocol that QEMU supports. These two
> documents should get you started on understanding what QMP is and how it
> works:
>
>
> https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/qmp-intro.txt
>
>
> https://gitlab.com/qemu-project/qemu/-/blob/master/docs/interop/qmp-spec.txt
>
> There's a python script (scripts/qmp/qmp-shell) that can be used to
> issue an unsupported* short-hand syntax that translates into "real" QMP.
> Or, as qmp-intro.txt says, you can use telnet or socat to copy/paste
> JSON straight into the socket.
>
> The commands that QMP accepts are defined by QAPI; those are defined in
> ./qapi/*.json -- a component called the QAPI generator digests this
> information and generates the runtime server code that handles parsing
> and dispatch.
>
>
> 2) About QMP Events
>
> Normally, after the initial handshake, QMP is a call-and-response
> protocol. The client sends an RPC request, the server executes the
> command and sends the response.
>
> QMP also supports asynchronous events, though. At any point, QMP can
> send an unprompted status message that informs the client of some state
> change within QEMU. This is seen most often for changes in the system
> emulator runstate, i.e. if QEMU is paused or resumed, etc.
>
> (It's also used for reporting block device errors, long-running task
> status updates like backup, etc.)
>
> It's likely that Phil wants to use this functionality to send
> information about GPIO state changes such that a client can render a
> meaningful visualization.
>


> 3) About AQMP
>
> QEMU today has a QMP library written in Python at ./python/qemu/qmp.py.
> This is a synchronous library that blocks execution until the command is
> done executing on the server.
>
> We have augmented it over the years to support caching events we
> received while waiting for execution to finish (for later retrieval),
> but it requires the caller to go back and check those cached events. It
> does not offer event handling callbacks. It is a very low-level library
> that is prone to race conditions depending on how the test using it is
> written.
>
> qmp-shell uses the old qmp library: it does not show you incoming events
> as they happen. You have to press "enter" with an empty buffer to coax
> the shell to check for new events and print them for you, which can be a
> little annoying if you want live updates.
>
> I started writing AQMP using Python's asyncio/await keywords to create a
> more modern, flexible QMP library to replace it. One of the hopes I have
> for this library is that it will handle asynchronous events much more
> nicely. It's my hope that qmp-shell can be upgraded to use this newer
> library instead.
>
> The AQMP library is about half finished: It supports all of the basic
> functionality of the protocol, but needs work on the callback API for
> dispatching event responses. It also likely needs a lot of testing and
> polish that's likely to become obvious as anyone tries to integrate it
> into a real program like a theoretical qmp-shell-2.0.
>

Thank you so much for the detailed answer. This helped a lot.
After reading the docs and messing around with simple commands.
I am confused about few things.

1) What is the difference between QMPShell and HMPShell? My understanding
after going through code and doco is both talk to QEMU using QMP and
basically
do the same thing but HMPShell is a subset of QMP shell which is
more human-friendly
compared to QMPShell. Is that right?

2) When I press <CTRL-A> - C in QEMU I get a monitor prompt, after
reading the man page I go to know that I can use telnet or socat to control
as well. Is this another interface to QEMU which uses QMP?

4) Understanding my project
>
> If you're still interested in my project, I'd recommend trying out
> qmp-shell against a running QEMU instance and issuing a few basic,
> boring commands ("query-status" is a good candidate) and seeing how that
> works.
>
> Then, I'd take a look at some of the other projects I mentioned
> (mitmproxy, irssi) to get a sense of what the work is here. This is
> largely a UI/API programming task, and there's real work to do on the
> AQMP library, but it's probably closer to the surface than the deep
> technical internals of QEMU.
>
> It might be a good introductory project that helps you get a better
> overview of the internals of QEMU if you're interested in more
> hardware-related aspects, but it still requires you have at least some
> interested in UI programming and API design.
>
> Phil's project might involve hardware specifics quite a bit more than
> mine, while still teaching you some overview of QMP as a necessity of
> the project.
>

TBH UI and Async are both quite new to me, I have only done CLI
stuff since there isn't much UI in low-level dev. I also wanted to try out
async
dev at a serious scale but never got an opportunity to do so. My rationale
behind choosing Phil's project is it would let me learn about the hardware
emulation and more importantly the visualization stuff using QEMU. But since
your project involves also async stuff I would love to work on your project
if
you allow me to ;).


> If you remain interested after the above, I can point you towards some
> more concrete tasks that need doing for you to get a fairly concrete
> sense of what the project entails.
>

Please, If you have any small tasks pending, I would like to work on them.


>
> > I would like to work on these projects even outside of GSoC if someone
> > is ready to
> > mentor in their free time :).
> >
>

Thanks,
Niteesh.

> Feel free to join #qemu-gsoc on irc.oftc.net. If you've not joined an
> IRC channel before, it's kind of like a prehistoric slack channel.
>
> Linux GUI: xchat, hexchat
> Linux TUI: irssi, WeeChat
>
> OSX GUI: LimeChat, Colloquy (I've never used either)
> OSX TUI: irssi and weechat should be available via ports (Not tried.)
>
> Windows GUI: mIRC, XChat
>
>
> I'm jsnow on OFTC. You can use my nickname at the start of a message
> ("jsnow: Hello, this is Niteesh from the mailing list") and it will show
> me a notification -- but the hours I am paying attention to IRC are
> around 10AM - 7PM EST. (15:00 - 00:00 GMT)
>
> I can be around later by request (00:00 - 05:00 GMT) if you give me some
> advance notice.
>
> > Thanks
> > Niteesh.
> >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 15573 bytes --]

Re: Interested in contributing to QEMU

[John Snow]

On 2/9/21 8:57 AM, Niteesh G. S. wrote:
> Hello John,
> On Mon, Feb 8, 2021 at 8:44 PM John Snow <jsnow@redhat.com 
> <mailto:jsnow@redhat.com>> wrote:
>     On 2/7/21 12:01 PM, Niteesh G. S. wrote:

[...]

> 
> 
> Which one in your opinion will yield a faster response, Mailing list or IRC?
> My problem with IRC is, I don't have an IRC bouncer set up so I miss
> some conversations. Which one is simpler to use? There are many options
> available. Is there something that will not require me to host a server?
> 

Indeed one of the problems with IRC :)

IRC can be faster if you happen to join at the right time, when people 
are around and paying attention. We generally conduct interviews and 
meetings for GSoC/Outreachy at scheduled times on IRC.

If you don't get an answer fairly quickly on IRC, send a mail. If you'd 
like to chat more interactively than mail allows, try and ask for an 
explicit time for someone to meet with you on IRC.

>     See the end of this mail for instructions on how to join, if you
>     need them.
> 
> I am already part of the QEMU IRC channel. But thanks for these 
> instructions.
> 

Sorry, I wasn't sure! I try to be aware that IRC is increasingly a thing 
that only weird old coots use :p

[...]

> 
> 
> Thank you so much for the detailed answer. This helped a lot.
> After reading the docs and messing around with simple commands.
> I am confused about few things.
> 
> 1) What is the difference between QMPShell and HMPShell? My understanding
> after going through code and doco is both talk to QEMU using QMP and 
> basically
> do the same thing but HMPShell is a subset of QMP shell which is 
> more human-friendly
> compared to QMPShell. Is that right?
> 

"It's complicated ..."

HMP shell is an older protocol that QEMU has that is meant to be more 
human-friendly, yes. Many of its commands are implemented in terms of 
QMP commands, but not all of them. It isn't right to call it a "subset".

QMP was introduced to create an API that was more machine-friendly than 
HMP, but we didn't finish porting everything from HMP to QMP, for a few 
reasons.

(1) HMP does not have any backwards compatibility promises; this makes 
it more appealing for various debugging commands and alpha/beta versions 
of commands. QMP is meant to have fairly strict backwards compatibility 
and a reliable API.

(2) QMP is designed as a control channel and not a data transfer 
channel, so some commands that may transmit large amounts of data or 
take indeterminate amounts of time have remained stuck in HMP without a 
proper QMP equivalent.

Naturally, you can use QMP to issue HMP-only commands if you want. There 
are a few uses of this in ./tests/qemu-iotests to do various "unsafe" 
things.

Daniel Berrange is working on porting "savevm" from HMP to QMP as one of 
our last holdouts that isn't just debugging/query stuff.

Whether or not we want to sunset HMP long-term is, AFAIK, an unresolved 
question. There are obvious uses for a user-friendly shell. Part of the 
qmp-shell revamp project is to investigate the viability of a 
user-friendly shell backed only by QMP, where the "friendly" bits are 
outside of QEMU proper and not subject to backwards compatibility promises.

> 2) When I press <CTRL-A> - C in QEMU I get a monitor prompt, after
> reading the man page I go to know that I can use telnet or socat to control
> as well. Is this another interface to QEMU which uses QMP?
> 

I'm not sure I know what this keypress does! If you see a prompt, 
though, it's HMP. Type "help" and send a newline.

The QMP protocol is described somewhere in /docs/ like I pointed out in 
my reply. You should get a greeting and you should be able to post a qmp 
capabilities handshake and get a reply.

>     4) Understanding my project
> 
>     If you're still interested in my project, I'd recommend trying out
>     qmp-shell against a running QEMU instance and issuing a few basic,
>     boring commands ("query-status" is a good candidate) and seeing how
>     that
>     works.
> 
>     Then, I'd take a look at some of the other projects I mentioned
>     (mitmproxy, irssi) to get a sense of what the work is here. This is
>     largely a UI/API programming task, and there's real work to do on the
>     AQMP library, but it's probably closer to the surface than the deep
>     technical internals of QEMU.
> 
>     It might be a good introductory project that helps you get a better
>     overview of the internals of QEMU if you're interested in more
>     hardware-related aspects, but it still requires you have at least some
>     interested in UI programming and API design.
> 
>     Phil's project might involve hardware specifics quite a bit more than
>     mine, while still teaching you some overview of QMP as a necessity of
>     the project.
> 
> TBH UI and Async are both quite new to me, I have only done CLI
> stuff since there isn't much UI in low-level dev. I also wanted to try 
> out async
> dev at a serious scale but never got an opportunity to do so. My rationale
> behind choosing Phil's project is it would let me learn about the hardware
> emulation and more importantly the visualization stuff using QEMU. But since
> your project involves also async stuff I would love to work on your 
> project if
> you allow me to ;).
> 

Yup, understood.

>     If you remain interested after the above, I can point you towards some
>     more concrete tasks that need doing for you to get a fairly concrete
>     sense of what the project entails.
> 
> Please, If you have any small tasks pending, I would like to work on them.
> 

OK: I think I need to be careful about "issuing" work to someone who 
isn't (yet) accepted into the program -- I shouldn't misrepresent this 
arrangement -- but I can give you some more research tips that may help 
you find your footing.

We can work on getting to know QMP a bit better; it sounds like it'd be 
relevant for both projects.

Try using '-qmp qmp.sock,server,nowait' when you boot up QEMU and then 
open the qmp.sock file with socat and try messing with it.

Try going to ./qapi/ in the source tree and "git grep event" to find 
some event definitions. try grepping for that event name in the QEMU 
tree at large and try to work out how QEMU emits these events.

Try *adding* an event somewhere in ./qapi/ and modifying QEMU to emit 
this event. Try using rlwrap and socat to connect to this QMP socket and 
verify that your event is being sent over the wire. Depending on where 
you add the event, it may be helpful to start QEMU in a paused state and 
issue a resume command from QMP to make sure you have time to connect to 
the socket.

For more hardware-oriented avenues of exploration, I'd encourage 
reaching out to phil, who seems to have a knack for finding weird 
embedded devices to babysit :)

> 
>      > I would like to work on these projects even outside of GSoC if
>     someone
>      > is ready to
>      > mentor in their free time :).
>      >
> 
> Thanks,
> Niteesh.
> 
>     Feel free to join #qemu-gsoc on irc.oftc.net <http://irc.oftc.net>.
>     If you've not joined an
>     IRC channel before, it's kind of like a prehistoric slack channel.
> 
>     Linux GUI: xchat, hexchat
>     Linux TUI: irssi, WeeChat
> 
>     OSX GUI: LimeChat, Colloquy (I've never used either)
>     OSX TUI: irssi and weechat should be available via ports (Not tried.)
> 
>     Windows GUI: mIRC, XChat
> 
> 
>     I'm jsnow on OFTC. You can use my nickname at the start of a message
>     ("jsnow: Hello, this is Niteesh from the mailing list") and it will
>     show
>     me a notification -- but the hours I am paying attention to IRC are
>     around 10AM - 7PM EST. (15:00 - 00:00 GMT)
> 
>     I can be around later by request (00:00 - 05:00 GMT) if you give me
>     some
>     advance notice.
> 
>      > Thanks
>      > Niteesh.
>      >
>      >
> 

Re: Interested in contributing to QEMU

[Niteesh G. S.]

[-- Attachment #1: Type: text/plain, Size: 10549 bytes --]

Hello John,

On Wed, Feb 10, 2021 at 10:33 PM John Snow <jsnow@redhat.com> wrote:

> On 2/9/21 8:57 AM, Niteesh G. S. wrote:
> > Hello John,
> > On Mon, Feb 8, 2021 at 8:44 PM John Snow <jsnow@redhat.com
> > <mailto:jsnow@redhat.com>> wrote:
> >     On 2/7/21 12:01 PM, Niteesh G. S. wrote:
>
> [...]
>
> >
> >
> > Which one in your opinion will yield a faster response, Mailing list or
> IRC?
> > My problem with IRC is, I don't have an IRC bouncer set up so I miss
> > some conversations. Which one is simpler to use? There are many options
> > available. Is there something that will not require me to host a server?
> >
>
> Indeed one of the problems with IRC :)
>
> IRC can be faster if you happen to join at the right time, when people
> are around and paying attention. We generally conduct interviews and
> meetings for GSoC/Outreachy at scheduled times on IRC.
>
> If you don't get an answer fairly quickly on IRC, send a mail. If you'd
> like to chat more interactively than mail allows, try and ask for an
> explicit time for someone to meet with you on IRC.
>
OK.

> >     See the end of this mail for instructions on how to join, if you
> >     need them.
> >
> > I am already part of the QEMU IRC channel. But thanks for these
> > instructions.
> >
>
> Sorry, I wasn't sure! I try to be aware that IRC is increasingly a thing
> that only weird old coots use :p
>
> [...]
>
> >
> >
> > Thank you so much for the detailed answer. This helped a lot.
> > After reading the docs and messing around with simple commands.
> > I am confused about few things.
> >
> > 1) What is the difference between QMPShell and HMPShell? My understanding
> > after going through code and doco is both talk to QEMU using QMP and
> > basically
> > do the same thing but HMPShell is a subset of QMP shell which is
> > more human-friendly
> > compared to QMPShell. Is that right?
> >
>
> "It's complicated ..."
>
> HMP shell is an older protocol that QEMU has that is meant to be more
> human-friendly, yes. Many of its commands are implemented in terms of
> QMP commands, but not all of them. It isn't right to call it a "subset".
>
> QMP was introduced to create an API that was more machine-friendly than
> HMP, but we didn't finish porting everything from HMP to QMP, for a few
> reasons.
>
> (1) HMP does not have any backwards compatibility promises; this makes
> it more appealing for various debugging commands and alpha/beta versions
> of commands. QMP is meant to have fairly strict backwards compatibility
> and a reliable API.
>
> (2) QMP is designed as a control channel and not a data transfer
> channel, so some commands that may transmit large amounts of data or
> take indeterminate amounts of time have remained stuck in HMP without a
> proper QMP equivalent.
>
> Naturally, you can use QMP to issue HMP-only commands if you want. There
> are a few uses of this in ./tests/qemu-iotests to do various "unsafe"
> things.
>
> Daniel Berrange is working on porting "savevm" from HMP to QMP as one of
> our last holdouts that isn't just debugging/query stuff.
>
> Whether or not we want to sunset HMP long-term is, AFAIK, an unresolved
> question. There are obvious uses for a user-friendly shell. Part of the
> qmp-shell revamp project is to investigate the viability of a
> user-friendly shell backed only by QMP, where the "friendly" bits are
> outside of QEMU proper and not subject to backwards compatibility promises.
>
> > 2) When I press <CTRL-A> - C in QEMU I get a monitor prompt, after
> > reading the man page I go to know that I can use telnet or socat to
> control
> > as well. Is this another interface to QEMU which uses QMP?
> >
>
> I'm not sure I know what this keypress does! If you see a prompt,
> though, it's HMP. Type "help" and send a newline.
>
> The QMP protocol is described somewhere in /docs/ like I pointed out in
> my reply. You should get a greeting and you should be able to post a qmp
> capabilities handshake and get a reply.
>
Thank you for the explanation.

>
> >     4) Understanding my project
> >
> >     If you're still interested in my project, I'd recommend trying out
> >     qmp-shell against a running QEMU instance and issuing a few basic,
> >     boring commands ("query-status" is a good candidate) and seeing how
> >     that
> >     works.
> >
> >     Then, I'd take a look at some of the other projects I mentioned
> >     (mitmproxy, irssi) to get a sense of what the work is here. This is
> >     largely a UI/API programming task, and there's real work to do on the
> >     AQMP library, but it's probably closer to the surface than the deep
> >     technical internals of QEMU.
> >
> >     It might be a good introductory project that helps you get a better
> >     overview of the internals of QEMU if you're interested in more
> >     hardware-related aspects, but it still requires you have at least
> some
> >     interested in UI programming and API design.
> >
> >     Phil's project might involve hardware specifics quite a bit more than
> >     mine, while still teaching you some overview of QMP as a necessity of
> >     the project.
> >
> > TBH UI and Async are both quite new to me, I have only done CLI
> > stuff since there isn't much UI in low-level dev. I also wanted to try
> > out async
> > dev at a serious scale but never got an opportunity to do so. My
> rationale
> > behind choosing Phil's project is it would let me learn about the
> hardware
> > emulation and more importantly the visualization stuff using QEMU. But
> since
> > your project involves also async stuff I would love to work on your
> > project if
> > you allow me to ;).
> >
>
> Yup, understood.
>
> >     If you remain interested after the above, I can point you towards
> some
> >     more concrete tasks that need doing for you to get a fairly concrete
> >     sense of what the project entails.
> >
> > Please, If you have any small tasks pending, I would like to work on
> them.
> >
>
> OK: I think I need to be careful about "issuing" work to someone who
> isn't (yet) accepted into the program -- I shouldn't misrepresent this
> arrangement -- but I can give you some more research tips that may help
> you find your footing.
>
> We can work on getting to know QMP a bit better; it sounds like it'd be
> relevant for both projects.
>
> Try using '-qmp qmp.sock,server,nowait' when you boot up QEMU and then
> open the qmp.sock file with socat and try messing with it.
>
> Try going to ./qapi/ in the source tree and "git grep event" to find
> some event definitions. try grepping for that event name in the QEMU
> tree at large and try to work out how QEMU emits these events.
>
> Try *adding* an event somewhere in ./qapi/ and modifying QEMU to emit
> this event. Try using rlwrap and socat to connect to this QMP socket and
> verify that your event is being sent over the wire. Depending on where
> you add the event, it may be helpful to start QEMU in a paused state and
> issue a resume command from QMP to make sure you have time to connect to
> the socket.
>

I added an event that is emitted every keypress. It also sends the qcode of
the key pressed.
https://github.com/gs-niteesh/qemu/pull/1/commits/1684f1f499e67ec4af5f75d0cc09eb1ecf9be709
After doing this, I think I have a basic understanding of how the event
stuff works and also
what happens when a command is issued from the shell.
I have summarized my understanding below, please correct me if I am wrong
anywhere.
1) The JSON files under qemu/qapi define the QMP messages. These JSON files
are then
compiled into their C representations during the build time.
2) These are then registered using qmp_register_command.
3) When a QMP request is initiated, the commands respective C function is
found from the
registered list of commands and is dispatched. One of the main functions
responsible for the
finding and dispatching of qmp_* functions is qmp_dispatch in
qapi/qmp-dispatch.c. Which
is invoked from monitor/qmp.c upon a QMP request. The result is then
published to the user
through monitor_qmp_respond -> qmp_send_response.
Similar stuff happens to events too, the difference being, the function
definitions are autogenerated
depending on their schema, whereas for commands only the function
prototypes are generated.
The events can be emitted through qapi_event_send_EVENT_NAME functions.
These functions
convert C to JSON and call qapi_event_emit which then finally calls the
monitor function to display an
event to the user.

Few things I couldn't figure out and understand are
1) Who informs the monitor that the command is been issued? Since there are
many ways to
connect to the monitor, I think there should be a generic API which informs
about input data.
But I can't find the code behind all this.
Before sending this mail, I decided to give this another try and found
qemu_chr_fe_set_handlers is responsible for register the handlers for input
and output. Is that
right? And also does chr_fe mean character_front_end?

2) I couldn't understand QEMU co-routines in general. In this case, I
couldn't understand the
coroutine code in qmp_dispatch in qapi/qmp-dispatch.c. Also please explain
OOB in simple
terms? How are they different from normal commands? And also their relation
with coroutines.

Thanks,
Niteesh.

>
> For more hardware-oriented avenues of exploration, I'd encourage
> reaching out to phil, who seems to have a knack for finding weird
> embedded devices to babysit :)
>
> >
> >      > I would like to work on these projects even outside of GSoC if
> >     someone
> >      > is ready to
> >      > mentor in their free time :).
> >      >
> >
> > Thanks,
> > Niteesh.
> >
> >     Feel free to join #qemu-gsoc on irc.oftc.net <http://irc.oftc.net>.
> >     If you've not joined an
> >     IRC channel before, it's kind of like a prehistoric slack channel.
> >
> >     Linux GUI: xchat, hexchat
> >     Linux TUI: irssi, WeeChat
> >
> >     OSX GUI: LimeChat, Colloquy (I've never used either)
> >     OSX TUI: irssi and weechat should be available via ports (Not tried.)
> >
> >     Windows GUI: mIRC, XChat
> >
> >
> >     I'm jsnow on OFTC. You can use my nickname at the start of a message
> >     ("jsnow: Hello, this is Niteesh from the mailing list") and it will
> >     show
> >     me a notification -- but the hours I am paying attention to IRC are
> >     around 10AM - 7PM EST. (15:00 - 00:00 GMT)
> >
> >     I can be around later by request (00:00 - 05:00 GMT) if you give me
> >     some
> >     advance notice.
> >
> >      > Thanks
> >      > Niteesh.
> >      >
> >      >
> >
>
>

[-- Attachment #2: Type: text/html, Size: 15285 bytes --]

Re: Interested in contributing to QEMU

[John Snow]

On 2/12/21 7:10 AM, Niteesh G. S. wrote:
> Hello John,
> 
> On Wed, Feb 10, 2021 at 10:33 PM John Snow <jsnow@redhat.com 
> <mailto:jsnow@redhat.com>> wrote:
> 

[...]

> 
>     OK: I think I need to be careful about "issuing" work to someone who
>     isn't (yet) accepted into the program -- I shouldn't misrepresent this
>     arrangement -- but I can give you some more research tips that may help
>     you find your footing.
> 
>     We can work on getting to know QMP a bit better; it sounds like it'd be
>     relevant for both projects.
> 
>     Try using '-qmp qmp.sock,server,nowait' when you boot up QEMU and then
>     open the qmp.sock file with socat and try messing with it.
> 
>     Try going to ./qapi/ in the source tree and "git grep event" to find
>     some event definitions. try grepping for that event name in the QEMU
>     tree at large and try to work out how QEMU emits these events.
> 
>     Try *adding* an event somewhere in ./qapi/ and modifying QEMU to emit
>     this event. Try using rlwrap and socat to connect to this QMP socket
>     and
>     verify that your event is being sent over the wire. Depending on where
>     you add the event, it may be helpful to start QEMU in a paused state
>     and
>     issue a resume command from QMP to make sure you have time to
>     connect to
>     the socket.
> 
> I added an event that is emitted every keypress. It also sends the qcode 
> of the key pressed.
> https://github.com/gs-niteesh/qemu/pull/1/commits/1684f1f499e67ec4af5f75d0cc09eb1ecf9be709 
> <https://github.com/gs-niteesh/qemu/pull/1/commits/1684f1f499e67ec4af5f75d0cc09eb1ecf9be709>
> After doing this, I think I have a basic understanding of how the event 
> stuff works and also
> what happens when a command is issued from the shell.

Good!

> I have summarized my understanding below, please correct me if I am 
> wrong anywhere.

> 1) The JSON files under qemu/qapi define the QMP messages. These JSON 
> files are then
> compiled into their C representations during the build time.

Yep! They are processed using the scripts/qapi/ python package, the QAPI 
generator, which builds those definitions into concrete C types, wires 
up the command handlers for the QMP socket, and so on.

> 2) These are then registered using qmp_register_command.

Yeah, but in most cases the code generator actually handles this part 
for you. Every QMP command we have is (to my knowledge) defined as a 
'command' in the JSON files.

All one has to do as the implementor is to write a function named 
"qmp_my_command_name" with the correct arguments and types. (And make 
sure it gets linked into the final binary!)

> 3) When a QMP request is initiated, the commands respective C function 
> is found from the
> registered list of commands and is dispatched. One of the main functions 
> responsible for the
> finding and dispatching of qmp_* functions is qmp_dispatch in 
> qapi/qmp-dispatch.c. Which
> is invoked from monitor/qmp.c upon a QMP request. The result is then 
> published to the user
> through monitor_qmp_respond -> qmp_send_response.

I believe so. I haven't actually looked too closely at how the API 
wrangler/dispatcher code works. The high-level abstraction is actually 
pretty good in most cases.

> Similar stuff happens to events too, the difference being, the function 
> definitions are autogenerated
> depending on their schema, whereas for commands only the function 
> prototypes are generated.
> The events can be emitted through qapi_event_send_EVENT_NAME functions. 
> These functions
> convert C to JSON and call qapi_event_emit which then finally calls the 
> monitor function to display an
> event to the user.
> 

Yep, exactly.

> Few things I couldn't figure out and understand are
> 1) Who informs the monitor that the command is been issued? Since there 
> are many ways to
> connect to the monitor, I think there should be a generic API which 
> informs about input data.
> But I can't find the code behind all this.

Yes. monitor/qmp.c handles the implementation of the monitor, but 
doesn't specify the actual transports for how the data arrives.

We rely on the chardev for that. See the signature for monitor_init_qmp, 
which receives a "chardev" and then uses it to start listening for data.

> Before sending this mail, I decided to give this another try and found
> qemu_chr_fe_set_handlers is responsible for register the handlers for 
> input and output. Is that
> right? And also does chr_fe mean character_front_end?
> 

QEMU likes to separate out the idea of a frontend and a backend to a 
device. A virtual device presented to a guest as some specific model of 
hardware is the frontend.

The way we achieve the functionality of that device on the user's 
physical machine is the device backend. For character devices, 
(chardev), the backend might just be stdio on your terminal.

...QMP is acting kind of like a frontend device in that it's a user of a 
backend, but it's not really a guest-facing device. FE/BE distinctions 
get kind of weird at times for that reason.

> 2) I couldn't understand QEMU co-routines in general. In this case, I 
> couldn't understand the
> coroutine code in qmp_dispatch in qapi/qmp-dispatch.c. Also please 
> explain OOB in simple
> terms? How are they different from normal commands? And also their 
> relation with coroutines.
> 

Coroutines are a type of "cooperative" user scheduling. In Linux, this 
means that the user process itself switching between tasks at its own 
discretion, independent of the Linux scheduler.

It is "cooperative" because each task needs to indicate when it is ready 
to yield, they are not pre-empted. When a task needs to wait for an 
operation to be ready, it can yield to allow another task to run.

In C on Linux, you could implement this with stackswaps (makecontext, 
swapcontext, setcontext and so on), but there are other ways, and other 
OSes that need an implementation. Look at `coroutine-ucontext.c` for one 
possible implementation.

QMP uses them to dispatch certain commands; those marked as such in 
those qapi JSON files. Look at the definition for 'block_resize'.

OOB commands are complicated, check the docs/devel/qmp-spec.txt file for 
more info on those. Simply (with lots of caveats and gotchas) is that 
QMP commands are processed one at a time in FIFO order. OOB commands, 
however, will "skip the queue" and may be processed immediately. That's 
roughly why "OOB" exists; it's to send critical commands like "cancel" 
or "exit" to try and un-jam QEMU when it gets stuck processing a 
previous command.

Because of this, the handling over what threading context we execute 
those commands in is handled a little differently / more carefully.

> Thanks,
> Niteesh.
> 

--js