python asyncio QMP library (AQMP)

python asyncio QMP library (AQMP)

[John Snow]

I've been doodling a new Asyncio-based QMP library that might fix some 
of the problems[1][2] with our older, non-async QMP library and provide 
a better basis for a proper distributable python package for people to 
write their own toy scripts to control QEMU.

It's a very early prototype, but I think I have the basics down and 
working now. I figured I would float my extremely early copy of it in 
case interested parties wanted to take a peek. (Or just to formally 
announce that "Hey, I'm working on this!")

https://gitlab.com/jsnow/aqmp/-/tree/devel

The design uses two independent coroutines in the bottom half, a reader 
and a writer. execute() works something like this:

                        +---------+
      +---------------- |execute()| <----------+
      |                 +---------+            |
      |                                        |
-----------------------------------------------------------
      v                                        |
+----+----+    +-----------+           +------+-------+
|Mailboxes|    |Event Queue|           |Outbound Queue|
+----+----+    +------+----+           +------+-------+
      |                |                       ^
      v                v                       |
   +--+----------------+---+       +-----------+-----------+
   | Reader Task/Coroutine |       | Writer Task/Coroutine |
   +-----------+-----------+       +-----------+-----------+
               |                               ^
               v                               |
         +-----+------+                  +-----+------+
         |StreamReader|                  |StreamWriter|
         +------------+                  +------------+


Execute deposits a message in the outbound queue; the writer issues it 
to the server. Execute goes to sleep waiting for mail in its inbox. The 
reader awakens upon new data available in the stream, the new data is 
placed in the mailbox, which awakens the caller.

The Reader/Writer are each executed by _task_runner, which ensures that 
any error that occurs in the BH will create a disconnection Task that 
will quiesce everything in the bottom half.

Any pending execute actions will be cancelled by the disconnection task; 
and will raise "DisconnectedError" to the caller.

Upon learning of a disconnection event, the client code responsible for 
managing the overall state of the connection can call disconnect(), 
which will raise any exceptions that occurred in the bottom halves that 
caused the unscheduled disconnect. From there, the client can decide to 
reconnect or abort, depending.


classes and things of interest:

aqmp.py - Mailbox: Simple data structure pairing an asyncio.Event with a 
Message. This is done to allow the disconnect task to wake up any 
waiting clients even if no message arrived. Python's asyncio.Queue does 
not offer a cancel primitive, so this approximates it.

aqmp.py - AQMP: The core of the loop logic is written here; the 
public-facing methods are connect(), disconnect(), execute() and 
execute_obj(), with a "make_execute_msg" class method available for 
creating Messages that can be executed later.

message.py - Message: This class comprises a JSON Object as read off the 
wire; putting it in a class like this allows me to write stricter type 
guarantees and offer _serialize/_deserialize class methods.

models.py - This module offers strictly typed and validated 
deserializations of core QMP structures that are not (generally) defined 
by the QAPI schema; i.e., it offers types, de/serialization, and 
validation for things such as "Greeting", "SuccessResponse", 
"AsynchronousEvent", etc.


What's left to do?

- Work on Exception naming and hierarchy,
   more helpful/pretty display of errors.
- Expand the event handling system to be more useful (Maybe?)
- Add QGA support
- Re-add socket creation support (Presently, it relies on the socket 
being created already.)
- Documentation, tests, etc.


Might be nice:

- Look into creating a "Server" variant to be used for testing the 
library, or even other clients.
- Look into synchronous wrappers that offer thread-safe primitives that 
can be used from traditional, synchronous contexts. (preventing "async 
creep".)


Long term goal:

- Get it merged in-tree; replace the legacy QMP module over time.

- Create an auto-generated SDK layer based on our QAPI schema that adds 
a library of type-safe commands, responses, and events that uses this 
QAPI-agnostic QMP library as its core.

- Add my "JobRunner" patches from earlier this year as a full-fledged 
jobs API that is designed to make running long-running tasks very 
trivial from a python console / qmp-shell.


--js


[1] qmp-shell is not capable of displaying events as they arrive 
asynchronously very well.
[2] Our use of non-blocking sockets and socket timeouts in qmp.py relies 
quite a bit on undefined behavior that thankfully has not broken yet.