On Wed, Sep 15, 2021 at 12:30 PM John Snow <jsnow@redhat.com> wrote:

> GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-async-qmp-aqmp
> CI: https://gitlab.com/jsnow/qemu/-/pipelines/371343890
> Docs: https://people.redhat.com/~jsnow/sphinx/html/qemu.aqmp.html
> Based-on: <20210915154031.321592-2-jsnow@redhat.com>
>
> (This is a quick V4 to add a few tiny edits and get the proper mboxes
> out on the list. My current plan is to send a PR for this series this
> week. Don't panic, it isn't used by default anywhere yet, nothing should
> break.)
>
> Hi!
>
> This patch series adds an Asynchronous QMP package to the Python
> library. It offers a few improvements over the previous library:
>
> - out-of-band support
> - true asynchronous event support
> - avoids undocumented interfaces abusing non-blocking sockets
> - unit tests!
> - documentation!
>
> This library serves as the basis for a new qmp-shell program that will
> offer improved reconnection support, true asynchronous display of
> events, VM and job status update notifiers, and so on.
>
> My intent is to eventually publish this library directly to PyPI as a
> standalone package. I would like to phase out our usage of the old QMP
> library over time; eventually replacing it entirely with this
> one. (Since v2 of this series, I have authored a compatibility shim not
> included in this series that can be used to run all of iotests on this
> new library successfully with very minimal churn.)
>
> This series looks big by line count, but it's *mostly*
> docstrings. Seriously!
>
> This package has *no* external dependencies whatsoever.
>
> Notes & Design
> ==============
>
> Here are some notes on the design of how the library works, to serve as
> a primer for review; however I also **highly recommend** browsing the
> generated Sphinx documentation for this series.
>
> Here's that link again:
> https://people.redhat.com/~jsnow/sphinx/html/qemu.aqmp.html
>
> The core machinery is split between the AsyncProtocol and QMPClient
> classes. AsyncProtocol provides the generic machinery, while QMPClient
> provides the QMP-specific details.
>
> The design uses two independent coroutines that act as the "bottom
> half", a writer task and a reader task. These tasks run for the duration
> of the connection and independently send and receive messages,
> respectively.
>
> A third task, disconnect, is scheduled asynchronously whenever an
> unrecoverable error occurs and facilitates coalescing of the other two
> tasks.
>
> This diagram for how execute() operates may be helpful for understanding
> how AsyncProtocol is laid out. The arrows indicate the direction of a
> QMP message; the long horizontal dash indicates the separation between
> the upper and lower halves of the event loop. The queue mechanisms
> between both dashes serve as the intermediaries between the upper and
> lower halves.
>
>                        +---------+
>                        | caller  |
>                        +---------+
>                            ^ |
>                            | v
>                        +---------+
>      +---------------> |execute()| -----------+
>      |                 +---------+            |
>      |                                        |
> [-----------------------------------------------------------]
>      |                                        |
>      |                                        v
> +----+------+    +----------------+    +------+-------+
> | ExecQueue |    | EventListeners |    |Outbound Queue|
> +----+------+    +----+-----------+    +------+-------+
>      ^                ^                       |
>      |                |                       |
> [-----------------------------------------------------------]
>      |                |                       |
>      |                |                       v
>   +--+----------------+---+       +-----------+-----------+
>   | Reader Task/Coroutine |       | Writer Task/Coroutine |
>   +-----------+-----------+       +-----------+-----------+
>               ^                               |
>               |                               v
>         +-----+------+                  +-----+------+
>         |StreamReader|                  |StreamWriter|
>         +------------+                  +------------+
>
> The caller will invoke execute(), which in turn will deposit a message
> in the outbound send queue. This will wake up the writer task, which
> well send the message over the wire.
>
> The execute() method will then yield to wait for a reply delivered to an
> execution queue created solely for that execute statement.
>
> When a message arrives, the Reader task will unblock and route the
> message either to the EventListener subsystem, or place it in the
> appropriate pending execution queue.
>
> Once a message is placed in the pending execution queue, execute() will
> unblock and the execution will conclude, returning the result of the RPC
> call to the caller.
>
> Patch Layout
> ============
>
> Patches 1-4   add tiny pre-requisites, utilities, etc.
> Patches 5-12  add a generic async message-based protocol class,
>               AsyncProtocol. They are split fairly small and should
>               be reasonably self-contained.
> Patches 13-15 check in more QMP-centric components.
> Patches 16-21 add qmp_client.py, with a new 'QMPClient()' class.
>               They're split into reasonably tiny pieces here.
> Patches 22-23 add a few finishing touches, they are small patches.
> Patches 24-27 adds unit tests. They're a little messy still, but
>               they've been quite helpful to me so far. Coverage of
>               protocol.py is at about ~86%.
>
> Future Work
> ===========
>
> These items are in progress:
>
> - A synchronous QMP wrapper that allows this library to be easily used
>   from non-async code; this will also allow me to prove it works well by
>   demoing its replacement throughout iotests.
>
>   This work is feature-complete, but needs polish. All of iotests is now
>   passing with Async QMP and this Sync wrapper. This will be its own
>   follow-up series.
>
> - A QMP server class; to facilitate writing of unit tests. An early
>   version is done, but possibly not feature complete. More polish and
>   tests are warranted. This will be its own follow-up series.
>
> - More unit tests for qmp_client.py, qmp_server.py and other modules.
>
> Changelog
> =========
>
> V4:
>
> - (06) Minor typo fix in comment (Eric Blake)
> - (25) Removed stale null_protocol.py file.
> - (26, 27) New - increase test coverage and add coverage.py support.
>
> V3:
>
> - (02, 05) Typo fixes (Eric Blake)
> - (04) Rewrote the "wait_closed" compatibility function for Python 3.6;
>   the older version raised unwanted exceptions in error pathways.
>   (Niteesh)
> - (04, 05, 06, 08) Rewrote _bh_disconnect fairly substantially again;
>   the problem is that exceptions can surface during both flushing of the
>   stream and when waiting for the stream to close. These errors can be
>   new, primary causes of failure or secondary failures. Distinguishing
>   between them is tricky. The new disconnection method takes much
>   greater pains to ensure that even if Exceptions occur, disconnect
>   *will* complete. This adds robustness to cases exposed by iotests
>   where one or more endpoints might segfault or abort and cleanup can be
>   challenged.
> - (11) Fixed logging hook names (Niteesh)
> - (24, 25) Bumped avocado dependency to v90; It added support for async
>   test functions which made my prior workaround non-suitable. The
>   choices were to mandate <90 and keep the workarounds or mandate >=90
>   and drop the workarounds. I went with the latter.
>
> V2:
>
> Renamed classes/methods:
>
> - Renamed qmp_protocol.py to qmp_client.py
> - Renamed 'QMP' class to 'QMPClient'
> - Renamed _begin_new_session() to _establish_session()
> - Split _establish_connection() out from _new_session().
> - Removed _results() method
>
> Bugfixes:
>
> - Suppress duplicate Exceptions when attempting to drain the
>   StreamWriter
> - Delay initialization of asyncio.Queue and asyncio.Event variables to
>   _new_session or later -- they must not be created outside of the loop,
>   even if they are not async functions.
> - Rework runstate_changed events to guarantee visibility of events to
>   waiters
> - Improve connect()/accept() cleanup to work with
>   asyncio.CancelledError, asyncio.TimeoutError
> - No-argument form of Message() now succeeds properly.
> - flush utility will correctly yield when data is below the "high water
>   mark", giving the stream a chance to actually flush.
> - Increase read buffer size to accommodate query-qmp-schema (Thanks
>   Niteesh)
>
> Ugly bits from V1 removed:
>
> - Remove tertiary filtering from EventListener (for now), accompanying
>   documentation removed from events.py
> - Use asyncio.wait() instead of custom wait_task_done()
> - MultiException is removed in favor of just raising the first Exception
>   that occurs in the bottom half; other Exceptions if any are logged
>   instead.
>
> Improvements:
>
> - QMPClient now allows ID-less execution statements via the _raw()
>   interface.
> - Add tests that grant ~86% coverage of protocol.py to the avocado test
>   suite.
> - Removed 'force' parameter from _bh_disconnect; the disconnection
>   routine determines for itself if we are in the error pathway or not
>   instead now.  This removes any chance of duplicate calls to
>   _schedule_disconnect accidentally dropping the 'force' setting.
>
> Debugging/Testing changes:
>
> - Add debug: bool parameter to asyncio_run utility wrapper
> - Improve error messages for '@require' decorator
> - Add debugging message for state change events
> - Avoid flushing the StreamWriter if we don't have one (This
>   circumstance only arises in testing, but it's helpful.)
> - Improved __repr__ method for AsyncProtocol, and removed __str__
>   method.  enforcing eval(__repr__(x)) == x does not make sense for
>   AsyncProtocol.
> - Misc logging message changes
> - Add a suite of fancy Task debugging utilities.
> - Most tracebacks now log at the DEBUG level instead of
>   CRITICAL/ERROR/WARNING; In those error cases, a one-line summary is
>   logged instead.
>
> Misc. aesthetic changes:
>
> - Misc docstring fixes, whitespace, etc.
> - Reordered the definition of some methods to try and keep similar
>   methods near each other (Moved _cleanup near _bh_disconnect in
>   QMPClient.)
>
> John Snow (27):
>   python/aqmp: add asynchronous QMP (AQMP) subpackage
>   python/aqmp: add error classes
>   python/pylint: Add exception for TypeVar names ('T')
>   python/aqmp: add asyncio compatibility wrappers
>   python/aqmp: add generic async message-based protocol support
>   python/aqmp: add runstate state machine to AsyncProtocol
>   python/aqmp: Add logging utility helpers
>   python/aqmp: add logging to AsyncProtocol
>   python/aqmp: add AsyncProtocol.accept() method
>   python/aqmp: add configurable read buffer limit
>   python/aqmp: add _cb_inbound and _cb_outbound logging hooks
>   python/aqmp: add AsyncProtocol._readline() method
>   python/aqmp: add QMP Message format
>   python/aqmp: add well-known QMP object models
>   python/aqmp: add QMP event support
>   python/pylint: disable too-many-function-args
>   python/aqmp: add QMP protocol support
>   python/pylint: disable no-member check
>   python/aqmp: Add message routing to QMP protocol
>   python/aqmp: add execute() interfaces
>   python/aqmp: add _raw() execution interface
>   python/aqmp: add asyncio_run compatibility wrapper
>   python/aqmp: add scary message
>   python: bump avocado to v90.0
>   python/aqmp: add AsyncProtocol unit tests
>   python/aqmp: add LineProtocol tests
>   python/aqmp: Add Coverage.py support
>
>  python/.gitignore              |   5 +
>  python/Makefile                |   9 +
>  python/Pipfile.lock            |   8 +-
>  python/avocado.cfg             |   3 +
>  python/qemu/aqmp/__init__.py   |  59 +++
>  python/qemu/aqmp/error.py      |  50 ++
>  python/qemu/aqmp/events.py     | 706 ++++++++++++++++++++++++++
>  python/qemu/aqmp/message.py    | 209 ++++++++
>  python/qemu/aqmp/models.py     | 133 +++++
>  python/qemu/aqmp/protocol.py   | 902 +++++++++++++++++++++++++++++++++
>  python/qemu/aqmp/py.typed      |   0
>  python/qemu/aqmp/qmp_client.py | 621 +++++++++++++++++++++++
>  python/qemu/aqmp/util.py       | 217 ++++++++
>  python/setup.cfg               |  17 +-
>  python/tests/protocol.py       | 583 +++++++++++++++++++++
>  15 files changed, 3516 insertions(+), 6 deletions(-)
>  create mode 100644 python/qemu/aqmp/__init__.py
>  create mode 100644 python/qemu/aqmp/error.py
>  create mode 100644 python/qemu/aqmp/events.py
>  create mode 100644 python/qemu/aqmp/message.py
>  create mode 100644 python/qemu/aqmp/models.py
>  create mode 100644 python/qemu/aqmp/protocol.py
>  create mode 100644 python/qemu/aqmp/py.typed
>  create mode 100644 python/qemu/aqmp/qmp_client.py
>  create mode 100644 python/qemu/aqmp/util.py
>  create mode 100644 python/tests/protocol.py
>
> --
> 2.31.1
>
>
>
Thanks, I have preliminarily staged this to my python branch:
https://gitlab.com/jsnow/qemu/-/commits/python

--js