From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-10.5 required=3.0 tests=BAYES_00,DKIM_INVALID, DKIM_SIGNED,HEADER_FROM_DIFFERENT_DOMAINS,HTML_MESSAGE,INCLUDES_PATCH, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 38023C433F5 for ; Mon, 20 Sep 2021 19:52:53 +0000 (UTC) Received: from lists.gnu.org (lists.gnu.org [209.51.188.17]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by mail.kernel.org (Postfix) with ESMTPS id 9DA3F611ED for ; Mon, 20 Sep 2021 19:52:52 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.4.1 mail.kernel.org 9DA3F611ED Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=redhat.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=nongnu.org Received: from localhost ([::1]:52608 helo=lists1p.gnu.org) by lists.gnu.org with esmtp (Exim 4.90_1) (envelope-from ) id 1mSPLP-0000Gh-Ox for qemu-devel@archiver.kernel.org; Mon, 20 Sep 2021 15:52:51 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]:46028) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mSPKN-00082u-J8 for qemu-devel@nongnu.org; Mon, 20 Sep 2021 15:51:47 -0400 Received: from us-smtp-delivery-124.mimecast.com ([216.205.24.124]:46423) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from ) id 1mSPKF-0002Ew-BW for qemu-devel@nongnu.org; Mon, 20 Sep 2021 15:51:46 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1632167496; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=qm2f1U0hocOPREXT454RFvqfUcjj1NNhAycEnaGgQ/c=; b=cLvjDQ3C/oFB6npgrnXoxglPy5pRMnP3xCOzWXbCb5uKlNZ0h1QQQgy1HbMehl8yopvEVj w+JaOOi1kVtTdO9wNz6SNC697N2K9p/ZW5L/Zvlx4Pg1h4mrvPJ4GWBa6q9yN/N2kHoa/F RgVrmrN7pD5yHvj4tNNbI9bXN9MYaNA= Received: from mail-oi1-f198.google.com (mail-oi1-f198.google.com [209.85.167.198]) (Using TLS) by relay.mimecast.com with ESMTP id us-mta-245-pkerYHAdOx6B6XuBycV3QQ-1; Mon, 20 Sep 2021 15:51:32 -0400 X-MC-Unique: pkerYHAdOx6B6XuBycV3QQ-1 Received: by mail-oi1-f198.google.com with SMTP id o8-20020a0568080bc800b0026bf78d5d98so37204341oik.19 for ; Mon, 20 Sep 2021 12:51:32 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=qm2f1U0hocOPREXT454RFvqfUcjj1NNhAycEnaGgQ/c=; b=HtvqWhMcUZnTOyGvmVGSSCoMkr4MQU5j/TkeYb8jy+4COv0Dxk7SZrhwPhM8gqhTuV 7v69zSvdpFHp7gJXlMT50H5icpiXYG+twaUww2IpWWow9SEQS5z9kgWR0FFLQLzOyHRT +ldQpqbMCzM5cqUz190QLjjQvLnBPQqLCbH32jOuothI2667zfR7k+PE+hO1fSyc2ftg aITrU2t3+/0IhvjtLDfsfXT95RDFiXBo8+izLBs3a6f8nAqffSPUyTLpYs7VK6gLwsmT eHb3WsUZw8nPJCLW2hKTdaY6CwNaIJlrcFl8G7112xNNlK9OXz25PPQEJmck5IvK+qnr 8yKw== X-Gm-Message-State: AOAM530to4W8Ay07XtyCS6CG817Uvro+EU893zXBcKx6O0O8uFCxmOCy FHmzNiQay2YJUUvTA5gIQ6e1e0qwovQHLIM1MQI1GAz9Llhv9JhSZB/XTX95f5UEycpFTAipoqF /oaeLCFoyKz2TYYduV82DifsghkubXQA= X-Received: by 2002:aca:f257:: with SMTP id q84mr641676oih.52.1632167491005; Mon, 20 Sep 2021 12:51:31 -0700 (PDT) X-Google-Smtp-Source: ABdhPJyssLkcgTK/RuovqXUeJAzm+YHjcFDeVpDB5k9C3w+CKKRtceJeKivzhc0XG4uoDigCzEhmFJZ6GCk29P1v4uI= X-Received: by 2002:aca:f257:: with SMTP id q84mr641654oih.52.1632167490695; Mon, 20 Sep 2021 12:51:30 -0700 (PDT) MIME-Version: 1.0 References: <20210915162955.333025-1-jsnow@redhat.com> In-Reply-To: <20210915162955.333025-1-jsnow@redhat.com> From: John Snow Date: Mon, 20 Sep 2021 15:51:19 -0400 Message-ID: Subject: Re: [PATCH v4 00/27] python: introduce Asynchronous QMP package To: qemu-devel Authentication-Results: relay.mimecast.com; auth=pass smtp.auth=CUSA124A263 smtp.mailfrom=jsnow@redhat.com X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: multipart/alternative; boundary="0000000000007f441405cc729b3e" Received-SPF: pass client-ip=216.205.24.124; envelope-from=jsnow@redhat.com; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -42 X-Spam_score: -4.3 X-Spam_bar: ---- X-Spam_report: (-4.3 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-1.475, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=0.001, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H2=-0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-BeenThere: qemu-devel@nongnu.org X-Mailman-Version: 2.1.23 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Cc: Eduardo Habkost , Stefan Hajnoczi , Markus Armbruster , Wainer dos Santos Moschetta , "Niteesh G . S ." , Willian Rampazzo , Cleber Rosa , Eric Blake Errors-To: qemu-devel-bounces+qemu-devel=archiver.kernel.org@nongnu.org Sender: "Qemu-devel" --0000000000007f441405cc729b3e Content-Type: text/plain; charset="UTF-8" On Wed, Sep 15, 2021 at 12:30 PM John Snow wrote: > GitLab: https://gitlab.com/jsnow/qemu/-/commits/python-async-qmp-aqmp > CI: https://gitlab.com/jsnow/qemu/-/pipelines/371343890 > Docs: https://people.redhat.com/~jsnow/sphinx/html/qemu.aqmp.html > Based-on: <20210915154031.321592-2-jsnow@redhat.com> > > (This is a quick V4 to add a few tiny edits and get the proper mboxes > out on the list. My current plan is to send a PR for this series this > week. Don't panic, it isn't used by default anywhere yet, nothing should > break.) > > Hi! > > This patch series adds an Asynchronous QMP package to the Python > library. It offers a few improvements over the previous library: > > - out-of-band support > - true asynchronous event support > - avoids undocumented interfaces abusing non-blocking sockets > - unit tests! > - documentation! > > This library serves as the basis for a new qmp-shell program that will > offer improved reconnection support, true asynchronous display of > events, VM and job status update notifiers, and so on. > > My intent is to eventually publish this library directly to PyPI as a > standalone package. I would like to phase out our usage of the old QMP > library over time; eventually replacing it entirely with this > one. (Since v2 of this series, I have authored a compatibility shim not > included in this series that can be used to run all of iotests on this > new library successfully with very minimal churn.) > > This series looks big by line count, but it's *mostly* > docstrings. Seriously! > > This package has *no* external dependencies whatsoever. > > Notes & Design > ============== > > Here are some notes on the design of how the library works, to serve as > a primer for review; however I also **highly recommend** browsing the > generated Sphinx documentation for this series. > > Here's that link again: > https://people.redhat.com/~jsnow/sphinx/html/qemu.aqmp.html > > The core machinery is split between the AsyncProtocol and QMPClient > classes. AsyncProtocol provides the generic machinery, while QMPClient > provides the QMP-specific details. > > The design uses two independent coroutines that act as the "bottom > half", a writer task and a reader task. These tasks run for the duration > of the connection and independently send and receive messages, > respectively. > > A third task, disconnect, is scheduled asynchronously whenever an > unrecoverable error occurs and facilitates coalescing of the other two > tasks. > > This diagram for how execute() operates may be helpful for understanding > how AsyncProtocol is laid out. The arrows indicate the direction of a > QMP message; the long horizontal dash indicates the separation between > the upper and lower halves of the event loop. The queue mechanisms > between both dashes serve as the intermediaries between the upper and > lower halves. > > +---------+ > | caller | > +---------+ > ^ | > | v > +---------+ > +---------------> |execute()| -----------+ > | +---------+ | > | | > [-----------------------------------------------------------] > | | > | v > +----+------+ +----------------+ +------+-------+ > | ExecQueue | | EventListeners | |Outbound Queue| > +----+------+ +----+-----------+ +------+-------+ > ^ ^ | > | | | > [-----------------------------------------------------------] > | | | > | | v > +--+----------------+---+ +-----------+-----------+ > | Reader Task/Coroutine | | Writer Task/Coroutine | > +-----------+-----------+ +-----------+-----------+ > ^ | > | v > +-----+------+ +-----+------+ > |StreamReader| |StreamWriter| > +------------+ +------------+ > > The caller will invoke execute(), which in turn will deposit a message > in the outbound send queue. This will wake up the writer task, which > well send the message over the wire. > > The execute() method will then yield to wait for a reply delivered to an > execution queue created solely for that execute statement. > > When a message arrives, the Reader task will unblock and route the > message either to the EventListener subsystem, or place it in the > appropriate pending execution queue. > > Once a message is placed in the pending execution queue, execute() will > unblock and the execution will conclude, returning the result of the RPC > call to the caller. > > Patch Layout > ============ > > Patches 1-4 add tiny pre-requisites, utilities, etc. > Patches 5-12 add a generic async message-based protocol class, > AsyncProtocol. They are split fairly small and should > be reasonably self-contained. > Patches 13-15 check in more QMP-centric components. > Patches 16-21 add qmp_client.py, with a new 'QMPClient()' class. > They're split into reasonably tiny pieces here. > Patches 22-23 add a few finishing touches, they are small patches. > Patches 24-27 adds unit tests. They're a little messy still, but > they've been quite helpful to me so far. Coverage of > protocol.py is at about ~86%. > > Future Work > =========== > > These items are in progress: > > - A synchronous QMP wrapper that allows this library to be easily used > from non-async code; this will also allow me to prove it works well by > demoing its replacement throughout iotests. > > This work is feature-complete, but needs polish. All of iotests is now > passing with Async QMP and this Sync wrapper. This will be its own > follow-up series. > > - A QMP server class; to facilitate writing of unit tests. An early > version is done, but possibly not feature complete. More polish and > tests are warranted. This will be its own follow-up series. > > - More unit tests for qmp_client.py, qmp_server.py and other modules. > > Changelog > ========= > > V4: > > - (06) Minor typo fix in comment (Eric Blake) > - (25) Removed stale null_protocol.py file. > - (26, 27) New - increase test coverage and add coverage.py support. > > V3: > > - (02, 05) Typo fixes (Eric Blake) > - (04) Rewrote the "wait_closed" compatibility function for Python 3.6; > the older version raised unwanted exceptions in error pathways. > (Niteesh) > - (04, 05, 06, 08) Rewrote _bh_disconnect fairly substantially again; > the problem is that exceptions can surface during both flushing of the > stream and when waiting for the stream to close. These errors can be > new, primary causes of failure or secondary failures. Distinguishing > between them is tricky. The new disconnection method takes much > greater pains to ensure that even if Exceptions occur, disconnect > *will* complete. This adds robustness to cases exposed by iotests > where one or more endpoints might segfault or abort and cleanup can be > challenged. > - (11) Fixed logging hook names (Niteesh) > - (24, 25) Bumped avocado dependency to v90; It added support for async > test functions which made my prior workaround non-suitable. The > choices were to mandate <90 and keep the workarounds or mandate >=90 > and drop the workarounds. I went with the latter. > > V2: > > Renamed classes/methods: > > - Renamed qmp_protocol.py to qmp_client.py > - Renamed 'QMP' class to 'QMPClient' > - Renamed _begin_new_session() to _establish_session() > - Split _establish_connection() out from _new_session(). > - Removed _results() method > > Bugfixes: > > - Suppress duplicate Exceptions when attempting to drain the > StreamWriter > - Delay initialization of asyncio.Queue and asyncio.Event variables to > _new_session or later -- they must not be created outside of the loop, > even if they are not async functions. > - Rework runstate_changed events to guarantee visibility of events to > waiters > - Improve connect()/accept() cleanup to work with > asyncio.CancelledError, asyncio.TimeoutError > - No-argument form of Message() now succeeds properly. > - flush utility will correctly yield when data is below the "high water > mark", giving the stream a chance to actually flush. > - Increase read buffer size to accommodate query-qmp-schema (Thanks > Niteesh) > > Ugly bits from V1 removed: > > - Remove tertiary filtering from EventListener (for now), accompanying > documentation removed from events.py > - Use asyncio.wait() instead of custom wait_task_done() > - MultiException is removed in favor of just raising the first Exception > that occurs in the bottom half; other Exceptions if any are logged > instead. > > Improvements: > > - QMPClient now allows ID-less execution statements via the _raw() > interface. > - Add tests that grant ~86% coverage of protocol.py to the avocado test > suite. > - Removed 'force' parameter from _bh_disconnect; the disconnection > routine determines for itself if we are in the error pathway or not > instead now. This removes any chance of duplicate calls to > _schedule_disconnect accidentally dropping the 'force' setting. > > Debugging/Testing changes: > > - Add debug: bool parameter to asyncio_run utility wrapper > - Improve error messages for '@require' decorator > - Add debugging message for state change events > - Avoid flushing the StreamWriter if we don't have one (This > circumstance only arises in testing, but it's helpful.) > - Improved __repr__ method for AsyncProtocol, and removed __str__ > method. enforcing eval(__repr__(x)) == x does not make sense for > AsyncProtocol. > - Misc logging message changes > - Add a suite of fancy Task debugging utilities. > - Most tracebacks now log at the DEBUG level instead of > CRITICAL/ERROR/WARNING; In those error cases, a one-line summary is > logged instead. > > Misc. aesthetic changes: > > - Misc docstring fixes, whitespace, etc. > - Reordered the definition of some methods to try and keep similar > methods near each other (Moved _cleanup near _bh_disconnect in > QMPClient.) > > John Snow (27): > python/aqmp: add asynchronous QMP (AQMP) subpackage > python/aqmp: add error classes > python/pylint: Add exception for TypeVar names ('T') > python/aqmp: add asyncio compatibility wrappers > python/aqmp: add generic async message-based protocol support > python/aqmp: add runstate state machine to AsyncProtocol > python/aqmp: Add logging utility helpers > python/aqmp: add logging to AsyncProtocol > python/aqmp: add AsyncProtocol.accept() method > python/aqmp: add configurable read buffer limit > python/aqmp: add _cb_inbound and _cb_outbound logging hooks > python/aqmp: add AsyncProtocol._readline() method > python/aqmp: add QMP Message format > python/aqmp: add well-known QMP object models > python/aqmp: add QMP event support > python/pylint: disable too-many-function-args > python/aqmp: add QMP protocol support > python/pylint: disable no-member check > python/aqmp: Add message routing to QMP protocol > python/aqmp: add execute() interfaces > python/aqmp: add _raw() execution interface > python/aqmp: add asyncio_run compatibility wrapper > python/aqmp: add scary message > python: bump avocado to v90.0 > python/aqmp: add AsyncProtocol unit tests > python/aqmp: add LineProtocol tests > python/aqmp: Add Coverage.py support > > python/.gitignore | 5 + > python/Makefile | 9 + > python/Pipfile.lock | 8 +- > python/avocado.cfg | 3 + > python/qemu/aqmp/__init__.py | 59 +++ > python/qemu/aqmp/error.py | 50 ++ > python/qemu/aqmp/events.py | 706 ++++++++++++++++++++++++++ > python/qemu/aqmp/message.py | 209 ++++++++ > python/qemu/aqmp/models.py | 133 +++++ > python/qemu/aqmp/protocol.py | 902 +++++++++++++++++++++++++++++++++ > python/qemu/aqmp/py.typed | 0 > python/qemu/aqmp/qmp_client.py | 621 +++++++++++++++++++++++ > python/qemu/aqmp/util.py | 217 ++++++++ > python/setup.cfg | 17 +- > python/tests/protocol.py | 583 +++++++++++++++++++++ > 15 files changed, 3516 insertions(+), 6 deletions(-) > create mode 100644 python/qemu/aqmp/__init__.py > create mode 100644 python/qemu/aqmp/error.py > create mode 100644 python/qemu/aqmp/events.py > create mode 100644 python/qemu/aqmp/message.py > create mode 100644 python/qemu/aqmp/models.py > create mode 100644 python/qemu/aqmp/protocol.py > create mode 100644 python/qemu/aqmp/py.typed > create mode 100644 python/qemu/aqmp/qmp_client.py > create mode 100644 python/qemu/aqmp/util.py > create mode 100644 python/tests/protocol.py > > -- > 2.31.1 > > > Thanks, I have preliminarily staged this to my python branch: https://gitlab.com/jsnow/qemu/-/commits/python --js --0000000000007f441405cc729b3e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable


=
On Wed, Sep 15, 2021 at 12:30 PM John= Snow <jsnow@redhat.com> wrot= e:
GitLab: https://gitlab.com/jsnow/qemu/-/commits/pyth= on-async-qmp-aqmp
CI: https://gitlab.com/jsnow/qemu/-/pipelines/371= 343890
Docs: https://people.redhat.com/~jsnow/sp= hinx/html/qemu.aqmp.html
Based-on: <20210915154031.321592-2-jsnow@redhat.com>

(This is a quick V4 to add a few tiny edits and get the proper mboxes
out on the list. My current plan is to send a PR for this series this
week. Don't panic, it isn't used by default anywhere yet, nothing s= hould
break.)

Hi!

This patch series adds an Asynchronous QMP package to the Python
library. It offers a few improvements over the previous library:

- out-of-band support
- true asynchronous event support
- avoids undocumented interfaces abusing non-blocking sockets
- unit tests!
- documentation!

This library serves as the basis for a new qmp-shell program that will
offer improved reconnection support, true asynchronous display of
events, VM and job status update notifiers, and so on.

My intent is to eventually publish this library directly to PyPI as a
standalone package. I would like to phase out our usage of the old QMP
library over time; eventually replacing it entirely with this
one. (Since v2 of this series, I have authored a compatibility shim not
included in this series that can be used to run all of iotests on this
new library successfully with very minimal churn.)

This series looks big by line count, but it's *mostly*
docstrings. Seriously!

This package has *no* external dependencies whatsoever.

Notes & Design
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Here are some notes on the design of how the library works, to serve as
a primer for review; however I also **highly recommend** browsing the
generated Sphinx documentation for this series.

Here's that link again:
https://people.redhat.com/~jsnow/sphinx/h= tml/qemu.aqmp.html

The core machinery is split between the AsyncProtocol and QMPClient
classes. AsyncProtocol provides the generic machinery, while QMPClient
provides the QMP-specific details.

The design uses two independent coroutines that act as the "bottom
half", a writer task and a reader task. These tasks run for the durati= on
of the connection and independently send and receive messages,
respectively.

A third task, disconnect, is scheduled asynchronously whenever an
unrecoverable error occurs and facilitates coalescing of the other two
tasks.

This diagram for how execute() operates may be helpful for understanding how AsyncProtocol is laid out. The arrows indicate the direction of a
QMP message; the long horizontal dash indicates the separation between
the upper and lower halves of the event loop. The queue mechanisms
between both dashes serve as the intermediaries between the upper and
lower halves.

=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0+---------+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0| caller=C2=A0 |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0+---------+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0^ |
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0| v
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0+---------+
=C2=A0 =C2=A0 =C2=A0+---------------> |execute()| -----------+
=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0+---------+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |
=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 |
[-----------------------------------------------------------]
=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 |
=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0 v
+----+------+=C2=A0 =C2=A0 +----------------+=C2=A0 =C2=A0 +------+-------+=
| ExecQueue |=C2=A0 =C2=A0 | EventListeners |=C2=A0 =C2=A0 |Outbound Queue|=
+----+------+=C2=A0 =C2=A0 +----+-----------+=C2=A0 =C2=A0 +------+-------+=
=C2=A0 =C2=A0 =C2=A0^=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 ^=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0|
=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0|
[-----------------------------------------------------------]
=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0|
=C2=A0 =C2=A0 =C2=A0|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0v
=C2=A0 +--+----------------+---+=C2=A0 =C2=A0 =C2=A0 =C2=A0+-----------+---= --------+
=C2=A0 | Reader Task/Coroutine |=C2=A0 =C2=A0 =C2=A0 =C2=A0| Writer Task/Co= routine |
=C2=A0 +-----------+-----------+=C2=A0 =C2=A0 =C2=A0 =C2=A0+-----------+---= --------+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 ^=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 = =C2=A0 =C2=A0v
=C2=A0 =C2=A0 =C2=A0 =C2=A0 +-----+------+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 +-----+------+
=C2=A0 =C2=A0 =C2=A0 =C2=A0 |StreamReader|=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |StreamWriter|
=C2=A0 =C2=A0 =C2=A0 =C2=A0 +------------+=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 +------------+

The caller will invoke execute(), which in turn will deposit a message
in the outbound send queue. This will wake up the writer task, which
well send the message over the wire.

The execute() method will then yield to wait for a reply delivered to an execution queue created solely for that execute statement.

When a message arrives, the Reader task will unblock and route the
message either to the EventListener subsystem, or place it in the
appropriate pending execution queue.

Once a message is placed in the pending execution queue, execute() will
unblock and the execution will conclude, returning the result of the RPC call to the caller.

Patch Layout
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

Patches 1-4=C2=A0 =C2=A0add tiny pre-requisites, utilities, etc.
Patches 5-12=C2=A0 add a generic async message-based protocol class,
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 AsyncProtocol. They are sp= lit fairly small and should
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 be reasonably self-contain= ed.
Patches 13-15 check in more QMP-centric components.
Patches 16-21 add qmp_client.py, with a new 'QMPClient()' class. =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 They're split into rea= sonably tiny pieces here.
Patches 22-23 add a few finishing touches, they are small patches.
Patches 24-27 adds unit tests. They're a little messy still, but
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 they've been quite hel= pful to me so far. Coverage of
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 protocol.py is at about ~8= 6%.

Future Work
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D

These items are in progress:

- A synchronous QMP wrapper that allows this library to be easily used
=C2=A0 from non-async code; this will also allow me to prove it works well = by
=C2=A0 demoing its replacement throughout iotests.

=C2=A0 This work is feature-complete, but needs polish. All of iotests is n= ow
=C2=A0 passing with Async QMP and this Sync wrapper. This will be its own =C2=A0 follow-up series.

- A QMP server class; to facilitate writing of unit tests. An early
=C2=A0 version is done, but possibly not feature complete. More polish and<= br> =C2=A0 tests are warranted. This will be its own follow-up series.

- More unit tests for qmp_client.py, qmp_server.py and other modules.

Changelog
=3D=3D=3D=3D=3D=3D=3D=3D=3D

V4:

- (06) Minor typo fix in comment (Eric Blake)
- (25) Removed stale null_protocol.py file.
- (26, 27) New - increase test coverage and add coverage.py support.

V3:

- (02, 05) Typo fixes (Eric Blake)
- (04) Rewrote the "wait_closed" compatibility function for Pytho= n 3.6;
=C2=A0 the older version raised unwanted exceptions in error pathways.
=C2=A0 (Niteesh)
- (04, 05, 06, 08) Rewrote _bh_disconnect fairly substantially again;
=C2=A0 the problem is that exceptions can surface during both flushing of t= he
=C2=A0 stream and when waiting for the stream to close. These errors can be=
=C2=A0 new, primary causes of failure or secondary failures. Distinguishing=
=C2=A0 between them is tricky. The new disconnection method takes much
=C2=A0 greater pains to ensure that even if Exceptions occur, disconnect =C2=A0 *will* complete. This adds robustness to cases exposed by iotests =C2=A0 where one or more endpoints might segfault or abort and cleanup can = be
=C2=A0 challenged.
- (11) Fixed logging hook names (Niteesh)
- (24, 25) Bumped avocado dependency to v90; It added support for async
=C2=A0 test functions which made my prior workaround non-suitable. The
=C2=A0 choices were to mandate <90 and keep the workarounds or mandate &= gt;=3D90
=C2=A0 and drop the workarounds. I went with the latter.

V2:

Renamed classes/methods:

- Renamed qmp_protocol.py to qmp_client.py
- Renamed 'QMP' class to 'QMPClient'
- Renamed _begin_new_session() to _establish_session()
- Split _establish_connection() out from _new_session().
- Removed _results() method

Bugfixes:

- Suppress duplicate Exceptions when attempting to drain the
=C2=A0 StreamWriter
- Delay initialization of asyncio.Queue and asyncio.Event variables to
=C2=A0 _new_session or later -- they must not be created outside of the loo= p,
=C2=A0 even if they are not async functions.
- Rework runstate_changed events to guarantee visibility of events to
=C2=A0 waiters
- Improve connect()/accept() cleanup to work with
=C2=A0 asyncio.CancelledError, asyncio.TimeoutError
- No-argument form of Message() now succeeds properly.
- flush utility will correctly yield when data is below the "high wate= r
=C2=A0 mark", giving the stream a chance to actually flush.
- Increase read buffer size to accommodate query-qmp-schema (Thanks
=C2=A0 Niteesh)

Ugly bits from V1 removed:

- Remove tertiary filtering from EventListener (for now), accompanying
=C2=A0 documentation removed from events.py
- Use asyncio.wait() instead of custom wait_task_done()
- MultiException is removed in favor of just raising the first Exception =C2=A0 that occurs in the bottom half; other Exceptions if any are logged =C2=A0 instead.

Improvements:

- QMPClient now allows ID-less execution statements via the _raw()
=C2=A0 interface.
- Add tests that grant ~86% coverage of protocol.py to the avocado test
=C2=A0 suite.
- Removed 'force' parameter from _bh_disconnect; the disconnection<= br> =C2=A0 routine determines for itself if we are in the error pathway or not<= br> =C2=A0 instead now.=C2=A0 This removes any chance of duplicate calls to
=C2=A0 _schedule_disconnect accidentally dropping the 'force' setti= ng.

Debugging/Testing changes:

- Add debug: bool parameter to asyncio_run utility wrapper
- Improve error messages for '@require' decorator
- Add debugging message for state change events
- Avoid flushing the StreamWriter if we don't have one (This
=C2=A0 circumstance only arises in testing, but it's helpful.)
- Improved __repr__ method for AsyncProtocol, and removed __str__
=C2=A0 method.=C2=A0 enforcing eval(__repr__(x)) =3D=3D x does not make sen= se for
=C2=A0 AsyncProtocol.
- Misc logging message changes
- Add a suite of fancy Task debugging utilities.
- Most tracebacks now log at the DEBUG level instead of
=C2=A0 CRITICAL/ERROR/WARNING; In those error cases, a one-line summary is<= br> =C2=A0 logged instead.

Misc. aesthetic changes:

- Misc docstring fixes, whitespace, etc.
- Reordered the definition of some methods to try and keep similar
=C2=A0 methods near each other (Moved _cleanup near _bh_disconnect in
=C2=A0 QMPClient.)

John Snow (27):
=C2=A0 python/aqmp: add asynchronous QMP (AQMP) subpackage
=C2=A0 python/aqmp: add error classes
=C2=A0 python/pylint: Add exception for TypeVar names ('T')
=C2=A0 python/aqmp: add asyncio compatibility wrappers
=C2=A0 python/aqmp: add generic async message-based protocol support
=C2=A0 python/aqmp: add runstate state machine to AsyncProtocol
=C2=A0 python/aqmp: Add logging utility helpers
=C2=A0 python/aqmp: add logging to AsyncProtocol
=C2=A0 python/aqmp: add AsyncProtocol.accept() method
=C2=A0 python/aqmp: add configurable read buffer limit
=C2=A0 python/aqmp: add _cb_inbound and _cb_outbound logging hooks
=C2=A0 python/aqmp: add AsyncProtocol._readline() method
=C2=A0 python/aqmp: add QMP Message format
=C2=A0 python/aqmp: add well-known QMP object models
=C2=A0 python/aqmp: add QMP event support
=C2=A0 python/pylint: disable too-many-function-args
=C2=A0 python/aqmp: add QMP protocol support
=C2=A0 python/pylint: disable no-member check
=C2=A0 python/aqmp: Add message routing to QMP protocol
=C2=A0 python/aqmp: add execute() interfaces
=C2=A0 python/aqmp: add _raw() execution interface
=C2=A0 python/aqmp: add asyncio_run compatibility wrapper
=C2=A0 python/aqmp: add scary message
=C2=A0 python: bump avocado to v90.0
=C2=A0 python/aqmp: add AsyncProtocol unit tests
=C2=A0 python/aqmp: add LineProtocol tests
=C2=A0 python/aqmp: Add Coverage.py support

=C2=A0python/.gitignore=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |= =C2=A0 =C2=A05 +
=C2=A0python/Makefile=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0 |=C2=A0 =C2=A09 +
=C2=A0python/Pipfile.lock=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 |=C2=A0 = =C2=A08 +-
=C2=A0python/avocado.cfg=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0|= =C2=A0 =C2=A03 +
=C2=A0python/qemu/aqmp/__init__.py=C2=A0 =C2=A0|=C2=A0 59 +++
=C2=A0python/qemu/aqmp/error.py=C2=A0 =C2=A0 =C2=A0 |=C2=A0 50 ++
=C2=A0python/qemu/aqmp/events.py=C2=A0 =C2=A0 =C2=A0| 706 +++++++++++++++++= +++++++++
=C2=A0python/qemu/aqmp/message.py=C2=A0 =C2=A0 | 209 ++++++++
=C2=A0python/qemu/aqmp/models.py=C2=A0 =C2=A0 =C2=A0| 133 +++++
=C2=A0python/qemu/aqmp/protocol.py=C2=A0 =C2=A0| 902 ++++++++++++++++++++++= +++++++++++
=C2=A0python/qemu/aqmp/py.typed=C2=A0 =C2=A0 =C2=A0 |=C2=A0 =C2=A00
=C2=A0python/qemu/aqmp/qmp_client.py | 621 +++++++++++++++++++++++
=C2=A0python/qemu/aqmp/util.py=C2=A0 =C2=A0 =C2=A0 =C2=A0| 217 ++++++++
=C2=A0python/setup.cfg=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2= =A0|=C2=A0 17 +-
=C2=A0python/tests/protocol.py=C2=A0 =C2=A0 =C2=A0 =C2=A0| 583 ++++++++++++= +++++++++
=C2=A015 files changed, 3516 insertions(+), 6 deletions(-)
=C2=A0create mode 100644 python/qemu/aqmp/__init__.py
=C2=A0create mode 100644 python/qemu/aqmp/error.py
=C2=A0create mode 100644 python/qemu/aqmp/events.py
=C2=A0create mode 100644 python/qemu/aqmp/message.py
=C2=A0create mode 100644 python/qemu/aqmp/models.py
=C2=A0create mode 100644 python/qemu/aqmp/protocol.py
=C2=A0create mode 100644 python/qemu/aqmp/py.typed
=C2=A0create mode 100644 python/qemu/aqmp/qmp_client.py
=C2=A0create mode 100644 python/qemu/aqmp/util.py
=C2=A0create mode 100644 python/tests/protocol.py

--
2.31.1



Thanks, I have preliminarily staged th= is to my python branch: https://gitlab.com/jsnow/qemu/-/commits/python

--js
--0000000000007f441405cc729b3e--