[PULL 00/17] Python patches

[PULL 00/17] Python patches

[John Snow]

The following changes since commit 5e9d14f2bea6df89c0675df953f9c839560d2266:

  Merge remote-tracking branch 'remotes/alistair/tags/pull-riscv-to-apply-20220121-1' into staging (2022-01-21 10:31:25 +0000)

are available in the Git repository at:

  https://gitlab.com/jsnow/qemu.git tags/python-pull-request

for you to fetch changes up to 05908602429cf9d6fce9b60704b8395f6d295441:

  scripts/render-block-graph: switch to AQMP (2022-01-21 16:01:31 -0500)

----------------------------------------------------------------
Python patches

A few fixes to the Python CI tests, a few fixes to the (async) QMP
library, and a set of patches that begin to shift us towards using the
new qmp lib.

----------------------------------------------------------------

John Snow (17):
  python: pin setuptools below v60.0.0
  python: use avocado's "new" runner
  python/aqmp: fix docstring typo
  python/aqmp: add __del__ method to legacy interface
  python/aqmp: handle asyncio.TimeoutError on execute()
  python/aqmp: copy type definitions from qmp
  python/aqmp: add SocketAddrT to package root
  python/aqmp: rename AQMPError to QMPError
  python/qemu-ga-client: don't use deprecated CLI syntax in usage
    comment
  python/qmp: switch qemu-ga-client to AQMP
  python/qmp: switch qom tools to AQMP
  python/qmp: switch qmp-shell to AQMP
  python: move qmp utilities to python/qemu/utils
  python: move qmp-shell under the AQMP package
  scripts/cpu-x86-uarch-abi: fix CLI parsing
  scripts/cpu-x86-uarch-abi: switch to AQMP
  scripts/render-block-graph: switch to AQMP

 python/README.rst                            |  2 +-
 python/Makefile                              |  2 +
 python/avocado.cfg                           |  2 +-
 python/qemu/aqmp/__init__.py                 | 16 ++++++--
 python/qemu/aqmp/error.py                    | 12 +++---
 python/qemu/aqmp/events.py                   |  4 +-
 python/qemu/aqmp/legacy.py                   | 41 +++++++++++++++++++-
 python/qemu/aqmp/protocol.py                 | 24 +++++++-----
 python/qemu/aqmp/qmp_client.py               | 16 +++++---
 python/qemu/{qmp => aqmp}/qmp_shell.py       | 31 ++++++++-------
 python/qemu/{qmp => utils}/qemu_ga_client.py | 24 ++++++------
 python/qemu/{qmp => utils}/qom.py            |  5 ++-
 python/qemu/{qmp => utils}/qom_common.py     |  3 +-
 python/qemu/{qmp => utils}/qom_fuse.py       | 11 +++---
 python/setup.cfg                             | 19 ++++-----
 scripts/cpu-x86-uarch-abi.py                 |  7 ++--
 scripts/qmp/qemu-ga-client                   |  2 +-
 scripts/qmp/qmp-shell                        |  2 +-
 scripts/qmp/qom-fuse                         |  2 +-
 scripts/qmp/qom-get                          |  2 +-
 scripts/qmp/qom-list                         |  2 +-
 scripts/qmp/qom-set                          |  2 +-
 scripts/qmp/qom-tree                         |  2 +-
 scripts/render_block_graph.py                |  8 ++--
 24 files changed, 151 insertions(+), 90 deletions(-)
 rename python/qemu/{qmp => aqmp}/qmp_shell.py (96%)
 rename python/qemu/{qmp => utils}/qemu_ga_client.py (94%)
 rename python/qemu/{qmp => utils}/qom.py (98%)
 rename python/qemu/{qmp => utils}/qom_common.py (98%)
 rename python/qemu/{qmp => utils}/qom_fuse.py (97%)

-- 
2.31.1

[PULL 01/17] python: pin setuptools below v60.0.0

[John Snow]

setuptools is a package that replaces the python stdlib 'distutils'. It
is generally installed by all venv-creating tools "by default". It isn't
actually needed at runtime for the qemu package, so our own setup.cfg
does not mention it as a dependency.

However, tox will create virtual environments that include it, and will
upgrade it to the very latest version. the 'venv' tool will also include
whichever version your host system happens to have.

Unfortunately, setuptools version 60.0.0 and above include a hack to
forcibly overwrite python's built-in distutils. The pylint tool that we
use to run code analysis checks on this package relies on distutils and
suffers regressions when setuptools >= 60.0.0 is present at all, see
https://github.com/PyCQA/pylint/issues/5704

Instruct tox and the 'check-dev' targets to avoid setuptools packages
that are too new, for now. Pipenv is unaffected, because setuptools 60
does not offer Python 3.6 support, and our pipenv config is pinned
against Python 3.6.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Cleber Rosa <crosa@redhat.com>
Message-id: 20220121005221.142236-1-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/Makefile  | 2 ++
 python/setup.cfg | 1 +
 2 files changed, 3 insertions(+)

diff --git a/python/Makefile b/python/Makefile
index 3334311362..949c472624 100644
--- a/python/Makefile
+++ b/python/Makefile
@@ -68,6 +68,8 @@ $(QEMU_VENV_DIR) $(QEMU_VENV_DIR)/bin/activate: setup.cfg
 		echo "ACTIVATE $(QEMU_VENV_DIR)";		\
 		. $(QEMU_VENV_DIR)/bin/activate;		\
 		echo "INSTALL qemu[devel] $(QEMU_VENV_DIR)";	\
+		pip install --disable-pip-version-check		\
+			"setuptools<60.0.0" 1>/dev/null;	\
 		make develop 1>/dev/null;			\
 	)
 	@touch $(QEMU_VENV_DIR)
diff --git a/python/setup.cfg b/python/setup.cfg
index 417e937839..aa238d8bc9 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -163,6 +163,7 @@ deps =
     .[devel]
     .[fuse]  # Workaround to trigger tox venv rebuild
     .[tui]   # Workaround to trigger tox venv rebuild
+    setuptools < 60  # Workaround, please see commit msg.
 commands =
     make check
 
-- 
2.31.1

[PULL 02/17] python: use avocado's "new" runner

[John Snow]

The old legacy runner no longer seems to work with output logging, so we
can't see failure logs when a test case fails. The new runner doesn't
(seem to) support Coverage.py yet, but seeing error output is a more
important feature.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Message-id: 20220119193916.4138217-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/avocado.cfg | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/python/avocado.cfg b/python/avocado.cfg
index c7722e7ecd..a460420059 100644
--- a/python/avocado.cfg
+++ b/python/avocado.cfg
@@ -1,5 +1,5 @@
 [run]
-test_runner = runner
+test_runner = nrunner
 
 [simpletests]
 # Don't show stdout/stderr in the test *summary*
-- 
2.31.1

[PULL 03/17] python/aqmp: fix docstring typo

[John Snow]

Reported-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 python/qemu/aqmp/__init__.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py
index 880d5b6fa7..173556404d 100644
--- a/python/qemu/aqmp/__init__.py
+++ b/python/qemu/aqmp/__init__.py
@@ -6,7 +6,7 @@
 QEMU Guest Agent, and the QEMU Storage Daemon.
 
 `QMPClient` provides the main functionality of this package. All errors
-raised by this library dervive from `AQMPError`, see `aqmp.error` for
+raised by this library derive from `AQMPError`, see `aqmp.error` for
 additional detail. See `aqmp.events` for an in-depth tutorial on
 managing QMP events.
 """
-- 
2.31.1

[PULL 04/17] python/aqmp: add __del__ method to legacy interface

[John Snow]

asyncio can complain *very* loudly if you forget to back out of things
gracefully before the garbage collector starts destroying objects that
contain live references to asyncio Tasks.

The usual fix is just to remember to call aqmp.disconnect(), but for the
sake of the legacy wrapper and quick, one-off scripts where a graceful
shutdown is not necessarily of paramount imporance, add a courtesy
cleanup that will trigger prior to seeing screenfuls of confusing
asyncio tracebacks.

Note that we can't *always* save you from yourself; depending on when
the GC runs, you might just seriously be out of luck. The best we can do
in this case is to gently remind you to clean up after yourself.

(Still much better than multiple pages of incomprehensible python
warnings for the crime of forgetting to put your toys away.)

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 python/qemu/aqmp/legacy.py | 18 ++++++++++++++++++
 1 file changed, 18 insertions(+)

diff --git a/python/qemu/aqmp/legacy.py b/python/qemu/aqmp/legacy.py
index 9e7b9fb80b..2ccb136b02 100644
--- a/python/qemu/aqmp/legacy.py
+++ b/python/qemu/aqmp/legacy.py
@@ -16,6 +16,8 @@
 import qemu.qmp
 from qemu.qmp import QMPMessage, QMPReturnValue, SocketAddrT
 
+from .error import AQMPError
+from .protocol import Runstate
 from .qmp_client import QMPClient
 
 
@@ -136,3 +138,19 @@ def settimeout(self, timeout: Optional[float]) -> None:
 
     def send_fd_scm(self, fd: int) -> None:
         self._aqmp.send_fd_scm(fd)
+
+    def __del__(self) -> None:
+        if self._aqmp.runstate == Runstate.IDLE:
+            return
+
+        if not self._aloop.is_running():
+            self.close()
+        else:
+            # Garbage collection ran while the event loop was running.
+            # Nothing we can do about it now, but if we don't raise our
+            # own error, the user will be treated to a lot of traceback
+            # they might not understand.
+            raise AQMPError(
+                "QEMUMonitorProtocol.close()"
+                " was not called before object was garbage collected"
+            )
-- 
2.31.1

[PULL 05/17] python/aqmp: handle asyncio.TimeoutError on execute()

[John Snow]

This exception can be injected into any await statement. If we are
canceled via timeout, we want to clear the pending execution record on
our way out.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 python/qemu/aqmp/qmp_client.py | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/python/qemu/aqmp/qmp_client.py b/python/qemu/aqmp/qmp_client.py
index 8105e29fa8..6a985ffe30 100644
--- a/python/qemu/aqmp/qmp_client.py
+++ b/python/qemu/aqmp/qmp_client.py
@@ -435,7 +435,11 @@ async def _issue(self, msg: Message) -> Union[None, str]:
             msg_id = msg['id']
 
         self._pending[msg_id] = asyncio.Queue(maxsize=1)
-        await self._outgoing.put(msg)
+        try:
+            await self._outgoing.put(msg)
+        except:
+            del self._pending[msg_id]
+            raise
 
         return msg_id
 
@@ -452,9 +456,9 @@ async def _reply(self, msg_id: Union[str, None]) -> Message:
             was lost, or some other problem.
         """
         queue = self._pending[msg_id]
-        result = await queue.get()
 
         try:
+            result = await queue.get()
             if isinstance(result, ExecInterruptedError):
                 raise result
             return result
-- 
2.31.1

[PULL 06/17] python/aqmp: copy type definitions from qmp

[John Snow]

Copy the remaining type definitions from QMP into the qemu.aqmp.legacy
module. Now, users that require the legacy interface don't need to
import anything else but qemu.aqmp.legacy wrapper.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 python/qemu/aqmp/legacy.py   | 22 ++++++++++++++++++++--
 python/qemu/aqmp/protocol.py | 16 ++++++++++------
 2 files changed, 30 insertions(+), 8 deletions(-)

diff --git a/python/qemu/aqmp/legacy.py b/python/qemu/aqmp/legacy.py
index 2ccb136b02..9431fe9330 100644
--- a/python/qemu/aqmp/legacy.py
+++ b/python/qemu/aqmp/legacy.py
@@ -6,7 +6,9 @@
 
 import asyncio
 from typing import (
+    Any,
     Awaitable,
+    Dict,
     List,
     Optional,
     TypeVar,
@@ -14,13 +16,29 @@
 )
 
 import qemu.qmp
-from qemu.qmp import QMPMessage, QMPReturnValue, SocketAddrT
 
 from .error import AQMPError
-from .protocol import Runstate
+from .protocol import Runstate, SocketAddrT
 from .qmp_client import QMPClient
 
 
+#: QMPMessage is an entire QMP message of any kind.
+QMPMessage = Dict[str, Any]
+
+#: QMPReturnValue is the 'return' value of a command.
+QMPReturnValue = object
+
+#: QMPObject is any object in a QMP message.
+QMPObject = Dict[str, object]
+
+# QMPMessage can be outgoing commands or incoming events/returns.
+# QMPReturnValue is usually a dict/json object, but due to QAPI's
+# 'returns-whitelist', it can actually be anything.
+#
+# {'return': {}} is a QMPMessage,
+# {} is the QMPReturnValue.
+
+
 # pylint: disable=missing-docstring
 
 
diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py
index c4fbe35a0e..5b4f2f0d0a 100644
--- a/python/qemu/aqmp/protocol.py
+++ b/python/qemu/aqmp/protocol.py
@@ -46,6 +46,10 @@
 _U = TypeVar('_U')
 _TaskFN = Callable[[], Awaitable[None]]  # aka ``async def func() -> None``
 
+InternetAddrT = Tuple[str, int]
+UnixAddrT = str
+SocketAddrT = Union[UnixAddrT, InternetAddrT]
+
 
 class Runstate(Enum):
     """Protocol session runstate."""
@@ -257,7 +261,7 @@ async def runstate_changed(self) -> Runstate:
 
     @upper_half
     @require(Runstate.IDLE)
-    async def accept(self, address: Union[str, Tuple[str, int]],
+    async def accept(self, address: SocketAddrT,
                      ssl: Optional[SSLContext] = None) -> None:
         """
         Accept a connection and begin processing message queues.
@@ -275,7 +279,7 @@ async def accept(self, address: Union[str, Tuple[str, int]],
 
     @upper_half
     @require(Runstate.IDLE)
-    async def connect(self, address: Union[str, Tuple[str, int]],
+    async def connect(self, address: SocketAddrT,
                       ssl: Optional[SSLContext] = None) -> None:
         """
         Connect to the server and begin processing message queues.
@@ -337,7 +341,7 @@ def _set_state(self, state: Runstate) -> None:
 
     @upper_half
     async def _new_session(self,
-                           address: Union[str, Tuple[str, int]],
+                           address: SocketAddrT,
                            ssl: Optional[SSLContext] = None,
                            accept: bool = False) -> None:
         """
@@ -397,7 +401,7 @@ async def _new_session(self,
     @upper_half
     async def _establish_connection(
             self,
-            address: Union[str, Tuple[str, int]],
+            address: SocketAddrT,
             ssl: Optional[SSLContext] = None,
             accept: bool = False
     ) -> None:
@@ -424,7 +428,7 @@ async def _establish_connection(
             await self._do_connect(address, ssl)
 
     @upper_half
-    async def _do_accept(self, address: Union[str, Tuple[str, int]],
+    async def _do_accept(self, address: SocketAddrT,
                          ssl: Optional[SSLContext] = None) -> None:
         """
         Acting as the transport server, accept a single connection.
@@ -482,7 +486,7 @@ async def _client_connected_cb(reader: asyncio.StreamReader,
         self.logger.debug("Connection accepted.")
 
     @upper_half
-    async def _do_connect(self, address: Union[str, Tuple[str, int]],
+    async def _do_connect(self, address: SocketAddrT,
                           ssl: Optional[SSLContext] = None) -> None:
         """
         Acting as the transport client, initiate a connection to a server.
-- 
2.31.1

[PULL 07/17] python/aqmp: add SocketAddrT to package root

[John Snow]

It's a commonly needed definition, it can be re-exported by the root.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 python/qemu/aqmp/__init__.py | 10 +++++++++-
 1 file changed, 9 insertions(+), 1 deletion(-)

diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py
index 173556404d..05f467c141 100644
--- a/python/qemu/aqmp/__init__.py
+++ b/python/qemu/aqmp/__init__.py
@@ -26,7 +26,12 @@
 from .error import AQMPError
 from .events import EventListener
 from .message import Message
-from .protocol import ConnectError, Runstate, StateError
+from .protocol import (
+    ConnectError,
+    Runstate,
+    SocketAddrT,
+    StateError,
+)
 from .qmp_client import ExecInterruptedError, ExecuteError, QMPClient
 
 
@@ -48,4 +53,7 @@
     'ConnectError',
     'ExecuteError',
     'ExecInterruptedError',
+
+    # Type aliases
+    'SocketAddrT',
 )
-- 
2.31.1

[PULL 08/17] python/aqmp: rename AQMPError to QMPError

[John Snow]

This is in preparation for renaming qemu.aqmp to qemu.qmp. I should have
done this from this from the very beginning, but it's a convenient time
to make sure this churn is taken care of.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 python/qemu/aqmp/__init__.py   |  6 +++---
 python/qemu/aqmp/error.py      | 12 ++++++------
 python/qemu/aqmp/events.py     |  4 ++--
 python/qemu/aqmp/legacy.py     |  4 ++--
 python/qemu/aqmp/protocol.py   |  8 ++++----
 python/qemu/aqmp/qmp_client.py |  8 ++++----
 6 files changed, 21 insertions(+), 21 deletions(-)

diff --git a/python/qemu/aqmp/__init__.py b/python/qemu/aqmp/__init__.py
index 05f467c141..4c22c38079 100644
--- a/python/qemu/aqmp/__init__.py
+++ b/python/qemu/aqmp/__init__.py
@@ -6,7 +6,7 @@
 QEMU Guest Agent, and the QEMU Storage Daemon.
 
 `QMPClient` provides the main functionality of this package. All errors
-raised by this library derive from `AQMPError`, see `aqmp.error` for
+raised by this library derive from `QMPError`, see `aqmp.error` for
 additional detail. See `aqmp.events` for an in-depth tutorial on
 managing QMP events.
 """
@@ -23,7 +23,7 @@
 
 import logging
 
-from .error import AQMPError
+from .error import QMPError
 from .events import EventListener
 from .message import Message
 from .protocol import (
@@ -48,7 +48,7 @@
     'Runstate',
 
     # Exceptions, most generic to most explicit
-    'AQMPError',
+    'QMPError',
     'StateError',
     'ConnectError',
     'ExecuteError',
diff --git a/python/qemu/aqmp/error.py b/python/qemu/aqmp/error.py
index 781f49b008..24ba4d5054 100644
--- a/python/qemu/aqmp/error.py
+++ b/python/qemu/aqmp/error.py
@@ -1,21 +1,21 @@
 """
-AQMP Error Classes
+QMP Error Classes
 
 This package seeks to provide semantic error classes that are intended
 to be used directly by clients when they would like to handle particular
 semantic failures (e.g. "failed to connect") without needing to know the
 enumeration of possible reasons for that failure.
 
-AQMPError serves as the ancestor for all exceptions raised by this
+QMPError serves as the ancestor for all exceptions raised by this
 package, and is suitable for use in handling semantic errors from this
 library. In most cases, individual public methods will attempt to catch
 and re-encapsulate various exceptions to provide a semantic
 error-handling interface.
 
-.. admonition:: AQMP Exception Hierarchy Reference
+.. admonition:: QMP Exception Hierarchy Reference
 
  |   `Exception`
- |    +-- `AQMPError`
+ |    +-- `QMPError`
  |         +-- `ConnectError`
  |         +-- `StateError`
  |         +-- `ExecInterruptedError`
@@ -31,11 +31,11 @@
 """
 
 
-class AQMPError(Exception):
+class QMPError(Exception):
     """Abstract error class for all errors originating from this package."""
 
 
-class ProtocolError(AQMPError):
+class ProtocolError(QMPError):
     """
     Abstract error class for protocol failures.
 
diff --git a/python/qemu/aqmp/events.py b/python/qemu/aqmp/events.py
index 5f7150c78d..f3d4e2b5e8 100644
--- a/python/qemu/aqmp/events.py
+++ b/python/qemu/aqmp/events.py
@@ -443,7 +443,7 @@ def accept(self, event) -> bool:
     Union,
 )
 
-from .error import AQMPError
+from .error import QMPError
 from .message import Message
 
 
@@ -451,7 +451,7 @@ def accept(self, event) -> bool:
 EventFilter = Callable[[Message], bool]
 
 
-class ListenerError(AQMPError):
+class ListenerError(QMPError):
     """
     Generic error class for `EventListener`-related problems.
     """
diff --git a/python/qemu/aqmp/legacy.py b/python/qemu/aqmp/legacy.py
index 9431fe9330..27df22818a 100644
--- a/python/qemu/aqmp/legacy.py
+++ b/python/qemu/aqmp/legacy.py
@@ -17,7 +17,7 @@
 
 import qemu.qmp
 
-from .error import AQMPError
+from .error import QMPError
 from .protocol import Runstate, SocketAddrT
 from .qmp_client import QMPClient
 
@@ -168,7 +168,7 @@ def __del__(self) -> None:
             # Nothing we can do about it now, but if we don't raise our
             # own error, the user will be treated to a lot of traceback
             # they might not understand.
-            raise AQMPError(
+            raise QMPError(
                 "QEMUMonitorProtocol.close()"
                 " was not called before object was garbage collected"
             )
diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py
index 5b4f2f0d0a..50e973c2f2 100644
--- a/python/qemu/aqmp/protocol.py
+++ b/python/qemu/aqmp/protocol.py
@@ -29,7 +29,7 @@
     cast,
 )
 
-from .error import AQMPError
+from .error import QMPError
 from .util import (
     bottom_half,
     create_task,
@@ -65,7 +65,7 @@ class Runstate(Enum):
     DISCONNECTING = 3
 
 
-class ConnectError(AQMPError):
+class ConnectError(QMPError):
     """
     Raised when the initial connection process has failed.
 
@@ -90,7 +90,7 @@ def __str__(self) -> str:
         return f"{self.error_message}: {cause}"
 
 
-class StateError(AQMPError):
+class StateError(QMPError):
     """
     An API command (connect, execute, etc) was issued at an inappropriate time.
 
@@ -363,7 +363,7 @@ async def _new_session(self,
             This exception will wrap a more concrete one. In most cases,
             the wrapped exception will be `OSError` or `EOFError`. If a
             protocol-level failure occurs while establishing a new
-            session, the wrapped error may also be an `AQMPError`.
+            session, the wrapped error may also be an `QMPError`.
         """
         assert self.runstate == Runstate.IDLE
 
diff --git a/python/qemu/aqmp/qmp_client.py b/python/qemu/aqmp/qmp_client.py
index 6a985ffe30..f1a845cc82 100644
--- a/python/qemu/aqmp/qmp_client.py
+++ b/python/qemu/aqmp/qmp_client.py
@@ -20,7 +20,7 @@
     cast,
 )
 
-from .error import AQMPError, ProtocolError
+from .error import ProtocolError, QMPError
 from .events import Events
 from .message import Message
 from .models import ErrorResponse, Greeting
@@ -66,7 +66,7 @@ class NegotiationError(_WrappedProtocolError):
     """
 
 
-class ExecuteError(AQMPError):
+class ExecuteError(QMPError):
     """
     Exception raised by `QMPClient.execute()` on RPC failure.
 
@@ -87,7 +87,7 @@ def __init__(self, error_response: ErrorResponse,
         self.error_class: str = error_response.error.class_
 
 
-class ExecInterruptedError(AQMPError):
+class ExecInterruptedError(QMPError):
     """
     Exception raised by `execute()` (et al) when an RPC is interrupted.
 
@@ -641,7 +641,7 @@ def send_fd_scm(self, fd: int) -> None:
         sock = self._writer.transport.get_extra_info('socket')
 
         if sock.family != socket.AF_UNIX:
-            raise AQMPError("Sending file descriptors requires a UNIX socket.")
+            raise QMPError("Sending file descriptors requires a UNIX socket.")
 
         if not hasattr(sock, 'sendmsg'):
             # We need to void the warranty sticker.
-- 
2.31.1

[PULL 09/17] python/qemu-ga-client: don't use deprecated CLI syntax in usage comment

[John Snow]

Cleanup related to commit ccd3b3b8112b670f, "qemu-option: warn for
short-form boolean options".

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
---
 python/qemu/qmp/qemu_ga_client.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/python/qemu/qmp/qemu_ga_client.py b/python/qemu/qmp/qemu_ga_client.py
index 67ac0b4211..b3e1d98c9e 100644
--- a/python/qemu/qmp/qemu_ga_client.py
+++ b/python/qemu/qmp/qemu_ga_client.py
@@ -5,7 +5,7 @@
 
 Start QEMU with:
 
-# qemu [...] -chardev socket,path=/tmp/qga.sock,server,wait=off,id=qga0 \
+# qemu [...] -chardev socket,path=/tmp/qga.sock,server=on,wait=off,id=qga0 \
   -device virtio-serial \
   -device virtserialport,chardev=qga0,name=org.qemu.guest_agent.0
 
-- 
2.31.1

[PULL 10/17] python/qmp: switch qemu-ga-client to AQMP

[John Snow]

Async QMP always raises a "ConnectError" on any connection error which
houses the cause in a second exception. We can check if this root cause
was python's ConnectionError to determine a fairly similar condition to
the original error check here.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 python/qemu/qmp/qemu_ga_client.py | 22 +++++++++++-----------
 1 file changed, 11 insertions(+), 11 deletions(-)

diff --git a/python/qemu/qmp/qemu_ga_client.py b/python/qemu/qmp/qemu_ga_client.py
index b3e1d98c9e..15ed430c61 100644
--- a/python/qemu/qmp/qemu_ga_client.py
+++ b/python/qemu/qmp/qemu_ga_client.py
@@ -37,8 +37,8 @@
 # the COPYING file in the top-level directory.
 
 import argparse
+import asyncio
 import base64
-import errno
 import os
 import random
 import sys
@@ -50,8 +50,8 @@
     Sequence,
 )
 
-from qemu import qmp
-from qemu.qmp import SocketAddrT
+from qemu.aqmp import ConnectError, SocketAddrT
+from qemu.aqmp.legacy import QEMUMonitorProtocol
 
 
 # This script has not seen many patches or careful attention in quite
@@ -61,7 +61,7 @@
 # pylint: disable=missing-docstring
 
 
-class QemuGuestAgent(qmp.QEMUMonitorProtocol):
+class QemuGuestAgent(QEMUMonitorProtocol):
     def __getattr__(self, name: str) -> Callable[..., Any]:
         def wrapper(**kwds: object) -> object:
             return self.command('guest-' + name.replace('_', '-'), **kwds)
@@ -149,7 +149,7 @@ def ping(self, timeout: Optional[float]) -> bool:
         self.qga.settimeout(timeout)
         try:
             self.qga.ping()
-        except TimeoutError:
+        except asyncio.TimeoutError:
             return False
         return True
 
@@ -172,7 +172,7 @@ def suspend(self, mode: str) -> None:
         try:
             getattr(self.qga, 'suspend' + '_' + mode)()
             # On error exception will raise
-        except TimeoutError:
+        except asyncio.TimeoutError:
             # On success command will timed out
             return
 
@@ -182,7 +182,7 @@ def shutdown(self, mode: str = 'powerdown') -> None:
 
         try:
             self.qga.shutdown(mode=mode)
-        except TimeoutError:
+        except asyncio.TimeoutError:
             pass
 
 
@@ -277,7 +277,7 @@ def _cmd_reboot(client: QemuGuestAgentClient, args: Sequence[str]) -> None:
 
 def send_command(address: str, cmd: str, args: Sequence[str]) -> None:
     if not os.path.exists(address):
-        print('%s not found' % address)
+        print(f"'{address}' not found. (Is QEMU running?)")
         sys.exit(1)
 
     if cmd not in commands:
@@ -287,10 +287,10 @@ def send_command(address: str, cmd: str, args: Sequence[str]) -> None:
 
     try:
         client = QemuGuestAgentClient(address)
-    except OSError as err:
+    except ConnectError as err:
         print(err)
-        if err.errno == errno.ECONNREFUSED:
-            print('Hint: qemu is not running?')
+        if isinstance(err.exc, ConnectionError):
+            print('(Is QEMU running?)')
         sys.exit(1)
 
     if cmd == 'fsfreeze' and args[0] == 'freeze':
-- 
2.31.1

[PULL 11/17] python/qmp: switch qom tools to AQMP

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 python/qemu/qmp/qom.py        |  5 +++--
 python/qemu/qmp/qom_common.py |  3 ++-
 python/qemu/qmp/qom_fuse.py   | 11 ++++++-----
 3 files changed, 11 insertions(+), 8 deletions(-)

diff --git a/python/qemu/qmp/qom.py b/python/qemu/qmp/qom.py
index 8ff28a8343..bb5d1a78f5 100644
--- a/python/qemu/qmp/qom.py
+++ b/python/qemu/qmp/qom.py
@@ -32,7 +32,8 @@
 
 import argparse
 
-from . import QMPResponseError
+from qemu.aqmp import ExecuteError
+
 from .qom_common import QOMCommand
 
 
@@ -233,7 +234,7 @@ def _list_node(self, path: str) -> None:
                 rsp = self.qmp.command('qom-get', path=path,
                                        property=item.name)
                 print(f"  {item.name}: {rsp} ({item.type})")
-            except QMPResponseError as err:
+            except ExecuteError as err:
                 print(f"  {item.name}: <EXCEPTION: {err!s}> ({item.type})")
         print('')
         for item in items:
diff --git a/python/qemu/qmp/qom_common.py b/python/qemu/qmp/qom_common.py
index 2e4c741f77..e034a6f247 100644
--- a/python/qemu/qmp/qom_common.py
+++ b/python/qemu/qmp/qom_common.py
@@ -27,7 +27,8 @@
     TypeVar,
 )
 
-from . import QEMUMonitorProtocol, QMPError
+from qemu.aqmp import QMPError
+from qemu.aqmp.legacy import QEMUMonitorProtocol
 
 
 class ObjectPropertyInfo:
diff --git a/python/qemu/qmp/qom_fuse.py b/python/qemu/qmp/qom_fuse.py
index 43f4671fdb..653a76b93b 100644
--- a/python/qemu/qmp/qom_fuse.py
+++ b/python/qemu/qmp/qom_fuse.py
@@ -48,7 +48,8 @@
 import fuse
 from fuse import FUSE, FuseOSError, Operations
 
-from . import QMPResponseError
+from qemu.aqmp import ExecuteError
+
 from .qom_common import QOMCommand
 
 
@@ -99,7 +100,7 @@ def is_object(self, path: str) -> bool:
         try:
             self.qom_list(path)
             return True
-        except QMPResponseError:
+        except ExecuteError:
             return False
 
     def is_property(self, path: str) -> bool:
@@ -112,7 +113,7 @@ def is_property(self, path: str) -> bool:
                 if item.name == prop:
                     return True
             return False
-        except QMPResponseError:
+        except ExecuteError:
             return False
 
     def is_link(self, path: str) -> bool:
@@ -125,7 +126,7 @@ def is_link(self, path: str) -> bool:
                 if item.name == prop and item.link:
                     return True
             return False
-        except QMPResponseError:
+        except ExecuteError:
             return False
 
     def read(self, path: str, size: int, offset: int, fh: IO[bytes]) -> bytes:
@@ -138,7 +139,7 @@ def read(self, path: str, size: int, offset: int, fh: IO[bytes]) -> bytes:
         try:
             data = str(self.qmp.command('qom-get', path=path, property=prop))
             data += '\n'  # make values shell friendly
-        except QMPResponseError as err:
+        except ExecuteError as err:
             raise FuseOSError(EPERM) from err
 
         if offset > len(data):
-- 
2.31.1

[PULL 12/17] python/qmp: switch qmp-shell to AQMP

[John Snow]

We have a replacement for async QMP, but it doesn't have feature parity
yet. For now, then, port the old tool onto the new backend.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 python/qemu/aqmp/legacy.py   |  3 +++
 python/qemu/qmp/qmp_shell.py | 31 +++++++++++++++++--------------
 2 files changed, 20 insertions(+), 14 deletions(-)

diff --git a/python/qemu/aqmp/legacy.py b/python/qemu/aqmp/legacy.py
index 27df22818a..0890f95b16 100644
--- a/python/qemu/aqmp/legacy.py
+++ b/python/qemu/aqmp/legacy.py
@@ -22,6 +22,9 @@
 from .qmp_client import QMPClient
 
 
+# (Temporarily) Re-export QMPBadPortError
+QMPBadPortError = qemu.qmp.QMPBadPortError
+
 #: QMPMessage is an entire QMP message of any kind.
 QMPMessage = Dict[str, Any]
 
diff --git a/python/qemu/qmp/qmp_shell.py b/python/qemu/qmp/qmp_shell.py
index e7d7eb18f1..d11bf54b00 100644
--- a/python/qemu/qmp/qmp_shell.py
+++ b/python/qemu/qmp/qmp_shell.py
@@ -95,8 +95,13 @@
     Sequence,
 )
 
-from qemu import qmp
-from qemu.qmp import QMPMessage
+from qemu.aqmp import ConnectError, QMPError, SocketAddrT
+from qemu.aqmp.legacy import (
+    QEMUMonitorProtocol,
+    QMPBadPortError,
+    QMPMessage,
+    QMPObject,
+)
 
 
 LOG = logging.getLogger(__name__)
@@ -125,7 +130,7 @@ def complete(self, text: str, state: int) -> Optional[str]:
         return None
 
 
-class QMPShellError(qmp.QMPError):
+class QMPShellError(QMPError):
     """
     QMP Shell Base error class.
     """
@@ -153,7 +158,7 @@ def visit_Name(cls,  # pylint: disable=invalid-name
         return node
 
 
-class QMPShell(qmp.QEMUMonitorProtocol):
+class QMPShell(QEMUMonitorProtocol):
     """
     QMPShell provides a basic readline-based QMP shell.
 
@@ -161,7 +166,7 @@ class QMPShell(qmp.QEMUMonitorProtocol):
     :param pretty: Pretty-print QMP messages.
     :param verbose: Echo outgoing QMP messages to console.
     """
-    def __init__(self, address: qmp.SocketAddrT,
+    def __init__(self, address: SocketAddrT,
                  pretty: bool = False, verbose: bool = False):
         super().__init__(address)
         self._greeting: Optional[QMPMessage] = None
@@ -237,7 +242,7 @@ def _parse_value(cls, val: str) -> object:
 
     def _cli_expr(self,
                   tokens: Sequence[str],
-                  parent: qmp.QMPObject) -> None:
+                  parent: QMPObject) -> None:
         for arg in tokens:
             (key, sep, val) = arg.partition('=')
             if sep != '=':
@@ -403,7 +408,7 @@ class HMPShell(QMPShell):
     :param pretty: Pretty-print QMP messages.
     :param verbose: Echo outgoing QMP messages to console.
     """
-    def __init__(self, address: qmp.SocketAddrT,
+    def __init__(self, address: SocketAddrT,
                  pretty: bool = False, verbose: bool = False):
         super().__init__(address, pretty, verbose)
         self._cpu_index = 0
@@ -512,19 +517,17 @@ def main() -> None:
 
     try:
         address = shell_class.parse_address(args.qmp_server)
-    except qmp.QMPBadPortError:
+    except QMPBadPortError:
         parser.error(f"Bad port number: {args.qmp_server}")
         return  # pycharm doesn't know error() is noreturn
 
     with shell_class(address, args.pretty, args.verbose) as qemu:
         try:
             qemu.connect(negotiate=not args.skip_negotiation)
-        except qmp.QMPConnectError:
-            die("Didn't get QMP greeting message")
-        except qmp.QMPCapabilitiesError:
-            die("Couldn't negotiate capabilities")
-        except OSError as err:
-            die(f"Couldn't connect to {args.qmp_server}: {err!s}")
+        except ConnectError as err:
+            if isinstance(err.exc, OSError):
+                die(f"Couldn't connect to {args.qmp_server}: {err!s}")
+            die(str(err))
 
         for _ in qemu.repl():
             pass
-- 
2.31.1

[PULL 13/17] python: move qmp utilities to python/qemu/utils

[John Snow]

In order to upload a QMP package to PyPI, I want to remove any scripts
that I am not 100% confident I want to support upstream, beyond our
castle walls.

Move most of our QMP utilities into the utils package so we can split
them out from the PyPI upload.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 python/qemu/{qmp => utils}/qemu_ga_client.py |  0
 python/qemu/{qmp => utils}/qom.py            |  0
 python/qemu/{qmp => utils}/qom_common.py     |  0
 python/qemu/{qmp => utils}/qom_fuse.py       |  0
 python/setup.cfg                             | 16 ++++++++--------
 scripts/qmp/qemu-ga-client                   |  2 +-
 scripts/qmp/qom-fuse                         |  2 +-
 scripts/qmp/qom-get                          |  2 +-
 scripts/qmp/qom-list                         |  2 +-
 scripts/qmp/qom-set                          |  2 +-
 scripts/qmp/qom-tree                         |  2 +-
 11 files changed, 14 insertions(+), 14 deletions(-)
 rename python/qemu/{qmp => utils}/qemu_ga_client.py (100%)
 rename python/qemu/{qmp => utils}/qom.py (100%)
 rename python/qemu/{qmp => utils}/qom_common.py (100%)
 rename python/qemu/{qmp => utils}/qom_fuse.py (100%)

diff --git a/python/qemu/qmp/qemu_ga_client.py b/python/qemu/utils/qemu_ga_client.py
similarity index 100%
rename from python/qemu/qmp/qemu_ga_client.py
rename to python/qemu/utils/qemu_ga_client.py
diff --git a/python/qemu/qmp/qom.py b/python/qemu/utils/qom.py
similarity index 100%
rename from python/qemu/qmp/qom.py
rename to python/qemu/utils/qom.py
diff --git a/python/qemu/qmp/qom_common.py b/python/qemu/utils/qom_common.py
similarity index 100%
rename from python/qemu/qmp/qom_common.py
rename to python/qemu/utils/qom_common.py
diff --git a/python/qemu/qmp/qom_fuse.py b/python/qemu/utils/qom_fuse.py
similarity index 100%
rename from python/qemu/qmp/qom_fuse.py
rename to python/qemu/utils/qom_fuse.py
diff --git a/python/setup.cfg b/python/setup.cfg
index aa238d8bc9..04a41ef1a0 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -60,13 +60,13 @@ tui =
 
 [options.entry_points]
 console_scripts =
-    qom = qemu.qmp.qom:main
-    qom-set = qemu.qmp.qom:QOMSet.entry_point
-    qom-get = qemu.qmp.qom:QOMGet.entry_point
-    qom-list = qemu.qmp.qom:QOMList.entry_point
-    qom-tree = qemu.qmp.qom:QOMTree.entry_point
-    qom-fuse = qemu.qmp.qom_fuse:QOMFuse.entry_point [fuse]
-    qemu-ga-client = qemu.qmp.qemu_ga_client:main
+    qom = qemu.utils.qom:main
+    qom-set = qemu.utils.qom:QOMSet.entry_point
+    qom-get = qemu.utils.qom:QOMGet.entry_point
+    qom-list = qemu.utils.qom:QOMList.entry_point
+    qom-tree = qemu.utils.qom:QOMTree.entry_point
+    qom-fuse = qemu.utils.qom_fuse:QOMFuse.entry_point [fuse]
+    qemu-ga-client = qemu.utils.qemu_ga_client:main
     qmp-shell = qemu.qmp.qmp_shell:main
     aqmp-tui = qemu.aqmp.aqmp_tui:main [tui]
 
@@ -80,7 +80,7 @@ python_version = 3.6
 warn_unused_configs = True
 namespace_packages = True
 
-[mypy-qemu.qmp.qom_fuse]
+[mypy-qemu.utils.qom_fuse]
 # fusepy has no type stubs:
 allow_subclassing_any = True
 
diff --git a/scripts/qmp/qemu-ga-client b/scripts/qmp/qemu-ga-client
index 102fd2cad9..56edd0234a 100755
--- a/scripts/qmp/qemu-ga-client
+++ b/scripts/qmp/qemu-ga-client
@@ -4,7 +4,7 @@ import os
 import sys
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import qemu_ga_client
+from qemu.utils import qemu_ga_client
 
 
 if __name__ == '__main__':
diff --git a/scripts/qmp/qom-fuse b/scripts/qmp/qom-fuse
index a58c8ef979..d453807b27 100755
--- a/scripts/qmp/qom-fuse
+++ b/scripts/qmp/qom-fuse
@@ -4,7 +4,7 @@ import os
 import sys
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp.qom_fuse import QOMFuse
+from qemu.utils.qom_fuse import QOMFuse
 
 
 if __name__ == '__main__':
diff --git a/scripts/qmp/qom-get b/scripts/qmp/qom-get
index e4f3e0c013..04ebe052e8 100755
--- a/scripts/qmp/qom-get
+++ b/scripts/qmp/qom-get
@@ -4,7 +4,7 @@ import os
 import sys
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp.qom import QOMGet
+from qemu.utils.qom import QOMGet
 
 
 if __name__ == '__main__':
diff --git a/scripts/qmp/qom-list b/scripts/qmp/qom-list
index 7a071a54e1..853b85a8d3 100755
--- a/scripts/qmp/qom-list
+++ b/scripts/qmp/qom-list
@@ -4,7 +4,7 @@ import os
 import sys
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp.qom import QOMList
+from qemu.utils.qom import QOMList
 
 
 if __name__ == '__main__':
diff --git a/scripts/qmp/qom-set b/scripts/qmp/qom-set
index 9ca9e2ba10..06820feec4 100755
--- a/scripts/qmp/qom-set
+++ b/scripts/qmp/qom-set
@@ -4,7 +4,7 @@ import os
 import sys
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp.qom import QOMSet
+from qemu.utils.qom import QOMSet
 
 
 if __name__ == '__main__':
diff --git a/scripts/qmp/qom-tree b/scripts/qmp/qom-tree
index 7d0ccca3a4..760e172277 100755
--- a/scripts/qmp/qom-tree
+++ b/scripts/qmp/qom-tree
@@ -4,7 +4,7 @@ import os
 import sys
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp.qom import QOMTree
+from qemu.utils.qom import QOMTree
 
 
 if __name__ == '__main__':
-- 
2.31.1

[PULL 14/17] python: move qmp-shell under the AQMP package

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 python/README.rst                      | 2 +-
 python/qemu/{qmp => aqmp}/qmp_shell.py | 0
 python/setup.cfg                       | 2 +-
 scripts/qmp/qmp-shell                  | 2 +-
 4 files changed, 3 insertions(+), 3 deletions(-)
 rename python/qemu/{qmp => aqmp}/qmp_shell.py (100%)

diff --git a/python/README.rst b/python/README.rst
index 9c1fceaee7..fcf74f69ea 100644
--- a/python/README.rst
+++ b/python/README.rst
@@ -59,7 +59,7 @@ Package installation also normally provides executable console scripts,
 so that tools like ``qmp-shell`` are always available via $PATH. To
 invoke them without installation, you can invoke e.g.:
 
-``> PYTHONPATH=~/src/qemu/python python3 -m qemu.qmp.qmp_shell``
+``> PYTHONPATH=~/src/qemu/python python3 -m qemu.aqmp.qmp_shell``
 
 The mappings between console script name and python module path can be
 found in ``setup.cfg``.
diff --git a/python/qemu/qmp/qmp_shell.py b/python/qemu/aqmp/qmp_shell.py
similarity index 100%
rename from python/qemu/qmp/qmp_shell.py
rename to python/qemu/aqmp/qmp_shell.py
diff --git a/python/setup.cfg b/python/setup.cfg
index 04a41ef1a0..3fb18f845d 100644
--- a/python/setup.cfg
+++ b/python/setup.cfg
@@ -67,7 +67,7 @@ console_scripts =
     qom-tree = qemu.utils.qom:QOMTree.entry_point
     qom-fuse = qemu.utils.qom_fuse:QOMFuse.entry_point [fuse]
     qemu-ga-client = qemu.utils.qemu_ga_client:main
-    qmp-shell = qemu.qmp.qmp_shell:main
+    qmp-shell = qemu.aqmp.qmp_shell:main
     aqmp-tui = qemu.aqmp.aqmp_tui:main [tui]
 
 [flake8]
diff --git a/scripts/qmp/qmp-shell b/scripts/qmp/qmp-shell
index 4a20f97db7..31b19d73e2 100755
--- a/scripts/qmp/qmp-shell
+++ b/scripts/qmp/qmp-shell
@@ -4,7 +4,7 @@ import os
 import sys
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
-from qemu.qmp import qmp_shell
+from qemu.aqmp import qmp_shell
 
 
 if __name__ == '__main__':
-- 
2.31.1

[PULL 15/17] scripts/cpu-x86-uarch-abi: fix CLI parsing

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 scripts/cpu-x86-uarch-abi.py | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/scripts/cpu-x86-uarch-abi.py b/scripts/cpu-x86-uarch-abi.py
index 08acc52a81..8963d90f0b 100644
--- a/scripts/cpu-x86-uarch-abi.py
+++ b/scripts/cpu-x86-uarch-abi.py
@@ -9,7 +9,7 @@
 from qemu import qmp
 import sys
 
-if len(sys.argv) != 1:
+if len(sys.argv) != 2:
     print("syntax: %s QMP-SOCK\n\n" % __file__ +
           "Where QMP-SOCK points to a QEMU process such as\n\n" +
           " # qemu-system-x86_64 -qmp unix:/tmp/qmp,server,nowait " +
@@ -66,7 +66,6 @@
 
 
 sock = sys.argv[1]
-cmd = sys.argv[2]
 shell = qmp.QEMUMonitorProtocol(sock)
 shell.connect()
 
-- 
2.31.1

[PULL 16/17] scripts/cpu-x86-uarch-abi: switch to AQMP

[John Snow]

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 scripts/cpu-x86-uarch-abi.py | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/scripts/cpu-x86-uarch-abi.py b/scripts/cpu-x86-uarch-abi.py
index 8963d90f0b..c262d2f027 100644
--- a/scripts/cpu-x86-uarch-abi.py
+++ b/scripts/cpu-x86-uarch-abi.py
@@ -6,7 +6,7 @@
 # compatibility levels for each CPU model.
 #
 
-from qemu import qmp
+from qemu.aqmp.legacy import QEMUMonitorProtocol
 import sys
 
 if len(sys.argv) != 2:
@@ -66,7 +66,7 @@
 
 
 sock = sys.argv[1]
-shell = qmp.QEMUMonitorProtocol(sock)
+shell = QEMUMonitorProtocol(sock)
 shell.connect()
 
 models = shell.cmd("query-cpu-definitions")
-- 
2.31.1

[PULL 17/17] scripts/render-block-graph: switch to AQMP

[John Snow]

Creating an instance of qemu.aqmp.ExecuteError is too involved here, so
just drop the specificity down to a generic QMPError.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
---
 scripts/render_block_graph.py | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/scripts/render_block_graph.py b/scripts/render_block_graph.py
index 42288a3cfb..b33fb70d5e 100755
--- a/scripts/render_block_graph.py
+++ b/scripts/render_block_graph.py
@@ -25,10 +25,8 @@
 from graphviz import Digraph
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', 'python'))
-from qemu.qmp import (
-    QEMUMonitorProtocol,
-    QMPResponseError,
-)
+from qemu.aqmp import QMPError
+from qemu.aqmp.legacy import QEMUMonitorProtocol
 
 
 def perm(arr):
@@ -104,7 +102,7 @@ def command(self, cmd):
         reply = json.loads(subprocess.check_output(ar))
 
         if 'error' in reply:
-            raise QMPResponseError(reply)
+            raise QMPError(reply)
 
         return reply['return']
 
-- 
2.31.1

Re: [PULL 00/17] Python patches

[Peter Maydell]

On Sat, 22 Jan 2022 at 00:09, John Snow <jsnow@redhat.com> wrote:
>
> The following changes since commit 5e9d14f2bea6df89c0675df953f9c839560d2266:
>
>   Merge remote-tracking branch 'remotes/alistair/tags/pull-riscv-to-apply-20220121-1' into staging (2022-01-21 10:31:25 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/jsnow/qemu.git tags/python-pull-request
>
> for you to fetch changes up to 05908602429cf9d6fce9b60704b8395f6d295441:
>
>   scripts/render-block-graph: switch to AQMP (2022-01-21 16:01:31 -0500)
>
> ----------------------------------------------------------------
> Python patches
>
> A few fixes to the Python CI tests, a few fixes to the (async) QMP
> library, and a set of patches that begin to shift us towards using the
> new qmp lib.
>
> ----------------------------------------------------------------

Was this set of patches supposed to fix the NetBSD VM intermittents?

I still see that failure, though it looks a little different now:

  TEST   iotest-qcow2: 040 [fail]
QEMU          --
"/home/qemu/qemu-test.ArLruP/build/tests/qemu-iotests/../../qemu-system-aarch64"
-nodefaults -display none -accel qtest -machine virt
QEMU_IMG      --
"/home/qemu/qemu-test.ArLruP/build/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/qemu/qemu-test.ArLruP/build/tests/qemu-iotests/../../qemu-io"
--cache writeback --aio threads -f qcow2
QEMU_NBD      --
"/home/qemu/qemu-test.ArLruP/build/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2
IMGPROTO      -- file
PLATFORM      -- NetBSD/amd64 localhost 9.2
TEST_DIR      -- /home/qemu/qemu-test.ArLruP/build/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmpb3k9h89o
GDB_OPTIONS   --
VALGRIND_QEMU --
PRINT_QEMU_OUTPUT --

--- /home/qemu/qemu-test.ArLruP/src/tests/qemu-iotests/040.out
+++ 040.out.bad
@@ -1,5 +1,30 @@
-.................................................................
+............ERROR:qemu.aqmp.qmp_client.qemu-11749:Failed to establish
connection: concurrent.futures._base.CancelledError
+E....................................................
+======================================================================
+ERROR: test_top_invalid (__main__.TestActiveZeroLengthImage)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "/home/qemu/qemu-test.ArLruP/src/tests/qemu-iotests/040", line
94, in setUp
+    self.vm.launch()
+  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/machine/machine.py",
line 399, in launch
+    self._launch()
+  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/machine/machine.py",
line 434, in _launch
+    self._post_launch()
+  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/machine/qtest.py",
line 147, in _post_launch
+    super()._post_launch()
+  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/machine/machine.py",
line 340, in _post_launch
+    self._qmp.accept(self._qmp_timer)
+  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/aqmp/legacy.py",
line 92, in accept
+    timeout
+  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/aqmp/legacy.py",
line 65, in _sync
+    asyncio.wait_for(future, timeout=timeout)
+  File "/usr/pkg/lib/python3.7/asyncio/base_events.py", line 587, in
run_until_complete
+    return future.result()
+  File "/usr/pkg/lib/python3.7/asyncio/tasks.py", line 449, in wait_for
+    raise futures.TimeoutError()
+concurrent.futures._base.TimeoutError
+
 ----------------------------------------------------------------------
 Ran 65 tests

-OK
+FAILED (errors=1)

thanks
-- PMM

Re: [PULL 00/17] Python patches

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 4033 bytes --]

On Sat, Jan 22, 2022, 8:58 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Sat, 22 Jan 2022 at 00:09, John Snow <jsnow@redhat.com> wrote:
> >
> > The following changes since commit
> 5e9d14f2bea6df89c0675df953f9c839560d2266:
> >
> >   Merge remote-tracking branch
> 'remotes/alistair/tags/pull-riscv-to-apply-20220121-1' into staging
> (2022-01-21 10:31:25 +0000)
> >
> > are available in the Git repository at:
> >
> >   https://gitlab.com/jsnow/qemu.git tags/python-pull-request
> >
> > for you to fetch changes up to 05908602429cf9d6fce9b60704b8395f6d295441:
> >
> >   scripts/render-block-graph: switch to AQMP (2022-01-21 16:01:31 -0500)
> >
> > ----------------------------------------------------------------
> > Python patches
> >
> > A few fixes to the Python CI tests, a few fixes to the (async) QMP
> > library, and a set of patches that begin to shift us towards using the
> > new qmp lib.
> >
> > ----------------------------------------------------------------
>
> Was this set of patches supposed to fix the NetBSD VM intermittents?
>

No, nobody reviewed or tested that series yet, so I didn't pull it.

(Is that too conservative ...?)

If you'd like, I can include them anyway in a v2 pull here and you can test
it as part of your merge.


> I still see that failure, though it looks a little different now:
>
>   TEST   iotest-qcow2: 040 [fail]
> QEMU          --
>
> "/home/qemu/qemu-test.ArLruP/build/tests/qemu-iotests/../../qemu-system-aarch64"
> -nodefaults -display none -accel qtest -machine virt
> QEMU_IMG      --
> "/home/qemu/qemu-test.ArLruP/build/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/qemu/qemu-test.ArLruP/build/tests/qemu-iotests/../../qemu-io"
> --cache writeback --aio threads -f qcow2
> QEMU_NBD      --
> "/home/qemu/qemu-test.ArLruP/build/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2
> IMGPROTO      -- file
> PLATFORM      -- NetBSD/amd64 localhost 9.2
> TEST_DIR      --
> /home/qemu/qemu-test.ArLruP/build/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmpb3k9h89o
> GDB_OPTIONS   --
> VALGRIND_QEMU --
> PRINT_QEMU_OUTPUT --
>
> --- /home/qemu/qemu-test.ArLruP/src/tests/qemu-iotests/040.out
> +++ 040.out.bad
> @@ -1,5 +1,30 @@
> -.................................................................
> +............ERROR:qemu.aqmp.qmp_client.qemu-11749:Failed to establish
> connection: concurrent.futures._base.CancelledError
> +E....................................................
> +======================================================================
> +ERROR: test_top_invalid (__main__.TestActiveZeroLengthImage)
> +----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "/home/qemu/qemu-test.ArLruP/src/tests/qemu-iotests/040", line
> 94, in setUp
> +    self.vm.launch()
> +  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/machine/machine.py",
> line 399, in launch
> +    self._launch()
> +  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/machine/machine.py",
> line 434, in _launch
> +    self._post_launch()
> +  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/machine/qtest.py",
> line 147, in _post_launch
> +    super()._post_launch()
> +  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/machine/machine.py",
> line 340, in _post_launch
> +    self._qmp.accept(self._qmp_timer)
> +  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/aqmp/legacy.py",
> line 92, in accept
> +    timeout
> +  File "/home/qemu/qemu-test.ArLruP/src/python/qemu/aqmp/legacy.py",
> line 65, in _sync
> +    asyncio.wait_for(future, timeout=timeout)
> +  File "/usr/pkg/lib/python3.7/asyncio/base_events.py", line 587, in
> run_until_complete
> +    return future.result()
> +  File "/usr/pkg/lib/python3.7/asyncio/tasks.py", line 449, in wait_for
> +    raise futures.TimeoutError()
> +concurrent.futures._base.TimeoutError
> +
>  ----------------------------------------------------------------------
>  Ran 65 tests
>
> -OK
> +FAILED (errors=1)
>
> thanks
> -- PMM
>
>

[-- Attachment #2: Type: text/html, Size: 5375 bytes --]

Re: [PULL 00/17] Python patches

[Peter Maydell]

On Sat, 22 Jan 2022 at 17:06, John Snow <jsnow@redhat.com> wrote:
>
>
>
> On Sat, Jan 22, 2022, 8:58 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>>
>> On Sat, 22 Jan 2022 at 00:09, John Snow <jsnow@redhat.com> wrote:
>> >
>> > The following changes since commit 5e9d14f2bea6df89c0675df953f9c839560d2266:
>> >
>> >   Merge remote-tracking branch 'remotes/alistair/tags/pull-riscv-to-apply-20220121-1' into staging (2022-01-21 10:31:25 +0000)
>> >
>> > are available in the Git repository at:
>> >
>> >   https://gitlab.com/jsnow/qemu.git tags/python-pull-request
>> >
>> > for you to fetch changes up to 05908602429cf9d6fce9b60704b8395f6d295441:
>> >
>> >   scripts/render-block-graph: switch to AQMP (2022-01-21 16:01:31 -0500)
>> >
>> > ----------------------------------------------------------------
>> > Python patches
>> >
>> > A few fixes to the Python CI tests, a few fixes to the (async) QMP
>> > library, and a set of patches that begin to shift us towards using the
>> > new qmp lib.
>> >
>> > ----------------------------------------------------------------
>>
>> Was this set of patches supposed to fix the NetBSD VM intermittents?
>
>
> No, nobody reviewed or tested that series yet, so I didn't pull it.
>
> (Is that too conservative ...?)
>
> If you'd like, I can include them anyway in a v2 pull here and you can test it as part of your merge.

No, that's fine -- I just hadn't been keeping track of which patches
were intended as the fix.


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/7.0
for any user-visible changes.

-- PMM

Re: [PULL 00/17] Python patches

[Richard Henderson]

On 4/21/22 08:15, John Snow wrote:
> The following changes since commit b1efff6bf031a93b5b8bf3912ddc720cc1653a61:
> 
>    Merge tag 'pull-ppc-20220420-2' of https://gitlab.com/danielhb/qemu into staging (2022-04-20 21:54:24 -0700)
> 
> are available in the Git repository at:
> 
>    https://gitlab.com/jsnow/qemu.git tags/python-pull-request
> 
> for you to fetch changes up to 47430775ed1a48d7beb2c7b8d7feaab73104ec46:
> 
>    python/qmp: remove pylint workaround from legacy.py (2022-04-21 11:01:00 -0400)
> 
> ----------------------------------------------------------------
> Python patches
> 
> This PR finalizes the switch from Luiz's QMP library to mine.

Applied, thanks.  Please update https://wiki.qemu.org/ChangeLog/7.1 as appropriate.


r~



> 
> ----------------------------------------------------------------
> 
> John Snow (17):
>    python/machine: permanently switch to AQMP
>    scripts/bench-block-job: switch to AQMP
>    iotests/mirror-top-perms: switch to AQMP
>    iotests: switch to AQMP
>    python/aqmp: add explicit GPLv2 license to legacy.py
>    python/aqmp: relicense as LGPLv2+
>    python/qmp-shell: relicense as LGPLv2+
>    python/aqmp-tui: relicense as LGPLv2+
>    python: temporarily silence pylint duplicate-code warnings
>    python/aqmp: take QMPBadPortError and parse_address from qemu.qmp
>    python/aqmp: fully separate from qmp.QEMUMonitorProtocol
>    python/aqmp: copy qmp docstrings to qemu.aqmp.legacy
>    python: remove the old QMP package
>    python: re-enable pylint duplicate-code warnings
>    python: rename qemu.aqmp to qemu.qmp
>    python: rename 'aqmp-tui' to 'qmp-tui'
>    python/qmp: remove pylint workaround from legacy.py
> 
>   python/README.rst                             |   2 +-
>   python/qemu/qmp/README.rst                    |   9 -
>   python/qemu/aqmp/__init__.py                  |  59 ---
>   python/qemu/aqmp/legacy.py                    | 177 -------
>   python/qemu/aqmp/py.typed                     |   0
>   python/qemu/machine/machine.py                |  18 +-
>   python/qemu/machine/qtest.py                  |   2 +-
>   python/qemu/qmp/__init__.py                   | 445 ++----------------
>   python/qemu/{aqmp => qmp}/error.py            |   0
>   python/qemu/{aqmp => qmp}/events.py           |   2 +-
>   python/qemu/qmp/legacy.py                     | 315 +++++++++++++
>   python/qemu/{aqmp => qmp}/message.py          |   0
>   python/qemu/{aqmp => qmp}/models.py           |   0
>   python/qemu/{aqmp => qmp}/protocol.py         |   4 +-
>   python/qemu/{aqmp => qmp}/qmp_client.py       |  16 +-
>   python/qemu/{aqmp => qmp}/qmp_shell.py        |  11 +-
>   .../qemu/{aqmp/aqmp_tui.py => qmp/qmp_tui.py} |  17 +-
>   python/qemu/{aqmp => qmp}/util.py             |   0
>   python/qemu/utils/qemu_ga_client.py           |   4 +-
>   python/qemu/utils/qom.py                      |   2 +-
>   python/qemu/utils/qom_common.py               |   4 +-
>   python/qemu/utils/qom_fuse.py                 |   2 +-
>   python/setup.cfg                              |  11 +-
>   python/tests/protocol.py                      |  14 +-
>   scripts/cpu-x86-uarch-abi.py                  |   2 +-
>   scripts/device-crash-test                     |   4 +-
>   scripts/qmp/qmp-shell                         |   2 +-
>   scripts/qmp/qmp-shell-wrap                    |   2 +-
>   scripts/render_block_graph.py                 |   4 +-
>   scripts/simplebench/bench_block_job.py        |   5 +-
>   tests/qemu-iotests/iotests.py                 |   3 +-
>   tests/qemu-iotests/tests/mirror-top-perms     |  11 +-
>   32 files changed, 422 insertions(+), 725 deletions(-)
>   delete mode 100644 python/qemu/qmp/README.rst
>   delete mode 100644 python/qemu/aqmp/__init__.py
>   delete mode 100644 python/qemu/aqmp/legacy.py
>   delete mode 100644 python/qemu/aqmp/py.typed
>   rename python/qemu/{aqmp => qmp}/error.py (100%)
>   rename python/qemu/{aqmp => qmp}/events.py (99%)
>   create mode 100644 python/qemu/qmp/legacy.py
>   rename python/qemu/{aqmp => qmp}/message.py (100%)
>   rename python/qemu/{aqmp => qmp}/models.py (100%)
>   rename python/qemu/{aqmp => qmp}/protocol.py (99%)
>   rename python/qemu/{aqmp => qmp}/qmp_client.py (97%)
>   rename python/qemu/{aqmp => qmp}/qmp_shell.py (98%)
>   rename python/qemu/{aqmp/aqmp_tui.py => qmp/qmp_tui.py} (98%)
>   rename python/qemu/{aqmp => qmp}/util.py (100%)
> 

[PULL 00/17] Python patches

[John Snow]

The following changes since commit b1efff6bf031a93b5b8bf3912ddc720cc1653a61:

  Merge tag 'pull-ppc-20220420-2' of https://gitlab.com/danielhb/qemu into staging (2022-04-20 21:54:24 -0700)

are available in the Git repository at:

  https://gitlab.com/jsnow/qemu.git tags/python-pull-request

for you to fetch changes up to 47430775ed1a48d7beb2c7b8d7feaab73104ec46:

  python/qmp: remove pylint workaround from legacy.py (2022-04-21 11:01:00 -0400)

----------------------------------------------------------------
Python patches

This PR finalizes the switch from Luiz's QMP library to mine.

----------------------------------------------------------------

John Snow (17):
  python/machine: permanently switch to AQMP
  scripts/bench-block-job: switch to AQMP
  iotests/mirror-top-perms: switch to AQMP
  iotests: switch to AQMP
  python/aqmp: add explicit GPLv2 license to legacy.py
  python/aqmp: relicense as LGPLv2+
  python/qmp-shell: relicense as LGPLv2+
  python/aqmp-tui: relicense as LGPLv2+
  python: temporarily silence pylint duplicate-code warnings
  python/aqmp: take QMPBadPortError and parse_address from qemu.qmp
  python/aqmp: fully separate from qmp.QEMUMonitorProtocol
  python/aqmp: copy qmp docstrings to qemu.aqmp.legacy
  python: remove the old QMP package
  python: re-enable pylint duplicate-code warnings
  python: rename qemu.aqmp to qemu.qmp
  python: rename 'aqmp-tui' to 'qmp-tui'
  python/qmp: remove pylint workaround from legacy.py

 python/README.rst                             |   2 +-
 python/qemu/qmp/README.rst                    |   9 -
 python/qemu/aqmp/__init__.py                  |  59 ---
 python/qemu/aqmp/legacy.py                    | 177 -------
 python/qemu/aqmp/py.typed                     |   0
 python/qemu/machine/machine.py                |  18 +-
 python/qemu/machine/qtest.py                  |   2 +-
 python/qemu/qmp/__init__.py                   | 445 ++----------------
 python/qemu/{aqmp => qmp}/error.py            |   0
 python/qemu/{aqmp => qmp}/events.py           |   2 +-
 python/qemu/qmp/legacy.py                     | 315 +++++++++++++
 python/qemu/{aqmp => qmp}/message.py          |   0
 python/qemu/{aqmp => qmp}/models.py           |   0
 python/qemu/{aqmp => qmp}/protocol.py         |   4 +-
 python/qemu/{aqmp => qmp}/qmp_client.py       |  16 +-
 python/qemu/{aqmp => qmp}/qmp_shell.py        |  11 +-
 .../qemu/{aqmp/aqmp_tui.py => qmp/qmp_tui.py} |  17 +-
 python/qemu/{aqmp => qmp}/util.py             |   0
 python/qemu/utils/qemu_ga_client.py           |   4 +-
 python/qemu/utils/qom.py                      |   2 +-
 python/qemu/utils/qom_common.py               |   4 +-
 python/qemu/utils/qom_fuse.py                 |   2 +-
 python/setup.cfg                              |  11 +-
 python/tests/protocol.py                      |  14 +-
 scripts/cpu-x86-uarch-abi.py                  |   2 +-
 scripts/device-crash-test                     |   4 +-
 scripts/qmp/qmp-shell                         |   2 +-
 scripts/qmp/qmp-shell-wrap                    |   2 +-
 scripts/render_block_graph.py                 |   4 +-
 scripts/simplebench/bench_block_job.py        |   5 +-
 tests/qemu-iotests/iotests.py                 |   3 +-
 tests/qemu-iotests/tests/mirror-top-perms     |  11 +-
 32 files changed, 422 insertions(+), 725 deletions(-)
 delete mode 100644 python/qemu/qmp/README.rst
 delete mode 100644 python/qemu/aqmp/__init__.py
 delete mode 100644 python/qemu/aqmp/legacy.py
 delete mode 100644 python/qemu/aqmp/py.typed
 rename python/qemu/{aqmp => qmp}/error.py (100%)
 rename python/qemu/{aqmp => qmp}/events.py (99%)
 create mode 100644 python/qemu/qmp/legacy.py
 rename python/qemu/{aqmp => qmp}/message.py (100%)
 rename python/qemu/{aqmp => qmp}/models.py (100%)
 rename python/qemu/{aqmp => qmp}/protocol.py (99%)
 rename python/qemu/{aqmp => qmp}/qmp_client.py (97%)
 rename python/qemu/{aqmp => qmp}/qmp_shell.py (98%)
 rename python/qemu/{aqmp/aqmp_tui.py => qmp/qmp_tui.py} (98%)
 rename python/qemu/{aqmp => qmp}/util.py (100%)

-- 
2.34.1