[PULL 0/4] Python patches

[PULL 0/4] Python patches

[John Snow]

The following changes since commit de3f5223fa4cf8bfc5e3fe1fd495ddf468edcdf7:

  Merge remote-tracking branch 'remotes/vivier/tags/m68k-for-7.0-pull-request' into staging (2022-01-10 14:43:03 +0000)

are available in the Git repository at:

  https://gitlab.com/jsnow/qemu.git tags/python-pull-request

for you to fetch changes up to 9ebfc5a583d8aa94bf1bc37c1f71559187fd809c:

  simplebench: Fix Python syntax error (reported by LGTM) (2022-01-10 18:23:10 -0500)

----------------------------------------------------------------
Python pull request

Fixes for the tests that broke during vacation, plus a simple syntax fix
for a python script.

----------------------------------------------------------------

John Snow (3):
  python/aqmp: use absolute import statement
  Python/aqmp: fix type definitions for mypy 0.920
  python: update type hints for mypy 0.930

Stefan Weil (1):
  simplebench: Fix Python syntax error (reported by LGTM)

 python/qemu/aqmp/aqmp_tui.py         | 3 ++-
 python/qemu/aqmp/protocol.py         | 5 +++--
 python/qemu/qmp/qom_common.py        | 6 +-----
 scripts/simplebench/bench-example.py | 2 +-
 4 files changed, 7 insertions(+), 9 deletions(-)

-- 
2.31.1

[PULL 1/4] python/aqmp: use absolute import statement

[John Snow]

pylint's dependency astroid appears to have bugs in 2.9.1 and 2.9.2 (Dec
31 and Jan 3) that appear to erroneously expect the qemu namespace to
have an __init__.py file. astroid 2.9.3 (Jan 9) avoids that problem, but
appears to not understand a relative import within a namespace package.

Update the relative import - it was worth changing anyway, because these
packages will eventually be packaged and distributed separately.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Message-id: 20220110191349.1841027-2-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/aqmp/aqmp_tui.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/python/qemu/aqmp/aqmp_tui.py b/python/qemu/aqmp/aqmp_tui.py
index a2929f771c..f1e926dd75 100644
--- a/python/qemu/aqmp/aqmp_tui.py
+++ b/python/qemu/aqmp/aqmp_tui.py
@@ -35,7 +35,8 @@
 import urwid
 import urwid_readline
 
-from ..qmp import QEMUMonitorProtocol, QMPBadPortError
+from qemu.qmp import QEMUMonitorProtocol, QMPBadPortError
+
 from .error import ProtocolError
 from .message import DeserializationError, Message, UnexpectedTypeError
 from .protocol import ConnectError, Runstate
-- 
2.31.1

[PULL 2/4] Python/aqmp: fix type definitions for mypy 0.920

[John Snow]

0.920 (Released 2021-12-15) is not entirely happy with the
way that I was defining _FutureT:

qemu/aqmp/protocol.py:601: error: Item "object" of the upper bound
"Optional[Future[Any]]" of type variable "_FutureT" has no attribute
"done"

Update it with something a little mechanically simpler that works better
across a wider array of mypy versions.

Signed-off-by: John Snow <jsnow@redhat.com>
Message-id: 20220110191349.1841027-3-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/aqmp/protocol.py | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/python/qemu/aqmp/protocol.py b/python/qemu/aqmp/protocol.py
index 5190b33b13..c4fbe35a0e 100644
--- a/python/qemu/aqmp/protocol.py
+++ b/python/qemu/aqmp/protocol.py
@@ -43,8 +43,8 @@
 
 
 T = TypeVar('T')
+_U = TypeVar('_U')
 _TaskFN = Callable[[], Awaitable[None]]  # aka ``async def func() -> None``
-_FutureT = TypeVar('_FutureT', bound=Optional['asyncio.Future[Any]'])
 
 
 class Runstate(Enum):
@@ -591,7 +591,8 @@ def _cleanup(self) -> None:
         """
         Fully reset this object to a clean state and return to `IDLE`.
         """
-        def _paranoid_task_erase(task: _FutureT) -> Optional[_FutureT]:
+        def _paranoid_task_erase(task: Optional['asyncio.Future[_U]']
+                                 ) -> Optional['asyncio.Future[_U]']:
             # Help to erase a task, ENSURING it is fully quiesced first.
             assert (task is None) or task.done()
             return None if (task and task.done()) else task
-- 
2.31.1

[PULL 3/4] python: update type hints for mypy 0.930

[John Snow]

Mypy 0.930, released Dec 22, changes the way argparse objects are
considered. Crafting a definition that works under Python 3.6 and an
older mypy alongside newer versions simultaneously is ... difficult,
so... eh. Stub it out with an 'Any' definition to get the CI moving
again.

Oh well.

Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Beraldo Leal <bleal@redhat.com>
Message-id: 20220110191349.1841027-4-jsnow@redhat.com
Signed-off-by: John Snow <jsnow@redhat.com>
---
 python/qemu/qmp/qom_common.py | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/python/qemu/qmp/qom_common.py b/python/qemu/qmp/qom_common.py
index a59ae1a2a1..2e4c741f77 100644
--- a/python/qemu/qmp/qom_common.py
+++ b/python/qemu/qmp/qom_common.py
@@ -30,10 +30,6 @@
 from . import QEMUMonitorProtocol, QMPError
 
 
-# The following is needed only for a type alias.
-Subparsers = argparse._SubParsersAction  # pylint: disable=protected-access
-
-
 class ObjectPropertyInfo:
     """
     Represents the return type from e.g. qom-list.
@@ -89,7 +85,7 @@ def __init__(self, args: argparse.Namespace):
         self.qmp.connect()
 
     @classmethod
-    def register(cls, subparsers: Subparsers) -> None:
+    def register(cls, subparsers: Any) -> None:
         """
         Register this command with the argument parser.
 
-- 
2.31.1

[PULL 4/4] simplebench: Fix Python syntax error (reported by LGTM)

[John Snow]

From: Stefan Weil <sw@weilnetz.de>

Fixes: b2fcb0c5754c2554b8406376e99a75e9e0a6b7bd
Signed-off-by: Stefan Weil <sw@weilnetz.de>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: John Snow <jsnow@redhat.com>
Message-id: 20220107153019.504124-1-sw@weilnetz.de
Signed-off-by: John Snow <jsnow@redhat.com>
---
 scripts/simplebench/bench-example.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/simplebench/bench-example.py b/scripts/simplebench/bench-example.py
index 4864435f39..fc370691e0 100644
--- a/scripts/simplebench/bench-example.py
+++ b/scripts/simplebench/bench-example.py
@@ -25,7 +25,7 @@
 
 def bench_func(env, case):
     """ Handle one "cell" of benchmarking table. """
-    return bench_block_copy(env['qemu_binary'], env['cmd'], {}
+    return bench_block_copy(env['qemu_binary'], env['cmd'], {},
                             case['source'], case['target'])
 
 
-- 
2.31.1

Re: [PULL 0/4] Python patches

[Peter Maydell]

On Mon, 10 Jan 2022 at 23:25, John Snow <jsnow@redhat.com> wrote:
>
> The following changes since commit de3f5223fa4cf8bfc5e3fe1fd495ddf468edcdf7:
>
>   Merge remote-tracking branch 'remotes/vivier/tags/m68k-for-7.0-pull-request' into staging (2022-01-10 14:43:03 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/jsnow/qemu.git tags/python-pull-request
>
> for you to fetch changes up to 9ebfc5a583d8aa94bf1bc37c1f71559187fd809c:
>
>   simplebench: Fix Python syntax error (reported by LGTM) (2022-01-10 18:23:10 -0500)
>
> ----------------------------------------------------------------
> Python pull request
>
> Fixes for the tests that broke during vacation, plus a simple syntax fix
> for a python script.


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/7.0
for any user-visible changes.

-- PMM

Re: [PULL 0/4] Python patches

[John Snow]

On Tue, Feb 15, 2022 at 1:01 PM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Tue, 15 Feb 2022 at 17:46, John Snow <jsnow@redhat.com> wrote:
> > Just so I don't leave this thread hanging, I filed a GitLab issue and
> > I'm working on it, but this one isn't as quick to solve as the other.
> >
> > https://gitlab.com/qemu-project/qemu/-/issues/874
>
> Is there anything particular to NetBSD that means it happens
> more often there, or is it just random luck that we hit
> the race there and haven't seen it elsewhere ?
>
> -- PMM

Complete random luck, something jostled loose by the scheduler.

I need to change the interface in the async library entirely to make
the process more granular -- We don't need the granularity in a truly
async mode, but the sync wrapper that allows the existing iotests
corpus to use the library in a synchronous manner *requires* a more
granular connection API, so I have to write one. It's in progress, it
just might be a few more days; verifying and testing the error
pathways has been slow work.

(In detail: python's asyncio.create_unix_server() call combines bind()
+ listen() + accept() into a single discrete step. A synchronous
client, though, needs to have a reprieve from all of those blocking
steps to launch the QEMU process after listen() but before accept() so
it can launch the QEMU process. I was able to pull the bind() step
out, but the async listen() + accept() steps the way I initially wrote
it are inseparable. Live and learn.)

In the meantime, there *IS* a way to use the old library, but I don't
think the environment variable in question is routed down into the VM
tests. I can look at (as a very quick fix) amending the VM launcher to
pass along that environment variable if it sees it set in the host
environment -- that should get you on the old, tried-and-true library
when you want it, and the test should pass.

--js

Re: [PULL 0/4] Python patches

[Peter Maydell]

On Tue, 15 Feb 2022 at 17:46, John Snow <jsnow@redhat.com> wrote:
> Just so I don't leave this thread hanging, I filed a GitLab issue and
> I'm working on it, but this one isn't as quick to solve as the other.
>
> https://gitlab.com/qemu-project/qemu/-/issues/874

Is there anything particular to NetBSD that means it happens
more often there, or is it just random luck that we hit
the race there and haven't seen it elsewhere ?

-- PMM

Re: [PULL 0/4] Python patches

[John Snow]

On Tue, Feb 8, 2022 at 9:40 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Thu, 3 Feb 2022 at 23:22, John Snow <jsnow@redhat.com> wrote:
> >
> > On Thu, Feb 3, 2022 at 11:52 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> > >
> > > On Thu, 3 Feb 2022 at 16:38, John Snow <jsnow@redhat.com> wrote:
> > >
> > > > On Thu, Feb 3, 2022, 11:20 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> > > >> Summary of Failures:
> > > >>
> > > >> 1/1 qemu:block / qemu-iotests qcow2 ERROR          243.14s   exit status 1
> >
> > I'm not too familiar with this new test runner, yet. (Is this error
> > even anything to do with the python lib? I guess I can't rule it
> > out...)
> > I just got a clean run of 'make vm-build-netbsd', so I'm using that
> > output as reference and making some guesses.
>
> Rerunning on the netbsd VM with Paolo's "revert the iotests
> conversion" patch, here's the output from a failing run, where
> iotest 041 failed:
>
> TEST   iotest-qcow2: 041 [fail]
> QEMU          --
> "/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/../../qemu-system-aarch64"
> -nodefaults -display none -accel qtest -machine virt
> QEMU_IMG      --
> "/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/../../qemu-io"
> --cache writeback --aio threads -f qcow2
> QEMU_NBD      --
> "/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2
> IMGPROTO      -- file
> PLATFORM      -- NetBSD/amd64 localhost 9.2
> TEST_DIR      -- /home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp6fiu68sr
> GDB_OPTIONS   --
> VALGRIND_QEMU --
> PRINT_QEMU_OUTPUT --
>
> --- /home/qemu/qemu-test.Kywnb7/src/tests/qemu-iotests/041.out
> +++ 041.out.bad
> @@ -1,5 +1,44 @@
> -...........................................................................................................
> +........................................ERROR:qemu.aqmp.qmp_client.qemu-14411:Failed
> to establish connection: concurrent.futures._base.CancelledError
> +E..................................................................
> +======================================================================
> +ERROR: test_mirror_to_self (__main__.TestSingleBlockdev)
> +----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/machine/machine.py",
> line 428, in launch
> +    self._launch()
> +  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/machine/machine.py",
> line 467, in _launch
> +    self._post_launch()
> +  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/machine/qtest.py",
> line 147, in _post_launch
> +    super()._post_launch()
> +  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/machine/machine.py",
> line 369, in _post_launch
> +    self._qmp.accept(self._qmp_timer)
> +  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/aqmp/legacy.py",
> line 95, in accept
> +    timeout
> +  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/aqmp/legacy.py",
> line 68, in _sync
> +    asyncio.wait_for(future, timeout=timeout)
> +  File "/usr/pkg/lib/python3.7/asyncio/base_events.py", line 587, in
> run_until_complete
> +    return future.result()
> +  File "/usr/pkg/lib/python3.7/asyncio/tasks.py", line 449, in wait_for
> +    raise futures.TimeoutError()
> +concurrent.futures._base.TimeoutError
> +
> +The above exception was the direct cause of the following exception:
> +
> +Traceback (most recent call last):
> +  File "/home/qemu/qemu-test.Kywnb7/src/tests/qemu-iotests/041", line
> 233, in setUp
> +    TestSingleDrive.setUp(self)
> +  File "/home/qemu/qemu-test.Kywnb7/src/tests/qemu-iotests/041", line
> 54, in setUp
> +    self.vm.launch()
> +  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/machine/machine.py",
> line 445, in launch
> +    ) from exc
> +qemu.machine.machine.VMLaunchFailure: TimeoutError
> +       Exit code: 1
> +       Command:
> /home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/../../qemu-system-aarch64
> -display none -vga none -chardev
> socket,id=mon,path=/tmp/tmp6fiu68sr/qemu-14411-monitor.sock -mon
> chardev=mon,mode=control -qtest
> unix:path=/tmp/tmp6fiu68sr/qemu-14411-qtest.sock -accel qtest
> -nodefaults -display none -accel qtest -machine virt -drive
> if=virtio,id=drive0,file=/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/scratch/test.img,format=qcow2,cache=writeback,aio=threads,node-name=top,backing.node-name=base
> +       Output: qemu-system-aarch64: -chardev
> socket,id=mon,path=/tmp/tmp6fiu68sr/qemu-14411-monitor.sock: Failed to
> connect to '/tmp/tmp6fiu68sr/qemu-14411-monitor.sock': Connection
> refused
> +
> +
> +
>  ----------------------------------------------------------------------
>  Ran 107 tests
>
> -OK
> +FAILED (errors=1)
>
>
> thanks
> -- PMM
>

Just so I don't leave this thread hanging, I filed a GitLab issue and
I'm working on it, but this one isn't as quick to solve as the other.

https://gitlab.com/qemu-project/qemu/-/issues/874

--js

Re: [PULL 0/4] Python patches

[Peter Maydell]

On Thu, 3 Feb 2022 at 23:22, John Snow <jsnow@redhat.com> wrote:
>
> On Thu, Feb 3, 2022 at 11:52 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> >
> > On Thu, 3 Feb 2022 at 16:38, John Snow <jsnow@redhat.com> wrote:
> >
> > > On Thu, Feb 3, 2022, 11:20 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> > >> Summary of Failures:
> > >>
> > >> 1/1 qemu:block / qemu-iotests qcow2 ERROR          243.14s   exit status 1
>
> I'm not too familiar with this new test runner, yet. (Is this error
> even anything to do with the python lib? I guess I can't rule it
> out...)
> I just got a clean run of 'make vm-build-netbsd', so I'm using that
> output as reference and making some guesses.

Rerunning on the netbsd VM with Paolo's "revert the iotests
conversion" patch, here's the output from a failing run, where
iotest 041 failed:

TEST   iotest-qcow2: 041 [fail]
QEMU          --
"/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/../../qemu-system-aarch64"
-nodefaults -display none -accel qtest -machine virt
QEMU_IMG      --
"/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/../../qemu-io"
--cache writeback --aio threads -f qcow2
QEMU_NBD      --
"/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2
IMGPROTO      -- file
PLATFORM      -- NetBSD/amd64 localhost 9.2
TEST_DIR      -- /home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp6fiu68sr
GDB_OPTIONS   --
VALGRIND_QEMU --
PRINT_QEMU_OUTPUT --

--- /home/qemu/qemu-test.Kywnb7/src/tests/qemu-iotests/041.out
+++ 041.out.bad
@@ -1,5 +1,44 @@
-...........................................................................................................
+........................................ERROR:qemu.aqmp.qmp_client.qemu-14411:Failed
to establish connection: concurrent.futures._base.CancelledError
+E..................................................................
+======================================================================
+ERROR: test_mirror_to_self (__main__.TestSingleBlockdev)
+----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/machine/machine.py",
line 428, in launch
+    self._launch()
+  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/machine/machine.py",
line 467, in _launch
+    self._post_launch()
+  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/machine/qtest.py",
line 147, in _post_launch
+    super()._post_launch()
+  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/machine/machine.py",
line 369, in _post_launch
+    self._qmp.accept(self._qmp_timer)
+  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/aqmp/legacy.py",
line 95, in accept
+    timeout
+  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/aqmp/legacy.py",
line 68, in _sync
+    asyncio.wait_for(future, timeout=timeout)
+  File "/usr/pkg/lib/python3.7/asyncio/base_events.py", line 587, in
run_until_complete
+    return future.result()
+  File "/usr/pkg/lib/python3.7/asyncio/tasks.py", line 449, in wait_for
+    raise futures.TimeoutError()
+concurrent.futures._base.TimeoutError
+
+The above exception was the direct cause of the following exception:
+
+Traceback (most recent call last):
+  File "/home/qemu/qemu-test.Kywnb7/src/tests/qemu-iotests/041", line
233, in setUp
+    TestSingleDrive.setUp(self)
+  File "/home/qemu/qemu-test.Kywnb7/src/tests/qemu-iotests/041", line
54, in setUp
+    self.vm.launch()
+  File "/home/qemu/qemu-test.Kywnb7/src/python/qemu/machine/machine.py",
line 445, in launch
+    ) from exc
+qemu.machine.machine.VMLaunchFailure: TimeoutError
+       Exit code: 1
+       Command:
/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/../../qemu-system-aarch64
-display none -vga none -chardev
socket,id=mon,path=/tmp/tmp6fiu68sr/qemu-14411-monitor.sock -mon
chardev=mon,mode=control -qtest
unix:path=/tmp/tmp6fiu68sr/qemu-14411-qtest.sock -accel qtest
-nodefaults -display none -accel qtest -machine virt -drive
if=virtio,id=drive0,file=/home/qemu/qemu-test.Kywnb7/build/tests/qemu-iotests/scratch/test.img,format=qcow2,cache=writeback,aio=threads,node-name=top,backing.node-name=base
+       Output: qemu-system-aarch64: -chardev
socket,id=mon,path=/tmp/tmp6fiu68sr/qemu-14411-monitor.sock: Failed to
connect to '/tmp/tmp6fiu68sr/qemu-14411-monitor.sock': Connection
refused
+
+
+
 ----------------------------------------------------------------------
 Ran 107 tests

-OK
+FAILED (errors=1)


thanks
-- PMM

Re: [PULL 0/4] Python patches

[Peter Maydell]

On Thu, 3 Feb 2022 at 01:59, John Snow <jsnow@redhat.com> wrote:
>
> The following changes since commit 47cc1a3655135b89fa75c2824fbddd29df874612:
>
>   Merge remote-tracking branch 'remotes/kwolf-gitlab/tags/for-upstream' into staging (2022-02-01 19:48:15 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/jsnow/qemu.git tags/python-pull-request
>
> for you to fetch changes up to b0b662bb2b340d63529672b5bdae596a6243c4d0:
>
>   python/aqmp: add socket bind step to legacy.py (2022-02-02 14:12:22 -0500)
>
> ----------------------------------------------------------------
> Python patches
>
> Peter: I expect this to address the iotest 040,041 failures you observed
> on NetBSD. If it doesn't, let me know.
>
> ----------------------------------------------------------------



Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/7.0
for any user-visible changes.

-- PMM

Re: [PULL 0/4] Python patches

[John Snow]

On Thu, Feb 3, 2022 at 11:52 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>
> On Thu, 3 Feb 2022 at 16:38, John Snow <jsnow@redhat.com> wrote:
>
> > On Thu, Feb 3, 2022, 11:20 AM Peter Maydell <peter.maydell@linaro.org> wrote:
> >> Summary of Failures:
> >>
> >> 1/1 qemu:block / qemu-iotests qcow2 ERROR          243.14s   exit status 1

I'm not too familiar with this new test runner, yet. (Is this error
even anything to do with the python lib? I guess I can't rule it
out...)
I just got a clean run of 'make vm-build-netbsd', so I'm using that
output as reference and making some guesses.

If I search the output for 'qcow2', I see the following output (with
possibly many lines between each hit):

1/1 qemu:block / qemu-iotests qcow2        RUNNING
>>> MALLOC_PERTURB_=205 PYTHON=/usr/pkg/bin/python3.7 /bin/sh /home/qemu/qemu-test.lj6FNa/build/../src/tests/qemu-iotests/../check-block.sh qcow2
▶ 1/1 qcow2 001                            OK
▶ 1/1 qcow2 002                            OK
▶ 1/1 qcow2 004                            OK

... and so on and so forth ...

▶ 1/1 qcow2 299                            OK
▶ 1/1 qcow2 313                            SKIP
▶ 1/1 qcow2 nbd-qemu-allocation            SKIP
▶ 1/1 qcow2 qsd-jobs                       OK
1/1 qemu:block / qemu-iotests qcow2        OK             176.35s   74
subtests passed


I tried modifying 040 to fail on purpose, and I see:

▶ 1/1 qcow2 039                            OK
▶ 1/1 qcow2 040                            FAIL
▶ 1/1 qcow2 041                            OK

[...]

▶ 1/1 qcow2 nbd-qemu-allocation            OK
▶ 1/1 qcow2 qsd-jobs                       OK
1/1 qemu:block / qemu-iotests qcow2        ERROR          106.06s
exit status 1
Summary of Failures:
1/1 qemu:block / qemu-iotests qcow2 ERROR          106.06s   exit status 1


I don't think I see it on the output you mailed, but can you point out
which test is failing, at least? Grepping for 'FAIL' should be
helpful.

--js

Re: [PULL 0/4] Python patches

[Peter Maydell]

On Thu, 3 Feb 2022 at 16:38, John Snow <jsnow@redhat.com> wrote:

> On Thu, Feb 3, 2022, 11:20 AM Peter Maydell <peter.maydell@linaro.org> wrote:
>> Summary of Failures:
>>
>> 1/1 qemu:block / qemu-iotests qcow2 ERROR          243.14s   exit status 1
>>
>>
>> Ok:                 0
>> Expected Fail:      0
>> Fail:               1
>> Unexpected Pass:    0
>> Skipped:            0
>> Timeout:            0
>>
>> Full log written to /home/qemu/qemu-test.yiYr4m/build/meson-logs/iotestslog.txt
>> ▶ 147/704 /bdrv-drain/deletion/drain
>>            OK
>> ▶ 178/704 /crypto/task/complete
>>            OK
>> ▶ 178/704 /crypto/task/datafree
>>            OK
>> [etc]

> Any chance of seeing that meson-logs/iotestslog.txt file?

Sorry, no. The VM runs, and it produces output to stdout, and
then it goes away again. The test cases and test harnesses
*must* output to standard output any information that might
be useful for diagnosing problems. The same scenario applies
for the gitlab CI jobs -- all we get is the job's output.

-- PMM

Re: [PULL 0/4] Python patches

[John Snow]

[-- Attachment #1: Type: text/plain, Size: 3504 bytes --]

On Thu, Feb 3, 2022, 11:20 AM Peter Maydell <peter.maydell@linaro.org>
wrote:

> On Thu, 3 Feb 2022 at 01:59, John Snow <jsnow@redhat.com> wrote:
> >
> > The following changes since commit
> 47cc1a3655135b89fa75c2824fbddd29df874612:
> >
> >   Merge remote-tracking branch 'remotes/kwolf-gitlab/tags/for-upstream'
> into staging (2022-02-01 19:48:15 +0000)
> >
> > are available in the Git repository at:
> >
> >   https://gitlab.com/jsnow/qemu.git tags/python-pull-request
> >
> > for you to fetch changes up to b0b662bb2b340d63529672b5bdae596a6243c4d0:
> >
> >   python/aqmp: add socket bind step to legacy.py (2022-02-02 14:12:22
> -0500)
> >
> > ----------------------------------------------------------------
> > Python patches
> >
> > Peter: I expect this to address the iotest 040,041 failures you observed
> > on NetBSD. If it doesn't, let me know.
>
> I still see this one, which is different from the 040,041 stuff,
> and where 'make check' is for some reason giving a lot less useful
> detail. (This is a prexisting intermittent from before this patchset).
>

I'm assuming there's less detail because of the meson test-runner doing
some io redirection into the logfile.

[etc]
>
▶ 175/704 /io/channel/pipe/sync
>            OK
> ▶ 175/704 /io/channel/pipe/async
>            OK
> 175/704 qemu:unit / test-io-channel-file
>            OK              0.11s   5 subtests passed
>
> 177/704 qemu:unit / test-io-channel-tls
>            RUNNING
> >>> G_TEST_BUILDDIR=/home/qemu/qemu-test.yiYr4m/build/tests/unit
> MALLOC_PERTURB_=5 G_TEST_SRCDIR=/home/qemu/qemu-test.yiYr4m/src/tests/unit
> /home/
> qemu/qemu-test.yiYr4m/build/tests/unit/test-io-channel-tls --tap -k
> ▶ 176/704 /io/channel/socket/ipv4-sync
>            OK
> ▶ 176/704 /io/channel/socket/ipv4-async
>            OK
> ▶ 176/704 /io/channel/socket/ipv4-fd
>            OK
> ▶ 176/704 /io/channel/socket/ipv6-sync
>            OK
> ▶ 176/704 /io/channel/socket/ipv6-async
>            OK
> ▶ 176/704 /io/channel/socket/unix-sync
>            OK
> ▶ 176/704 /io/channel/socket/unix-async
>            OK
> ▶ 176/704 /io/channel/socket/unix-fd-pass
>            OK
> ▶ 176/704 /io/channel/socket/unix-listen-cleanup
>            OK
> 176/704 qemu:unit / test-io-channel-socket
>            OK              0.13s   9 subtests passed
>
> ▶ 1/1 qcow2 qsd-jobs                OK
> 1/1 qemu:block / qemu-iotests qcow2 ERROR          243.14s   exit status 1
>
> 178/704 qemu:unit / test-io-task
>            RUNNING
> >>> G_TEST_BUILDDIR=/home/qemu/qemu-test.yiYr4m/build/tests/unit
> MALLOC_PERTURB_=194
> G_TEST_SRCDIR=/home/qemu/qemu-test.yiYr4m/src/tests/unit /hom
> e/qemu/qemu-test.yiYr4m/build/tests/unit/test-io-task --tap -k
> ▶ 147/704 /bdrv-drain/blockjob/iothread/error/drain_subtree
>            OK
>
> Summary of Failures:
>
> 1/1 qemu:block / qemu-iotests qcow2 ERROR          243.14s   exit status 1
>
>
> Ok:                 0
> Expected Fail:      0
> Fail:               1
> Unexpected Pass:    0
> Skipped:            0
> Timeout:            0
>
> Full log written to
> /home/qemu/qemu-test.yiYr4m/build/meson-logs/iotestslog.txt
> ▶ 147/704 /bdrv-drain/deletion/drain
>            OK
> ▶ 178/704 /crypto/task/complete
>            OK
> ▶ 178/704 /crypto/task/datafree
>            OK
> [etc]
>
> thanks
> -- PMM
>

Any chance of seeing that meson-logs/iotestslog.txt file?

>

[-- Attachment #2: Type: text/html, Size: 4959 bytes --]

Re: [PULL 0/4] Python patches

[Peter Maydell]

On Thu, 3 Feb 2022 at 01:59, John Snow <jsnow@redhat.com> wrote:
>
> The following changes since commit 47cc1a3655135b89fa75c2824fbddd29df874612:
>
>   Merge remote-tracking branch 'remotes/kwolf-gitlab/tags/for-upstream' into staging (2022-02-01 19:48:15 +0000)
>
> are available in the Git repository at:
>
>   https://gitlab.com/jsnow/qemu.git tags/python-pull-request
>
> for you to fetch changes up to b0b662bb2b340d63529672b5bdae596a6243c4d0:
>
>   python/aqmp: add socket bind step to legacy.py (2022-02-02 14:12:22 -0500)
>
> ----------------------------------------------------------------
> Python patches
>
> Peter: I expect this to address the iotest 040,041 failures you observed
> on NetBSD. If it doesn't, let me know.

I still see this one, which is different from the 040,041 stuff,
and where 'make check' is for some reason giving a lot less useful
detail. (This is a prexisting intermittent from before this patchset).



[etc]
▶ 175/704 /io/channel/pipe/sync
           OK
▶ 175/704 /io/channel/pipe/async
           OK
175/704 qemu:unit / test-io-channel-file
           OK              0.11s   5 subtests passed

177/704 qemu:unit / test-io-channel-tls
           RUNNING
>>> G_TEST_BUILDDIR=/home/qemu/qemu-test.yiYr4m/build/tests/unit MALLOC_PERTURB_=5 G_TEST_SRCDIR=/home/qemu/qemu-test.yiYr4m/src/tests/unit /home/
qemu/qemu-test.yiYr4m/build/tests/unit/test-io-channel-tls --tap -k
▶ 176/704 /io/channel/socket/ipv4-sync
           OK
▶ 176/704 /io/channel/socket/ipv4-async
           OK
▶ 176/704 /io/channel/socket/ipv4-fd
           OK
▶ 176/704 /io/channel/socket/ipv6-sync
           OK
▶ 176/704 /io/channel/socket/ipv6-async
           OK
▶ 176/704 /io/channel/socket/unix-sync
           OK
▶ 176/704 /io/channel/socket/unix-async
           OK
▶ 176/704 /io/channel/socket/unix-fd-pass
           OK
▶ 176/704 /io/channel/socket/unix-listen-cleanup
           OK
176/704 qemu:unit / test-io-channel-socket
           OK              0.13s   9 subtests passed

▶ 1/1 qcow2 qsd-jobs                OK
1/1 qemu:block / qemu-iotests qcow2 ERROR          243.14s   exit status 1

178/704 qemu:unit / test-io-task
           RUNNING
>>> G_TEST_BUILDDIR=/home/qemu/qemu-test.yiYr4m/build/tests/unit MALLOC_PERTURB_=194 G_TEST_SRCDIR=/home/qemu/qemu-test.yiYr4m/src/tests/unit /hom
e/qemu/qemu-test.yiYr4m/build/tests/unit/test-io-task --tap -k
▶ 147/704 /bdrv-drain/blockjob/iothread/error/drain_subtree
           OK

Summary of Failures:

1/1 qemu:block / qemu-iotests qcow2 ERROR          243.14s   exit status 1


Ok:                 0
Expected Fail:      0
Fail:               1
Unexpected Pass:    0
Skipped:            0
Timeout:            0

Full log written to /home/qemu/qemu-test.yiYr4m/build/meson-logs/iotestslog.txt
▶ 147/704 /bdrv-drain/deletion/drain
           OK
▶ 178/704 /crypto/task/complete
           OK
▶ 178/704 /crypto/task/datafree
           OK
[etc]

thanks
-- PMM

[PULL 0/4] Python patches

[John Snow]

The following changes since commit 47cc1a3655135b89fa75c2824fbddd29df874612:

  Merge remote-tracking branch 'remotes/kwolf-gitlab/tags/for-upstream' into staging (2022-02-01 19:48:15 +0000)

are available in the Git repository at:

  https://gitlab.com/jsnow/qemu.git tags/python-pull-request

for you to fetch changes up to b0b662bb2b340d63529672b5bdae596a6243c4d0:

  python/aqmp: add socket bind step to legacy.py (2022-02-02 14:12:22 -0500)

----------------------------------------------------------------
Python patches

Peter: I expect this to address the iotest 040,041 failures you observed
on NetBSD. If it doesn't, let me know.

----------------------------------------------------------------

John Snow (4):
  python/aqmp: Fix negotiation with pre-"oob" QEMU
  python/machine: raise VMLaunchFailure exception from launch()
  python: upgrade mypy to 0.780
  python/aqmp: add socket bind step to legacy.py

 python/Pipfile.lock                       | 66 +++++++++++++----------
 python/qemu/aqmp/legacy.py                |  3 ++
 python/qemu/aqmp/protocol.py              | 41 ++++++++++++--
 python/qemu/aqmp/qmp_client.py            |  4 +-
 python/qemu/machine/machine.py            | 45 +++++++++++++---
 python/setup.cfg                          |  2 +-
 tests/qemu-iotests/tests/mirror-top-perms |  3 +-
 7 files changed, 123 insertions(+), 41 deletions(-)

-- 
2.31.1