[Qemu-devel] [PULL 00/15] Python queue, 2018-10-30

[Qemu-devel] [PULL 00/15] Python queue, 2018-10-30

[Eduardo Habkost]

Sorry for submitting this at the last minute.

The following changes since commit a2e002ff7913ce93aa0f7dbedd2123dce5f1a9cd:

  Merge remote-tracking branch 'remotes/vivier2/tags/qemu-trivial-for-3.1-pull-request' into staging (2018-10-30 15:49:55 +0000)

are available in the Git repository at:

  git://github.com/ehabkost/qemu.git tags/python-next-pull-request

for you to fetch changes up to e301e65c97142c4d2a2adf35f5a4fa73d65d8a4f:

  scripts/qemu.py: use a more consistent docstring style (2018-10-30 21:13:54 -0300)

----------------------------------------------------------------
Python queue, 2018-10-30

* Makefile rule for running acceptance tests
  (make check-acceptance) (Cleber Rosa)
* Make iotests compatible with Python 3
  (Max Reitz)
* device-crash-test whitelist update (Thomas Huth)
* Misc cleanups (Cleber Rosa)

----------------------------------------------------------------

Cleber Rosa (5):
  Bootstrap Python venv for tests
  Acceptance tests: add make rule for running them
  Travis support for the acceptance tests
  scripts/decodetree.py: fix reference to attributes
  scripts/qemu.py: use a more consistent docstring style

Max Reitz (9):
  iotests: Make nbd-fault-injector flush
  iotests: Flush in iotests.py's QemuIoInteractive
  iotests: Use Python byte strings where appropriate
  iotests: Use // for Python integer division
  iotests: Different iterator behavior in Python 3
  iotests: Explicitly bequeath FDs in Python
  iotests: 'new' module replacement in 169
  iotests: Modify imports for Python 3
  iotests: Unify log outputs between Python 2 and 3

Thomas Huth (1):
  scripts/device-crash-test: Remove devices that are not user_creatable
    anymore

 docs/devel/testing.rst                   |  43 +-
 tests/requirements.txt                   |   4 +
 scripts/decodetree.py                    |   2 +-
 scripts/qemu.py                          |  99 +++--
 scripts/qtest.py                         |   2 +-
 .travis.yml                              |   8 +
 scripts/device-crash-test                |  17 -
 tests/Makefile.include                   |  43 ++
 tests/qemu-iotests/030                   |   2 +-
 tests/qemu-iotests/040                   |   4 +-
 tests/qemu-iotests/041                   |   4 +-
 tests/qemu-iotests/044                   |  24 +-
 tests/qemu-iotests/045                   |   2 +-
 tests/qemu-iotests/056                   |   2 +-
 tests/qemu-iotests/065                   |   4 +-
 tests/qemu-iotests/083.out               |   9 +
 tests/qemu-iotests/093                   |  18 +-
 tests/qemu-iotests/124                   |   4 +-
 tests/qemu-iotests/136                   |   2 +-
 tests/qemu-iotests/139                   |   2 +-
 tests/qemu-iotests/147                   |   2 +-
 tests/qemu-iotests/149                   |  14 +-
 tests/qemu-iotests/151                   |  12 +-
 tests/qemu-iotests/163                   |  13 +-
 tests/qemu-iotests/169                   |   3 +-
 tests/qemu-iotests/194.out               |  22 +-
 tests/qemu-iotests/202.out               |  12 +-
 tests/qemu-iotests/203.out               |  14 +-
 tests/qemu-iotests/206.out               | 218 +++++-----
 tests/qemu-iotests/207                   |   6 +-
 tests/qemu-iotests/207.out               |  72 ++--
 tests/qemu-iotests/208.out               |   8 +-
 tests/qemu-iotests/210.out               |  94 ++--
 tests/qemu-iotests/211.out               | 102 ++---
 tests/qemu-iotests/212.out               | 174 ++++----
 tests/qemu-iotests/213.out               | 182 ++++----
 tests/qemu-iotests/216.out               |   4 +-
 tests/qemu-iotests/218.out               |  20 +-
 tests/qemu-iotests/219.out               | 526 +++++++++++------------
 tests/qemu-iotests/222.out               |  24 +-
 tests/qemu-iotests/iotests.py            |  37 +-
 tests/qemu-iotests/nbd-fault-injector.py |  12 +-
 tests/qemu-iotests/qcow2.py              |  10 +-
 tests/qemu-iotests/qed.py                |   6 +-
 44 files changed, 1014 insertions(+), 868 deletions(-)
 create mode 100644 tests/requirements.txt

-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 01/15] scripts/device-crash-test: Remove devices that are not user_creatable anymore

[Eduardo Habkost]

From: Thomas Huth <thuth@redhat.com>

Devices that are derived from TYPE_SYS_BUS_DEVICE are not user_creatable
anymore by default, and some others have been marked as non-user_creatable
manually, so we can remove these devices from the "ignore"-list in the
device-crash-test script.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <1538729067-7944-1-git-send-email-thuth@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/device-crash-test | 17 -----------------
 1 file changed, 17 deletions(-)

diff --git a/scripts/device-crash-test b/scripts/device-crash-test
index 930200b034..e93a7c0c84 100755
--- a/scripts/device-crash-test
+++ b/scripts/device-crash-test
@@ -72,21 +72,6 @@ ERROR_WHITELIST = [
     # devices that don't work out of the box because they require extra options to "-device DEV":
     #            DEVICE                                    | ERROR MESSAGE
     {'device':'.*-(i386|x86_64)-cpu', 'expected':True},    # CPU socket-id is not set
-    {'device':'ARM,bitband-memory', 'expected':True},      # source-memory property not set
-    {'device':'arm.cortex-a9-global-timer', 'expected':True}, # a9_gtimer_realize: num-cpu must be between 1 and 4
-    {'device':'arm_mptimer', 'expected':True},             # num-cpu must be between 1 and 4
-    {'device':'armv7m', 'expected':True},                  # memory property was not set
-    {'device':'aspeed.scu', 'expected':True},              # Unknown silicon revision: 0x0
-    {'device':'aspeed.sdmc', 'expected':True},             # Unknown silicon revision: 0x0
-    {'device':'bcm2835-dma', 'expected':True},             # bcm2835_dma_realize: required dma-mr link not found: Property '.dma-mr' not found
-    {'device':'bcm2835-fb', 'expected':True},              # bcm2835_fb_realize: required vcram-base property not set
-    {'device':'bcm2835-mbox', 'expected':True},            # bcm2835_mbox_realize: required mbox-mr link not found: Property '.mbox-mr' not found
-    {'device':'bcm2835-peripherals', 'expected':True},     # bcm2835_peripherals_realize: required ram link not found: Property '.ram' not found
-    {'device':'bcm2835-property', 'expected':True},        # bcm2835_property_realize: required fb link not found: Property '.fb' not found
-    {'device':'bcm2835_gpio', 'expected':True},            # bcm2835_gpio_realize: required sdhci link not found: Property '.sdbus-sdhci' not found
-    {'device':'bcm2836', 'expected':True},                 # bcm2836_realize: required ram link not found: Property '.ram' not found
-    {'device':'cfi.pflash01', 'expected':True},            # attribute "sector-length" not specified or zero.
-    {'device':'cfi.pflash02', 'expected':True},            # attribute "sector-length" not specified or zero.
     {'device':'icp', 'expected':True},                     # icp_realize: required link 'xics' not found: Property '.xics' not found
     {'device':'ics', 'expected':True},                     # ics_base_realize: required link 'xics' not found: Property '.xics' not found
     # "-device ide-cd" does work on more recent QEMU versions, so it doesn't have expected=True
@@ -108,7 +93,6 @@ ERROR_WHITELIST = [
     {'device':'pc-dimm', 'expected':True},                 # 'memdev' property is not set
     {'device':'pci-bridge', 'expected':True},              # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
     {'device':'pci-bridge-seat', 'expected':True},         # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
-    {'device':'pxa2xx-dma', 'expected':True},              # channels value invalid
     {'device':'pxb', 'expected':True},                     # Bridge chassis not specified. Each bridge is required to be assigned a unique chassis id > 0.
     {'device':'scsi-block', 'expected':True},              # drive property not set
     {'device':'scsi-disk', 'expected':True},               # drive property not set
@@ -145,7 +129,6 @@ ERROR_WHITELIST = [
     {'device':'virtio-input-host-pci', 'expected':True},   # evdev property is required
     {'device':'xen-pvdevice', 'expected':True},            # Device ID invalid, it must always be supplied
     {'device':'vhost-vsock-ccw', 'expected':True},         # guest-cid property must be greater than 2
-    {'device':'ALTR.timer', 'expected':True},              # "clock-frequency" property must be provided
     {'device':'zpci', 'expected':True},                    # target must be defined
     {'device':'pnv-(occ|icp|lpc)', 'expected':True},       # required link 'xics' not found: Property '.xics' not found
     {'device':'powernv-cpu-.*', 'expected':True},          # pnv_core_realize: required link 'xics' not found: Property '.xics' not found
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 02/15] iotests: Make nbd-fault-injector flush

[Eduardo Habkost]

From: Max Reitz <mreitz@redhat.com>

When closing a connection, make the nbd-fault-injector flush the socket.
Without this, the output is a bit unreliable with Python 3.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20181022135307.14398-2-mreitz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/qemu-iotests/083.out               | 9 +++++++++
 tests/qemu-iotests/nbd-fault-injector.py | 1 +
 2 files changed, 10 insertions(+)

diff --git a/tests/qemu-iotests/083.out b/tests/qemu-iotests/083.out
index be6079d27e..f9af8bb691 100644
--- a/tests/qemu-iotests/083.out
+++ b/tests/qemu-iotests/083.out
@@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
 
 === Check disconnect after neg2 ===
 
+Unable to read from socket: Connection reset by peer
 Connection closed
 read failed: Input/output error
 
@@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
 
 === Check disconnect before request ===
 
+Unable to read from socket: Connection reset by peer
 Connection closed
 read failed: Input/output error
 
@@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/
 
 === Check disconnect after neg-classic ===
 
+Unable to read from socket: Connection reset by peer
 Connection closed
 read failed: Input/output error
 
@@ -161,6 +164,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
 
 === Check disconnect after neg2 ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
 read failed: Input/output error
 
 === Check disconnect 8 neg2 ===
@@ -173,6 +178,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
 
 === Check disconnect before request ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
 read failed: Input/output error
 
 === Check disconnect after request ===
@@ -234,6 +241,8 @@ can't open device nbd+unix:///?socket=TEST_DIR/nbd.sock
 
 === Check disconnect after neg-classic ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
 read failed: Input/output error
 
 *** done
diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
index f9193c0fae..439a090eb6 100755
--- a/tests/qemu-iotests/nbd-fault-injector.py
+++ b/tests/qemu-iotests/nbd-fault-injector.py
@@ -112,6 +112,7 @@ class FaultInjectionSocket(object):
             if rule.match(event, io):
                 if rule.when == 0 or bufsize is None:
                     print('Closing connection on rule match %s' % rule.name)
+                    self.sock.flush()
                     sys.exit(0)
                 if rule.when != -1:
                     return rule.when
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 03/15] iotests: Flush in iotests.py's QemuIoInteractive

[Eduardo Habkost]

From: Max Reitz <mreitz@redhat.com>

After issuing a command, flush the pipe.  This does not change anything
in Python 2, but it makes a difference in Python 3.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20181022135307.14398-3-mreitz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/qemu-iotests/iotests.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 4e67fbbe96..10f2d17419 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -178,6 +178,7 @@ class QemuIoInteractive:
         cmd = cmd.strip()
         assert cmd != 'q' and cmd != 'quit'
         self._p.stdin.write(cmd + '\n')
+        self._p.stdin.flush()
         return self._read_output()
 
 
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 04/15] iotests: Use Python byte strings where appropriate

[Eduardo Habkost]

From: Max Reitz <mreitz@redhat.com>

Since byte strings are no longer the default in Python 3, we have to
explicitly use them where we need to, which is mostly when working with
structures.  It also means that we need to open a file in binary mode
when we want to use structures.

On the other hand, we have to accomodate for the fact that some
functions (still) work with byte strings but we want to use unicode
strings (in Python 3 at least, and it does not matter in Python 2).
This includes base64 encoding, but it is most notable when working with
the subprocess module: Either we set universal_newlines to True so that
the default streams are opened in text mode (hence this parameter is
aliased as "text" as of 3.7), or, if that is not possible, we have to
decode the output to a normal string.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Message-Id: <20181022135307.14398-4-mreitz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qtest.py                         |  2 +-
 tests/qemu-iotests/044                   |  8 ++++----
 tests/qemu-iotests/149                   |  8 +++++---
 tests/qemu-iotests/207                   |  4 ++--
 tests/qemu-iotests/iotests.py            | 11 +++++++----
 tests/qemu-iotests/nbd-fault-injector.py |  4 ++--
 tests/qemu-iotests/qcow2.py              | 10 +++++-----
 7 files changed, 26 insertions(+), 21 deletions(-)

diff --git a/scripts/qtest.py b/scripts/qtest.py
index df0daf26ca..adf1fe3f26 100644
--- a/scripts/qtest.py
+++ b/scripts/qtest.py
@@ -64,7 +64,7 @@ class QEMUQtestProtocol(object):
 
         @param qtest_cmd: qtest command text to be sent
         """
-        self._sock.sendall(qtest_cmd + "\n")
+        self._sock.sendall((qtest_cmd + "\n").encode('utf-8'))
 
     def close(self):
         self._sock.close()
diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 11ea0f4d35..69e736f687 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -53,21 +53,21 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             fd.seek(off_reftable)
 
             for i in xrange(0, h.refcount_table_clusters):
-                sector = ''.join(struct.pack('>Q',
+                sector = b''.join(struct.pack('>Q',
                     off_refblock + i * 64 * 512 + j * 512)
                     for j in xrange(0, 64))
                 fd.write(sector)
 
             # Write the refcount blocks
             assert(fd.tell() == off_refblock)
-            sector = ''.join(struct.pack('>H', 1) for j in xrange(0, 64 * 256))
+            sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256))
             for block in xrange(0, h.refcount_table_clusters):
                 fd.write(sector)
 
             # Write the L1 table
             assert(fd.tell() == off_l1)
             assert(off_l2 + 512 * h.l1_size == off_data)
-            table = ''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
+            table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
                 for j in xrange(0, h.l1_size))
             fd.write(table)
 
@@ -85,7 +85,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
                 remaining = remaining - 1024 * 512
                 off = off + 1024 * 512
 
-            table = ''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
+            table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
                 for j in xrange(0, remaining / 512))
             fd.write(table)
 
diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
index 9e0cad76f9..1225334cb8 100755
--- a/tests/qemu-iotests/149
+++ b/tests/qemu-iotests/149
@@ -79,7 +79,7 @@ class LUKSConfig(object):
 
     def first_password_base64(self):
         (pw, slot) = self.first_password()
-        return base64.b64encode(pw)
+        return base64.b64encode(pw.encode('ascii')).decode('ascii')
 
     def active_slots(self):
         slots = []
@@ -98,7 +98,8 @@ def verify_passwordless_sudo():
     proc = subprocess.Popen(args,
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
 
     msg = proc.communicate()[0]
 
@@ -116,7 +117,8 @@ def cryptsetup(args, password=None):
     proc = subprocess.Popen(fullargs,
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
 
     msg = proc.communicate(password)[0]
 
diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
index 444ae233ae..2d86a3da37 100755
--- a/tests/qemu-iotests/207
+++ b/tests/qemu-iotests/207
@@ -109,7 +109,7 @@ with iotests.FilePath('t.img') as disk_path, \
     md5_key = subprocess.check_output(
         'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
         'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
-        shell=True).rstrip()
+        shell=True).rstrip().decode('ascii')
 
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
@@ -147,7 +147,7 @@ with iotests.FilePath('t.img') as disk_path, \
     sha1_key = subprocess.check_output(
         'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
         'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
-        shell=True).rstrip()
+        shell=True).rstrip().decode('ascii')
 
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 10f2d17419..7290c0b159 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -104,7 +104,8 @@ def qemu_img_pipe(*args):
     '''Run qemu-img and return its output'''
     subp = subprocess.Popen(qemu_img_args + list(args),
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
     exitcode = subp.wait()
     if exitcode < 0:
         sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
@@ -128,7 +129,8 @@ def qemu_io(*args):
     '''Run qemu-io and return the stdout data'''
     args = qemu_io_args + list(args)
     subp = subprocess.Popen(args, stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
     exitcode = subp.wait()
     if exitcode < 0:
         sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args)))
@@ -149,7 +151,8 @@ class QemuIoInteractive:
         self.args = qemu_io_args + list(args)
         self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE,
                                    stdout=subprocess.PIPE,
-                                   stderr=subprocess.STDOUT)
+                                   stderr=subprocess.STDOUT,
+                                   universal_newlines=True)
         assert self._p.stdout.read(9) == 'qemu-io> '
 
     def close(self):
@@ -193,7 +196,7 @@ def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
 
 def create_image(name, size):
     '''Create a fully-allocated raw image with sector markers'''
-    file = open(name, 'w')
+    file = open(name, 'wb')
     i = 0
     while i < size:
         sector = struct.pack('>l504xl', i / 512, i / 512)
diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
index 439a090eb6..d45e2e0a6a 100755
--- a/tests/qemu-iotests/nbd-fault-injector.py
+++ b/tests/qemu-iotests/nbd-fault-injector.py
@@ -86,7 +86,7 @@ def recvall(sock, bufsize):
             raise Exception('unexpected disconnect')
         chunks.append(chunk)
         received += len(chunk)
-    return ''.join(chunks)
+    return b''.join(chunks)
 
 class Rule(object):
     def __init__(self, name, event, io, when):
@@ -177,7 +177,7 @@ def handle_connection(conn, use_export):
         req = read_request(conn)
         if req.type == NBD_CMD_READ:
             write_reply(conn, 0, req.handle)
-            conn.send('\0' * req.len, event='data')
+            conn.send(b'\0' * req.len, event='data')
         elif req.type == NBD_CMD_WRITE:
             _ = conn.recv(req.len, event='data')
             write_reply(conn, 0, req.handle)
diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py
index b95a837759..b392972d1b 100755
--- a/tests/qemu-iotests/qcow2.py
+++ b/tests/qemu-iotests/qcow2.py
@@ -10,7 +10,7 @@ class QcowHeaderExtension:
     def __init__(self, magic, length, data):
         if length % 8 != 0:
             padding = 8 - (length % 8)
-            data += "\0" * padding
+            data += b"\0" * padding
 
         self.magic  = magic
         self.length = length
@@ -103,7 +103,7 @@ class QcowHeader:
 
         fd.seek(self.header_length)
         extensions = self.extensions
-        extensions.append(QcowHeaderExtension(0, 0, ""))
+        extensions.append(QcowHeaderExtension(0, 0, b""))
         for ex in extensions:
             buf = struct.pack('>II', ex.magic, ex.length)
             fd.write(buf)
@@ -137,8 +137,8 @@ class QcowHeader:
         for ex in self.extensions:
 
             data = ex.data[:ex.length]
-            if all(c in string.printable for c in data):
-                data = "'%s'" % data
+            if all(c in string.printable.encode('ascii') for c in data):
+                data = "'%s'" % data.decode('ascii')
             else:
                 data = "<binary>"
 
@@ -178,7 +178,7 @@ def cmd_add_header_ext(fd, magic, data):
         sys.exit(1)
 
     h = QcowHeader(fd)
-    h.extensions.append(QcowHeaderExtension.create(magic, data))
+    h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii')))
     h.update(fd)
 
 def cmd_add_header_ext_stdio(fd, magic):
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 05/15] iotests: Use // for Python integer division

[Eduardo Habkost]

From: Max Reitz <mreitz@redhat.com>

In Python 3, / is always a floating-point division.  We usually do not
want this, and as Python 2.7 understands // as well, change all integer
divisions to use that.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20181022135307.14398-5-mreitz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/qemu-iotests/030        |  2 +-
 tests/qemu-iotests/040        |  4 ++--
 tests/qemu-iotests/041        |  4 ++--
 tests/qemu-iotests/044        |  2 +-
 tests/qemu-iotests/093        | 18 +++++++++---------
 tests/qemu-iotests/136        |  2 +-
 tests/qemu-iotests/149        |  6 +++---
 tests/qemu-iotests/151        | 12 ++++++------
 tests/qemu-iotests/163        |  2 +-
 tests/qemu-iotests/iotests.py |  2 +-
 tests/qemu-iotests/qed.py     |  6 +++---
 11 files changed, 30 insertions(+), 30 deletions(-)

diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
index 1dbc2ddc49..276e06b5ba 100755
--- a/tests/qemu-iotests/030
+++ b/tests/qemu-iotests/030
@@ -521,7 +521,7 @@ new_state = "2"
 state = "2"
 event = "%s"
 new_state = "1"
-''' % (event, errno, self.STREAM_BUFFER_SIZE / 512, event, event))
+''' % (event, errno, self.STREAM_BUFFER_SIZE // 512, event, event))
         file.close()
 
 class TestEIO(TestErrors):
diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index 1cb1ceeb33..b81133a474 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase):
 
         self.assert_no_active_block_jobs()
         result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
-                             base=backing_img, speed=(self.image_len / 4))
+                             base=backing_img, speed=(self.image_len // 4))
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp('device_del', id='scsi0')
         self.assert_qmp(result, 'return', {})
@@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase):
 
         self.assert_no_active_block_jobs()
         result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
-                             base=backing_img, speed=(self.image_len / 4))
+                             base=backing_img, speed=(self.image_len // 4))
         self.assert_qmp(result, 'return', {})
 
         result = self.vm.qmp('query-block')
diff --git a/tests/qemu-iotests/041 b/tests/qemu-iotests/041
index 9336ab6ff5..3615011d98 100755
--- a/tests/qemu-iotests/041
+++ b/tests/qemu-iotests/041
@@ -404,7 +404,7 @@ new_state = "2"
 state = "2"
 event = "%s"
 new_state = "1"
-''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event))
+''' % (event, errno, self.MIRROR_GRANULARITY // 512, event, event))
         file.close()
 
     def setUp(self):
@@ -569,7 +569,7 @@ new_state = "2"
 state = "2"
 event = "%s"
 new_state = "1"
-''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event))
+''' % (event, errno, self.MIRROR_GRANULARITY // 512, event, event))
         file.close()
 
     def setUp(self):
diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 69e736f687..7ef5e46fe9 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
                 off = off + 1024 * 512
 
             table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
-                for j in xrange(0, remaining / 512))
+                for j in xrange(0, remaining // 512))
             fd.write(table)
 
 
diff --git a/tests/qemu-iotests/093 b/tests/qemu-iotests/093
index 9d1971a56c..d88fbc182e 100755
--- a/tests/qemu-iotests/093
+++ b/tests/qemu-iotests/093
@@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase):
         # in. The throttled requests won't be executed until we
         # advance the virtual clock.
         rq_size = 512
-        rd_nr = max(params['bps'] / rq_size / 2,
-                    params['bps_rd'] / rq_size,
-                    params['iops'] / 2,
+        rd_nr = max(params['bps'] // rq_size // 2,
+                    params['bps_rd'] // rq_size,
+                    params['iops'] // 2,
                     params['iops_rd'])
         rd_nr *= seconds * 2
-        rd_nr /= ndrives
-        wr_nr = max(params['bps'] / rq_size / 2,
-                    params['bps_wr'] / rq_size,
-                    params['iops'] / 2,
+        rd_nr //= ndrives
+        wr_nr = max(params['bps'] // rq_size // 2,
+                    params['bps_wr'] // rq_size,
+                    params['iops'] // 2,
                     params['iops_wr'])
         wr_nr *= seconds * 2
-        wr_nr /= ndrives
+        wr_nr //= ndrives
 
         # Send I/O requests to all drives
         for i in range(rd_nr):
@@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
             self.configure_throttle(ndrives, settings)
 
             # Wait for the bucket to empty so we can do bursts
-            wait_ns = nsec_per_sec * burst_length * burst_rate / rate
+            wait_ns = nsec_per_sec * burst_length * burst_rate // rate
             self.vm.qtest("clock_step %d" % wait_ns)
 
             # Test I/O at the max burst rate
diff --git a/tests/qemu-iotests/136 b/tests/qemu-iotests/136
index a154d8ef9d..af7ffa4540 100755
--- a/tests/qemu-iotests/136
+++ b/tests/qemu-iotests/136
@@ -24,7 +24,7 @@ import os
 
 interval_length = 10
 nsec_per_sec = 1000000000
-op_latency = nsec_per_sec / 1000 # See qtest_latency_ns in accounting.c
+op_latency = nsec_per_sec // 1000 # See qtest_latency_ns in accounting.c
 bad_sector = 8192
 bad_offset = bad_sector * 512
 blkdebug_file = os.path.join(iotests.test_dir, 'blkdebug.conf')
diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
index 1225334cb8..4f363f295f 100755
--- a/tests/qemu-iotests/149
+++ b/tests/qemu-iotests/149
@@ -314,13 +314,13 @@ def test_once(config, qemu_img=False):
     image_size = 4 * oneTB
     if qemu_img:
         iotests.log("# Create image")
-        qemu_img_create(config, image_size / oneMB)
+        qemu_img_create(config, image_size // oneMB)
     else:
         iotests.log("# Create image")
-        create_image(config, image_size / oneMB)
+        create_image(config, image_size // oneMB)
 
     lowOffsetMB = 100
-    highOffsetMB = 3 * oneTB / oneMB
+    highOffsetMB = 3 * oneTB // oneMB
 
     try:
         if not qemu_img:
diff --git a/tests/qemu-iotests/151 b/tests/qemu-iotests/151
index fe53b9f446..1bb74d67c4 100755
--- a/tests/qemu-iotests/151
+++ b/tests/qemu-iotests/151
@@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase):
                             'write -P 1 0 %i' % self.image_len);
 
         # Start some background requests
-        for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
+        for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset)
-        for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
+        for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         # Start the block job
@@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase):
         self.assert_qmp(result, 'return', {})
 
         # Start some more requests
-        for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
+        for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
-        for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
+        for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         # Wait for the READY event
@@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase):
         # the source) should be settled using the active mechanism.
         # The mirror code itself asserts that the source BDS's dirty
         # bitmap will stay clean between READY and COMPLETED.
-        for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
+        for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
-        for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
+        for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         if sync_source_and_target:
diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
index 403842354e..5fd424761b 100755
--- a/tests/qemu-iotests/163
+++ b/tests/qemu-iotests/163
@@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         entry_bits = 3
         entry_size = 1 << entry_bits
         l1_mask = 0x00fffffffffffe00
-        div_roundup = lambda n, d: (n + d - 1) / d
+        div_roundup = lambda n, d: (n + d - 1) // d
 
         def split_by_n(data, n):
             for x in xrange(0, len(data), n):
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 7290c0b159..7ca94e9278 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -199,7 +199,7 @@ def create_image(name, size):
     file = open(name, 'wb')
     i = 0
     while i < size:
-        sector = struct.pack('>l504xl', i / 512, i / 512)
+        sector = struct.pack('>l504xl', i // 512, i // 512)
         file.write(sector)
         i = i + 512
     file.close()
diff --git a/tests/qemu-iotests/qed.py b/tests/qemu-iotests/qed.py
index ea469b9c48..8adaaf46c4 100755
--- a/tests/qemu-iotests/qed.py
+++ b/tests/qemu-iotests/qed.py
@@ -80,7 +80,7 @@ class QED(object):
 
     def load_l1_table(self):
         self.l1_table = self.read_table(self.header['l1_table_offset'])
-        self.table_nelems = self.header['table_size'] * self.header['cluster_size'] / table_elem_size
+        self.table_nelems = self.header['table_size'] * self.header['cluster_size'] // table_elem_size
 
     def write_table(self, offset, table):
         s = ''.join(pack_table_elem(x) for x in table)
@@ -167,14 +167,14 @@ def cmd_zero_cluster(qed, pos, *args):
         n = int(args[0])
 
     for i in xrange(n):
-        l1_index = pos / qed.header['cluster_size'] / len(qed.l1_table)
+        l1_index = pos // qed.header['cluster_size'] // len(qed.l1_table)
         if qed.l1_table[l1_index] == 0:
             err('no l2 table allocated')
 
         l2_offset = qed.l1_table[l1_index]
         l2_table = qed.read_table(l2_offset)
 
-        l2_index = (pos / qed.header['cluster_size']) % len(qed.l1_table)
+        l2_index = (pos // qed.header['cluster_size']) % len(qed.l1_table)
         l2_table[l2_index] = 1 # zero the data cluster
         qed.write_table(l2_offset, l2_table)
         pos += qed.header['cluster_size']
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 06/15] iotests: Different iterator behavior in Python 3

[Eduardo Habkost]

From: Max Reitz <mreitz@redhat.com>

In Python 3, several functions now return iterators instead of lists.
This includes range(), items(), map(), and filter().  This means that if
we really want a list, we have to wrap those instances with list().  But
then again, the two instances where this is the case for map() and
filter(), there are shorter expressions which work without either
function.

On the other hand, sometimes we do just want an iterator, in which case
we have sometimes used xrange() and iteritems() which no longer exist in
Python 3.  Just change these calls to be range() and items(), works in
both Python 2 and 3, and is really what we want in 3 (which is what
matters).  But because it is so simple to do (and to find and remove
once we completely switch to Python 3), make range() be an alias for
xrange() in the two affected tests (044 and 163).

In one instance, we only wanted the first instance of the result of a
filter() call.  Instead of using next(filter()) which would work only in
Python 3, or list(filter())[0] which would work everywhere but is a bit
weird, this instance is changed to use a generator expression with a
next() wrapped around, which works both in 2.7 and 3.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20181022135307.14398-6-mreitz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/qemu-iotests/044 | 16 ++++++++++------
 tests/qemu-iotests/056 |  2 +-
 tests/qemu-iotests/065 |  4 ++--
 tests/qemu-iotests/124 |  4 ++--
 tests/qemu-iotests/139 |  2 +-
 tests/qemu-iotests/163 | 11 +++++++----
 6 files changed, 23 insertions(+), 16 deletions(-)

diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 7ef5e46fe9..9ec3dba734 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -26,6 +26,10 @@ import iotests
 from iotests import qemu_img, qemu_img_verbose, qemu_io
 import struct
 import subprocess
+import sys
+
+if sys.version_info.major == 2:
+    range = xrange
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
 
@@ -52,23 +56,23 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             # Write a refcount table
             fd.seek(off_reftable)
 
-            for i in xrange(0, h.refcount_table_clusters):
+            for i in range(0, h.refcount_table_clusters):
                 sector = b''.join(struct.pack('>Q',
                     off_refblock + i * 64 * 512 + j * 512)
-                    for j in xrange(0, 64))
+                    for j in range(0, 64))
                 fd.write(sector)
 
             # Write the refcount blocks
             assert(fd.tell() == off_refblock)
             sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256))
-            for block in xrange(0, h.refcount_table_clusters):
+            for block in range(0, h.refcount_table_clusters):
                 fd.write(sector)
 
             # Write the L1 table
             assert(fd.tell() == off_l1)
             assert(off_l2 + 512 * h.l1_size == off_data)
             table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
-                for j in xrange(0, h.l1_size))
+                for j in range(0, h.l1_size))
             fd.write(table)
 
             # Write the L2 tables
@@ -79,14 +83,14 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             off = off_data
             while remaining > 1024 * 512:
                 pytable = list((1 << 63) | off + 512 * j
-                    for j in xrange(0, 1024))
+                    for j in range(0, 1024))
                 table = struct.pack('>1024Q', *pytable)
                 fd.write(table)
                 remaining = remaining - 1024 * 512
                 off = off + 1024 * 512
 
             table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
-                for j in xrange(0, remaining // 512))
+                for j in range(0, remaining // 512))
             fd.write(table)
 
 
diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
index 223292175a..3df323984d 100755
--- a/tests/qemu-iotests/056
+++ b/tests/qemu-iotests/056
@@ -32,7 +32,7 @@ target_img = os.path.join(iotests.test_dir, 'target.img')
 def img_create(img, fmt=iotests.imgfmt, size='64M', **kwargs):
     fullname = os.path.join(iotests.test_dir, '%s.%s' % (img, fmt))
     optargs = []
-    for k,v in kwargs.iteritems():
+    for k,v in kwargs.items():
         optargs = optargs + ['-o', '%s=%s' % (k,v)]
     args = ['create', '-f', fmt] + optargs + [fullname, size]
     iotests.qemu_img(*args)
diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 72aa9707c7..8bac383ea7 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific):
                     :data.index('')]
         for field in data:
             self.assertTrue(re.match('^ {4}[^ ]', field) is not None)
-        data = map(lambda line: line.strip(), data)
+        data = [line.strip() for line in data]
         self.assertEqual(data, self.human_compare)
 
 class TestQMP(TestImageInfoSpecific):
@@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific):
 
     def test_qmp(self):
         result = self.vm.qmp('query-block')['return']
-        drive = filter(lambda drive: drive['device'] == 'drive0', result)[0]
+        drive = next(drive for drive in result if drive['device'] == 'drive0')
         data = drive['inserted']['image']['format-specific']
         self.assertEqual(data['type'], iotests.imgfmt)
         self.assertEqual(data['data'], self.compare)
diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
index 3ea4ac53f5..9f189e3b54 100755
--- a/tests/qemu-iotests/124
+++ b/tests/qemu-iotests/124
@@ -39,7 +39,7 @@ def try_remove(img):
 def transaction_action(action, **kwargs):
     return {
         'type': action,
-        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems())
+        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items())
     }
 
 
@@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase):
     def img_create(self, img, fmt=iotests.imgfmt, size='64M',
                    parent=None, parentFormat=None, **kwargs):
         optargs = []
-        for k,v in kwargs.iteritems():
+        for k,v in kwargs.items():
             optargs = optargs + ['-o', '%s=%s' % (k,v)]
         args = ['create', '-f', fmt] + optargs + [img, size]
         if parent:
diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
index cc7fe337f3..62402c1c35 100755
--- a/tests/qemu-iotests/139
+++ b/tests/qemu-iotests/139
@@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
     # Check whether a BlockDriverState exists
     def checkBlockDriverState(self, node, must_exist = True):
         result = self.vm.qmp('query-named-block-nodes')
-        nodes = filter(lambda x: x['node-name'] == node, result['return'])
+        nodes = [x for x in result['return'] if x['node-name'] == node]
         self.assertLessEqual(len(nodes), 1)
         self.assertEqual(must_exist, len(nodes) == 1)
 
diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
index 5fd424761b..158ba5d092 100755
--- a/tests/qemu-iotests/163
+++ b/tests/qemu-iotests/163
@@ -18,9 +18,12 @@
 # along with this program.  If not, see <http://www.gnu.org/licenses/>.
 #
 
-import os, random, iotests, struct, qcow2
+import os, random, iotests, struct, qcow2, sys
 from iotests import qemu_img, qemu_io, image_size
 
+if sys.version_info.major == 2:
+    range = xrange
+
 test_img = os.path.join(iotests.test_dir, 'test.img')
 check_img = os.path.join(iotests.test_dir, 'check.img')
 
@@ -41,7 +44,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         div_roundup = lambda n, d: (n + d - 1) // d
 
         def split_by_n(data, n):
-            for x in xrange(0, len(data), n):
+            for x in range(0, len(data), n):
                 yield struct.unpack('>Q', data[x:x + n])[0] & l1_mask
 
         def check_l1_table(h, l1_data):
@@ -135,8 +138,8 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         self.image_verify()
 
     def test_random_write(self):
-        offs_list = range(0, size_to_int(self.image_len),
-                          size_to_int(self.chunk_size))
+        offs_list = list(range(0, size_to_int(self.image_len),
+                               size_to_int(self.chunk_size)))
         random.shuffle(offs_list)
         for offs in offs_list:
             qemu_io('-c', 'write -P 0xff %d %s' % (offs, self.chunk_size),
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 07/15] iotests: Explicitly bequeath FDs in Python

[Eduardo Habkost]

From: Max Reitz <mreitz@redhat.com>

Python 3.4 introduced the inheritable attribute for FDs.  At the same
time, it changed the default so that all FDs are not inheritable by
default, that only inheritable FDs are inherited to subprocesses, and
only if close_fds is explicitly set to False.

Adhere to this by setting close_fds to False when working with
subprocesses that may want to inherit FDs, and by trying to
set_inheritable() on FDs that we do want to bequeath to them.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20181022135307.14398-7-mreitz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py        | 34 +++++++++++++++++++++++++++++-----
 tests/qemu-iotests/045 |  2 +-
 tests/qemu-iotests/147 |  2 +-
 3 files changed, 31 insertions(+), 7 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index fd4249f7a8..768611f1de 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -148,11 +148,19 @@ class QEMUMachine(object):
         if opts:
             options.append(opts)
 
+        # This did not exist before 3.4, but since then it is
+        # mandatory for our purpose
+        if hasattr(os, 'set_inheritable'):
+            os.set_inheritable(fd, True)
+
         self._args.append('-add-fd')
         self._args.append(','.join(options))
         return self
 
-    def send_fd_scm(self, fd_file_path):
+    # Exactly one of fd and file_path must be given.
+    # (If it is file_path, the helper will open that file and pass its
+    # own fd)
+    def send_fd_scm(self, fd=None, file_path=None):
         # In iotest.py, the qmp should always use unix socket.
         assert self._qmp.is_scm_available()
         if self._socket_scm_helper is None:
@@ -160,12 +168,27 @@ class QEMUMachine(object):
         if not os.path.exists(self._socket_scm_helper):
             raise QEMUMachineError("%s does not exist" %
                                    self._socket_scm_helper)
+
+        # This did not exist before 3.4, but since then it is
+        # mandatory for our purpose
+        if hasattr(os, 'set_inheritable'):
+            os.set_inheritable(self._qmp.get_sock_fd(), True)
+            if fd is not None:
+                os.set_inheritable(fd, True)
+
         fd_param = ["%s" % self._socket_scm_helper,
-                    "%d" % self._qmp.get_sock_fd(),
-                    "%s" % fd_file_path]
+                    "%d" % self._qmp.get_sock_fd()]
+
+        if file_path is not None:
+            assert fd is None
+            fd_param.append(file_path)
+        else:
+            assert fd is not None
+            fd_param.append(str(fd))
+
         devnull = open(os.path.devnull, 'rb')
         proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
-                                stderr=subprocess.STDOUT)
+                                stderr=subprocess.STDOUT, close_fds=False)
         output = proc.communicate()[0]
         if output:
             LOG.debug(output)
@@ -286,7 +309,8 @@ class QEMUMachine(object):
                                        stdin=devnull,
                                        stdout=self._qemu_log_file,
                                        stderr=subprocess.STDOUT,
-                                       shell=False)
+                                       shell=False,
+                                       close_fds=False)
         self._post_launch()
 
     def wait(self):
diff --git a/tests/qemu-iotests/045 b/tests/qemu-iotests/045
index 6be8fc4912..55a5d31ca8 100755
--- a/tests/qemu-iotests/045
+++ b/tests/qemu-iotests/045
@@ -140,7 +140,7 @@ class TestSCMFd(iotests.QMPTestCase):
         os.remove(image0)
 
     def _send_fd_by_SCM(self):
-        ret = self.vm.send_fd_scm(image0)
+        ret = self.vm.send_fd_scm(file_path=image0)
         self.assertEqual(ret, 0, 'Failed to send fd with UNIX SCM')
 
     def test_add_fd(self):
diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
index d2081df84b..05b374b7d3 100755
--- a/tests/qemu-iotests/147
+++ b/tests/qemu-iotests/147
@@ -229,7 +229,7 @@ class BuiltinNBD(NBDBlockdevAddBase):
         sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
         sockfd.connect(unix_socket)
 
-        result = self.vm.send_fd_scm(str(sockfd.fileno()))
+        result = self.vm.send_fd_scm(fd=sockfd.fileno())
         self.assertEqual(result, 0, 'Failed to send socket FD')
 
         result = self.vm.qmp('getfd', fdname='nbd-fifo')
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 08/15] iotests: 'new' module replacement in 169

[Eduardo Habkost]

From: Max Reitz <mreitz@redhat.com>

iotest 169 uses the 'new' module to add methods to a class.  This module
no longer exists in Python 3.  Instead, we can use a lambda.  Best of
all, this works in 2.7 just as well.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20181022135307.14398-8-mreitz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/qemu-iotests/169 | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
index 69850c4c67..527aebd0cb 100755
--- a/tests/qemu-iotests/169
+++ b/tests/qemu-iotests/169
@@ -23,7 +23,6 @@ import iotests
 import time
 import itertools
 import operator
-import new
 import re
 from iotests import qemu_img
 
@@ -204,7 +203,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
 
 def inject_test_case(klass, name, method, *args, **kwargs):
     mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + method + name, new.instancemethod(mc, None, klass))
+    setattr(klass, 'test_' + method + name, lambda self: mc(self))
 
 for cmb in list(itertools.product((True, False), repeat=4)):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 09/15] iotests: Modify imports for Python 3

[Eduardo Habkost]

From: Max Reitz <mreitz@redhat.com>

There are two imports that need to be modified when running the iotests
under Python 3: One is StringIO, which no longer exists; instead, the
StringIO class comes from the io module, so import it from there (and
use the BytesIO class for Python 2).  The other is the ConfigParser,
which has just been renamed to configparser.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20181022135307.14398-9-mreitz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/qemu-iotests/iotests.py            | 13 +++++++++----
 tests/qemu-iotests/nbd-fault-injector.py |  7 +++++--
 2 files changed, 14 insertions(+), 6 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 7ca94e9278..a0f35e4b68 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -29,6 +29,7 @@ import json
 import signal
 import logging
 import atexit
+import io
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'scripts'))
 import qtest
@@ -681,15 +682,19 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
     verify_platform(supported_oses)
     verify_cache_mode(supported_cache_modes)
 
-    # We need to filter out the time taken from the output so that qemu-iotest
-    # can reliably diff the results against master output.
-    import StringIO
     if debug:
         output = sys.stdout
         verbosity = 2
         sys.argv.remove('-d')
     else:
-        output = StringIO.StringIO()
+        # We need to filter out the time taken from the output so that
+        # qemu-iotest can reliably diff the results against master output.
+        if sys.version_info.major >= 3:
+            output = io.StringIO()
+        else:
+            # io.StringIO is for unicode strings, which is not what
+            # 2.x's test runner emits.
+            output = io.BytesIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
index d45e2e0a6a..6b2d659dee 100755
--- a/tests/qemu-iotests/nbd-fault-injector.py
+++ b/tests/qemu-iotests/nbd-fault-injector.py
@@ -48,7 +48,10 @@ import sys
 import socket
 import struct
 import collections
-import ConfigParser
+if sys.version_info.major >= 3:
+    import configparser
+else:
+    import ConfigParser as configparser
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 
@@ -225,7 +228,7 @@ def parse_config(config):
     return rules
 
 def load_rules(filename):
-    config = ConfigParser.RawConfigParser()
+    config = configparser.RawConfigParser()
     with open(filename, 'rt') as f:
         config.readfp(f, filename)
     return parse_config(config)
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 10/15] iotests: Unify log outputs between Python 2 and 3

[Eduardo Habkost]

From: Max Reitz <mreitz@redhat.com>

When dumping an object into the log, there are differences between
Python 2 and 3.  First, unicode strings are prefixed by 'u' in Python 2
(they are no longer in 3, because unicode strings are the default
there).  Second, the order of keys in dicts may differ.  Third,
especially long numbers are longs in Python 2 and thus get an 'L'
suffix, which does not happen in Python 3.

We can get around all of these differences by dumping objects (lists and
dicts) in a language-independent format, namely JSON.  The JSON
generator even allows emitting dicts with their keys sorted
alphabetically.

This changes the output of all tests that use these logging functions
(dict keys are ordered now, strings in dicts are now enclosed in double
quotes instead of single quotes, the 'L' suffix of large integers is
dropped, and "true" and "false" are now in lower case).
The quote change necessitates a small change to a filter used in test
207.

Suggested-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Reviewed-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20181022135307.14398-10-mreitz@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/qemu-iotests/194.out    |  22 +-
 tests/qemu-iotests/202.out    |  12 +-
 tests/qemu-iotests/203.out    |  14 +-
 tests/qemu-iotests/206.out    | 218 +++++++-------
 tests/qemu-iotests/207        |   2 +-
 tests/qemu-iotests/207.out    |  72 ++---
 tests/qemu-iotests/208.out    |   8 +-
 tests/qemu-iotests/210.out    |  94 +++---
 tests/qemu-iotests/211.out    | 102 +++----
 tests/qemu-iotests/212.out    | 174 +++++------
 tests/qemu-iotests/213.out    | 182 ++++++------
 tests/qemu-iotests/216.out    |   4 +-
 tests/qemu-iotests/218.out    |  20 +-
 tests/qemu-iotests/219.out    | 526 +++++++++++++++++-----------------
 tests/qemu-iotests/222.out    |  24 +-
 tests/qemu-iotests/iotests.py |  10 +-
 16 files changed, 744 insertions(+), 740 deletions(-)

diff --git a/tests/qemu-iotests/194.out b/tests/qemu-iotests/194.out
index 50ac50da5e..71857853fb 100644
--- a/tests/qemu-iotests/194.out
+++ b/tests/qemu-iotests/194.out
@@ -1,18 +1,18 @@
 Launching VMs...
 Launching NBD server on destination...
-{u'return': {}}
-{u'return': {}}
+{"return": {}}
+{"return": {}}
 Starting `drive-mirror` on source...
-{u'return': {}}
+{"return": {}}
 Waiting for `drive-mirror` to complete...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_READY'}
+{"data": {"device": "mirror-job0", "len": 1073741824, "offset": 1073741824, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 Starting migration...
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'}
+{"return": {}}
+{"data": {"status": "setup"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"status": "active"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"status": "completed"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 Gracefully ending the `drive-mirror` job on source...
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_COMPLETED'}
+{"return": {}}
+{"data": {"device": "mirror-job0", "len": 1073741824, "offset": 1073741824, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 Stopping the NBD server on destination...
-{u'return': {}}
+{"return": {}}
diff --git a/tests/qemu-iotests/202.out b/tests/qemu-iotests/202.out
index d5ea374e17..9a8619e796 100644
--- a/tests/qemu-iotests/202.out
+++ b/tests/qemu-iotests/202.out
@@ -1,11 +1,11 @@
 Launching VM...
 Adding IOThread...
-{u'return': {}}
+{"return": {}}
 Adding blockdevs...
-{u'return': {}}
-{u'return': {}}
+{"return": {}}
+{"return": {}}
 Setting iothread...
-{u'return': {}}
-{u'return': {}}
+{"return": {}}
+{"return": {}}
 Creating external snapshots...
-{u'return': {}}
+{"return": {}}
diff --git a/tests/qemu-iotests/203.out b/tests/qemu-iotests/203.out
index 1a11f0975c..9d4abba8c5 100644
--- a/tests/qemu-iotests/203.out
+++ b/tests/qemu-iotests/203.out
@@ -1,11 +1,11 @@
 Launching VM...
 Setting IOThreads...
-{u'return': {}}
-{u'return': {}}
+{"return": {}}
+{"return": {}}
 Enabling migration QMP events...
-{u'return': {}}
+{"return": {}}
 Starting migration...
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'}
+{"return": {}}
+{"data": {"status": "setup"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"status": "active"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"status": "completed"}, "event": "MIGRATION", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
diff --git a/tests/qemu-iotests/206.out b/tests/qemu-iotests/206.out
index 789eebe57b..91f4db55d3 100644
--- a/tests/qemu-iotests/206.out
+++ b/tests/qemu-iotests/206.out
@@ -1,16 +1,16 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
-
-{'execute': 'blockdev-add', 'arguments': {'node_name': 'imgfile', 'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}}
-{u'return': {}}
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'imgfile', 'size': 134217728}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "node_name": "imgfile"}}
+{"return": {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "imgfile", "size": 134217728}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -24,15 +24,15 @@ Format specific information:
 
 === Successful image creation (inline blockdev-add, explicit defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'nocow': False, 'preallocation': 'off', 'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "nocow": false, "preallocation": "off", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 65536, 'refcount-bits': 16, 'version': 'v3', 'preallocation': 'off', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'lazy-refcounts': False, 'driver': 'qcow2', 'size': 67108864}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 65536, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "lazy-refcounts": false, "preallocation": "off", "refcount-bits": 16, "size": 67108864, "version": "v3"}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -46,15 +46,15 @@ Format specific information:
 
 === Successful image creation (v3 non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'nocow': True, 'preallocation': 'falloc', 'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "nocow": true, "preallocation": "falloc", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 2097152, 'refcount-bits': 1, 'version': 'v3', 'preallocation': 'metadata', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'lazy-refcounts': True, 'driver': 'qcow2', 'size': 33554432}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 2097152, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "lazy-refcounts": true, "preallocation": "metadata", "refcount-bits": 1, "size": 33554432, "version": "v3"}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -68,15 +68,15 @@ Format specific information:
 
 === Successful image creation (v2 non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 512, 'backing-fmt': 'qcow2', 'driver': 'qcow2', 'version': 'v2', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'backing-file': 'TEST_DIR/PID-t.qcow2.base', 'size': 33554432}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"backing-file": "TEST_DIR/PID-t.qcow2.base", "backing-fmt": "qcow2", "cluster-size": 512, "driver": "qcow2", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "size": 33554432, "version": "v2"}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -90,10 +90,10 @@ Format specific information:
 
 === Successful image creation (encrypted) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'encrypt': {'key-secret': 'keysec0', 'iter-time': 10, 'cipher-mode': 'ctr', 'ivgen-hash-alg': 'md5', 'cipher-alg': 'twofish-128', 'format': 'luks', 'ivgen-alg': 'plain64', 'hash-alg': 'sha1'}, 'driver': 'qcow2', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'size': 33554432}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "encrypt": {"cipher-alg": "twofish-128", "cipher-mode": "ctr", "format": "luks", "hash-alg": "sha1", "iter-time": 10, "ivgen-alg": "plain64", "ivgen-hash-alg": "md5", "key-secret": "keysec0"}, "file": {"driver": "file", "filename": "TEST_DIR/PID-t.qcow2"}, "size": 33554432}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -144,113 +144,113 @@ Format specific information:
 
 === Invalid BlockdevRef ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': "this doesn't exist", 'size': 33554432}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "this doesn't exist", "size": 33554432}}}
+{"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Invalid sizes ===
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 1234}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 1234}}}
+{"return": {}}
 Job failed: Image size must be a multiple of 512 bytes
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 18446744073709551104L}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 18446744073709551104}}}
+{"return": {}}
 Job failed: Could not resize image: Image size cannot be negative
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 9223372036854775808L}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 9223372036854775808}}}
+{"return": {}}
 Job failed: Could not resize image: Image size cannot be negative
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 9223372036854775296}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 9223372036854775296}}}
+{"return": {}}
 Job failed: Could not resize image: Failed to grow the L1 table: File too large
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Invalid version ===
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'version': 'v1', 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'error': {u'class': u'GenericError', u'desc': u"Invalid parameter 'v1'"}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "size": 67108864, "version": "v1"}}}
+{"error": {"class": "GenericError", "desc": "Invalid parameter 'v1'"}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'lazy-refcounts': True, 'version': 'v2', 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "lazy-refcounts": true, "size": 67108864, "version": "v2"}}}
+{"return": {}}
 Job failed: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater)
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'refcount-bits': 8, 'version': 'v2', 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 8, "size": 67108864, "version": "v2"}}}
+{"return": {}}
 Job failed: Different refcount widths than 16 bits require compatibility level 1.1 or above (use version=v3 or greater)
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Invalid backing file options ===
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'preallocation': 'full', 'driver': 'qcow2', 'backing-file': '/dev/null', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"backing-file": "/dev/null", "driver": "qcow2", "file": "node0", "preallocation": "full", "size": 67108864}}}
+{"return": {}}
 Job failed: Backing file and preallocation cannot be used at the same time
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'backing-fmt': 'qcow2', 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"backing-fmt": "qcow2", "driver": "qcow2", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Backing format cannot be used without backing file
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Invalid cluster size ===
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 1234, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 1234, "driver": "qcow2", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 128, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 128, "driver": "qcow2", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 4194304, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 4194304, "driver": "qcow2", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 0, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 0, "driver": "qcow2", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 512, 'driver': 'qcow2', 'file': 'node0', 'size': 281474976710656}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 512, "driver": "qcow2", "file": "node0", "size": 281474976710656}}}
+{"return": {}}
 Job failed: Could not resize image: Failed to grow the L1 table: File too large
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Invalid refcount width ===
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'refcount-bits': 128, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 128, "size": 67108864}}}
+{"return": {}}
 Job failed: Refcount width must be a power of two and may not exceed 64 bits
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'refcount-bits': 0, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 0, "size": 67108864}}}
+{"return": {}}
 Job failed: Refcount width must be a power of two and may not exceed 64 bits
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'refcount-bits': 7, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "qcow2", "file": "node0", "refcount-bits": 7, "size": 67108864}}}
+{"return": {}}
 Job failed: Refcount width must be a power of two and may not exceed 64 bits
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
index 2d86a3da37..c617ee7453 100755
--- a/tests/qemu-iotests/207
+++ b/tests/qemu-iotests/207
@@ -28,7 +28,7 @@ iotests.verify_image_format(supported_fmts=['raw'])
 iotests.verify_protocol(supported=['ssh'])
 
 def filter_hash(msg):
-    return re.sub("'hash': '[0-9a-f]+'", "'hash': HASH", msg)
+    return re.sub('"hash": "[0-9a-f]+"', '"hash": HASH', msg)
 
 def blockdev_create(vm, options):
     result = vm.qmp_log('blockdev-create', job_id='job0', options=options,
diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out
index 078b7e63cb..45ac7c2a8f 100644
--- a/tests/qemu-iotests/207.out
+++ b/tests/qemu-iotests/207.out
@@ -1,9 +1,9 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_IMG"}}
 file format: IMGFMT
@@ -16,49 +16,49 @@ virtual size: 4.0M (4194304 bytes)
 
 === Test host-key-check options ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'mode': 'none'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 8388608}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 8388608}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_IMG"}}
 file format: IMGFMT
 virtual size: 8.0M (8388608 bytes)
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'mode': 'known_hosts'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "known_hosts"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_IMG"}}
 file format: IMGFMT
 virtual size: 4.0M (4194304 bytes)
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'hash': 'wrong', 'type': 'md5', 'mode': 'hash'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 2097152}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "wrong", "mode": "hash", "type": "md5"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 2097152}}}
+{"return": {}}
 Job failed: remote host key does not match host_key_check 'wrong'
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'hash': HASH, 'type': 'md5', 'mode': 'hash'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 8388608}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": HASH, "mode": "hash", "type": "md5"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 8388608}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_IMG"}}
 file format: IMGFMT
 virtual size: 8.0M (8388608 bytes)
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'hash': 'wrong', 'type': 'sha1', 'mode': 'hash'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 2097152}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": "wrong", "mode": "hash", "type": "sha1"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 2097152}}}
+{"return": {}}
 Job failed: remote host key does not match host_key_check 'wrong'
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'hash': HASH, 'type': 'sha1', 'mode': 'hash'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"hash": HASH, "mode": "hash", "type": "sha1"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_IMG"}}
 file format: IMGFMT
@@ -66,15 +66,15 @@ virtual size: 4.0M (4194304 bytes)
 
 === Invalid path and user ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': '/this/is/not/an/existing/path', 'host-key-check': {'mode': 'none'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "/this/is/not/an/existing/path", "server": {"host": "127.0.0.1", "port": "22"}}, "size": 4194304}}}
+{"return": {}}
 Job failed: failed to open remote file '/this/is/not/an/existing/path': Failed opening remote file (libssh2 error code: -31)
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'mode': 'none'}, 'user': 'invalid user', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "ssh", "location": {"host-key-check": {"mode": "none"}, "path": "TEST_DIR/PID-t.img", "server": {"host": "127.0.0.1", "port": "22"}, "user": "invalid user"}, "size": 4194304}}}
+{"return": {}}
 Job failed: failed to authenticate using publickey authentication and the identities held by your ssh-agent
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
diff --git a/tests/qemu-iotests/208.out b/tests/qemu-iotests/208.out
index 3687e9d0dd..9ff2582a42 100644
--- a/tests/qemu-iotests/208.out
+++ b/tests/qemu-iotests/208.out
@@ -1,9 +1,9 @@
 Launching VM...
 Starting NBD server...
-{u'return': {}}
+{"return": {}}
 Adding NBD export...
-{u'return': {}}
+{"return": {}}
 Creating external snapshot...
-{u'return': {}}
+{"return": {}}
 Stopping NBD server...
-{u'return': {}}
+{"return": {}}
diff --git a/tests/qemu-iotests/210.out b/tests/qemu-iotests/210.out
index 078ba544a1..923cb05117 100644
--- a/tests/qemu-iotests/210.out
+++ b/tests/qemu-iotests/210.out
@@ -1,16 +1,16 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
-
-{'execute': 'blockdev-add', 'arguments': {'node_name': 'imgfile', 'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks'}}
-{u'return': {}}
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'iter-time': 10, 'driver': 'luks', 'file': 'imgfile', 'size': 134217728}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.luks", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.luks", "node_name": "imgfile"}}
+{"return": {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "imgfile", "iter-time": 10, "key-secret": "keysec0", "size": 134217728}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_IMG"}, "key-secret": "keysec0"}
 file format: IMGFMT
@@ -54,15 +54,15 @@ Format specific information:
 
 === Successful image creation (with non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.luks", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'hash-alg': 'sha1', 'cipher-mode': 'ctr', 'cipher-alg': 'twofish-128', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks'}, 'iter-time': 10, 'ivgen-alg': 'plain64', 'ivgen-hash-alg': 'md5', 'driver': 'luks', 'size': 67108864}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cipher-alg": "twofish-128", "cipher-mode": "ctr", "driver": "luks", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.luks"}, "hash-alg": "sha1", "iter-time": 10, "ivgen-alg": "plain64", "ivgen-hash-alg": "md5", "key-secret": "keysec0", "size": 67108864}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_IMG"}, "key-secret": "keysec0"}
 file format: IMGFMT
@@ -106,18 +106,18 @@ Format specific information:
 
 === Invalid BlockdevRef ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'luks', 'file': "this doesn't exist", 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "this doesn't exist", "size": 67108864}}}
+{"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Zero size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'iter-time': 10, 'driver': 'luks', 'file': 'node0', 'size': 0}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "node0", "iter-time": 10, "key-secret": "keysec0", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_IMG"}, "key-secret": "keysec0"}
 file format: IMGFMT
@@ -161,34 +161,34 @@ Format specific information:
 
 === Invalid sizes ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'driver': 'luks', 'file': 'node0', 'size': 18446744073709551104L}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "node0", "key-secret": "keysec0", "size": 18446744073709551104}}}
+{"return": {}}
 Job failed: The requested file size is too large
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'driver': 'luks', 'file': 'node0', 'size': 9223372036854775808L}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "node0", "key-secret": "keysec0", "size": 9223372036854775808}}}
+{"return": {}}
 Job failed: The requested file size is too large
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'driver': 'luks', 'file': 'node0', 'size': 9223372036854775296}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "luks", "file": "node0", "key-secret": "keysec0", "size": 9223372036854775296}}}
+{"return": {}}
 Job failed: The requested file size is too large
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Resize image with invalid sizes ===
 
-{'execute': 'block_resize', 'arguments': {'size': 9223372036854775296, 'node_name': 'node1'}}
-{u'error': {u'class': u'GenericError', u'desc': u'The requested file size is too large'}}
-{'execute': 'block_resize', 'arguments': {'size': 9223372036854775808L, 'node_name': 'node1'}}
-{u'error': {u'class': u'GenericError', u'desc': u"Invalid parameter type for 'size', expected: integer"}}
-{'execute': 'block_resize', 'arguments': {'size': 18446744073709551104L, 'node_name': 'node1'}}
-{u'error': {u'class': u'GenericError', u'desc': u"Invalid parameter type for 'size', expected: integer"}}
-{'execute': 'block_resize', 'arguments': {'size': -9223372036854775808, 'node_name': 'node1'}}
-{u'error': {u'class': u'GenericError', u'desc': u"Parameter 'size' expects a >0 size"}}
+{"execute": "block_resize", "arguments": {"node_name": "node1", "size": 9223372036854775296}}
+{"error": {"class": "GenericError", "desc": "The requested file size is too large"}}
+{"execute": "block_resize", "arguments": {"node_name": "node1", "size": 9223372036854775808}}
+{"error": {"class": "GenericError", "desc": "Invalid parameter type for 'size', expected: integer"}}
+{"execute": "block_resize", "arguments": {"node_name": "node1", "size": 18446744073709551104}}
+{"error": {"class": "GenericError", "desc": "Invalid parameter type for 'size', expected: integer"}}
+{"execute": "block_resize", "arguments": {"node_name": "node1", "size": -9223372036854775808}}
+{"error": {"class": "GenericError", "desc": "Parameter 'size' expects a >0 size"}}
 image: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_IMG"}, "key-secret": "keysec0"}
 file format: IMGFMT
 virtual size: 0 (0 bytes)
diff --git a/tests/qemu-iotests/211.out b/tests/qemu-iotests/211.out
index 6feaea3978..eebb0ea086 100644
--- a/tests/qemu-iotests/211.out
+++ b/tests/qemu-iotests/211.out
@@ -1,16 +1,16 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
-
-{'execute': 'blockdev-add', 'arguments': {'node_name': 'imgfile', 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}}
-{u'return': {}}
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'imgfile', 'size': 134217728}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "node_name": "imgfile"}}
+{"return": {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "imgfile", "size": 134217728}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -21,15 +21,15 @@ cluster_size: 1048576
 
 === Successful image creation (explicit defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'preallocation': 'off', 'driver': 'vdi', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}, 'size': 67108864}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi"}, "preallocation": "off", "size": 67108864}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -40,15 +40,15 @@ cluster_size: 1048576
 
 === Successful image creation (with non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'preallocation': 'metadata', 'driver': 'vdi', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}, 'size': 33554432}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vdi"}, "preallocation": "metadata", "size": 33554432}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -60,18 +60,18 @@ cluster_size: 1048576
 
 === Invalid BlockdevRef ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': "this doesn't exist", 'size': 33554432}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "this doesn't exist", "size": 33554432}}}
+{"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Zero size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 0}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -80,10 +80,10 @@ cluster_size: 1048576
 
 === Maximum size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 562949819203584}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 562949819203584}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -92,21 +92,21 @@ cluster_size: 1048576
 
 === Invalid sizes ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 18446744073709551104L}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 18446744073709551104}}}
+{"return": {}}
 Job failed: Unsupported VDI image size (size is 0xfffffffffffffe00, max supported is 0x1fffff8000000)
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 9223372036854775808L}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 9223372036854775808}}}
+{"return": {}}
 Job failed: Unsupported VDI image size (size is 0x8000000000000000, max supported is 0x1fffff8000000)
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 562949819203585}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vdi", "file": "node0", "size": 562949819203585}}}
+{"return": {}}
 Job failed: Unsupported VDI image size (size is 0x1fffff8000001, max supported is 0x1fffff8000000)
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
diff --git a/tests/qemu-iotests/212.out b/tests/qemu-iotests/212.out
index 9150da7a2c..01da467282 100644
--- a/tests/qemu-iotests/212.out
+++ b/tests/qemu-iotests/212.out
@@ -1,16 +1,16 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
-
-{'execute': 'blockdev-add', 'arguments': {'node_name': 'imgfile', 'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}}
-{u'return': {}}
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'imgfile', 'size': 134217728}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "node_name": "imgfile"}}
+{"return": {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "imgfile", "size": 134217728}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -18,15 +18,15 @@ virtual size: 128M (134217728 bytes)
 
 === Successful image creation (explicit defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 1048576, 'driver': 'parallels', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}, 'size': 67108864}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 1048576, "driver": "parallels", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels"}, "size": 67108864}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -34,15 +34,15 @@ virtual size: 64M (67108864 bytes)
 
 === Successful image creation (with non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 65536, 'driver': 'parallels', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}, 'size': 33554432}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 65536, "driver": "parallels", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.parallels"}, "size": 33554432}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -50,18 +50,18 @@ virtual size: 32M (33554432 bytes)
 
 === Invalid BlockdevRef ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': "this doesn't exist", 'size': 33554432}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "this doesn't exist", "size": 33554432}}}
+{"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Zero size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 0}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -69,10 +69,10 @@ virtual size: 0 (0 bytes)
 
 === Maximum size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 4503599627369984}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 4503599627369984}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -80,77 +80,77 @@ virtual size: 4096T (4503599627369984 bytes)
 
 === Invalid sizes ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 1234}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 1234}}}
+{"return": {}}
 Job failed: Image size must be a multiple of 512 bytes
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 18446744073709551104L}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 18446744073709551104}}}
+{"return": {}}
 Job failed: Image size is too large for this cluster size
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 9223372036854775808L}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 9223372036854775808}}}
+{"return": {}}
 Job failed: Image size is too large for this cluster size
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 9223372036854775296}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 9223372036854775296}}}
+{"return": {}}
 Job failed: Image size is too large for this cluster size
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 4503599627370497}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "parallels", "file": "node0", "size": 4503599627370497}}}
+{"return": {}}
 Job failed: Image size is too large for this cluster size
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Invalid cluster size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 1234, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 1234, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Cluster size must be a multiple of 512 bytes
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 128, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 128, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Cluster size must be a multiple of 512 bytes
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 4294967296, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 4294967296, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Cluster size is too large
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 9223372036854775808L, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 9223372036854775808, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Cluster size is too large
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 18446744073709551104L, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 18446744073709551104, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Cluster size is too large
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 0, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 0, "driver": "parallels", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Image size is too large for this cluster size
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 512, 'driver': 'parallels', 'file': 'node0', 'size': 281474976710656}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"cluster-size": 512, "driver": "parallels", "file": "node0", "size": 281474976710656}}}
+{"return": {}}
 Job failed: Image size is too large for this cluster size
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
diff --git a/tests/qemu-iotests/213.out b/tests/qemu-iotests/213.out
index e1dcd47201..0c9d65b2fe 100644
--- a/tests/qemu-iotests/213.out
+++ b/tests/qemu-iotests/213.out
@@ -1,16 +1,16 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
-
-{'execute': 'blockdev-add', 'arguments': {'node_name': 'imgfile', 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}}
-{u'return': {}}
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'imgfile', 'size': 134217728}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
+
+{"execute": "blockdev-add", "arguments": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "node_name": "imgfile"}}
+{"return": {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "imgfile", "size": 134217728}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -19,15 +19,15 @@ cluster_size: 8388608
 
 === Successful image creation (explicit defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 8388608, 'driver': 'vhdx', 'subformat': 'dynamic', 'log-size': 1048576, 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}, 'block-state-zero': True, 'size': 67108864}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 8388608, "block-state-zero": true, "driver": "vhdx", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx"}, "log-size": 1048576, "size": 67108864, "subformat": "dynamic"}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -36,15 +36,15 @@ cluster_size: 8388608
 
 === Successful image creation (with non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 268435456, 'driver': 'vhdx', 'subformat': 'fixed', 'log-size': 8388608, 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}, 'block-state-zero': False, 'size': 33554432}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 268435456, "block-state-zero": false, "driver": "vhdx", "file": {"driver": "file", "filename": "TEST_DIR/PID-t.vhdx"}, "log-size": 8388608, "size": 33554432, "subformat": "fixed"}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -53,18 +53,18 @@ cluster_size: 268435456
 
 === Invalid BlockdevRef ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': "this doesn't exist", 'size': 33554432}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "this doesn't exist", "size": 33554432}}}
+{"return": {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Zero size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 0}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 0}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -73,10 +73,10 @@ cluster_size: 8388608
 
 === Maximum size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 70368744177664}}}
-{u'return': {}}
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 70368744177664}}}
+{"return": {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -85,85 +85,85 @@ cluster_size: 67108864
 
 === Invalid sizes ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 18446744073709551104L}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 18446744073709551104}}}
+{"return": {}}
 Job failed: Image size too large; max of 64TB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 9223372036854775808L}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 9223372036854775808}}}
+{"return": {}}
 Job failed: Image size too large; max of 64TB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 9223372036854775296}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 9223372036854775296}}}
+{"return": {}}
 Job failed: Image size too large; max of 64TB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 70368744177665}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "size": 70368744177665}}}
+{"return": {}}
 Job failed: Image size too large; max of 64TB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Invalid block size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'block-size': 1234567, 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 1234567, "driver": "vhdx", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Block size must be a multiple of 1 MB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'block-size': 128, 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 128, "driver": "vhdx", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Block size must be a multiple of 1 MB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'block-size': 3145728, 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 3145728, "driver": "vhdx", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Block size must be a power of two
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'block-size': 536870912, 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 536870912, "driver": "vhdx", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Block size must not exceed 268435456
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'block-size': 0, 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"block-size": 0, "driver": "vhdx", "file": "node0", "size": 67108864}}}
+{"return": {}}
 Job failed: Block size must be a multiple of 1 MB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
 === Invalid log size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'log-size': 1234567, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 1234567, "size": 67108864}}}
+{"return": {}}
 Job failed: Log size must be a multiple of 1 MB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'log-size': 128, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 128, "size": 67108864}}}
+{"return": {}}
 Job failed: Log size must be a multiple of 1 MB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'log-size': 4294967296, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 4294967296, "size": 67108864}}}
+{"return": {}}
 Job failed: Log size must be smaller than 4 GB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'log-size': 0, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{"execute": "blockdev-create", "arguments": {"job_id": "job0", "options": {"driver": "vhdx", "file": "node0", "log-size": 0, "size": 67108864}}}
+{"return": {}}
 Job failed: Log size must be a multiple of 1 MB
-{'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{"execute": "job-dismiss", "arguments": {"id": "job0"}}
+{"return": {}}
 
diff --git a/tests/qemu-iotests/216.out b/tests/qemu-iotests/216.out
index 45ea857ee1..a70aa5cdae 100644
--- a/tests/qemu-iotests/216.out
+++ b/tests/qemu-iotests/216.out
@@ -7,8 +7,8 @@ Done
 
 --- Doing COR ---
 
-{u'return': {}}
-{u'return': u''}
+{"return": {}}
+{"return": ""}
 
 --- Checking COR result ---
 
diff --git a/tests/qemu-iotests/218.out b/tests/qemu-iotests/218.out
index 7dbf78e682..825a657081 100644
--- a/tests/qemu-iotests/218.out
+++ b/tests/qemu-iotests/218.out
@@ -4,27 +4,27 @@
 --- force=false ---
 
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 65536, u'len': 1048576, u'offset': 65536}, u'event': u'BLOCK_JOB_CANCELLED'}
+{"return": {}}
+{"data": {"device": "mirror", "len": 1048576, "offset": 65536, "speed": 65536, "type": "mirror"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 
 --- force=true ---
 
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 65536, u'len': 1048576, u'offset': 65536}, u'event': u'BLOCK_JOB_CANCELLED'}
+{"return": {}}
+{"data": {"device": "mirror", "len": 1048576, "offset": 65536, "speed": 65536, "type": "mirror"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 
 === Cancel mirror job after convergence ===
 
 --- force=false ---
 
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_READY'}
+{"data": {"device": "mirror", "len": 1048576, "offset": 1048576, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_COMPLETED'}
+{"return": {}}
+{"data": {"device": "mirror", "len": 1048576, "offset": 1048576, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_COMPLETED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 
 --- force=true ---
 
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_READY'}
+{"data": {"device": "mirror", "len": 1048576, "offset": 1048576, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_READY", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_CANCELLED'}
+{"return": {}}
+{"data": {"device": "mirror", "len": 1048576, "offset": 1048576, "speed": 0, "type": "mirror"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
diff --git a/tests/qemu-iotests/219.out b/tests/qemu-iotests/219.out
index 6dc07bc41e..8ebd3fee60 100644
--- a/tests/qemu-iotests/219.out
+++ b/tests/qemu-iotests/219.out
@@ -2,326 +2,326 @@ Launching VM...
 
 
 Starting block job: drive-mirror (auto-finalize: True; auto-dismiss: True)
-{u'return': {}}
-{u'return': [{u'status': u'running', u'current-progress': 'FILTERED', u'total-progress': 'FILTERED', u'id': u'job0', u'type': u'mirror'}]}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'created', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
+{"return": {}}
+{"return": [{"current-progress": "FILTERED", "id": "job0", "status": "running", "total-progress": "FILTERED", "type": "mirror"}]}
+{"data": {"id": "job0", "status": "created"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 
 Pause/resume in RUNNING
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 65536, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 65536, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "mirror"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 131072, "id": "job0", "status": "running", "total-progress": 4194304, "type": "mirror"}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 131072, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "mirror"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 196608, "id": "job0", "status": "running", "total-progress": 4194304, "type": "mirror"}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 196608, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "mirror"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 262144, "id": "job0", "status": "running", "total-progress": 4194304, "type": "mirror"}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 327680, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 262144, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "mirror"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 327680, "id": "job0", "status": "running", "total-progress": 4194304, "type": "mirror"}]}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{"return": {}}
 
 Waiting for READY state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'ready', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'ready', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{"data": {"id": "job0", "status": "ready"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "ready", "total-progress": 4194304, "type": "mirror"}]}
 
 Pause/resume in READY
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'standby', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'standby', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'ready', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'ready', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "standby"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "standby", "total-progress": 4194304, "type": "mirror"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "ready"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "ready", "total-progress": 4194304, "type": "mirror"}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'standby', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'standby', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'ready', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'ready', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "standby"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "standby", "total-progress": 4194304, "type": "mirror"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "ready"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "ready", "total-progress": 4194304, "type": "mirror"}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'standby', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'standby', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'ready', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'ready', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "standby"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "standby", "total-progress": 4194304, "type": "mirror"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "ready"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "ready", "total-progress": 4194304, "type": "mirror"}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'standby', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'standby', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'ready', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'ready', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'ready' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'ready' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'ready' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'ready' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{"return": {}}
+{"data": {"id": "job0", "status": "standby"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "standby", "total-progress": 4194304, "type": "mirror"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "ready"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "ready", "total-progress": 4194304, "type": "mirror"}]}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'ready' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'ready' cannot accept command verb 'dismiss'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'ready' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'ready' cannot accept command verb 'dismiss'"}}
+{"return": {}}
 
 Waiting for PENDING state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'waiting', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'pending', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'concluded', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'null', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': []}
+{"data": {"id": "job0", "status": "waiting"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "pending"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "concluded"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "null"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": []}
 
 
 Starting block job: drive-backup (auto-finalize: True; auto-dismiss: True)
-{u'return': {}}
-{u'return': [{u'status': u'running', u'current-progress': 'FILTERED', u'total-progress': 'FILTERED', u'id': u'job0', u'type': u'backup'}]}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'created', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
+{"return": {}}
+{"return": [{"current-progress": "FILTERED", "id": "job0", "status": "running", "total-progress": "FILTERED", "type": "backup"}]}
+{"data": {"id": "job0", "status": "created"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 
 Pause/resume in RUNNING
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 65536, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 65536, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 131072, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 131072, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 196608, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 196608, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 262144, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 327680, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 262144, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 327680, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{"return": {}}
 
 Waiting for PENDING state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'waiting', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'pending', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'concluded', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'null', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': []}
+{"data": {"id": "job0", "status": "waiting"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "pending"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "concluded"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "null"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": []}
 
 
 Starting block job: drive-backup (auto-finalize: True; auto-dismiss: False)
-{u'return': {}}
-{u'return': [{u'status': u'running', u'current-progress': 'FILTERED', u'total-progress': 'FILTERED', u'id': u'job0', u'type': u'backup'}]}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'created', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
+{"return": {}}
+{"return": [{"current-progress": "FILTERED", "id": "job0", "status": "running", "total-progress": "FILTERED", "type": "backup"}]}
+{"data": {"id": "job0", "status": "created"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 
 Pause/resume in RUNNING
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 65536, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 65536, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 131072, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 131072, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 196608, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 196608, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 262144, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 327680, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 262144, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 327680, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{"return": {}}
 
 Waiting for PENDING state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'waiting', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'pending', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'concluded', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'concluded', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'null', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': []}
+{"data": {"id": "job0", "status": "waiting"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "pending"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "concluded"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "concluded", "total-progress": 4194304, "type": "backup"}]}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
+{"return": {}}
+{"data": {"id": "job0", "status": "null"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": []}
 
 
 Starting block job: drive-backup (auto-finalize: False; auto-dismiss: True)
-{u'return': {}}
-{u'return': [{u'status': u'running', u'current-progress': 'FILTERED', u'total-progress': 'FILTERED', u'id': u'job0', u'type': u'backup'}]}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'created', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
+{"return": {}}
+{"return": [{"current-progress": "FILTERED", "id": "job0", "status": "running", "total-progress": "FILTERED", "type": "backup"}]}
+{"data": {"id": "job0", "status": "created"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 
 Pause/resume in RUNNING
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 65536, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 65536, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 131072, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 131072, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 196608, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 196608, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 262144, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 327680, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 262144, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 327680, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{"return": {}}
 
 Waiting for PENDING state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'waiting', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'pending', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'pending', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'concluded', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'null', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': []}
+{"data": {"id": "job0", "status": "waiting"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "pending"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "pending", "total-progress": 4194304, "type": "backup"}]}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
+{"return": {}}
+{"data": {"id": "job0", "status": "concluded"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "null"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": []}
 
 
 Starting block job: drive-backup (auto-finalize: False; auto-dismiss: False)
-{u'return': {}}
-{u'return': [{u'status': u'running', u'current-progress': 'FILTERED', u'total-progress': 'FILTERED', u'id': u'job0', u'type': u'backup'}]}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'created', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
+{"return": {}}
+{"return": [{"current-progress": "FILTERED", "id": "job0", "status": "running", "total-progress": "FILTERED", "type": "backup"}]}
+{"data": {"id": "job0", "status": "created"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
 
 Pause/resume in RUNNING
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 65536, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 65536, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 131072, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 131072, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 196608, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 196608, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 262144, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 327680, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{"return": {}}
+{"data": {"id": "job0", "status": "paused"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 262144, "id": "job0", "status": "paused", "total-progress": 4194304, "type": "backup"}]}
+{"return": {}}
+{"data": {"id": "job0", "status": "running"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 327680, "id": "job0", "status": "running", "total-progress": 4194304, "type": "backup"}]}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{"return": {}}
 
 Waiting for PENDING state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'waiting', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'pending', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'pending', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'concluded', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'concluded', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'null', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': []}
+{"data": {"id": "job0", "status": "waiting"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"data": {"id": "job0", "status": "pending"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "pending", "total-progress": 4194304, "type": "backup"}]}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
+{"return": {}}
+{"data": {"id": "job0", "status": "concluded"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": [{"current-progress": 4194304, "id": "job0", "status": "concluded", "total-progress": 4194304, "type": "backup"}]}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
+{"error": {"class": "GenericError", "desc": "Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
+{"return": {}}
+{"data": {"id": "job0", "status": "null"}, "event": "JOB_STATUS_CHANGE", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": []}
diff --git a/tests/qemu-iotests/222.out b/tests/qemu-iotests/222.out
index 48f336a02b..16643dde30 100644
--- a/tests/qemu-iotests/222.out
+++ b/tests/qemu-iotests/222.out
@@ -8,13 +8,13 @@ Done
 
 --- Setting up Fleecing Graph ---
 
-{u'return': {}}
-{u'return': {}}
+{"return": {}}
+{"return": {}}
 
 --- Setting up NBD Export ---
 
-{u'return': {}}
-{u'return': {}}
+{"return": {}}
+{"return": {}}
 
 --- Sanity Check ---
 
@@ -29,13 +29,13 @@ read -P0 0x3fe0000 64k
 --- Testing COW ---
 
 write -P0xab 0 64k
-{u'return': u''}
+{"return": ""}
 write -P0xad 0x00f8000 64k
-{u'return': u''}
+{"return": ""}
 write -P0x1d 0x2008000 64k
-{u'return': u''}
+{"return": ""}
 write -P0xea 0x3fe0000 64k
-{u'return': u''}
+{"return": ""}
 
 --- Verifying Data ---
 
@@ -49,10 +49,10 @@ read -P0 0x3fe0000 64k
 
 --- Cleanup ---
 
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'drive0', u'type': u'backup', u'speed': 0, u'len': 67108864, u'offset': 393216}, u'event': u'BLOCK_JOB_CANCELLED'}
-{u'return': {}}
-{u'return': {}}
+{"return": {}}
+{"data": {"device": "drive0", "len": 67108864, "offset": 393216, "speed": 0, "type": "backup"}, "event": "BLOCK_JOB_CANCELLED", "timestamp": {"microseconds": "USECS", "seconds": "SECS"}}
+{"return": {}}
+{"return": {}}
 
 --- Confirming writes ---
 
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index a0f35e4b68..27bb2b600c 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -254,7 +254,10 @@ def filter_img_info(output, filename):
 def log(msg, filters=[]):
     for flt in filters:
         msg = flt(msg)
-    print(msg)
+    if type(msg) is dict or type(msg) is list:
+        print(json.dumps(msg, sort_keys=True))
+    else:
+        print(msg)
 
 class Timeout:
     def __init__(self, seconds, errmsg = "Timeout"):
@@ -442,10 +445,11 @@ class VM(qtest.QEMUQtestMachine):
         return result
 
     def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs):
-        logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs)
+        logmsg = '{"execute": "%s", "arguments": %s}' % \
+            (cmd, json.dumps(kwargs, sort_keys=True))
         log(logmsg, filters)
         result = self.qmp(cmd, **kwargs)
-        log(str(result), filters)
+        log(json.dumps(result, sort_keys=True), filters)
         return result
 
     def run_job(self, job, auto_finalize=True, auto_dismiss=False):
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 11/15] Bootstrap Python venv for tests

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

A number of QEMU tests are written in Python, and may benefit
from an untainted Python venv.

By using make rules, tests that depend on specific Python libs
can set that rule as a requirement, along with rules that require
the presence or installation of specific libraries.

The tests/requirements.txt is supposed to contain the Python
requirements that should be added to the venv created by check-venv.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Message-Id: <20181018153134.8493-2-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 tests/requirements.txt |  3 +++
 tests/Makefile.include | 26 ++++++++++++++++++++++++++
 2 files changed, 29 insertions(+)
 create mode 100644 tests/requirements.txt

diff --git a/tests/requirements.txt b/tests/requirements.txt
new file mode 100644
index 0000000000..d39f9d1576
--- /dev/null
+++ b/tests/requirements.txt
@@ -0,0 +1,3 @@
+# Add Python module requirements, one per line, to be installed
+# in the tests/venv Python virtual environment. For more info,
+# refer to: https://pip.pypa.io/en/stable/user_guide/#id1
diff --git a/tests/Makefile.include b/tests/Makefile.include
index f77a495109..eabc1da2f3 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -12,6 +12,7 @@ check-help:
 	@echo " $(MAKE) check-block          Run block tests"
 	@echo " $(MAKE) check-tcg            Run TCG tests"
 	@echo " $(MAKE) check-report.html    Generates an HTML test report"
+	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean          Clean the tests"
 	@echo
 	@echo "Please note that HTML reports do not regenerate if the unit tests"
@@ -899,6 +900,30 @@ check-decodetree:
           ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
           TEST, decodetree.py)
 
+# Python venv for running tests
+
+.PHONY: check-venv
+
+TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
+TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
+
+$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
+ifeq ($(.SHELLSTATUS),0)
+$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
+	$(call quiet-command, \
+            $(PYTHON) -m venv --system-site-packages $@, \
+            VENV, $@)
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
+            PIP, $(TESTS_VENV_REQ))
+	$(call quiet-command, touch $@)
+else
+$(TESTS_VENV_DIR):
+	$(error "venv directory for tests requires Python 3")
+endif
+
+check-venv: $(TESTS_VENV_DIR)
+
 # Consolidated targets
 
 .PHONY: check-qapi-schema check-qtest check-unit check check-clean
@@ -912,6 +937,7 @@ check-clean:
 	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
 	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
 	rm -f tests/test-qapi-gen-timestamp
+	rm -rf $(TESTS_VENV_DIR)
 
 clean: check-clean
 
-- 
2.18.0.rc1.1.g3f1ff2140

Re: [Qemu-devel] [PULL 11/15] Bootstrap Python venv for tests

[Peter Maydell]

On 31 October 2018 at 00:31, Eduardo Habkost <ehabkost@redhat.com> wrote:
> From: Cleber Rosa <crosa@redhat.com>
>
> A number of QEMU tests are written in Python, and may benefit
> from an untainted Python venv.
>
> By using make rules, tests that depend on specific Python libs
> can set that rule as a requirement, along with rules that require
> the presence or installation of specific libraries.
>
> The tests/requirements.txt is supposed to contain the Python
> requirements that should be added to the venv created by check-venv.
>
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
> Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> Message-Id: <20181018153134.8493-2-crosa@redhat.com>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>

> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -12,6 +12,7 @@ check-help:
>         @echo " $(MAKE) check-block          Run block tests"
>         @echo " $(MAKE) check-tcg            Run TCG tests"
>         @echo " $(MAKE) check-report.html    Generates an HTML test report"
> +       @echo " $(MAKE) check-venv           Creates a Python venv for tests"
>         @echo " $(MAKE) check-clean          Clean the tests"
>         @echo
>         @echo "Please note that HTML reports do not regenerate if the unit tests"
> @@ -899,6 +900,30 @@ check-decodetree:
>            ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>            TEST, decodetree.py)
>
> +# Python venv for running tests
> +
> +.PHONY: check-venv
> +
> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> +TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> +
> +$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> +ifeq ($(.SHELLSTATUS),0)
> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> +       $(call quiet-command, \
> +            $(PYTHON) -m venv --system-site-packages $@, \
> +            VENV, $@)
> +       $(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
> +            PIP, $(TESTS_VENV_REQ))
> +       $(call quiet-command, touch $@)
> +else
> +$(TESTS_VENV_DIR):
> +       $(error "venv directory for tests requires Python 3")
> +endif
> +
> +check-venv: $(TESTS_VENV_DIR)

Hi -- this seems to be causing one of the travis configs to fail:

https://travis-ci.org/qemu/qemu/jobs/451311466

The config includes "--python=/usr/bin/python3", but the build
fails with
     CHK version_gen.h
/home/travis/build/qemu/qemu/tests/Makefile.include:928: *** "venv
directory for tests requires Python 3". Stop.

Would you mind having a look at what's happening there ?

thanks
-- PMM

Re: [Qemu-devel] [PULL 11/15] Bootstrap Python venv for tests

[Philippe Mathieu-Daudé]

On 6/11/18 14:10, Peter Maydell wrote:
> On 31 October 2018 at 00:31, Eduardo Habkost <ehabkost@redhat.com> wrote:
>> From: Cleber Rosa <crosa@redhat.com>
>>
>> A number of QEMU tests are written in Python, and may benefit
>> from an untainted Python venv.
>>
>> By using make rules, tests that depend on specific Python libs
>> can set that rule as a requirement, along with rules that require
>> the presence or installation of specific libraries.
>>
>> The tests/requirements.txt is supposed to contain the Python
>> requirements that should be added to the venv created by check-venv.
>>
>> Signed-off-by: Cleber Rosa <crosa@redhat.com>
>> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
>> Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
>> Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
>> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
>> Message-Id: <20181018153134.8493-2-crosa@redhat.com>
>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> 
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -12,6 +12,7 @@ check-help:
>>          @echo " $(MAKE) check-block          Run block tests"
>>          @echo " $(MAKE) check-tcg            Run TCG tests"
>>          @echo " $(MAKE) check-report.html    Generates an HTML test report"
>> +       @echo " $(MAKE) check-venv           Creates a Python venv for tests"
>>          @echo " $(MAKE) check-clean          Clean the tests"
>>          @echo
>>          @echo "Please note that HTML reports do not regenerate if the unit tests"
>> @@ -899,6 +900,30 @@ check-decodetree:
>>             ./check.sh "$(PYTHON)" "$(SRC_PATH)/scripts/decodetree.py", \
>>             TEST, decodetree.py)
>>
>> +# Python venv for running tests
>> +
>> +.PHONY: check-venv
>> +
>> +TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>> +TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
>> +
>> +$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>> +ifeq ($(.SHELLSTATUS),0)
>> +$(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>> +       $(call quiet-command, \
>> +            $(PYTHON) -m venv --system-site-packages $@, \
>> +            VENV, $@)
>> +       $(call quiet-command, \
>> +            $(TESTS_VENV_DIR)/bin/python -m pip -q install -r $(TESTS_VENV_REQ), \
>> +            PIP, $(TESTS_VENV_REQ))
>> +       $(call quiet-command, touch $@)
>> +else
>> +$(TESTS_VENV_DIR):
>> +       $(error "venv directory for tests requires Python 3")
>> +endif
>> +
>> +check-venv: $(TESTS_VENV_DIR)
> 
> Hi -- this seems to be causing one of the travis configs to fail:
> 
> https://travis-ci.org/qemu/qemu/jobs/451311466
> 
> The config includes "--python=/usr/bin/python3", but the build
> fails with
>       CHK version_gen.h
> /home/travis/build/qemu/qemu/tests/Makefile.include:928: *** "venv
> directory for tests requires Python 3". Stop.
> 
> Would you mind having a look at what's happening there ?

I tested the failure using 'make check-venv PYTHON=python2' and the 
success using 'make check-venv PYTHON=python3' but didn't think of the 
default...

The quicker fix is to force PYTHON in .travis.yml, I'll prepare a patch.

> 
> thanks
> -- PMM
> 

[Qemu-devel] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Eduardo Habkost]

The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
seems to provide an older version.  Change the existing rules to
use command output instead of exit code, to make it compatible
with older GNU make versions.

Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
I think that's the cause of the Travis failures.  I have
submitted a test job right now, at:
  https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
Let's see if it fixes the issue.
---
 tests/Makefile.include | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/tests/Makefile.include b/tests/Makefile.include
index d2e577eabb..074eece558 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
 # information please refer to "avocado --help".
 AVOCADO_SHOW=none
 
-$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
-ifeq ($(.SHELLSTATUS),0)
+PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
+ifeq ($(PYTHON3), 1)
 $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
 	$(call quiet-command, \
             $(PYTHON) -m venv --system-site-packages $@, \
-- 
2.18.0.rc1.1.g3f1ff2140

Re: [Qemu-devel] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Philippe Mathieu-Daudé]

On 6/11/18 15:13, Eduardo Habkost wrote:
> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> seems to provide an older version.  Change the existing rules to
> use command output instead of exit code, to make it compatible
> with older GNU make versions.

You were quicker, I just found out :)

> 
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
> I think that's the cause of the Travis failures.  I have
> submitted a test job right now, at:
>    https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> Let's see if it fixes the issue.

You can run it locally:

$ make docker-image-travis
...
$ docker run --rm -it -v $(pwd):$(pwd) -w $(pwd) \
     -u 0 --entrypoint=/bin/bash qemu:travis
root@ede09efe22fd:qemu# cd build/docker_travis/
root@ede09efe22fd:qemu/build/docker_travis# make AVOCADO_SHOW=app 
check-acceptance
   VENV    qemu/build/docker_travis/tests/venv
The virtual environment was not created successfully because ensurepip 
is not
available.  On Debian/Ubuntu systems, you need to install the python3-venv
package using the following command.

     apt-get install python3-venv

You may need to use sudo with that command.  After installing the 
python3-venv
package, recreate your virtual environment.

root@ede09efe22fd:qemu/build/docker_travis# apt install python3-pip 
python3.4-venv
root@ede09efe22fd:qemu/build/docker_travis# make AVOCADO_SHOW=app 
check-acceptance
   VENV    qemu/build/docker_travis/tests/venv
   PIP     qemu/tests/requirements.txt
   AVOCADO tests/acceptance
JOB ID     : 5e5529e79456c388e80323acdc71f3887341a498
JOB LOG    : 
qemu/build/docker_travis/tests/results/job-2018-11-06T14.26-5e5529e/job.log
  (1/6) 
qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: 
CANCEL: No QEMU binary defined or found in the source tree (0.01 s)
  (2/6) 
qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version: 
CANCEL: No QEMU binary defined or found in the source tree (0.01 s)
  (3/6) qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL: No QEMU 
binary defined or found in the source tree (0.00 s)
  (4/6) qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: 
CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
  (5/6) 
qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: 
CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
  (6/6) qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password: 
CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
CANCEL 6
JOB TIME   : 0.45 s

Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> ---
>   tests/Makefile.include | 4 ++--
>   1 file changed, 2 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index d2e577eabb..074eece558 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>   # information please refer to "avocado --help".
>   AVOCADO_SHOW=none
>   
> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> -ifeq ($(.SHELLSTATUS),0)
> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
> +ifeq ($(PYTHON3), 1)
>   $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>   	$(call quiet-command, \
>               $(PYTHON) -m venv --system-site-packages $@, \
> 

Re: [Qemu-devel] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Philippe Mathieu-Daudé]

Hi Peter,

Can you apply this patch as a CI bug-fix?

Thanks,

Phil.

On 6/11/18 15:27, Philippe Mathieu-Daudé wrote:
> On 6/11/18 15:13, Eduardo Habkost wrote:
>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
>> seems to provide an older version.  Change the existing rules to
>> use command output instead of exit code, to make it compatible
>> with older GNU make versions.
> 
> You were quicker, I just found out :)
> 
>>
>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>> ---
>> I think that's the cause of the Travis failures.  I have
>> submitted a test job right now, at:
>>    https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
>> Let's see if it fixes the issue.
> 
> You can run it locally:
> 
> $ make docker-image-travis
> ...
> $ docker run --rm -it -v $(pwd):$(pwd) -w $(pwd) \
>      -u 0 --entrypoint=/bin/bash qemu:travis
> root@ede09efe22fd:qemu# cd build/docker_travis/
> root@ede09efe22fd:qemu/build/docker_travis# make AVOCADO_SHOW=app 
> check-acceptance
>    VENV    qemu/build/docker_travis/tests/venv
> The virtual environment was not created successfully because ensurepip 
> is not
> available.  On Debian/Ubuntu systems, you need to install the python3-venv
> package using the following command.
> 
>      apt-get install python3-venv
> 
> You may need to use sudo with that command.  After installing the 
> python3-venv
> package, recreate your virtual environment.
> 
> root@ede09efe22fd:qemu/build/docker_travis# apt install python3-pip 
> python3.4-venv
> root@ede09efe22fd:qemu/build/docker_travis# make AVOCADO_SHOW=app 
> check-acceptance
>    VENV    qemu/build/docker_travis/tests/venv
>    PIP     qemu/tests/requirements.txt
>    AVOCADO tests/acceptance
> JOB ID     : 5e5529e79456c388e80323acdc71f3887341a498
> JOB LOG    : 
> qemu/build/docker_travis/tests/results/job-2018-11-06T14.26-5e5529e/job.log
>   (1/6) 
> qemu/tests/acceptance/boot_linux_console.py:BootLinuxConsole.test: 
> CANCEL: No QEMU binary defined or found in the source tree (0.01 s)
>   (2/6) 
> qemu/tests/acceptance/version.py:Version.test_qmp_human_info_version: 
> CANCEL: No QEMU binary defined or found in the source tree (0.01 s)
>   (3/6) qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc: CANCEL: No QEMU 
> binary defined or found in the source tree (0.00 s)
>   (4/6) qemu/tests/acceptance/vnc.py:Vnc.test_no_vnc_change_password: 
> CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
>   (5/6) 
> qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password_requires_a_password: 
> CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
>   (6/6) qemu/tests/acceptance/vnc.py:Vnc.test_vnc_change_password: 
> CANCEL: No QEMU binary defined or found in the source tree (0.00 s)
> RESULTS    : PASS 0 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | 
> CANCEL 6
> JOB TIME   : 0.45 s
> 
> Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
> 
>> ---
>>   tests/Makefile.include | 4 ++--
>>   1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index d2e577eabb..074eece558 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>   # information please refer to "avocado --help".
>>   AVOCADO_SHOW=none
>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' 
>> >/dev/null 2>&1)
>> -ifeq ($(.SHELLSTATUS),0)
>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if 
>> sys.version_info >= (3, 0) else 0)')
>> +ifeq ($(PYTHON3), 1)
>>   $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>       $(call quiet-command, \
>>               $(PYTHON) -m venv --system-site-packages $@, \
>>

Re: [Qemu-devel] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Peter Maydell]

On 6 November 2018 at 14:13, Eduardo Habkost <ehabkost@redhat.com> wrote:
> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> seems to provide an older version.  Change the existing rules to
> use command output instead of exit code, to make it compatible
> with older GNU make versions.
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
> I think that's the cause of the Travis failures.  I have
> submitted a test job right now, at:
>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> Let's see if it fixes the issue.
> ---
>  tests/Makefile.include | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index d2e577eabb..074eece558 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>  # information please refer to "avocado --help".
>  AVOCADO_SHOW=none
>
> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> -ifeq ($(.SHELLSTATUS),0)
> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
> +ifeq ($(PYTHON3), 1)
>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>         $(call quiet-command, \
>              $(PYTHON) -m venv --system-site-packages $@, \
> --

Thanks, applied to master as a build fix.

-- PMM

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Markus Armbruster]

Eduardo Habkost <ehabkost@redhat.com> writes:

> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> seems to provide an older version.  Change the existing rules to
> use command output instead of exit code, to make it compatible
> with older GNU make versions.
>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
> I think that's the cause of the Travis failures.  I have
> submitted a test job right now, at:
>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> Let's see if it fixes the issue.
> ---
>  tests/Makefile.include | 4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
>
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index d2e577eabb..074eece558 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>  # information please refer to "avocado --help".
>  AVOCADO_SHOW=none
>  
> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> -ifeq ($(.SHELLSTATUS),0)
> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
> +ifeq ($(PYTHON3), 1)
>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>  	$(call quiet-command, \
>              $(PYTHON) -m venv --system-site-packages $@, \

PEP 394 recommends software distributions install Python 3 into the
default path as python3, and users use that instead of python, except
for programs that are source compatible with both 2 and 3.  So, is
finding out whether python is a Python 3 really appropriate?  Why can't
we just use python3 and be done with it?

If we can't: isn't this a configure problem?

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Peter Maydell]

On 7 November 2018 at 06:05, Markus Armbruster <armbru@redhat.com> wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
>
>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
>> seems to provide an older version.  Change the existing rules to
>> use command output instead of exit code, to make it compatible
>> with older GNU make versions.
>>
>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>> ---
>> I think that's the cause of the Travis failures.  I have
>> submitted a test job right now, at:
>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
>> Let's see if it fixes the issue.
>> ---
>>  tests/Makefile.include | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index d2e577eabb..074eece558 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>  # information please refer to "avocado --help".
>>  AVOCADO_SHOW=none
>>
>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>> -ifeq ($(.SHELLSTATUS),0)
>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
>> +ifeq ($(PYTHON3), 1)
>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>       $(call quiet-command, \
>>              $(PYTHON) -m venv --system-site-packages $@, \
>
> PEP 394 recommends software distributions install Python 3 into the
> default path as python3, and users use that instead of python, except
> for programs that are source compatible with both 2 and 3.  So, is
> finding out whether python is a Python 3 really appropriate?  Why can't
> we just use python3 and be done with it?
>
> If we can't: isn't this a configure problem?

You can't just use python3 and be done with it because python3
might not exist, and because (as with python 2) the user might
want to tell us the path to it. You could have configure detect
whether python3 exists and set a PYTHON3 as well as a PYTHON
(plus I guess support for the user to say "my python3 is this
binary"), and then have this code in Makefile.include handle
"PYTHON3 is not set" to mean "python 3 isn't available".

thanks
-- PMM

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Eduardo Habkost]

On Wed, Nov 07, 2018 at 07:05:03AM +0100, Markus Armbruster wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
> > The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> > seems to provide an older version.  Change the existing rules to
> > use command output instead of exit code, to make it compatible
> > with older GNU make versions.
> >
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > ---
> > I think that's the cause of the Travis failures.  I have
> > submitted a test job right now, at:
> >   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> > Let's see if it fixes the issue.
> > ---
> >  tests/Makefile.include | 4 ++--
> >  1 file changed, 2 insertions(+), 2 deletions(-)
> >
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index d2e577eabb..074eece558 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> >  # information please refer to "avocado --help".
> >  AVOCADO_SHOW=none
> >  
> > -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> > -ifeq ($(.SHELLSTATUS),0)
> > +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
> > +ifeq ($(PYTHON3), 1)
> >  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> >  	$(call quiet-command, \
> >              $(PYTHON) -m venv --system-site-packages $@, \
> 
> PEP 394 recommends software distributions install Python 3 into the
> default path as python3, and users use that instead of python, except
> for programs that are source compatible with both 2 and 3.  So, is
> finding out whether python is a Python 3 really appropriate?  Why can't
> we just use python3 and be done with it?

Because './configure --with-python=...' exists, and I didn't want
to break it.

Now, why do we need --with-python, and why do we need to use
$(PYTHON) when running tests?  If somebody wants to use a
different Python binary when running tests, they can already use
$PATH for that.

(That's the same argument I used for iotests a while ago:
https://www.mail-archive.com/qemu-devel@nongnu.org/msg566631.html)

> 
> If we can't: isn't this a configure problem?

It is, and I think Cleber mentioned that he planned to do it in
./configure, a while ago.  But just using the python3 binary from
$PATH would be even better.

-- 
Eduardo

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Peter Maydell]

On 7 November 2018 at 12:49, Eduardo Habkost <ehabkost@redhat.com> wrote:
> Now, why do we need --with-python, and why do we need to use
> $(PYTHON) when running tests?  If somebody wants to use a
> different Python binary when running tests, they can already use
> $PATH for that.
>
> (That's the same argument I used for iotests a while ago:
> https://www.mail-archive.com/qemu-devel@nongnu.org/msg566631.html)

I'm not a great fan of requiring the user to mess with their PATH
to get configure to work. Also, the first python on the path
might be the wrong one, and we don't pass PATH from configure
to make so you end up having to make sure you specify it
right in both places.

Plus we already have --with-python, so if you want to drop
it you need to deprecate it first, and you need a justification
that's strong enough to outweigh breaking users' existing
build/packaging setups and scripts...

thanks
-- PMM

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Eduardo Habkost]

On Wed, Nov 07, 2018 at 01:45:35PM +0000, Peter Maydell wrote:
> On 7 November 2018 at 12:49, Eduardo Habkost <ehabkost@redhat.com> wrote:
> > Now, why do we need --with-python, and why do we need to use
> > $(PYTHON) when running tests?  If somebody wants to use a
> > different Python binary when running tests, they can already use
> > $PATH for that.
> >
> > (That's the same argument I used for iotests a while ago:
> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg566631.html)
> 
> I'm not a great fan of requiring the user to mess with their PATH
> to get configure to work. Also, the first python on the path
> might be the wrong one, and we don't pass PATH from configure
> to make so you end up having to make sure you specify it
> right in both places.

You're assuming that this will actually require some people to
mess with their $PATH because they currently don't have Python on
their $PATH.  I don't see any evidence that this is expected to
happen.  Do you?

> 
> Plus we already have --with-python, so if you want to drop
> it you need to deprecate it first, and you need a justification
> that's strong enough to outweigh breaking users' existing
> build/packaging setups and scripts...

I would really like to remove the option as soon as we start
requiring Python 3.  Let's stop reinventing solutions to problems
already addressed by PEP 394.

-- 
Eduardo

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Markus Armbruster]

Eduardo Habkost <ehabkost@redhat.com> writes:

> On Wed, Nov 07, 2018 at 01:45:35PM +0000, Peter Maydell wrote:
>> On 7 November 2018 at 12:49, Eduardo Habkost <ehabkost@redhat.com> wrote:
>> > Now, why do we need --with-python, and why do we need to use
>> > $(PYTHON) when running tests?  If somebody wants to use a
>> > different Python binary when running tests, they can already use
>> > $PATH for that.
>> >
>> > (That's the same argument I used for iotests a while ago:
>> > https://www.mail-archive.com/qemu-devel@nongnu.org/msg566631.html)
>> 
>> I'm not a great fan of requiring the user to mess with their PATH
>> to get configure to work. Also, the first python on the path
>> might be the wrong one, and we don't pass PATH from configure
>> to make so you end up having to make sure you specify it
>> right in both places.
>
> You're assuming that this will actually require some people to
> mess with their $PATH because they currently don't have Python on
> their $PATH.  I don't see any evidence that this is expected to
> happen.  Do you?

What makes Python so special we must provide special means to find
it off the PATH?  Why not other tools?

>> Plus we already have --with-python, so if you want to drop
>> it you need to deprecate it first, and you need a justification
>> that's strong enough to outweigh breaking users' existing
>> build/packaging setups and scripts...
>
> I would really like to remove the option as soon as we start
> requiring Python 3.  Let's stop reinventing solutions to problems
> already addressed by PEP 394.

Concur.

We should use python3 wherever we need a Python 3.  There's no point in
letting configure check whether python3 is actually Python 3.

We should use python2 wherever we need a Python 2.  This case needs to
go away (if it isn't gone already).

We may use any of python3, python, python2 wherever we can work with
both Python 3 and 2.  Simply using python3 works on sufficiently recent
hosts.  For maximum portability, we can let configure try python3, then
python.  I wouldn't bother trying python2 as well.

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Cleber Rosa]

On 11/7/18 1:05 AM, Markus Armbruster wrote:
> Eduardo Habkost <ehabkost@redhat.com> writes:
> 
>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
>> seems to provide an older version.  Change the existing rules to
>> use command output instead of exit code, to make it compatible
>> with older GNU make versions.
>>
>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>> ---
>> I think that's the cause of the Travis failures.  I have
>> submitted a test job right now, at:
>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
>> Let's see if it fixes the issue.
>> ---
>>  tests/Makefile.include | 4 ++--
>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>
>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>> index d2e577eabb..074eece558 100644
>> --- a/tests/Makefile.include
>> +++ b/tests/Makefile.include
>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>  # information please refer to "avocado --help".
>>  AVOCADO_SHOW=none
>>  
>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>> -ifeq ($(.SHELLSTATUS),0)
>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
>> +ifeq ($(PYTHON3), 1)
>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>  	$(call quiet-command, \
>>              $(PYTHON) -m venv --system-site-packages $@, \
> 
> PEP 394 recommends software distributions install Python 3 into the
> default path as python3, and users use that instead of python, except
> for programs that are source compatible with both 2 and 3.  So, is
> finding out whether python is a Python 3 really appropriate?  Why can't
> we just use python3 and be done with it?
> 

I mentioned that before, when you pointed out the issue you fix here,
that configure may be the best place to get the Python version (not only
the major version) and make it available elsewhere.  Even if not used
for other purposes, it is IMO important information to show on the
resulting "configure" output.

I'm sending patches to do that in a few.

> If we can't: isn't this a configure problem?
> 

I believe adhering to PEP394 is a good thing, but even that document
recognizes that the real world is not a perfect place: "however, end
users should be aware that python refers to python3 on at least Arch
Linux".  And it only covers *nix systems, so if we hope to care for
non-*nix systems, we have to check the Python version manually.

So, I guess the safest approach from QEMU's side is to check for the
version indeed.

- Cleber.

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Markus Armbruster]

Cleber Rosa <crosa@redhat.com> writes:

> On 11/7/18 1:05 AM, Markus Armbruster wrote:
>> Eduardo Habkost <ehabkost@redhat.com> writes:
>> 
>>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
>>> seems to provide an older version.  Change the existing rules to
>>> use command output instead of exit code, to make it compatible
>>> with older GNU make versions.
>>>
>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>> ---
>>> I think that's the cause of the Travis failures.  I have
>>> submitted a test job right now, at:
>>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
>>> Let's see if it fixes the issue.
>>> ---
>>>  tests/Makefile.include | 4 ++--
>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>>
>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>> index d2e577eabb..074eece558 100644
>>> --- a/tests/Makefile.include
>>> +++ b/tests/Makefile.include
>>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>>  # information please refer to "avocado --help".
>>>  AVOCADO_SHOW=none
>>>  
>>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>>> -ifeq ($(.SHELLSTATUS),0)
>>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
>>> +ifeq ($(PYTHON3), 1)
>>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>>  	$(call quiet-command, \
>>>              $(PYTHON) -m venv --system-site-packages $@, \
>> 
>> PEP 394 recommends software distributions install Python 3 into the
>> default path as python3, and users use that instead of python, except
>> for programs that are source compatible with both 2 and 3.  So, is
>> finding out whether python is a Python 3 really appropriate?  Why can't
>> we just use python3 and be done with it?
>> 
>
> I mentioned that before, when you pointed out the issue you fix here,
> that configure may be the best place to get the Python version (not only
> the major version) and make it available elsewhere.  Even if not used
> for other purposes, it is IMO important information to show on the
> resulting "configure" output.
>
> I'm sending patches to do that in a few.
>
>> If we can't: isn't this a configure problem?
>> 
>
> I believe adhering to PEP394 is a good thing, but even that document
> recognizes that the real world is not a perfect place: "however, end
> users should be aware that python refers to python3 on at least Arch
> Linux".  And it only covers *nix systems, so if we hope to care for
> non-*nix systems, we have to check the Python version manually.
>
> So, I guess the safest approach from QEMU's side is to check for the
> version indeed.

If somebody can point to a system people still use where python3 doesn't
get you a Python 3, but python does, catering for such (crappy) systems
in configure makes sense.  Until then, it's a waste of brain waves and
configure run time.

PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
recent version of Arch Linux that's still crappy in this regard?

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Philippe Mathieu-Daudé]

Hi Markus,


Le jeu. 8 nov. 2018 09:46, Markus Armbruster <armbru@redhat.com> a écrit :

> Cleber Rosa <crosa@redhat.com> writes:
>
> > On 11/7/18 1:05 AM, Markus Armbruster wrote:
> >> Eduardo Habkost <ehabkost@redhat.com> writes:
> >>
> >>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> >>> seems to provide an older version.  Change the existing rules to
> >>> use command output instead of exit code, to make it compatible
> >>> with older GNU make versions.
> >>>
> >>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> >>> ---
> >>> I think that's the cause of the Travis failures.  I have
> >>> submitted a test job right now, at:
> >>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> >>> Let's see if it fixes the issue.
> >>> ---
> >>>  tests/Makefile.include | 4 ++--
> >>>  1 file changed, 2 insertions(+), 2 deletions(-)
> >>>
> >>> diff --git a/tests/Makefile.include b/tests/Makefile.include
> >>> index d2e577eabb..074eece558 100644
> >>> --- a/tests/Makefile.include
> >>> +++ b/tests/Makefile.include
> >>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> >>>  # information please refer to "avocado --help".
> >>>  AVOCADO_SHOW=none
> >>>
> >>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)'
> >/dev/null 2>&1)
> >>> -ifeq ($(.SHELLSTATUS),0)
> >>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if
> sys.version_info >= (3, 0) else 0)')
> >>> +ifeq ($(PYTHON3), 1)
> >>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> >>>     $(call quiet-command, \
> >>>              $(PYTHON) -m venv --system-site-packages $@, \
> >>
> >> PEP 394 recommends software distributions install Python 3 into the
> >> default path as python3, and users use that instead of python, except
> >> for programs that are source compatible with both 2 and 3.  So, is
> >> finding out whether python is a Python 3 really appropriate?  Why can't
> >> we just use python3 and be done with it?
> >>
> >
> > I mentioned that before, when you pointed out the issue you fix here,
> > that configure may be the best place to get the Python version (not only
> > the major version) and make it available elsewhere.  Even if not used
> > for other purposes, it is IMO important information to show on the
> > resulting "configure" output.
> >
> > I'm sending patches to do that in a few.
> >
> >> If we can't: isn't this a configure problem?
> >>
> >
> > I believe adhering to PEP394 is a good thing, but even that document
> > recognizes that the real world is not a perfect place: "however, end
> > users should be aware that python refers to python3 on at least Arch
> > Linux".  And it only covers *nix systems, so if we hope to care for
> > non-*nix systems, we have to check the Python version manually.
> >
> > So, I guess the safest approach from QEMU's side is to check for the
> > version indeed.
>
> If somebody can point to a system people still use where python3 doesn't
> get you a Python 3, but python does, catering for such (crappy) systems
> in configure makes sense.  Until then, it's a waste of brain waves and
> configure run time.
>
> PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
> recent version of Arch Linux that's still crappy in this regard?
>

Arch doesn't provide python2 by default, thus python points to python3.
I think what's crappy is scripts expecting python to be python2.

>

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Markus Armbruster]

Philippe Mathieu-Daudé <f4bug@amsat.org> writes:

> Hi Markus,
>
>
> Le jeu. 8 nov. 2018 09:46, Markus Armbruster <armbru@redhat.com> a écrit :
>
>> Cleber Rosa <crosa@redhat.com> writes:
>>
>> > On 11/7/18 1:05 AM, Markus Armbruster wrote:
[...]
>> >> PEP 394 recommends software distributions install Python 3 into the
>> >> default path as python3, and users use that instead of python, except
>> >> for programs that are source compatible with both 2 and 3.  So, is
>> >> finding out whether python is a Python 3 really appropriate?  Why can't
>> >> we just use python3 and be done with it?
>> >>
>> >
>> > I mentioned that before, when you pointed out the issue you fix here,
>> > that configure may be the best place to get the Python version (not only
>> > the major version) and make it available elsewhere.  Even if not used
>> > for other purposes, it is IMO important information to show on the
>> > resulting "configure" output.
>> >
>> > I'm sending patches to do that in a few.
>> >
>> >> If we can't: isn't this a configure problem?
>> >>
>> >
>> > I believe adhering to PEP394 is a good thing, but even that document
>> > recognizes that the real world is not a perfect place: "however, end
>> > users should be aware that python refers to python3 on at least Arch
>> > Linux".  And it only covers *nix systems, so if we hope to care for
>> > non-*nix systems, we have to check the Python version manually.
>> >
>> > So, I guess the safest approach from QEMU's side is to check for the
>> > version indeed.
>>
>> If somebody can point to a system people still use where python3 doesn't
>> get you a Python 3, but python does, catering for such (crappy) systems
>> in configure makes sense.  Until then, it's a waste of brain waves and
>> configure run time.
>>
>> PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
>> recent version of Arch Linux that's still crappy in this regard?
>>
>
> Arch doesn't provide python2 by default, thus python points to python3.

That's non-crappy as long as python3 also exists, as PEP 394 recommends.
Does it?

> I think what's crappy is scripts expecting python to be python2.

No argument.

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Max Reitz]

On 08.11.18 13:43, Markus Armbruster wrote:
> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
> 
>> Hi Markus,
>>
>>
>> Le jeu. 8 nov. 2018 09:46, Markus Armbruster <armbru@redhat.com> a écrit :
>>
>>> Cleber Rosa <crosa@redhat.com> writes:
>>>
>>>> On 11/7/18 1:05 AM, Markus Armbruster wrote:
> [...]
>>>>> PEP 394 recommends software distributions install Python 3 into the
>>>>> default path as python3, and users use that instead of python, except
>>>>> for programs that are source compatible with both 2 and 3.  So, is
>>>>> finding out whether python is a Python 3 really appropriate?  Why can't
>>>>> we just use python3 and be done with it?
>>>>>
>>>>
>>>> I mentioned that before, when you pointed out the issue you fix here,
>>>> that configure may be the best place to get the Python version (not only
>>>> the major version) and make it available elsewhere.  Even if not used
>>>> for other purposes, it is IMO important information to show on the
>>>> resulting "configure" output.
>>>>
>>>> I'm sending patches to do that in a few.
>>>>
>>>>> If we can't: isn't this a configure problem?
>>>>>
>>>>
>>>> I believe adhering to PEP394 is a good thing, but even that document
>>>> recognizes that the real world is not a perfect place: "however, end
>>>> users should be aware that python refers to python3 on at least Arch
>>>> Linux".  And it only covers *nix systems, so if we hope to care for
>>>> non-*nix systems, we have to check the Python version manually.
>>>>
>>>> So, I guess the safest approach from QEMU's side is to check for the
>>>> version indeed.
>>>
>>> If somebody can point to a system people still use where python3 doesn't
>>> get you a Python 3, but python does, catering for such (crappy) systems
>>> in configure makes sense.  Until then, it's a waste of brain waves and
>>> configure run time.
>>>
>>> PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
>>> recent version of Arch Linux that's still crappy in this regard?
>>>
>>
>> Arch doesn't provide python2 by default, thus python points to python3.
> 
> That's non-crappy as long as python3 also exists, as PEP 394 recommends.
> Does it?

Sure it does.

Arch is just problematic in how it handles "python" itself.  I don't
think there is any system that has Python 3 but no "python3".

Max

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Markus Armbruster]

Max Reitz <mreitz@redhat.com> writes:

> On 08.11.18 13:43, Markus Armbruster wrote:
>> Philippe Mathieu-Daudé <f4bug@amsat.org> writes:
>> 
>>> Hi Markus,
>>>
>>>
>>> Le jeu. 8 nov. 2018 09:46, Markus Armbruster <armbru@redhat.com> a écrit :
>>>
>>>> Cleber Rosa <crosa@redhat.com> writes:
>>>>
>>>>> On 11/7/18 1:05 AM, Markus Armbruster wrote:
>> [...]
>>>>>> PEP 394 recommends software distributions install Python 3 into the
>>>>>> default path as python3, and users use that instead of python, except
>>>>>> for programs that are source compatible with both 2 and 3.  So, is
>>>>>> finding out whether python is a Python 3 really appropriate?  Why can't
>>>>>> we just use python3 and be done with it?
>>>>>>
>>>>>
>>>>> I mentioned that before, when you pointed out the issue you fix here,
>>>>> that configure may be the best place to get the Python version (not only
>>>>> the major version) and make it available elsewhere.  Even if not used
>>>>> for other purposes, it is IMO important information to show on the
>>>>> resulting "configure" output.
>>>>>
>>>>> I'm sending patches to do that in a few.
>>>>>
>>>>>> If we can't: isn't this a configure problem?
>>>>>>
>>>>>
>>>>> I believe adhering to PEP394 is a good thing, but even that document
>>>>> recognizes that the real world is not a perfect place: "however, end
>>>>> users should be aware that python refers to python3 on at least Arch
>>>>> Linux".  And it only covers *nix systems, so if we hope to care for
>>>>> non-*nix systems, we have to check the Python version manually.
>>>>>
>>>>> So, I guess the safest approach from QEMU's side is to check for the
>>>>> version indeed.
>>>>
>>>> If somebody can point to a system people still use where python3 doesn't
>>>> get you a Python 3, but python does, catering for such (crappy) systems
>>>> in configure makes sense.  Until then, it's a waste of brain waves and
>>>> configure run time.
>>>>
>>>> PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
>>>> recent version of Arch Linux that's still crappy in this regard?
>>>>
>>>
>>> Arch doesn't provide python2 by default, thus python points to python3.
>> 
>> That's non-crappy as long as python3 also exists, as PEP 394 recommends.
>> Does it?
>
> Sure it does.
>
> Arch is just problematic in how it handles "python" itself.  I don't
> think there is any system that has Python 3 but no "python3".

Bottom line: we can safely assume that a host that has Python 3 provides
it as python3.

Corollary:

* When python3 doesn't exist, checking whether python is Python 3 is
  futile.

* Checking whether python3 really is Python 3 is silly.

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Cleber Rosa]

On 11/8/18 3:45 AM, Markus Armbruster wrote:
> Cleber Rosa <crosa@redhat.com> writes:
> 
>> On 11/7/18 1:05 AM, Markus Armbruster wrote:
>>> Eduardo Habkost <ehabkost@redhat.com> writes:
>>>
>>>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
>>>> seems to provide an older version.  Change the existing rules to
>>>> use command output instead of exit code, to make it compatible
>>>> with older GNU make versions.
>>>>
>>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
>>>> ---
>>>> I think that's the cause of the Travis failures.  I have
>>>> submitted a test job right now, at:
>>>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
>>>> Let's see if it fixes the issue.
>>>> ---
>>>>  tests/Makefile.include | 4 ++--
>>>>  1 file changed, 2 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
>>>> index d2e577eabb..074eece558 100644
>>>> --- a/tests/Makefile.include
>>>> +++ b/tests/Makefile.include
>>>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
>>>>  # information please refer to "avocado --help".
>>>>  AVOCADO_SHOW=none
>>>>  
>>>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>>>> -ifeq ($(.SHELLSTATUS),0)
>>>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
>>>> +ifeq ($(PYTHON3), 1)
>>>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
>>>>  	$(call quiet-command, \
>>>>              $(PYTHON) -m venv --system-site-packages $@, \
>>>
>>> PEP 394 recommends software distributions install Python 3 into the
>>> default path as python3, and users use that instead of python, except
>>> for programs that are source compatible with both 2 and 3.  So, is
>>> finding out whether python is a Python 3 really appropriate?  Why can't
>>> we just use python3 and be done with it?
>>>
>>
>> I mentioned that before, when you pointed out the issue you fix here,
>> that configure may be the best place to get the Python version (not only
>> the major version) and make it available elsewhere.  Even if not used
>> for other purposes, it is IMO important information to show on the
>> resulting "configure" output.
>>
>> I'm sending patches to do that in a few.
>>
>>> If we can't: isn't this a configure problem?
>>>
>>
>> I believe adhering to PEP394 is a good thing, but even that document
>> recognizes that the real world is not a perfect place: "however, end
>> users should be aware that python refers to python3 on at least Arch
>> Linux".  And it only covers *nix systems, so if we hope to care for
>> non-*nix systems, we have to check the Python version manually.
>>
>> So, I guess the safest approach from QEMU's side is to check for the
>> version indeed.
> 
> If somebody can point to a system people still use where python3 doesn't
> get you a Python 3, but python does, catering for such (crappy) systems
> in configure makes sense.  Until then, it's a waste of brain waves and
> configure run time.
> 
> PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
> recent version of Arch Linux that's still crappy in this regard?
> 

I'm not taking the mission of looking for those odd distros, I agree
it's not productive.  My point is that the Python version is an
important thing to keep track of, specially with the advent of running
Python dependent tests on environments we don't fully control.

Supposing the Python version is captured and tracked by configure, then
the "python" binary name becomes a non-issue.

- Cleber.

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Eduardo Habkost]

On Thu, Nov 08, 2018 at 11:06:45AM -0500, Cleber Rosa wrote:
> 
> 
> On 11/8/18 3:45 AM, Markus Armbruster wrote:
> > Cleber Rosa <crosa@redhat.com> writes:
> > 
> >> On 11/7/18 1:05 AM, Markus Armbruster wrote:
> >>> Eduardo Habkost <ehabkost@redhat.com> writes:
> >>>
> >>>> The $(SHELLSTATUS) variable requires GNU make >= 4.2, but Travis
> >>>> seems to provide an older version.  Change the existing rules to
> >>>> use command output instead of exit code, to make it compatible
> >>>> with older GNU make versions.
> >>>>
> >>>> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> >>>> ---
> >>>> I think that's the cause of the Travis failures.  I have
> >>>> submitted a test job right now, at:
> >>>>   https://travis-ci.org/ehabkost/qemu-hacks/jobs/451387962
> >>>> Let's see if it fixes the issue.
> >>>> ---
> >>>>  tests/Makefile.include | 4 ++--
> >>>>  1 file changed, 2 insertions(+), 2 deletions(-)
> >>>>
> >>>> diff --git a/tests/Makefile.include b/tests/Makefile.include
> >>>> index d2e577eabb..074eece558 100644
> >>>> --- a/tests/Makefile.include
> >>>> +++ b/tests/Makefile.include
> >>>> @@ -913,8 +913,8 @@ TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> >>>>  # information please refer to "avocado --help".
> >>>>  AVOCADO_SHOW=none
> >>>>  
> >>>> -$(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> >>>> -ifeq ($(.SHELLSTATUS),0)
> >>>> +PYTHON3 = $(shell $(PYTHON) -c 'import sys; print(1 if sys.version_info >= (3, 0) else 0)')
> >>>> +ifeq ($(PYTHON3), 1)
> >>>>  $(TESTS_VENV_DIR): $(TESTS_VENV_REQ)
> >>>>  	$(call quiet-command, \
> >>>>              $(PYTHON) -m venv --system-site-packages $@, \
> >>>
> >>> PEP 394 recommends software distributions install Python 3 into the
> >>> default path as python3, and users use that instead of python, except
> >>> for programs that are source compatible with both 2 and 3.  So, is
> >>> finding out whether python is a Python 3 really appropriate?  Why can't
> >>> we just use python3 and be done with it?
> >>>
> >>
> >> I mentioned that before, when you pointed out the issue you fix here,
> >> that configure may be the best place to get the Python version (not only
> >> the major version) and make it available elsewhere.  Even if not used
> >> for other purposes, it is IMO important information to show on the
> >> resulting "configure" output.
> >>
> >> I'm sending patches to do that in a few.
> >>
> >>> If we can't: isn't this a configure problem?
> >>>
> >>
> >> I believe adhering to PEP394 is a good thing, but even that document
> >> recognizes that the real world is not a perfect place: "however, end
> >> users should be aware that python refers to python3 on at least Arch
> >> Linux".  And it only covers *nix systems, so if we hope to care for
> >> non-*nix systems, we have to check the Python version manually.
> >>
> >> So, I guess the safest approach from QEMU's side is to check for the
> >> version indeed.
> > 
> > If somebody can point to a system people still use where python3 doesn't
> > get you a Python 3, but python does, catering for such (crappy) systems
> > in configure makes sense.  Until then, it's a waste of brain waves and
> > configure run time.
> > 
> > PEP 394 mentions Arch Linux.  It's been seven years.  What's the most
> > recent version of Arch Linux that's still crappy in this regard?
> > 
> 
> I'm not taking the mission of looking for those odd distros, I agree
> it's not productive.  My point is that the Python version is an
> important thing to keep track of, specially with the advent of running
> Python dependent tests on environments we don't fully control.
> 
> Supposing the Python version is captured and tracked by configure, then
> the "python" binary name becomes a non-issue.

I'm not sure I agree with the "is an important thing to keep
track of" part.  I don't think we'll have any need to keep track
of the Python version in shell script or makefiles after we start
requiring Python 3.

Extra cleanups (like moving version checks to ./configure) are
still welcome, but keep in mind that this will probably be thrown
away once we drop Python 2 support.

-- 
Eduardo

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Cleber Rosa]

On 11/8/18 11:51 AM, Eduardo Habkost wrote:

> I'm not sure I agree with the "is an important thing to keep
> track of" part.  I don't think we'll have any need to keep track
> of the Python version in shell script or makefiles after we start
> requiring Python 3.
> 

Well, "Python 3" is not a uniform thing.  There are many changes across
the 3.x spectrum that can bite you.  I'm speaking out of the experience
with Avocado, that supports Python 3.4, 3.5 and 3.6 (and that reminds me
we should add tests for 3.7).

> Extra cleanups (like moving version checks to ./configure) are
> still welcome, but keep in mind that this will probably be thrown
> away once we drop Python 2 support.
> 

I don't think it should.  Let me give the example of a "Python 3.0" job
on Travis, related to the following snippet:

    # Python builds
    - env: CONFIG="--target-list=x86_64-softmmu"
      python:
        - "3.0"

If you look at https://travis-ci.org/clebergnu/qemu/jobs/452033247#L983
you'll see that the intended goal was missed.  That has to do with how
Travis makes the requested Python version available, and it was only
obvious because this branch prints the Python version used.

Developers writing Python code, for instance tests, may assume a given
API that doesn't exist or behave like they expect, and not knowing the
Python version used on a remote environment makes debugging extra hard.

- Cleber.

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Eduardo Habkost]

On Thu, Nov 08, 2018 at 12:36:37PM -0500, Cleber Rosa wrote:
> 
> 
> On 11/8/18 11:51 AM, Eduardo Habkost wrote:
> 
> > I'm not sure I agree with the "is an important thing to keep
> > track of" part.  I don't think we'll have any need to keep track
> > of the Python version in shell script or makefiles after we start
> > requiring Python 3.
> > 
> 
> Well, "Python 3" is not a uniform thing.  There are many changes across
> the 3.x spectrum that can bite you.  I'm speaking out of the experience
> with Avocado, that supports Python 3.4, 3.5 and 3.6 (and that reminds me
> we should add tests for 3.7).

Do you expect us to need to keep track of the exact Python
version before running the Python interpreter?  Code written in
Python doesn't count, because it can simply use sys.version_info.


> 
> > Extra cleanups (like moving version checks to ./configure) are
> > still welcome, but keep in mind that this will probably be thrown
> > away once we drop Python 2 support.
> > 
> 
> I don't think it should.  Let me give the example of a "Python 3.0" job
> on Travis, related to the following snippet:
> 
>     # Python builds
>     - env: CONFIG="--target-list=x86_64-softmmu"
>       python:
>         - "3.0"
> 
> If you look at https://travis-ci.org/clebergnu/qemu/jobs/452033247#L983
> you'll see that the intended goal was missed.  That has to do with how
> Travis makes the requested Python version available, and it was only
> obvious because this branch prints the Python version used.
> 
> Developers writing Python code, for instance tests, may assume a given
> API that doesn't exist or behave like they expect, and not knowing the
> Python version used on a remote environment makes debugging extra hard.

I'm not sure I get your point.  We can surely add diagnostic
messages to show the Python version, and we can make ./configure
fail if Python 3 is unavailable.  I'm talking about adding code
to ./configure to save Python version info for makefile rules and
shell scripts.  I expect the extra code to be unnecessary in the
future.

-- 
Eduardo

Re: [Qemu-devel] [Qemu-block] [PATCH] tests: Fix Python 3 detection on older GNU make versions

[Cleber Rosa]

On 11/8/18 1:26 PM, Eduardo Habkost wrote:
> On Thu, Nov 08, 2018 at 12:36:37PM -0500, Cleber Rosa wrote:
>>
>>
>> On 11/8/18 11:51 AM, Eduardo Habkost wrote:
>>
>>> I'm not sure I agree with the "is an important thing to keep
>>> track of" part.  I don't think we'll have any need to keep track
>>> of the Python version in shell script or makefiles after we start
>>> requiring Python 3.
>>>
>>
>> Well, "Python 3" is not a uniform thing.  There are many changes across
>> the 3.x spectrum that can bite you.  I'm speaking out of the experience
>> with Avocado, that supports Python 3.4, 3.5 and 3.6 (and that reminds me
>> we should add tests for 3.7).
> 
> Do you expect us to need to keep track of the exact Python
> version before running the Python interpreter?  Code written in
> Python doesn't count, because it can simply use sys.version_info.
> 
> 

You're right, Python code can do that.  My proposal is really to
facilitate other parts of the test automation, and as described later,
the debugging on remote systems.

>>
>>> Extra cleanups (like moving version checks to ./configure) are
>>> still welcome, but keep in mind that this will probably be thrown
>>> away once we drop Python 2 support.
>>>
>>
>> I don't think it should.  Let me give the example of a "Python 3.0" job
>> on Travis, related to the following snippet:
>>
>>     # Python builds
>>     - env: CONFIG="--target-list=x86_64-softmmu"
>>       python:
>>         - "3.0"
>>
>> If you look at https://travis-ci.org/clebergnu/qemu/jobs/452033247#L983
>> you'll see that the intended goal was missed.  That has to do with how
>> Travis makes the requested Python version available, and it was only
>> obvious because this branch prints the Python version used.
>>
>> Developers writing Python code, for instance tests, may assume a given
>> API that doesn't exist or behave like they expect, and not knowing the
>> Python version used on a remote environment makes debugging extra hard.
> 
> I'm not sure I get your point.  We can surely add diagnostic
> messages to show the Python version, and we can make ./configure
> fail if Python 3 is unavailable.  I'm talking about adding code
> to ./configure to save Python version info for makefile rules and
> shell scripts.  I expect the extra code to be unnecessary in the
> future.
> 

I'm definitely happy if you agree with printing the Python version,
that's where most of the value is indeed.  Then, if we need to keep the
version in the generate Makefile is something to be seen.

- Cleber.

[Qemu-devel] [PULL 12/15] Acceptance tests: add make rule for running them

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

The acceptance (aka functional, aka Avocado-based) tests are
Python files located in "tests/acceptance" that need to be run
with the Avocado libs and test runner.

Let's provide a convenient way for QEMU developers to run them,
by making use of the tests-venv with the required setup.

Also, while the Avocado test runner will take care of creating a
location to save test results to, it was understood that it's better
if the results are kept within the build tree.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Message-Id: <20181018153134.8493-3-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 docs/devel/testing.rst | 43 +++++++++++++++++++++++++++++++++++++-----
 tests/requirements.txt |  1 +
 tests/Makefile.include | 21 +++++++++++++++++++--
 3 files changed, 58 insertions(+), 7 deletions(-)

diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
index a227754f86..18e2c0868a 100644
--- a/docs/devel/testing.rst
+++ b/docs/devel/testing.rst
@@ -545,10 +545,39 @@ Tests based on ``avocado_qemu.Test`` can easily:
    - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
    - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
 
-Installation
-------------
+Running tests
+-------------
+
+You can run the acceptance tests simply by executing:
+
+.. code::
+
+  make check-acceptance
+
+This involves the automatic creation of Python virtual environment
+within the build tree (at ``tests/venv``) which will have all the
+right dependencies, and will save tests results also within the
+build tree (at ``tests/results``).
 
-To install Avocado and its dependencies, run:
+Note: the build environment must be using a Python 3 stack, and have
+the ``venv`` and ``pip`` packages installed.  If necessary, make sure
+``configure`` is called with ``--python=`` and that those modules are
+available.  On Debian and Ubuntu based systems, depending on the
+specific version, they may be on packages named ``python3-venv`` and
+``python3-pip``.
+
+The scripts installed inside the virtual environment may be used
+without an "activation".  For instance, the Avocado test runner
+may be invoked by running:
+
+ .. code::
+
+  tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
+
+Manual Installation
+-------------------
+
+To manually install Avocado and its dependencies, run:
 
 .. code::
 
@@ -689,11 +718,15 @@ The exact QEMU binary to be used on QEMUMachine.
 Uninstalling Avocado
 --------------------
 
-If you've followed the installation instructions above, you can easily
-uninstall Avocado.  Start by listing the packages you have installed::
+If you've followed the manual installation instructions above, you can
+easily uninstall Avocado.  Start by listing the packages you have
+installed::
 
   pip list --user
 
 And remove any package you want with::
 
   pip uninstall <package_name>
+
+If you've used ``make check-acceptance``, the Python virtual environment where
+Avocado is installed will be cleaned up as part of ``make check-clean``.
diff --git a/tests/requirements.txt b/tests/requirements.txt
index d39f9d1576..64c6e27a94 100644
--- a/tests/requirements.txt
+++ b/tests/requirements.txt
@@ -1,3 +1,4 @@
 # Add Python module requirements, one per line, to be installed
 # in the tests/venv Python virtual environment. For more info,
 # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
+avocado-framework==65.0
diff --git a/tests/Makefile.include b/tests/Makefile.include
index eabc1da2f3..d2e577eabb 100644
--- a/tests/Makefile.include
+++ b/tests/Makefile.include
@@ -11,6 +11,7 @@ check-help:
 	@echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
 	@echo " $(MAKE) check-block          Run block tests"
 	@echo " $(MAKE) check-tcg            Run TCG tests"
+	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
 	@echo " $(MAKE) check-report.html    Generates an HTML test report"
 	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
 	@echo " $(MAKE) check-clean          Clean the tests"
@@ -902,10 +903,15 @@ check-decodetree:
 
 # Python venv for running tests
 
-.PHONY: check-venv
+.PHONY: check-venv check-acceptance
 
 TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
 TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
+TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
+# Controls the output generated by Avocado when running tests.
+# Any number of command separated loggers are accepted.  For more
+# information please refer to "avocado --help".
+AVOCADO_SHOW=none
 
 $(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
 ifeq ($(.SHELLSTATUS),0)
@@ -922,8 +928,19 @@ $(TESTS_VENV_DIR):
 	$(error "venv directory for tests requires Python 3")
 endif
 
+$(TESTS_RESULTS_DIR):
+	$(call quiet-command, mkdir -p $@, \
+            MKDIR, $@)
+
 check-venv: $(TESTS_VENV_DIR)
 
+check-acceptance: check-venv $(TESTS_RESULTS_DIR)
+	$(call quiet-command, \
+            $(TESTS_VENV_DIR)/bin/python -m avocado \
+            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
+            --failfast=on $(SRC_PATH)/tests/acceptance, \
+            "AVOCADO", "tests/acceptance")
+
 # Consolidated targets
 
 .PHONY: check-qapi-schema check-qtest check-unit check check-clean
@@ -937,7 +954,7 @@ check-clean:
 	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
 	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
 	rm -f tests/test-qapi-gen-timestamp
-	rm -rf $(TESTS_VENV_DIR)
+	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
 
 clean: check-clean
 
-- 
2.18.0.rc1.1.g3f1ff2140

Re: [Qemu-devel] [PULL 12/15] Acceptance tests: add make rule for running them

[Paolo Bonzini]

On 31/10/2018 01:31, Eduardo Habkost wrote:
> From: Cleber Rosa <crosa@redhat.com>
> 
> The acceptance (aka functional, aka Avocado-based) tests are
> Python files located in "tests/acceptance" that need to be run
> with the Avocado libs and test runner.
> 
> Let's provide a convenient way for QEMU developers to run them,
> by making use of the tests-venv with the required setup.
> 
> Also, while the Avocado test runner will take care of creating a
> location to save test results to, it was understood that it's better
> if the results are kept within the build tree.
> 
> Signed-off-by: Cleber Rosa <crosa@redhat.com>
> Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
> Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> Message-Id: <20181018153134.8493-3-crosa@redhat.com>
> Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> ---
>  docs/devel/testing.rst | 43 +++++++++++++++++++++++++++++++++++++-----
>  tests/requirements.txt |  1 +
>  tests/Makefile.include | 21 +++++++++++++++++++--
>  3 files changed, 58 insertions(+), 7 deletions(-)
> 
> diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> index a227754f86..18e2c0868a 100644
> --- a/docs/devel/testing.rst
> +++ b/docs/devel/testing.rst
> @@ -545,10 +545,39 @@ Tests based on ``avocado_qemu.Test`` can easily:
>     - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
>     - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
>  
> -Installation
> -------------
> +Running tests
> +-------------
> +
> +You can run the acceptance tests simply by executing:
> +
> +.. code::
> +
> +  make check-acceptance
> +
> +This involves the automatic creation of Python virtual environment
> +within the build tree (at ``tests/venv``) which will have all the
> +right dependencies, and will save tests results also within the
> +build tree (at ``tests/results``).
>  
> -To install Avocado and its dependencies, run:
> +Note: the build environment must be using a Python 3 stack, and have
> +the ``venv`` and ``pip`` packages installed.  If necessary, make sure
> +``configure`` is called with ``--python=`` and that those modules are
> +available.  On Debian and Ubuntu based systems, depending on the
> +specific version, they may be on packages named ``python3-venv`` and
> +``python3-pip``.
> +
> +The scripts installed inside the virtual environment may be used
> +without an "activation".  For instance, the Avocado test runner
> +may be invoked by running:
> +
> + .. code::
> +
> +  tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
> +
> +Manual Installation
> +-------------------
> +
> +To manually install Avocado and its dependencies, run:
>  
>  .. code::
>  
> @@ -689,11 +718,15 @@ The exact QEMU binary to be used on QEMUMachine.
>  Uninstalling Avocado
>  --------------------
>  
> -If you've followed the installation instructions above, you can easily
> -uninstall Avocado.  Start by listing the packages you have installed::
> +If you've followed the manual installation instructions above, you can
> +easily uninstall Avocado.  Start by listing the packages you have
> +installed::
>  
>    pip list --user
>  
>  And remove any package you want with::
>  
>    pip uninstall <package_name>
> +
> +If you've used ``make check-acceptance``, the Python virtual environment where
> +Avocado is installed will be cleaned up as part of ``make check-clean``.
> diff --git a/tests/requirements.txt b/tests/requirements.txt
> index d39f9d1576..64c6e27a94 100644
> --- a/tests/requirements.txt
> +++ b/tests/requirements.txt
> @@ -1,3 +1,4 @@
>  # Add Python module requirements, one per line, to be installed
>  # in the tests/venv Python virtual environment. For more info,
>  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> +avocado-framework==65.0
> diff --git a/tests/Makefile.include b/tests/Makefile.include
> index eabc1da2f3..d2e577eabb 100644
> --- a/tests/Makefile.include
> +++ b/tests/Makefile.include
> @@ -11,6 +11,7 @@ check-help:
>  	@echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
>  	@echo " $(MAKE) check-block          Run block tests"
>  	@echo " $(MAKE) check-tcg            Run TCG tests"
> +	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
>  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
>  	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
>  	@echo " $(MAKE) check-clean          Clean the tests"
> @@ -902,10 +903,15 @@ check-decodetree:
>  
>  # Python venv for running tests
>  
> -.PHONY: check-venv
> +.PHONY: check-venv check-acceptance
>  
>  TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
>  TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> +# Controls the output generated by Avocado when running tests.
> +# Any number of command separated loggers are accepted.  For more
> +# information please refer to "avocado --help".
> +AVOCADO_SHOW=none
>  
>  $(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
>  ifeq ($(.SHELLSTATUS),0)
> @@ -922,8 +928,19 @@ $(TESTS_VENV_DIR):
>  	$(error "venv directory for tests requires Python 3")
>  endif
>  
> +$(TESTS_RESULTS_DIR):
> +	$(call quiet-command, mkdir -p $@, \
> +            MKDIR, $@)
> +
>  check-venv: $(TESTS_VENV_DIR)
>  
> +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
> +	$(call quiet-command, \
> +            $(TESTS_VENV_DIR)/bin/python -m avocado \
> +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> +            --failfast=on $(SRC_PATH)/tests/acceptance, \
> +            "AVOCADO", "tests/acceptance")

Right now it seems like anything that is target-dependent should be
developed under qtest, rather than as an acceptance test.  On the other
hand this results in qtests that actually do not use the qtest protocol
at all---only the libqtest function to interact with QMP etc.---and as
such would look like a perfect match for "make check-acceptance".
drive_del-test is already an example of this.

How hard would it be to make it run tests for all supported targets,
rather than just the host architecture?  Perhaps using a tag such as
"all-targets".

Paolo

>  # Consolidated targets
>  
>  .PHONY: check-qapi-schema check-qtest check-unit check check-clean
> @@ -937,7 +954,7 @@ check-clean:
>  	rm -rf $(check-unit-y) tests/*.o $(QEMU_IOTESTS_HELPERS-y)
>  	rm -rf $(sort $(foreach target,$(SYSEMU_TARGET_LIST), $(check-qtest-$(target)-y)) $(check-qtest-generic-y))
>  	rm -f tests/test-qapi-gen-timestamp
> -	rm -rf $(TESTS_VENV_DIR)
> +	rm -rf $(TESTS_VENV_DIR) $(TESTS_RESULTS_DIR)
>  
>  clean: check-clean
>  
> 

Re: [Qemu-devel] [PULL 12/15] Acceptance tests: add make rule for running them

[Eduardo Habkost]

On Wed, Nov 07, 2018 at 12:24:55AM +0100, Paolo Bonzini wrote:
> On 31/10/2018 01:31, Eduardo Habkost wrote:
> > From: Cleber Rosa <crosa@redhat.com>
> > 
> > The acceptance (aka functional, aka Avocado-based) tests are
> > Python files located in "tests/acceptance" that need to be run
> > with the Avocado libs and test runner.
> > 
> > Let's provide a convenient way for QEMU developers to run them,
> > by making use of the tests-venv with the required setup.
> > 
> > Also, while the Avocado test runner will take care of creating a
> > location to save test results to, it was understood that it's better
> > if the results are kept within the build tree.
> > 
> > Signed-off-by: Cleber Rosa <crosa@redhat.com>
> > Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
> > Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
> > Reviewed-by: Caio Carrara <ccarrara@redhat.com>
> > Message-Id: <20181018153134.8493-3-crosa@redhat.com>
> > Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
> > ---
> >  docs/devel/testing.rst | 43 +++++++++++++++++++++++++++++++++++++-----
> >  tests/requirements.txt |  1 +
> >  tests/Makefile.include | 21 +++++++++++++++++++--
> >  3 files changed, 58 insertions(+), 7 deletions(-)
> > 
> > diff --git a/docs/devel/testing.rst b/docs/devel/testing.rst
> > index a227754f86..18e2c0868a 100644
> > --- a/docs/devel/testing.rst
> > +++ b/docs/devel/testing.rst
> > @@ -545,10 +545,39 @@ Tests based on ``avocado_qemu.Test`` can easily:
> >     - http://avocado-framework.readthedocs.io/en/latest/api/test/avocado.html#avocado.Test
> >     - http://avocado-framework.readthedocs.io/en/latest/api/utils/avocado.utils.html
> >  
> > -Installation
> > -------------
> > +Running tests
> > +-------------
> > +
> > +You can run the acceptance tests simply by executing:
> > +
> > +.. code::
> > +
> > +  make check-acceptance
> > +
> > +This involves the automatic creation of Python virtual environment
> > +within the build tree (at ``tests/venv``) which will have all the
> > +right dependencies, and will save tests results also within the
> > +build tree (at ``tests/results``).
> >  
> > -To install Avocado and its dependencies, run:
> > +Note: the build environment must be using a Python 3 stack, and have
> > +the ``venv`` and ``pip`` packages installed.  If necessary, make sure
> > +``configure`` is called with ``--python=`` and that those modules are
> > +available.  On Debian and Ubuntu based systems, depending on the
> > +specific version, they may be on packages named ``python3-venv`` and
> > +``python3-pip``.
> > +
> > +The scripts installed inside the virtual environment may be used
> > +without an "activation".  For instance, the Avocado test runner
> > +may be invoked by running:
> > +
> > + .. code::
> > +
> > +  tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
> > +
> > +Manual Installation
> > +-------------------
> > +
> > +To manually install Avocado and its dependencies, run:
> >  
> >  .. code::
> >  
> > @@ -689,11 +718,15 @@ The exact QEMU binary to be used on QEMUMachine.
> >  Uninstalling Avocado
> >  --------------------
> >  
> > -If you've followed the installation instructions above, you can easily
> > -uninstall Avocado.  Start by listing the packages you have installed::
> > +If you've followed the manual installation instructions above, you can
> > +easily uninstall Avocado.  Start by listing the packages you have
> > +installed::
> >  
> >    pip list --user
> >  
> >  And remove any package you want with::
> >  
> >    pip uninstall <package_name>
> > +
> > +If you've used ``make check-acceptance``, the Python virtual environment where
> > +Avocado is installed will be cleaned up as part of ``make check-clean``.
> > diff --git a/tests/requirements.txt b/tests/requirements.txt
> > index d39f9d1576..64c6e27a94 100644
> > --- a/tests/requirements.txt
> > +++ b/tests/requirements.txt
> > @@ -1,3 +1,4 @@
> >  # Add Python module requirements, one per line, to be installed
> >  # in the tests/venv Python virtual environment. For more info,
> >  # refer to: https://pip.pypa.io/en/stable/user_guide/#id1
> > +avocado-framework==65.0
> > diff --git a/tests/Makefile.include b/tests/Makefile.include
> > index eabc1da2f3..d2e577eabb 100644
> > --- a/tests/Makefile.include
> > +++ b/tests/Makefile.include
> > @@ -11,6 +11,7 @@ check-help:
> >  	@echo " $(MAKE) check-qapi-schema    Run QAPI schema tests"
> >  	@echo " $(MAKE) check-block          Run block tests"
> >  	@echo " $(MAKE) check-tcg            Run TCG tests"
> > +	@echo " $(MAKE) check-acceptance     Run all acceptance (functional) tests"
> >  	@echo " $(MAKE) check-report.html    Generates an HTML test report"
> >  	@echo " $(MAKE) check-venv           Creates a Python venv for tests"
> >  	@echo " $(MAKE) check-clean          Clean the tests"
> > @@ -902,10 +903,15 @@ check-decodetree:
> >  
> >  # Python venv for running tests
> >  
> > -.PHONY: check-venv
> > +.PHONY: check-venv check-acceptance
> >  
> >  TESTS_VENV_DIR=$(BUILD_DIR)/tests/venv
> >  TESTS_VENV_REQ=$(SRC_PATH)/tests/requirements.txt
> > +TESTS_RESULTS_DIR=$(BUILD_DIR)/tests/results
> > +# Controls the output generated by Avocado when running tests.
> > +# Any number of command separated loggers are accepted.  For more
> > +# information please refer to "avocado --help".
> > +AVOCADO_SHOW=none
> >  
> >  $(shell $(PYTHON) -c 'import sys; assert sys.version_info >= (3,0)' >/dev/null 2>&1)
> >  ifeq ($(.SHELLSTATUS),0)
> > @@ -922,8 +928,19 @@ $(TESTS_VENV_DIR):
> >  	$(error "venv directory for tests requires Python 3")
> >  endif
> >  
> > +$(TESTS_RESULTS_DIR):
> > +	$(call quiet-command, mkdir -p $@, \
> > +            MKDIR, $@)
> > +
> >  check-venv: $(TESTS_VENV_DIR)
> >  
> > +check-acceptance: check-venv $(TESTS_RESULTS_DIR)
> > +	$(call quiet-command, \
> > +            $(TESTS_VENV_DIR)/bin/python -m avocado \
> > +            --show=$(AVOCADO_SHOW) run --job-results-dir=$(TESTS_RESULTS_DIR) \
> > +            --failfast=on $(SRC_PATH)/tests/acceptance, \
> > +            "AVOCADO", "tests/acceptance")
> 
> Right now it seems like anything that is target-dependent should be
> developed under qtest, rather than as an acceptance test.  On the other
> hand this results in qtests that actually do not use the qtest protocol
> at all---only the libqtest function to interact with QMP etc.---and as
> such would look like a perfect match for "make check-acceptance".
> drive_del-test is already an example of this.
> 
> How hard would it be to make it run tests for all supported targets,
> rather than just the host architecture?  Perhaps using a tag such as
> "all-targets".

I shouldn't be difficult, and I think Cleber and Philippe are
working on it.  See
<https://trello.com/c/CmxAb30X/56-make-check-acceptance-a-way-to-test-all-architectures>
and <https://trello.com/c/tWhq0eXT/54-basic-multi-arch-support>.

-- 
Eduardo

[Qemu-devel] [PULL 13/15] Travis support for the acceptance tests

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

This enables the execution of the acceptance tests on Travis.

Because the Travis environment is based on Ubuntu Trusty, it requires
the python3-pip and python3.4-venv packages.

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Acked-by: Wainer dos Santos Moschetta <wainersm@redhat.com>
Reviewed-by: Caio Carrara <ccarrara@redhat.com>
Message-Id: <20181018153134.8493-4-crosa@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 .travis.yml | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/.travis.yml b/.travis.yml
index 95be6ec59f..aa49c7b114 100644
--- a/.travis.yml
+++ b/.travis.yml
@@ -117,6 +117,14 @@ matrix:
     - env: CONFIG="--target-list=x86_64-softmmu"
       python:
         - "3.6"
+    # Acceptance (Functional) tests
+    - env: CONFIG="--python=/usr/bin/python3 --target-list=x86_64-softmmu"
+           TEST_CMD="make AVOCADO_SHOW=app check-acceptance"
+      addons:
+        apt:
+          packages:
+            - python3-pip
+            - python3.4-venv
     # Using newer GCC with sanitizers
     - addons:
         apt:
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 14/15] scripts/decodetree.py: fix reference to attributes

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20181004161852.11673-9-crosa@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/decodetree.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/decodetree.py b/scripts/decodetree.py
index 457cffea90..37c76b5507 100755
--- a/scripts/decodetree.py
+++ b/scripts/decodetree.py
@@ -298,7 +298,7 @@ class Field:
             s = 's'
         else:
             s = ''
-        return str(pos) + ':' + s + str(len)
+        return str(self.pos) + ':' + s + str(self.len)
 
     def str_extract(self):
         if self.sign:
-- 
2.18.0.rc1.1.g3f1ff2140

[Qemu-devel] [PULL 15/15] scripts/qemu.py: use a more consistent docstring style

[Eduardo Habkost]

From: Cleber Rosa <crosa@redhat.com>

Signed-off-by: Cleber Rosa <crosa@redhat.com>
Message-Id: <20181004161852.11673-10-crosa@redhat.com>
[ehabkost: reverted unintentional submodule update]
Signed-off-by: Eduardo Habkost <ehabkost@redhat.com>
---
 scripts/qemu.py | 65 +++++++++++++++++++++++++++++++------------------
 1 file changed, 41 insertions(+), 24 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index 768611f1de..6e3b0e6771 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -59,9 +59,9 @@ class QEMUMachineAddDeviceError(QEMUMachineError):
     """
 
 class MonitorResponseError(qmp.qmp.QMPError):
-    '''
+    """
     Represents erroneous QMP monitor reply
-    '''
+    """
     def __init__(self, reply):
         try:
             desc = reply["error"]["desc"]
@@ -72,14 +72,15 @@ class MonitorResponseError(qmp.qmp.QMPError):
 
 
 class QEMUMachine(object):
-    '''A QEMU VM
+    """
+    A QEMU VM
 
     Use this object as a context manager to ensure the QEMU process terminates::
 
         with VM(binary) as vm:
             ...
         # vm is guaranteed to be shut down here
-    '''
+    """
 
     def __init__(self, binary, args=None, wrapper=None, name=None,
                  test_dir="/var/tmp", monitor_address=None,
@@ -141,7 +142,9 @@ class QEMUMachine(object):
         self._args.append(args)
 
     def add_fd(self, fd, fdset, opaque, opts=''):
-        '''Pass a file descriptor to the VM'''
+        """
+        Pass a file descriptor to the VM
+        """
         options = ['fd=%d' % fd,
                    'set=%d' % fdset,
                    'opaque=%s' % opaque]
@@ -197,7 +200,9 @@ class QEMUMachine(object):
 
     @staticmethod
     def _remove_if_exists(path):
-        '''Remove file object at path if it exists'''
+        """
+        Remove file object at path if it exists
+        """
         try:
             os.remove(path)
         except OSError as exception:
@@ -300,7 +305,9 @@ class QEMUMachine(object):
             raise
 
     def _launch(self):
-        '''Launch the VM and establish a QMP connection'''
+        """
+        Launch the VM and establish a QMP connection
+        """
         devnull = open(os.path.devnull, 'rb')
         self._pre_launch()
         self._qemu_full_args = (self._wrapper + [self._binary] +
@@ -314,14 +321,18 @@ class QEMUMachine(object):
         self._post_launch()
 
     def wait(self):
-        '''Wait for the VM to power off'''
+        """
+        Wait for the VM to power off
+        """
         self._popen.wait()
         self._qmp.close()
         self._load_io_log()
         self._post_shutdown()
 
     def shutdown(self):
-        '''Terminate the VM and clean up'''
+        """
+        Terminate the VM and clean up
+        """
         if self.is_running():
             try:
                 self._qmp.cmd('quit')
@@ -345,7 +356,9 @@ class QEMUMachine(object):
         self._launched = False
 
     def qmp(self, cmd, conv_keys=True, **args):
-        '''Invoke a QMP command and return the response dict'''
+        """
+        Invoke a QMP command and return the response dict
+        """
         qmp_args = dict()
         for key, value in args.items():
             if conv_keys:
@@ -356,11 +369,11 @@ class QEMUMachine(object):
         return self._qmp.cmd(cmd, args=qmp_args)
 
     def command(self, cmd, conv_keys=True, **args):
-        '''
+        """
         Invoke a QMP command.
         On success return the response dict.
         On failure raise an exception.
-        '''
+        """
         reply = self.qmp(cmd, conv_keys, **args)
         if reply is None:
             raise qmp.qmp.QMPError("Monitor is closed")
@@ -369,13 +382,17 @@ class QEMUMachine(object):
         return reply["return"]
 
     def get_qmp_event(self, wait=False):
-        '''Poll for one queued QMP events and return it'''
+        """
+        Poll for one queued QMP events and return it
+        """
         if len(self._events) > 0:
             return self._events.pop(0)
         return self._qmp.pull_event(wait=wait)
 
     def get_qmp_events(self, wait=False):
-        '''Poll for queued QMP events and return a list of dicts'''
+        """
+        Poll for queued QMP events and return a list of dicts
+        """
         events = self._qmp.get_events(wait=wait)
         events.extend(self._events)
         del self._events[:]
@@ -383,7 +400,7 @@ class QEMUMachine(object):
         return events
 
     def event_wait(self, name, timeout=60.0, match=None):
-        '''
+        """
         Wait for specified timeout on named event in QMP; optionally filter
         results by match.
 
@@ -391,7 +408,7 @@ class QEMUMachine(object):
         branch processing on match's value None
            {"foo": {"bar": 1}} matches {"foo": None}
            {"foo": {"bar": 1}} does not matches {"foo": {"baz": None}}
-        '''
+        """
         def event_match(event, match=None):
             if match is None:
                 return True
@@ -424,29 +441,29 @@ class QEMUMachine(object):
         return None
 
     def get_log(self):
-        '''
+        """
         After self.shutdown or failed qemu execution, this returns the output
         of the qemu process.
-        '''
+        """
         return self._iolog
 
     def add_args(self, *args):
-        '''
+        """
         Adds to the list of extra arguments to be given to the QEMU binary
-        '''
+        """
         self._args.extend(args)
 
     def set_machine(self, machine_type):
-        '''
+        """
         Sets the machine type
 
         If set, the machine type will be added to the base arguments
         of the resulting QEMU command line.
-        '''
+        """
         self._machine = machine_type
 
     def set_console(self, device_type=None):
-        '''
+        """
         Sets the device type for a console device
 
         If set, the console device and a backing character device will
@@ -464,7 +481,7 @@ class QEMUMachine(object):
         @param device_type: the device type, such as "isa-serial"
         @raises: QEMUMachineAddDeviceError if the device type is not given
                  and can not be determined.
-        '''
+        """
         if device_type is None:
             if self._machine is None:
                 raise QEMUMachineAddDeviceError("Can not add a console device:"
-- 
2.18.0.rc1.1.g3f1ff2140

Re: [Qemu-devel] [PULL 00/15] Python queue, 2018-10-30

[no-reply]

Hi,

This series seems to have some coding style problems. See output below for
more information:

Type: series
Message-id: 20181031003120.26771-1-ehabkost@redhat.com
Subject: [Qemu-devel] [PULL 00/15] Python queue, 2018-10-30

=== TEST SCRIPT BEGIN ===
#!/bin/bash

BASE=base
n=1
total=$(git log --oneline $BASE.. | wc -l)
failed=0

git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram

commits="$(git log --format=%H --reverse $BASE..)"
for c in $commits; do
    echo "Checking PATCH $n/$total: $(git log -n 1 --format=%s $c)..."
    if ! git show $c --format=email | ./scripts/checkpatch.pl --mailback -; then
        failed=1
        echo
    fi
    n=$((n+1))
done

exit $failed
=== TEST SCRIPT END ===

Updating 3c8cf5a9c21ff8782164d1def7f44bd888713384
Switched to a new branch 'test'
ecdea7d559 scripts/qemu.py: use a more consistent docstring style
6ddc1452f7 scripts/decodetree.py: fix reference to attributes
3ad025e8ee Travis support for the acceptance tests
1d3c9f27c6 Acceptance tests: add make rule for running them
c3960877a7 Bootstrap Python venv for tests
8813f0cf44 iotests: Unify log outputs between Python 2 and 3
fcfed8478f iotests: Modify imports for Python 3
c55c4c4e7f iotests: 'new' module replacement in 169
df208ab7f4 iotests: Explicitly bequeath FDs in Python
f1b456a621 iotests: Different iterator behavior in Python 3
67d5fdbb30 iotests: Use // for Python integer division
4240e24793 iotests: Use Python byte strings where appropriate
c3f2f4635d iotests: Flush in iotests.py's QemuIoInteractive
08dbcfb140 iotests: Make nbd-fault-injector flush
90484ef565 scripts/device-crash-test: Remove devices that are not user_creatable anymore

=== OUTPUT BEGIN ===
Checking PATCH 1/15: scripts/device-crash-test: Remove devices that are not user_creatable anymore...
Checking PATCH 2/15: iotests: Make nbd-fault-injector flush...
Checking PATCH 3/15: iotests: Flush in iotests.py's QemuIoInteractive...
Checking PATCH 4/15: iotests: Use Python byte strings where appropriate...
Checking PATCH 5/15: iotests: Use // for Python integer division...
ERROR: line over 90 characters
#236: FILE: tests/qemu-iotests/qed.py:83:
+        self.table_nelems = self.header['table_size'] * self.header['cluster_size'] // table_elem_size

total: 1 errors, 0 warnings, 179 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

Checking PATCH 6/15: iotests: Different iterator behavior in Python 3...
Checking PATCH 7/15: iotests: Explicitly bequeath FDs in Python...
Checking PATCH 8/15: iotests: 'new' module replacement in 169...
Checking PATCH 9/15: iotests: Modify imports for Python 3...
Checking PATCH 10/15: iotests: Unify log outputs between Python 2 and 3...
Checking PATCH 11/15: Bootstrap Python venv for tests...
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#80: 
new file mode 100644

total: 0 errors, 1 warnings, 47 lines checked

Your patch has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
Checking PATCH 12/15: Acceptance tests: add make rule for running them...
Checking PATCH 13/15: Travis support for the acceptance tests...
Checking PATCH 14/15: scripts/decodetree.py: fix reference to attributes...
Checking PATCH 15/15: scripts/qemu.py: use a more consistent docstring style...
=== OUTPUT END ===

Test command exited with code: 1


---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

Re: [Qemu-devel] [PULL 00/15] Python queue, 2018-10-30

[Peter Maydell]

On 31 October 2018 at 00:31, Eduardo Habkost <ehabkost@redhat.com> wrote:
> Sorry for submitting this at the last minute.
>
> The following changes since commit a2e002ff7913ce93aa0f7dbedd2123dce5f1a9cd:
>
>   Merge remote-tracking branch 'remotes/vivier2/tags/qemu-trivial-for-3.1-pull-request' into staging (2018-10-30 15:49:55 +0000)
>
> are available in the Git repository at:
>
>   git://github.com/ehabkost/qemu.git tags/python-next-pull-request
>
> for you to fetch changes up to e301e65c97142c4d2a2adf35f5a4fa73d65d8a4f:
>
>   scripts/qemu.py: use a more consistent docstring style (2018-10-30 21:13:54 -0300)
>
> ----------------------------------------------------------------
> Python queue, 2018-10-30
>
> * Makefile rule for running acceptance tests
>   (make check-acceptance) (Cleber Rosa)
> * Make iotests compatible with Python 3
>   (Max Reitz)
> * device-crash-test whitelist update (Thomas Huth)
> * Misc cleanups (Cleber Rosa)
>
> ----------------------------------------------------------------

Applied, thanks.

-- PMM