[Qemu-devel] [PATCH 0/9] iotests: Make them work for both Python 2 and 3

[Qemu-devel] [PATCH 0/9] iotests: Make them work for both Python 2 and 3

[Max Reitz]

This series prepares the iotests to work with both Python 2 and 3.  In
some places, it adds version-specific code and decides what to do based
on the version (for instance, whether to import the StringIO class from
the 'io' or the 'StringIO' module), but most of the time, it just makes
code work for both versions in general.

And when we make the switch to make Python 3 mandatory, we can simply
drop the Python 2 branches.


Max Reitz (9):
  iotests: Make nbd-fault-injector flush
  iotests: Flush in iotests.py's QemuIoInteractive
  iotests: Use Python byte strings where appropriate
  iotests: Use // for Python integer division
  iotests: Different iterator behavior in Python 3
  iotests: Explicitly inherit FDs in Python
  iotests: 'new' module replacement in 169
  iotests: Modify imports for Python 3
  iotests: Unify log outputs between Python 2 and 3

 scripts/qemu.py                          |  13 +-
 scripts/qmp/qmp.py                       |   7 +
 scripts/qtest.py                         |   2 +-
 tests/qemu-iotests/040                   |   4 +-
 tests/qemu-iotests/044                   |  20 +-
 tests/qemu-iotests/056                   |   2 +-
 tests/qemu-iotests/065                   |   4 +-
 tests/qemu-iotests/083.out               |   9 +
 tests/qemu-iotests/093                   |  18 +-
 tests/qemu-iotests/124                   |   4 +-
 tests/qemu-iotests/139                   |   2 +-
 tests/qemu-iotests/147                   |   7 +
 tests/qemu-iotests/149                   |  14 +-
 tests/qemu-iotests/151                   |  12 +-
 tests/qemu-iotests/163                   |   8 +-
 tests/qemu-iotests/169                   |   3 +-
 tests/qemu-iotests/194.out               |  22 +-
 tests/qemu-iotests/202.out               |  12 +-
 tests/qemu-iotests/203.out               |  14 +-
 tests/qemu-iotests/206.out               | 144 +++----
 tests/qemu-iotests/207                   |   4 +-
 tests/qemu-iotests/207.out               |  52 +--
 tests/qemu-iotests/208.out               |   8 +-
 tests/qemu-iotests/210.out               |  72 ++--
 tests/qemu-iotests/211.out               |  66 +--
 tests/qemu-iotests/212.out               | 102 ++---
 tests/qemu-iotests/213.out               | 124 +++---
 tests/qemu-iotests/216.out               |   4 +-
 tests/qemu-iotests/218.out               |  20 +-
 tests/qemu-iotests/219.out               | 526 +++++++++++------------
 tests/qemu-iotests/222.out               |  24 +-
 tests/qemu-iotests/iotests.py            |  64 ++-
 tests/qemu-iotests/nbd-fault-injector.py |  12 +-
 tests/qemu-iotests/qcow2.py              |  10 +-
 34 files changed, 745 insertions(+), 664 deletions(-)

-- 
2.17.1

[Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush

[Max Reitz]

When closing a connection, make the nbd-fault-injector flush the socket.
Without this, the output is a bit unreliable with Python 3.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/083.out               | 9 +++++++++
 tests/qemu-iotests/nbd-fault-injector.py | 1 +
 2 files changed, 10 insertions(+)

diff --git a/tests/qemu-iotests/083.out b/tests/qemu-iotests/083.out
index be6079d27e..f9af8bb691 100644
--- a/tests/qemu-iotests/083.out
+++ b/tests/qemu-iotests/083.out
@@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
 
 === Check disconnect after neg2 ===
 
+Unable to read from socket: Connection reset by peer
 Connection closed
 read failed: Input/output error
 
@@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
 
 === Check disconnect before request ===
 
+Unable to read from socket: Connection reset by peer
 Connection closed
 read failed: Input/output error
 
@@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/
 
 === Check disconnect after neg-classic ===
 
+Unable to read from socket: Connection reset by peer
 Connection closed
 read failed: Input/output error
 
@@ -161,6 +164,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
 
 === Check disconnect after neg2 ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
 read failed: Input/output error
 
 === Check disconnect 8 neg2 ===
@@ -173,6 +178,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
 
 === Check disconnect before request ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
 read failed: Input/output error
 
 === Check disconnect after request ===
@@ -234,6 +241,8 @@ can't open device nbd+unix:///?socket=TEST_DIR/nbd.sock
 
 === Check disconnect after neg-classic ===
 
+Unable to read from socket: Connection reset by peer
+Connection closed
 read failed: Input/output error
 
 *** done
diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
index f9193c0fae..439a090eb6 100755
--- a/tests/qemu-iotests/nbd-fault-injector.py
+++ b/tests/qemu-iotests/nbd-fault-injector.py
@@ -112,6 +112,7 @@ class FaultInjectionSocket(object):
             if rule.match(event, io):
                 if rule.when == 0 or bufsize is None:
                     print('Closing connection on rule match %s' % rule.name)
+                    self.sock.flush()
                     sys.exit(0)
                 if rule.when != -1:
                     return rule.when
-- 
2.17.1

[Qemu-devel] [PATCH 2/9] iotests: Flush in iotests.py's QemuIoInteractive

[Max Reitz]

After issuing a command, flush the pipe.  This does not change anything
in Python 2, but it makes a difference in Python 3.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 1 +
 1 file changed, 1 insertion(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 4e67fbbe96..10f2d17419 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -178,6 +178,7 @@ class QemuIoInteractive:
         cmd = cmd.strip()
         assert cmd != 'q' and cmd != 'quit'
         self._p.stdin.write(cmd + '\n')
+        self._p.stdin.flush()
         return self._read_output()
 
 
-- 
2.17.1

[Qemu-devel] [PATCH 3/9] iotests: Use Python byte strings where appropriate

[Max Reitz]

Since byte strings are no longer the default in Python 3, we have to
explicitly use them where we need to, which is mostly when working with
structures.  It also means that we need to open a file in binary mode
when we want to use structures.

On the other hand, we have to accomodate for the fact that some
functions (still) work with byte strings but we want to use unicode
strings (in Python 3 at least, and it does not matter in Python 2).
This includes base64 encoding, but it is most notable when working with
the subprocess module: Either we set univeral_newlines to True so that
the default streams are opened in text mode (hence this parameter is
aliased as "text" as of 3.7), or, if that is not possible, we have to
decode the output to a normal string.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 scripts/qtest.py                         |  2 +-
 tests/qemu-iotests/044                   |  8 ++++----
 tests/qemu-iotests/149                   |  8 +++++---
 tests/qemu-iotests/207                   |  4 ++--
 tests/qemu-iotests/iotests.py            | 11 +++++++----
 tests/qemu-iotests/nbd-fault-injector.py |  4 ++--
 tests/qemu-iotests/qcow2.py              | 10 +++++-----
 7 files changed, 26 insertions(+), 21 deletions(-)

diff --git a/scripts/qtest.py b/scripts/qtest.py
index df0daf26ca..adf1fe3f26 100644
--- a/scripts/qtest.py
+++ b/scripts/qtest.py
@@ -64,7 +64,7 @@ class QEMUQtestProtocol(object):
 
         @param qtest_cmd: qtest command text to be sent
         """
-        self._sock.sendall(qtest_cmd + "\n")
+        self._sock.sendall((qtest_cmd + "\n").encode('utf-8'))
 
     def close(self):
         self._sock.close()
diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 11ea0f4d35..69e736f687 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -53,21 +53,21 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             fd.seek(off_reftable)
 
             for i in xrange(0, h.refcount_table_clusters):
-                sector = ''.join(struct.pack('>Q',
+                sector = b''.join(struct.pack('>Q',
                     off_refblock + i * 64 * 512 + j * 512)
                     for j in xrange(0, 64))
                 fd.write(sector)
 
             # Write the refcount blocks
             assert(fd.tell() == off_refblock)
-            sector = ''.join(struct.pack('>H', 1) for j in xrange(0, 64 * 256))
+            sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256))
             for block in xrange(0, h.refcount_table_clusters):
                 fd.write(sector)
 
             # Write the L1 table
             assert(fd.tell() == off_l1)
             assert(off_l2 + 512 * h.l1_size == off_data)
-            table = ''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
+            table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
                 for j in xrange(0, h.l1_size))
             fd.write(table)
 
@@ -85,7 +85,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
                 remaining = remaining - 1024 * 512
                 off = off + 1024 * 512
 
-            table = ''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
+            table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
                 for j in xrange(0, remaining / 512))
             fd.write(table)
 
diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
index 9e0cad76f9..1225334cb8 100755
--- a/tests/qemu-iotests/149
+++ b/tests/qemu-iotests/149
@@ -79,7 +79,7 @@ class LUKSConfig(object):
 
     def first_password_base64(self):
         (pw, slot) = self.first_password()
-        return base64.b64encode(pw)
+        return base64.b64encode(pw.encode('ascii')).decode('ascii')
 
     def active_slots(self):
         slots = []
@@ -98,7 +98,8 @@ def verify_passwordless_sudo():
     proc = subprocess.Popen(args,
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
 
     msg = proc.communicate()[0]
 
@@ -116,7 +117,8 @@ def cryptsetup(args, password=None):
     proc = subprocess.Popen(fullargs,
                             stdin=subprocess.PIPE,
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
 
     msg = proc.communicate(password)[0]
 
diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
index 444ae233ae..2d86a3da37 100755
--- a/tests/qemu-iotests/207
+++ b/tests/qemu-iotests/207
@@ -109,7 +109,7 @@ with iotests.FilePath('t.img') as disk_path, \
     md5_key = subprocess.check_output(
         'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
         'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
-        shell=True).rstrip()
+        shell=True).rstrip().decode('ascii')
 
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
@@ -147,7 +147,7 @@ with iotests.FilePath('t.img') as disk_path, \
     sha1_key = subprocess.check_output(
         'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
         'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
-        shell=True).rstrip()
+        shell=True).rstrip().decode('ascii')
 
     vm.launch()
     blockdev_create(vm, { 'driver': 'ssh',
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 10f2d17419..7290c0b159 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -104,7 +104,8 @@ def qemu_img_pipe(*args):
     '''Run qemu-img and return its output'''
     subp = subprocess.Popen(qemu_img_args + list(args),
                             stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
     exitcode = subp.wait()
     if exitcode < 0:
         sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
@@ -128,7 +129,8 @@ def qemu_io(*args):
     '''Run qemu-io and return the stdout data'''
     args = qemu_io_args + list(args)
     subp = subprocess.Popen(args, stdout=subprocess.PIPE,
-                            stderr=subprocess.STDOUT)
+                            stderr=subprocess.STDOUT,
+                            universal_newlines=True)
     exitcode = subp.wait()
     if exitcode < 0:
         sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args)))
@@ -149,7 +151,8 @@ class QemuIoInteractive:
         self.args = qemu_io_args + list(args)
         self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE,
                                    stdout=subprocess.PIPE,
-                                   stderr=subprocess.STDOUT)
+                                   stderr=subprocess.STDOUT,
+                                   universal_newlines=True)
         assert self._p.stdout.read(9) == 'qemu-io> '
 
     def close(self):
@@ -193,7 +196,7 @@ def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
 
 def create_image(name, size):
     '''Create a fully-allocated raw image with sector markers'''
-    file = open(name, 'w')
+    file = open(name, 'wb')
     i = 0
     while i < size:
         sector = struct.pack('>l504xl', i / 512, i / 512)
diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
index 439a090eb6..d45e2e0a6a 100755
--- a/tests/qemu-iotests/nbd-fault-injector.py
+++ b/tests/qemu-iotests/nbd-fault-injector.py
@@ -86,7 +86,7 @@ def recvall(sock, bufsize):
             raise Exception('unexpected disconnect')
         chunks.append(chunk)
         received += len(chunk)
-    return ''.join(chunks)
+    return b''.join(chunks)
 
 class Rule(object):
     def __init__(self, name, event, io, when):
@@ -177,7 +177,7 @@ def handle_connection(conn, use_export):
         req = read_request(conn)
         if req.type == NBD_CMD_READ:
             write_reply(conn, 0, req.handle)
-            conn.send('\0' * req.len, event='data')
+            conn.send(b'\0' * req.len, event='data')
         elif req.type == NBD_CMD_WRITE:
             _ = conn.recv(req.len, event='data')
             write_reply(conn, 0, req.handle)
diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py
index b95a837759..b392972d1b 100755
--- a/tests/qemu-iotests/qcow2.py
+++ b/tests/qemu-iotests/qcow2.py
@@ -10,7 +10,7 @@ class QcowHeaderExtension:
     def __init__(self, magic, length, data):
         if length % 8 != 0:
             padding = 8 - (length % 8)
-            data += "\0" * padding
+            data += b"\0" * padding
 
         self.magic  = magic
         self.length = length
@@ -103,7 +103,7 @@ class QcowHeader:
 
         fd.seek(self.header_length)
         extensions = self.extensions
-        extensions.append(QcowHeaderExtension(0, 0, ""))
+        extensions.append(QcowHeaderExtension(0, 0, b""))
         for ex in extensions:
             buf = struct.pack('>II', ex.magic, ex.length)
             fd.write(buf)
@@ -137,8 +137,8 @@ class QcowHeader:
         for ex in self.extensions:
 
             data = ex.data[:ex.length]
-            if all(c in string.printable for c in data):
-                data = "'%s'" % data
+            if all(c in string.printable.encode('ascii') for c in data):
+                data = "'%s'" % data.decode('ascii')
             else:
                 data = "<binary>"
 
@@ -178,7 +178,7 @@ def cmd_add_header_ext(fd, magic, data):
         sys.exit(1)
 
     h = QcowHeader(fd)
-    h.extensions.append(QcowHeaderExtension.create(magic, data))
+    h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii')))
     h.update(fd)
 
 def cmd_add_header_ext_stdio(fd, magic):
-- 
2.17.1

[Qemu-devel] [PATCH 4/9] iotests: Use // for Python integer division

[Max Reitz]

In Python 3, / is always a floating-point division.  We usually do not
want this, and as Python 2.7 understands // as well, change all integer
divisions to use that.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/040        |  4 ++--
 tests/qemu-iotests/044        |  2 +-
 tests/qemu-iotests/093        | 18 +++++++++---------
 tests/qemu-iotests/149        |  6 +++---
 tests/qemu-iotests/151        | 12 ++++++------
 tests/qemu-iotests/163        |  2 +-
 tests/qemu-iotests/iotests.py |  2 +-
 7 files changed, 23 insertions(+), 23 deletions(-)

diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index 1cb1ceeb33..b81133a474 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase):
 
         self.assert_no_active_block_jobs()
         result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
-                             base=backing_img, speed=(self.image_len / 4))
+                             base=backing_img, speed=(self.image_len // 4))
         self.assert_qmp(result, 'return', {})
         result = self.vm.qmp('device_del', id='scsi0')
         self.assert_qmp(result, 'return', {})
@@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase):
 
         self.assert_no_active_block_jobs()
         result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
-                             base=backing_img, speed=(self.image_len / 4))
+                             base=backing_img, speed=(self.image_len // 4))
         self.assert_qmp(result, 'return', {})
 
         result = self.vm.qmp('query-block')
diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 69e736f687..7ef5e46fe9 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
                 off = off + 1024 * 512
 
             table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
-                for j in xrange(0, remaining / 512))
+                for j in xrange(0, remaining // 512))
             fd.write(table)
 
 
diff --git a/tests/qemu-iotests/093 b/tests/qemu-iotests/093
index 9d1971a56c..d88fbc182e 100755
--- a/tests/qemu-iotests/093
+++ b/tests/qemu-iotests/093
@@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase):
         # in. The throttled requests won't be executed until we
         # advance the virtual clock.
         rq_size = 512
-        rd_nr = max(params['bps'] / rq_size / 2,
-                    params['bps_rd'] / rq_size,
-                    params['iops'] / 2,
+        rd_nr = max(params['bps'] // rq_size // 2,
+                    params['bps_rd'] // rq_size,
+                    params['iops'] // 2,
                     params['iops_rd'])
         rd_nr *= seconds * 2
-        rd_nr /= ndrives
-        wr_nr = max(params['bps'] / rq_size / 2,
-                    params['bps_wr'] / rq_size,
-                    params['iops'] / 2,
+        rd_nr //= ndrives
+        wr_nr = max(params['bps'] // rq_size // 2,
+                    params['bps_wr'] // rq_size,
+                    params['iops'] // 2,
                     params['iops_wr'])
         wr_nr *= seconds * 2
-        wr_nr /= ndrives
+        wr_nr //= ndrives
 
         # Send I/O requests to all drives
         for i in range(rd_nr):
@@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
             self.configure_throttle(ndrives, settings)
 
             # Wait for the bucket to empty so we can do bursts
-            wait_ns = nsec_per_sec * burst_length * burst_rate / rate
+            wait_ns = nsec_per_sec * burst_length * burst_rate // rate
             self.vm.qtest("clock_step %d" % wait_ns)
 
             # Test I/O at the max burst rate
diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
index 1225334cb8..4f363f295f 100755
--- a/tests/qemu-iotests/149
+++ b/tests/qemu-iotests/149
@@ -314,13 +314,13 @@ def test_once(config, qemu_img=False):
     image_size = 4 * oneTB
     if qemu_img:
         iotests.log("# Create image")
-        qemu_img_create(config, image_size / oneMB)
+        qemu_img_create(config, image_size // oneMB)
     else:
         iotests.log("# Create image")
-        create_image(config, image_size / oneMB)
+        create_image(config, image_size // oneMB)
 
     lowOffsetMB = 100
-    highOffsetMB = 3 * oneTB / oneMB
+    highOffsetMB = 3 * oneTB // oneMB
 
     try:
         if not qemu_img:
diff --git a/tests/qemu-iotests/151 b/tests/qemu-iotests/151
index fe53b9f446..1bb74d67c4 100755
--- a/tests/qemu-iotests/151
+++ b/tests/qemu-iotests/151
@@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase):
                             'write -P 1 0 %i' % self.image_len);
 
         # Start some background requests
-        for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
+        for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset)
-        for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
+        for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         # Start the block job
@@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase):
         self.assert_qmp(result, 'return', {})
 
         # Start some more requests
-        for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
+        for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
-        for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
+        for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         # Wait for the READY event
@@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase):
         # the source) should be settled using the active mechanism.
         # The mirror code itself asserts that the source BDS's dirty
         # bitmap will stay clean between READY and COMPLETED.
-        for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
+        for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
-        for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
+        for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
             self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
 
         if sync_source_and_target:
diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
index 403842354e..5fd424761b 100755
--- a/tests/qemu-iotests/163
+++ b/tests/qemu-iotests/163
@@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         entry_bits = 3
         entry_size = 1 << entry_bits
         l1_mask = 0x00fffffffffffe00
-        div_roundup = lambda n, d: (n + d - 1) / d
+        div_roundup = lambda n, d: (n + d - 1) // d
 
         def split_by_n(data, n):
             for x in xrange(0, len(data), n):
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 7290c0b159..7ca94e9278 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -199,7 +199,7 @@ def create_image(name, size):
     file = open(name, 'wb')
     i = 0
     while i < size:
-        sector = struct.pack('>l504xl', i / 512, i / 512)
+        sector = struct.pack('>l504xl', i // 512, i // 512)
         file.write(sector)
         i = i + 512
     file.close()
-- 
2.17.1

[Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3

[Max Reitz]

In Python 3, several functions now return iterators instead of lists.
This includes range(), items(), map(), and filter().  This means that if
we really want a list, we have to wrap those instances with list().  On
the other hand, sometimes we do just want an iterator, in which case we
have sometimes used xrange() and iteritems() which no longer exist in
Python 3.  Just change these calls to be range() and items(), which
costs a bit of performance in Python 2, but will do the right thing in
Python 3 (which is what is important).

In one instance, we only wanted the first instance of the result of a
filter() call.  Instead of using next(filter()) which would work only in
Python 3, or list(filter())[0] which would work everywhere but is a bit
weird, this instance is changed to a single-line for with next() wrapped
around, which works both in 2.7 and 3.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/044 | 12 ++++++------
 tests/qemu-iotests/056 |  2 +-
 tests/qemu-iotests/065 |  4 ++--
 tests/qemu-iotests/124 |  4 ++--
 tests/qemu-iotests/139 |  2 +-
 tests/qemu-iotests/163 |  6 +++---
 6 files changed, 15 insertions(+), 15 deletions(-)

diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 7ef5e46fe9..e2d6c9b189 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -52,23 +52,23 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             # Write a refcount table
             fd.seek(off_reftable)
 
-            for i in xrange(0, h.refcount_table_clusters):
+            for i in range(0, h.refcount_table_clusters):
                 sector = b''.join(struct.pack('>Q',
                     off_refblock + i * 64 * 512 + j * 512)
-                    for j in xrange(0, 64))
+                    for j in range(0, 64))
                 fd.write(sector)
 
             # Write the refcount blocks
             assert(fd.tell() == off_refblock)
             sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256))
-            for block in xrange(0, h.refcount_table_clusters):
+            for block in range(0, h.refcount_table_clusters):
                 fd.write(sector)
 
             # Write the L1 table
             assert(fd.tell() == off_l1)
             assert(off_l2 + 512 * h.l1_size == off_data)
             table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
-                for j in xrange(0, h.l1_size))
+                for j in range(0, h.l1_size))
             fd.write(table)
 
             # Write the L2 tables
@@ -79,14 +79,14 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
             off = off_data
             while remaining > 1024 * 512:
                 pytable = list((1 << 63) | off + 512 * j
-                    for j in xrange(0, 1024))
+                    for j in range(0, 1024))
                 table = struct.pack('>1024Q', *pytable)
                 fd.write(table)
                 remaining = remaining - 1024 * 512
                 off = off + 1024 * 512
 
             table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
-                for j in xrange(0, remaining // 512))
+                for j in range(0, remaining // 512))
             fd.write(table)
 
 
diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
index 223292175a..3df323984d 100755
--- a/tests/qemu-iotests/056
+++ b/tests/qemu-iotests/056
@@ -32,7 +32,7 @@ target_img = os.path.join(iotests.test_dir, 'target.img')
 def img_create(img, fmt=iotests.imgfmt, size='64M', **kwargs):
     fullname = os.path.join(iotests.test_dir, '%s.%s' % (img, fmt))
     optargs = []
-    for k,v in kwargs.iteritems():
+    for k,v in kwargs.items():
         optargs = optargs + ['-o', '%s=%s' % (k,v)]
     args = ['create', '-f', fmt] + optargs + [fullname, size]
     iotests.qemu_img(*args)
diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 72aa9707c7..a339bf6069 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific):
                     :data.index('')]
         for field in data:
             self.assertTrue(re.match('^ {4}[^ ]', field) is not None)
-        data = map(lambda line: line.strip(), data)
+        data = list(map(lambda line: line.strip(), data))
         self.assertEqual(data, self.human_compare)
 
 class TestQMP(TestImageInfoSpecific):
@@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific):
 
     def test_qmp(self):
         result = self.vm.qmp('query-block')['return']
-        drive = filter(lambda drive: drive['device'] == 'drive0', result)[0]
+        drive = next(drive for drive in result if drive['device'] == 'drive0')
         data = drive['inserted']['image']['format-specific']
         self.assertEqual(data['type'], iotests.imgfmt)
         self.assertEqual(data['data'], self.compare)
diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
index 3ea4ac53f5..9f189e3b54 100755
--- a/tests/qemu-iotests/124
+++ b/tests/qemu-iotests/124
@@ -39,7 +39,7 @@ def try_remove(img):
 def transaction_action(action, **kwargs):
     return {
         'type': action,
-        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems())
+        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items())
     }
 
 
@@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase):
     def img_create(self, img, fmt=iotests.imgfmt, size='64M',
                    parent=None, parentFormat=None, **kwargs):
         optargs = []
-        for k,v in kwargs.iteritems():
+        for k,v in kwargs.items():
             optargs = optargs + ['-o', '%s=%s' % (k,v)]
         args = ['create', '-f', fmt] + optargs + [img, size]
         if parent:
diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
index cc7fe337f3..e00f10b8c8 100755
--- a/tests/qemu-iotests/139
+++ b/tests/qemu-iotests/139
@@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
     # Check whether a BlockDriverState exists
     def checkBlockDriverState(self, node, must_exist = True):
         result = self.vm.qmp('query-named-block-nodes')
-        nodes = filter(lambda x: x['node-name'] == node, result['return'])
+        nodes = list(filter(lambda x: x['node-name'] == node, result['return']))
         self.assertLessEqual(len(nodes), 1)
         self.assertEqual(must_exist, len(nodes) == 1)
 
diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
index 5fd424761b..35c1a2bafc 100755
--- a/tests/qemu-iotests/163
+++ b/tests/qemu-iotests/163
@@ -41,7 +41,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         div_roundup = lambda n, d: (n + d - 1) // d
 
         def split_by_n(data, n):
-            for x in xrange(0, len(data), n):
+            for x in range(0, len(data), n):
                 yield struct.unpack('>Q', data[x:x + n])[0] & l1_mask
 
         def check_l1_table(h, l1_data):
@@ -135,8 +135,8 @@ class ShrinkBaseClass(iotests.QMPTestCase):
         self.image_verify()
 
     def test_random_write(self):
-        offs_list = range(0, size_to_int(self.image_len),
-                          size_to_int(self.chunk_size))
+        offs_list = list(range(0, size_to_int(self.image_len),
+                               size_to_int(self.chunk_size)))
         random.shuffle(offs_list)
         for offs in offs_list:
             qemu_io('-c', 'write -P 0xff %d %s' % (offs, self.chunk_size),
-- 
2.17.1

[Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python

[Max Reitz]

Python 3.2 introduced the inheritable attribute for FDs.  At the same
time, it changed the default so that all FDs are not inheritable by
default, that only inheritable FDs are inherited to subprocesses, and
only if close_fds is explicitly set to False.

Adhere to this by setting close_fds to False when working with
subprocesses that may want to inherit FDs, and by trying to
set_inheritable() on FDs that we do want to bequeath to them.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 scripts/qemu.py        | 13 +++++++++++--
 scripts/qmp/qmp.py     |  7 +++++++
 tests/qemu-iotests/147 |  7 +++++++
 3 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/scripts/qemu.py b/scripts/qemu.py
index f099ce7278..28366c4a67 100644
--- a/scripts/qemu.py
+++ b/scripts/qemu.py
@@ -142,10 +142,18 @@ class QEMUMachine(object):
         if opts:
             options.append(opts)
 
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(fd, True)
+        except AttributeError:
+            pass
+
         self._args.append('-add-fd')
         self._args.append(','.join(options))
         return self
 
+    # The caller needs to make sure the FD is inheritable
     def send_fd_scm(self, fd_file_path):
         # In iotest.py, the qmp should always use unix socket.
         assert self._qmp.is_scm_available()
@@ -159,7 +167,7 @@ class QEMUMachine(object):
                     "%s" % fd_file_path]
         devnull = open(os.path.devnull, 'rb')
         proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
-                                stderr=subprocess.STDOUT)
+                                stderr=subprocess.STDOUT, close_fds=False)
         output = proc.communicate()[0]
         if output:
             LOG.debug(output)
@@ -280,7 +288,8 @@ class QEMUMachine(object):
                                        stdin=devnull,
                                        stdout=self._qemu_log_file,
                                        stderr=subprocess.STDOUT,
-                                       shell=False)
+                                       shell=False,
+                                       close_fds=False)
         self._post_launch()
 
     def wait(self):
diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
index 5c8cf6a056..009be8345b 100644
--- a/scripts/qmp/qmp.py
+++ b/scripts/qmp/qmp.py
@@ -10,6 +10,7 @@
 
 import json
 import errno
+import os
 import socket
 import logging
 
@@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object):
         return self.__sock.fileno()
 
     def is_scm_available(self):
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(self.get_sock_fd(), True)
+        except AttributeError:
+            pass
         return self.__sock.family == socket.AF_UNIX
diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
index d2081df84b..b58455645b 100755
--- a/tests/qemu-iotests/147
+++ b/tests/qemu-iotests/147
@@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase):
         sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
         sockfd.connect(unix_socket)
 
+        # This did not exist before 3.2, but since then it is
+        # mandatory for our purpose
+        try:
+            os.set_inheritable(sockfd.fileno(), True)
+        except AttributeError:
+            pass
+
         result = self.vm.send_fd_scm(str(sockfd.fileno()))
         self.assertEqual(result, 0, 'Failed to send socket FD')
 
-- 
2.17.1

[Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

[Max Reitz]

iotest 169 uses the 'new' module to add methods to a class.  This module
no longer exists in Python 3.  Instead, we can use a lambda.  Best of
all, this works in 2.7 just as well.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/169 | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
index f243db9955..e5614b159d 100755
--- a/tests/qemu-iotests/169
+++ b/tests/qemu-iotests/169
@@ -23,7 +23,6 @@ import iotests
 import time
 import itertools
 import operator
-import new
 from iotests import qemu_img
 
 
@@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
 
 def inject_test_case(klass, name, method, *args, **kwargs):
     mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
+    setattr(klass, 'test_' + name, lambda self: mc(self))
 
 for cmb in list(itertools.product((True, False), repeat=4)):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
-- 
2.17.1

[Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

[Max Reitz]

There are two imports that need to be modified when running the iotests
under Python 3: One is StringIO, which no longer exists; instead, the
StringIO class comes from the io module, so import it from there.  The
other is the ConfigParser, which has just been renamed to configparser.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py            | 8 ++++++--
 tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
 2 files changed, 11 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 7ca94e9278..a64ea90fb4 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
 
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
-    import StringIO
+    if sys.version_info.major >= 3:
+        from io import StringIO
+    else:
+        from StringIO import StringIO
+
     if debug:
         output = sys.stdout
         verbosity = 2
         sys.argv.remove('-d')
     else:
-        output = StringIO.StringIO()
+        output = StringIO()
 
     logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
 
diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
index d45e2e0a6a..6b2d659dee 100755
--- a/tests/qemu-iotests/nbd-fault-injector.py
+++ b/tests/qemu-iotests/nbd-fault-injector.py
@@ -48,7 +48,10 @@ import sys
 import socket
 import struct
 import collections
-import ConfigParser
+if sys.version_info.major >= 3:
+    import configparser
+else:
+    import ConfigParser as configparser
 
 FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
 
@@ -225,7 +228,7 @@ def parse_config(config):
     return rules
 
 def load_rules(filename):
-    config = ConfigParser.RawConfigParser()
+    config = configparser.RawConfigParser()
     with open(filename, 'rt') as f:
         config.readfp(f, filename)
     return parse_config(config)
-- 
2.17.1

[Qemu-devel] [PATCH 9/9] iotests: Unify log outputs between Python 2 and 3

[Max Reitz]

When dumping an object into the log, there are differences between
Python 2 and 3.  First, unicode strings are prefixed by 'u' in Python 2
(they are no longer in 3, because unicode strings are the default
there).  Second, the order of keys in dicts may differ.  Third,
especially long numbers are longs in Python 2 and thus get an 'L'
suffix, which does not happen in Python 3.

To get around these differences, this patch introduces functions to
convert an object to a string that looks the same regardless of the
Python version: In Python 2, they decode unicode strings to byte strings
(normal str); in Python 3, they encode byte strings to unicode strings
(normal str).  They also manually convert lists and dicts to strings,
which allows sorting the dicts by key, so we are no longer at the mercy
of the internal implementation when it comes to how the keys appear in
the output.

This changes the output of all tests that use these logging functions.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/194.out    |  22 +-
 tests/qemu-iotests/202.out    |  12 +-
 tests/qemu-iotests/203.out    |  14 +-
 tests/qemu-iotests/206.out    | 144 +++++-----
 tests/qemu-iotests/207.out    |  52 ++--
 tests/qemu-iotests/208.out    |   8 +-
 tests/qemu-iotests/210.out    |  72 ++---
 tests/qemu-iotests/211.out    |  66 ++---
 tests/qemu-iotests/212.out    | 102 +++----
 tests/qemu-iotests/213.out    | 124 ++++----
 tests/qemu-iotests/216.out    |   4 +-
 tests/qemu-iotests/218.out    |  20 +-
 tests/qemu-iotests/219.out    | 526 +++++++++++++++++-----------------
 tests/qemu-iotests/222.out    |  24 +-
 tests/qemu-iotests/iotests.py |  42 ++-
 15 files changed, 634 insertions(+), 598 deletions(-)

diff --git a/tests/qemu-iotests/194.out b/tests/qemu-iotests/194.out
index 50ac50da5e..722af888f6 100644
--- a/tests/qemu-iotests/194.out
+++ b/tests/qemu-iotests/194.out
@@ -1,18 +1,18 @@
 Launching VMs...
 Launching NBD server on destination...
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 Starting `drive-mirror` on source...
-{u'return': {}}
+{'return': {}}
 Waiting for `drive-mirror` to complete...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_READY'}
+{'data': {'device': 'mirror-job0', 'len': 1073741824, 'offset': 1073741824, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_READY', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 Starting migration...
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'}
+{'return': {}}
+{'data': {'status': 'setup'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'status': 'active'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'status': 'completed'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 Gracefully ending the `drive-mirror` job on source...
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror-job0', u'type': u'mirror', u'speed': 0, u'len': 1073741824, u'offset': 1073741824}, u'event': u'BLOCK_JOB_COMPLETED'}
+{'return': {}}
+{'data': {'device': 'mirror-job0', 'len': 1073741824, 'offset': 1073741824, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_COMPLETED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 Stopping the NBD server on destination...
-{u'return': {}}
+{'return': {}}
diff --git a/tests/qemu-iotests/202.out b/tests/qemu-iotests/202.out
index d5ea374e17..d133f09fe2 100644
--- a/tests/qemu-iotests/202.out
+++ b/tests/qemu-iotests/202.out
@@ -1,11 +1,11 @@
 Launching VM...
 Adding IOThread...
-{u'return': {}}
+{'return': {}}
 Adding blockdevs...
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 Setting iothread...
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 Creating external snapshots...
-{u'return': {}}
+{'return': {}}
diff --git a/tests/qemu-iotests/203.out b/tests/qemu-iotests/203.out
index 1a11f0975c..b156388a0b 100644
--- a/tests/qemu-iotests/203.out
+++ b/tests/qemu-iotests/203.out
@@ -1,11 +1,11 @@
 Launching VM...
 Setting IOThreads...
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 Enabling migration QMP events...
-{u'return': {}}
+{'return': {}}
 Starting migration...
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'setup'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'active'}, u'event': u'MIGRATION'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'completed'}, u'event': u'MIGRATION'}
+{'return': {}}
+{'data': {'status': 'setup'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'status': 'active'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'status': 'completed'}, 'event': 'MIGRATION', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
diff --git a/tests/qemu-iotests/206.out b/tests/qemu-iotests/206.out
index 789eebe57b..8fb3228cac 100644
--- a/tests/qemu-iotests/206.out
+++ b/tests/qemu-iotests/206.out
@@ -1,16 +1,16 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-add', 'arguments': {'node_name': 'imgfile', 'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}}
-{u'return': {}}
+{'execute': 'blockdev-add', 'arguments': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2', 'node_name': 'imgfile'}}
+{'return': {}}
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'imgfile', 'size': 134217728}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -24,15 +24,15 @@ Format specific information:
 
 === Successful image creation (inline blockdev-add, explicit defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'nocow': False, 'preallocation': 'off', 'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2', 'nocow': False, 'preallocation': 'off', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 65536, 'refcount-bits': 16, 'version': 'v3', 'preallocation': 'off', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'lazy-refcounts': False, 'driver': 'qcow2', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 65536, 'driver': 'qcow2', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'lazy-refcounts': False, 'preallocation': 'off', 'refcount-bits': 16, 'size': 67108864, 'version': 'v3'}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -46,15 +46,15 @@ Format specific information:
 
 === Successful image creation (v3 non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'nocow': True, 'preallocation': 'falloc', 'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2', 'nocow': True, 'preallocation': 'falloc', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 2097152, 'refcount-bits': 1, 'version': 'v3', 'preallocation': 'metadata', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'lazy-refcounts': True, 'driver': 'qcow2', 'size': 33554432}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 2097152, 'driver': 'qcow2', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'lazy-refcounts': True, 'preallocation': 'metadata', 'refcount-bits': 1, 'size': 33554432, 'version': 'v3'}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -68,15 +68,15 @@ Format specific information:
 
 === Successful image creation (v2 non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 512, 'backing-fmt': 'qcow2', 'driver': 'qcow2', 'version': 'v2', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'backing-file': 'TEST_DIR/PID-t.qcow2.base', 'size': 33554432}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'backing-file': 'TEST_DIR/PID-t.qcow2.base', 'backing-fmt': 'qcow2', 'cluster-size': 512, 'driver': 'qcow2', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'size': 33554432, 'version': 'v2'}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -90,10 +90,10 @@ Format specific information:
 
 === Successful image creation (encrypted) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'encrypt': {'key-secret': 'keysec0', 'iter-time': 10, 'cipher-mode': 'ctr', 'ivgen-hash-alg': 'md5', 'cipher-alg': 'twofish-128', 'format': 'luks', 'ivgen-alg': 'plain64', 'hash-alg': 'sha1'}, 'driver': 'qcow2', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'size': 33554432}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'encrypt': {'cipher-alg': 'twofish-128', 'cipher-mode': 'ctr', 'format': 'luks', 'hash-alg': 'sha1', 'iter-time': 10, 'ivgen-alg': 'plain64', 'ivgen-hash-alg': 'md5', 'key-secret': 'keysec0'}, 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.qcow2'}, 'size': 33554432}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -145,112 +145,112 @@ Format specific information:
 === Invalid BlockdevRef ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': "this doesn't exist", 'size': 33554432}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Invalid sizes ===
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 1234}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Image size must be a multiple of 512 bytes
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 18446744073709551104L}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 18446744073709551104}}}
+{'return': {}}
 Job failed: Could not resize image: Image size cannot be negative
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 9223372036854775808L}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 9223372036854775808}}}
+{'return': {}}
 Job failed: Could not resize image: Image size cannot be negative
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 9223372036854775296}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Could not resize image: Failed to grow the L1 table: File too large
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Invalid version ===
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'version': 'v1', 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'error': {u'class': u'GenericError', u'desc': u"Invalid parameter 'v1'"}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'size': 67108864, 'version': 'v1'}}}
+{'error': {'class': 'GenericError', 'desc': "Invalid parameter 'v1'"}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'lazy-refcounts': True, 'version': 'v2', 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'lazy-refcounts': True, 'size': 67108864, 'version': 'v2'}}}
+{'return': {}}
 Job failed: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater)
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'refcount-bits': 8, 'version': 'v2', 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'refcount-bits': 8, 'size': 67108864, 'version': 'v2'}}}
+{'return': {}}
 Job failed: Different refcount widths than 16 bits require compatibility level 1.1 or above (use version=v3 or greater)
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Invalid backing file options ===
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'preallocation': 'full', 'driver': 'qcow2', 'backing-file': '/dev/null', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'backing-file': '/dev/null', 'driver': 'qcow2', 'file': 'node0', 'preallocation': 'full', 'size': 67108864}}}
+{'return': {}}
 Job failed: Backing file and preallocation cannot be used at the same time
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'backing-fmt': 'qcow2', 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Backing format cannot be used without backing file
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Invalid cluster size ===
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 1234, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 128, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 4194304, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 0, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cluster size must be a power of two between 512 and 2048k
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 512, 'driver': 'qcow2', 'file': 'node0', 'size': 281474976710656}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Could not resize image: Failed to grow the L1 table: File too large
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Invalid refcount width ===
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'refcount-bits': 128, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'refcount-bits': 128, 'size': 67108864}}}
+{'return': {}}
 Job failed: Refcount width must be a power of two and may not exceed 64 bits
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'refcount-bits': 0, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'refcount-bits': 0, 'size': 67108864}}}
+{'return': {}}
 Job failed: Refcount width must be a power of two and may not exceed 64 bits
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'refcount-bits': 7, 'driver': 'qcow2', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'qcow2', 'file': 'node0', 'refcount-bits': 7, 'size': 67108864}}}
+{'return': {}}
 Job failed: Refcount width must be a power of two and may not exceed 64 bits
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
diff --git a/tests/qemu-iotests/207.out b/tests/qemu-iotests/207.out
index 078b7e63cb..e18ff4e26a 100644
--- a/tests/qemu-iotests/207.out
+++ b/tests/qemu-iotests/207.out
@@ -1,9 +1,9 @@
 === Successful image creation (defaults) ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_IMG"}}
 file format: IMGFMT
@@ -16,49 +16,49 @@ virtual size: 4.0M (4194304 bytes)
 
 === Test host-key-check options ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'mode': 'none'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 8388608}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'host-key-check': {'mode': 'none'}, 'path': 'TEST_DIR/PID-t.img', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 8388608}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_IMG"}}
 file format: IMGFMT
 virtual size: 8.0M (8388608 bytes)
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'mode': 'known_hosts'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'host-key-check': {'mode': 'known_hosts'}, 'path': 'TEST_DIR/PID-t.img', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_IMG"}}
 file format: IMGFMT
 virtual size: 4.0M (4194304 bytes)
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'hash': 'wrong', 'type': 'md5', 'mode': 'hash'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 2097152}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'host-key-check': {'hash': 'wrong', 'mode': 'hash', 'type': 'md5'}, 'path': 'TEST_DIR/PID-t.img', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 2097152}}}
+{'return': {}}
 Job failed: remote host key does not match host_key_check 'wrong'
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'hash': HASH, 'type': 'md5', 'mode': 'hash'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 8388608}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'host-key-check': {'hash': HASH, 'mode': 'hash', 'type': 'md5'}, 'path': 'TEST_DIR/PID-t.img', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 8388608}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_IMG"}}
 file format: IMGFMT
 virtual size: 8.0M (8388608 bytes)
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'hash': 'wrong', 'type': 'sha1', 'mode': 'hash'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 2097152}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'host-key-check': {'hash': 'wrong', 'mode': 'hash', 'type': 'sha1'}, 'path': 'TEST_DIR/PID-t.img', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 2097152}}}
+{'return': {}}
 Job failed: remote host key does not match host_key_check 'wrong'
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'hash': HASH, 'type': 'sha1', 'mode': 'hash'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'host-key-check': {'hash': HASH, 'mode': 'hash', 'type': 'sha1'}, 'path': 'TEST_DIR/PID-t.img', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: json:{"driver": "IMGFMT", "file": {"server.host": "127.0.0.1", "server.port": "22", "driver": "ssh", "path": "TEST_IMG"}}
 file format: IMGFMT
@@ -66,15 +66,15 @@ virtual size: 4.0M (4194304 bytes)
 
 === Invalid path and user ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': '/this/is/not/an/existing/path', 'host-key-check': {'mode': 'none'}, 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'host-key-check': {'mode': 'none'}, 'path': '/this/is/not/an/existing/path', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
+{'return': {}}
 Job failed: failed to open remote file '/this/is/not/an/existing/path': Failed opening remote file (libssh2 error code: -31)
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'path': 'TEST_DIR/PID-t.img', 'host-key-check': {'mode': 'none'}, 'user': 'invalid user', 'server': {'host': '127.0.0.1', 'port': '22'}}, 'size': 4194304}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'ssh', 'location': {'host-key-check': {'mode': 'none'}, 'path': 'TEST_DIR/PID-t.img', 'server': {'host': '127.0.0.1', 'port': '22'}, 'user': 'invalid user'}, 'size': 4194304}}}
+{'return': {}}
 Job failed: failed to authenticate using publickey authentication and the identities held by your ssh-agent
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
diff --git a/tests/qemu-iotests/208.out b/tests/qemu-iotests/208.out
index 3687e9d0dd..38c26457eb 100644
--- a/tests/qemu-iotests/208.out
+++ b/tests/qemu-iotests/208.out
@@ -1,9 +1,9 @@
 Launching VM...
 Starting NBD server...
-{u'return': {}}
+{'return': {}}
 Adding NBD export...
-{u'return': {}}
+{'return': {}}
 Creating external snapshot...
-{u'return': {}}
+{'return': {}}
 Stopping NBD server...
-{u'return': {}}
+{'return': {}}
diff --git a/tests/qemu-iotests/210.out b/tests/qemu-iotests/210.out
index 078ba544a1..49a8dcf268 100644
--- a/tests/qemu-iotests/210.out
+++ b/tests/qemu-iotests/210.out
@@ -1,16 +1,16 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-add', 'arguments': {'node_name': 'imgfile', 'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks'}}
-{u'return': {}}
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'iter-time': 10, 'driver': 'luks', 'file': 'imgfile', 'size': 134217728}}}
-{u'return': {}}
+{'execute': 'blockdev-add', 'arguments': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks', 'node_name': 'imgfile'}}
+{'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'luks', 'file': 'imgfile', 'iter-time': 10, 'key-secret': 'keysec0', 'size': 134217728}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_IMG"}, "key-secret": "keysec0"}
 file format: IMGFMT
@@ -54,15 +54,15 @@ Format specific information:
 
 === Successful image creation (with non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'hash-alg': 'sha1', 'cipher-mode': 'ctr', 'cipher-alg': 'twofish-128', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks'}, 'iter-time': 10, 'ivgen-alg': 'plain64', 'ivgen-hash-alg': 'md5', 'driver': 'luks', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cipher-alg': 'twofish-128', 'cipher-mode': 'ctr', 'driver': 'luks', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.luks'}, 'hash-alg': 'sha1', 'iter-time': 10, 'ivgen-alg': 'plain64', 'ivgen-hash-alg': 'md5', 'key-secret': 'keysec0', 'size': 67108864}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_IMG"}, "key-secret": "keysec0"}
 file format: IMGFMT
@@ -107,17 +107,17 @@ Format specific information:
 === Invalid BlockdevRef ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'luks', 'file': "this doesn't exist", 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Zero size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'iter-time': 10, 'driver': 'luks', 'file': 'node0', 'size': 0}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'luks', 'file': 'node0', 'iter-time': 10, 'key-secret': 'keysec0', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_IMG"}, "key-secret": "keysec0"}
 file format: IMGFMT
@@ -161,34 +161,34 @@ Format specific information:
 
 === Invalid sizes ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'driver': 'luks', 'file': 'node0', 'size': 18446744073709551104L}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'luks', 'file': 'node0', 'key-secret': 'keysec0', 'size': 18446744073709551104}}}
+{'return': {}}
 Job failed: The requested file size is too large
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'driver': 'luks', 'file': 'node0', 'size': 9223372036854775808L}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'luks', 'file': 'node0', 'key-secret': 'keysec0', 'size': 9223372036854775808}}}
+{'return': {}}
 Job failed: The requested file size is too large
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'key-secret': 'keysec0', 'driver': 'luks', 'file': 'node0', 'size': 9223372036854775296}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'luks', 'file': 'node0', 'key-secret': 'keysec0', 'size': 9223372036854775296}}}
+{'return': {}}
 Job failed: The requested file size is too large
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Resize image with invalid sizes ===
 
-{'execute': 'block_resize', 'arguments': {'size': 9223372036854775296, 'node_name': 'node1'}}
-{u'error': {u'class': u'GenericError', u'desc': u'The requested file size is too large'}}
-{'execute': 'block_resize', 'arguments': {'size': 9223372036854775808L, 'node_name': 'node1'}}
-{u'error': {u'class': u'GenericError', u'desc': u"Invalid parameter type for 'size', expected: integer"}}
-{'execute': 'block_resize', 'arguments': {'size': 18446744073709551104L, 'node_name': 'node1'}}
-{u'error': {u'class': u'GenericError', u'desc': u"Invalid parameter type for 'size', expected: integer"}}
-{'execute': 'block_resize', 'arguments': {'size': -9223372036854775808, 'node_name': 'node1'}}
-{u'error': {u'class': u'GenericError', u'desc': u"Parameter 'size' expects a >0 size"}}
+{'execute': 'block_resize', 'arguments': {'node_name': 'node1', 'size': 9223372036854775296}}
+{'error': {'class': 'GenericError', 'desc': 'The requested file size is too large'}}
+{'execute': 'block_resize', 'arguments': {'node_name': 'node1', 'size': 9223372036854775808}}
+{'error': {'class': 'GenericError', 'desc': "Invalid parameter type for 'size', expected: integer"}}
+{'execute': 'block_resize', 'arguments': {'node_name': 'node1', 'size': 18446744073709551104}}
+{'error': {'class': 'GenericError', 'desc': "Invalid parameter type for 'size', expected: integer"}}
+{'execute': 'block_resize', 'arguments': {'node_name': 'node1', 'size': -9223372036854775808}}
+{'error': {'class': 'GenericError', 'desc': "Parameter 'size' expects a >0 size"}}
 image: json:{"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_IMG"}, "key-secret": "keysec0"}
 file format: IMGFMT
 virtual size: 0 (0 bytes)
diff --git a/tests/qemu-iotests/211.out b/tests/qemu-iotests/211.out
index 6feaea3978..3816d89b40 100644
--- a/tests/qemu-iotests/211.out
+++ b/tests/qemu-iotests/211.out
@@ -1,16 +1,16 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-add', 'arguments': {'node_name': 'imgfile', 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}}
-{u'return': {}}
+{'execute': 'blockdev-add', 'arguments': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi', 'node_name': 'imgfile'}}
+{'return': {}}
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'imgfile', 'size': 134217728}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -21,15 +21,15 @@ cluster_size: 1048576
 
 === Successful image creation (explicit defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'preallocation': 'off', 'driver': 'vdi', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}, 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}, 'preallocation': 'off', 'size': 67108864}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -40,15 +40,15 @@ cluster_size: 1048576
 
 === Successful image creation (with non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'preallocation': 'metadata', 'driver': 'vdi', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}, 'size': 33554432}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vdi'}, 'preallocation': 'metadata', 'size': 33554432}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -61,17 +61,17 @@ cluster_size: 1048576
 === Invalid BlockdevRef ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': "this doesn't exist", 'size': 33554432}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Zero size ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 0}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -81,9 +81,9 @@ cluster_size: 1048576
 === Maximum size ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 562949819203584}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -92,21 +92,21 @@ cluster_size: 1048576
 
 === Invalid sizes ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 18446744073709551104L}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 18446744073709551104}}}
+{'return': {}}
 Job failed: Unsupported VDI image size (size is 0xfffffffffffffe00, max supported is 0x1fffff8000000)
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 9223372036854775808L}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 9223372036854775808}}}
+{'return': {}}
 Job failed: Unsupported VDI image size (size is 0x8000000000000000, max supported is 0x1fffff8000000)
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vdi', 'file': 'node0', 'size': 562949819203585}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Unsupported VDI image size (size is 0x1fffff8000001, max supported is 0x1fffff8000000)
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
diff --git a/tests/qemu-iotests/212.out b/tests/qemu-iotests/212.out
index 9150da7a2c..9779d3a4cb 100644
--- a/tests/qemu-iotests/212.out
+++ b/tests/qemu-iotests/212.out
@@ -1,16 +1,16 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-add', 'arguments': {'node_name': 'imgfile', 'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}}
-{u'return': {}}
+{'execute': 'blockdev-add', 'arguments': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels', 'node_name': 'imgfile'}}
+{'return': {}}
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'imgfile', 'size': 134217728}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -18,15 +18,15 @@ virtual size: 128M (134217728 bytes)
 
 === Successful image creation (explicit defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 1048576, 'driver': 'parallels', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}, 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -34,15 +34,15 @@ virtual size: 64M (67108864 bytes)
 
 === Successful image creation (with non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 65536, 'driver': 'parallels', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.parallels'}, 'size': 33554432}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -51,17 +51,17 @@ virtual size: 32M (33554432 bytes)
 === Invalid BlockdevRef ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': "this doesn't exist", 'size': 33554432}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Zero size ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 0}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -70,9 +70,9 @@ virtual size: 0 (0 bytes)
 === Maximum size ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 4503599627369984}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -81,76 +81,76 @@ virtual size: 4096T (4503599627369984 bytes)
 === Invalid sizes ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 1234}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Image size must be a multiple of 512 bytes
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 18446744073709551104L}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 18446744073709551104}}}
+{'return': {}}
 Job failed: Image size is too large for this cluster size
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 9223372036854775808L}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 9223372036854775808}}}
+{'return': {}}
 Job failed: Image size is too large for this cluster size
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 9223372036854775296}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Image size is too large for this cluster size
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'parallels', 'file': 'node0', 'size': 4503599627370497}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Image size is too large for this cluster size
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Invalid cluster size ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 1234, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cluster size must be a multiple of 512 bytes
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 128, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cluster size must be a multiple of 512 bytes
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 4294967296, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cluster size is too large
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 9223372036854775808L, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 9223372036854775808, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
+{'return': {}}
 Job failed: Cluster size is too large
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 18446744073709551104L, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 18446744073709551104, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
+{'return': {}}
 Job failed: Cluster size is too large
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 0, 'driver': 'parallels', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Image size is too large for this cluster size
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'cluster-size': 512, 'driver': 'parallels', 'file': 'node0', 'size': 281474976710656}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Image size is too large for this cluster size
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
diff --git a/tests/qemu-iotests/213.out b/tests/qemu-iotests/213.out
index e1dcd47201..9e214d4fc5 100644
--- a/tests/qemu-iotests/213.out
+++ b/tests/qemu-iotests/213.out
@@ -1,16 +1,16 @@
 === Successful image creation (defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-add', 'arguments': {'node_name': 'imgfile', 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}}
-{u'return': {}}
+{'execute': 'blockdev-add', 'arguments': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx', 'node_name': 'imgfile'}}
+{'return': {}}
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'imgfile', 'size': 134217728}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -19,15 +19,15 @@ cluster_size: 8388608
 
 === Successful image creation (explicit defaults) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 8388608, 'driver': 'vhdx', 'subformat': 'dynamic', 'log-size': 1048576, 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}, 'block-state-zero': True, 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 8388608, 'block-state-zero': True, 'driver': 'vhdx', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}, 'log-size': 1048576, 'size': 67108864, 'subformat': 'dynamic'}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -36,15 +36,15 @@ cluster_size: 8388608
 
 === Successful image creation (with non-default options) ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'size': 0, 'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx', 'size': 0}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 268435456, 'driver': 'vhdx', 'subformat': 'fixed', 'log-size': 8388608, 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}, 'block-state-zero': False, 'size': 33554432}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 268435456, 'block-state-zero': False, 'driver': 'vhdx', 'file': {'driver': 'file', 'filename': 'TEST_DIR/PID-t.vhdx'}, 'log-size': 8388608, 'size': 33554432, 'subformat': 'fixed'}}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -54,17 +54,17 @@ cluster_size: 268435456
 === Invalid BlockdevRef ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': "this doesn't exist", 'size': 33554432}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exist
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Zero size ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 0}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -74,9 +74,9 @@ cluster_size: 8388608
 === Maximum size ===
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 70368744177664}}}
-{u'return': {}}
+{'return': {}}
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 image: TEST_IMG
 file format: IMGFMT
@@ -85,85 +85,85 @@ cluster_size: 67108864
 
 === Invalid sizes ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 18446744073709551104L}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 18446744073709551104}}}
+{'return': {}}
 Job failed: Image size too large; max of 64TB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 9223372036854775808L}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 9223372036854775808}}}
+{'return': {}}
 Job failed: Image size too large; max of 64TB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 9223372036854775296}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Image size too large; max of 64TB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 {'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'size': 70368744177665}}}
-{u'return': {}}
+{'return': {}}
 Job failed: Image size too large; max of 64TB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Invalid block size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'block-size': 1234567, 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 1234567, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
+{'return': {}}
 Job failed: Block size must be a multiple of 1 MB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'block-size': 128, 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 128, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
+{'return': {}}
 Job failed: Block size must be a multiple of 1 MB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'block-size': 3145728, 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 3145728, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
+{'return': {}}
 Job failed: Block size must be a power of two
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'block-size': 536870912, 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 536870912, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
+{'return': {}}
 Job failed: Block size must not exceed 268435456
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'block-size': 0, 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'block-size': 0, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
+{'return': {}}
 Job failed: Block size must be a multiple of 1 MB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
 === Invalid log size ===
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'log-size': 1234567, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'log-size': 1234567, 'size': 67108864}}}
+{'return': {}}
 Job failed: Log size must be a multiple of 1 MB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'log-size': 128, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'log-size': 128, 'size': 67108864}}}
+{'return': {}}
 Job failed: Log size must be a multiple of 1 MB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'log-size': 4294967296, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'log-size': 4294967296, 'size': 67108864}}}
+{'return': {}}
 Job failed: Log size must be smaller than 4 GB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
-{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'log-size': 0, 'driver': 'vhdx', 'file': 'node0', 'size': 67108864}}}
-{u'return': {}}
+{'execute': 'blockdev-create', 'arguments': {'job_id': 'job0', 'options': {'driver': 'vhdx', 'file': 'node0', 'log-size': 0, 'size': 67108864}}}
+{'return': {}}
 Job failed: Log size must be a multiple of 1 MB
 {'execute': 'job-dismiss', 'arguments': {'id': 'job0'}}
-{u'return': {}}
+{'return': {}}
 
diff --git a/tests/qemu-iotests/216.out b/tests/qemu-iotests/216.out
index 45ea857ee1..908f9314ac 100644
--- a/tests/qemu-iotests/216.out
+++ b/tests/qemu-iotests/216.out
@@ -7,8 +7,8 @@ Done
 
 --- Doing COR ---
 
-{u'return': {}}
-{u'return': u''}
+{'return': {}}
+{'return': ''}
 
 --- Checking COR result ---
 
diff --git a/tests/qemu-iotests/218.out b/tests/qemu-iotests/218.out
index 7dbf78e682..9d270e4834 100644
--- a/tests/qemu-iotests/218.out
+++ b/tests/qemu-iotests/218.out
@@ -4,27 +4,27 @@
 --- force=false ---
 
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 65536, u'len': 1048576, u'offset': 65536}, u'event': u'BLOCK_JOB_CANCELLED'}
+{'return': {}}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 65536, 'speed': 65536, 'type': 'mirror'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 --- force=true ---
 
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 65536, u'len': 1048576, u'offset': 65536}, u'event': u'BLOCK_JOB_CANCELLED'}
+{'return': {}}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 65536, 'speed': 65536, 'type': 'mirror'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 === Cancel mirror job after convergence ===
 
 --- force=false ---
 
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_READY'}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_READY', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_COMPLETED'}
+{'return': {}}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_COMPLETED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 --- force=true ---
 
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_READY'}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_READY', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 Cancelling job
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'mirror', u'type': u'mirror', u'speed': 0, u'len': 1048576, u'offset': 1048576}, u'event': u'BLOCK_JOB_CANCELLED'}
+{'return': {}}
+{'data': {'device': 'mirror', 'len': 1048576, 'offset': 1048576, 'speed': 0, 'type': 'mirror'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
diff --git a/tests/qemu-iotests/219.out b/tests/qemu-iotests/219.out
index 6dc07bc41e..9acf1e89ac 100644
--- a/tests/qemu-iotests/219.out
+++ b/tests/qemu-iotests/219.out
@@ -2,326 +2,326 @@ Launching VM...
 
 
 Starting block job: drive-mirror (auto-finalize: True; auto-dismiss: True)
-{u'return': {}}
-{u'return': [{u'status': u'running', u'current-progress': 'FILTERED', u'total-progress': 'FILTERED', u'id': u'job0', u'type': u'mirror'}]}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'created', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
+{'return': {}}
+{'return': [{'current-progress': 'FILTERED', 'id': 'job0', 'status': 'running', 'total-progress': 'FILTERED', 'type': 'mirror'}]}
+{'data': {'id': 'job0', 'status': 'created'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 Pause/resume in RUNNING
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 65536, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 65536, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 131072, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'mirror'}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 131072, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 196608, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'mirror'}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 196608, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 262144, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'mirror'}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 327680, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 262144, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 327680, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'mirror'}]}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{'return': {}}
 
 Waiting for READY state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'ready', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'ready', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{'data': {'id': 'job0', 'status': 'ready'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'ready', 'total-progress': 4194304, 'type': 'mirror'}]}
 
 Pause/resume in READY
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'standby', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'standby', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'ready', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'ready', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'standby'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'standby', 'total-progress': 4194304, 'type': 'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'ready'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'ready', 'total-progress': 4194304, 'type': 'mirror'}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'standby', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'standby', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'ready', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'ready', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'standby'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'standby', 'total-progress': 4194304, 'type': 'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'ready'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'ready', 'total-progress': 4194304, 'type': 'mirror'}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'standby', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'standby', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'ready', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'ready', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'standby'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'standby', 'total-progress': 4194304, 'type': 'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'ready'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'ready', 'total-progress': 4194304, 'type': 'mirror'}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'standby', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'standby', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'ready', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'ready', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'mirror'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'ready' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'ready' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'ready' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'ready' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'standby'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'standby', 'total-progress': 4194304, 'type': 'mirror'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'ready'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'ready', 'total-progress': 4194304, 'type': 'mirror'}]}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'ready' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'ready' cannot accept command verb 'dismiss'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'ready' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'ready' cannot accept command verb 'dismiss'"}}
+{'return': {}}
 
 Waiting for PENDING state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'waiting', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'pending', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'concluded', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'null', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': []}
+{'data': {'id': 'job0', 'status': 'waiting'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'pending'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'concluded'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'null'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': []}
 
 
 Starting block job: drive-backup (auto-finalize: True; auto-dismiss: True)
-{u'return': {}}
-{u'return': [{u'status': u'running', u'current-progress': 'FILTERED', u'total-progress': 'FILTERED', u'id': u'job0', u'type': u'backup'}]}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'created', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
+{'return': {}}
+{'return': [{'current-progress': 'FILTERED', 'id': 'job0', 'status': 'running', 'total-progress': 'FILTERED', 'type': 'backup'}]}
+{'data': {'id': 'job0', 'status': 'created'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 Pause/resume in RUNNING
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 65536, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 65536, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 131072, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 131072, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 196608, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 196608, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 262144, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 327680, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 262144, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 327680, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{'return': {}}
 
 Waiting for PENDING state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'waiting', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'pending', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'concluded', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'null', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': []}
+{'data': {'id': 'job0', 'status': 'waiting'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'pending'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'concluded'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'null'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': []}
 
 
 Starting block job: drive-backup (auto-finalize: True; auto-dismiss: False)
-{u'return': {}}
-{u'return': [{u'status': u'running', u'current-progress': 'FILTERED', u'total-progress': 'FILTERED', u'id': u'job0', u'type': u'backup'}]}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'created', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
+{'return': {}}
+{'return': [{'current-progress': 'FILTERED', 'id': 'job0', 'status': 'running', 'total-progress': 'FILTERED', 'type': 'backup'}]}
+{'data': {'id': 'job0', 'status': 'created'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 Pause/resume in RUNNING
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 65536, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 65536, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 131072, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 131072, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 196608, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 196608, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 262144, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 327680, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 262144, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 327680, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{'return': {}}
 
 Waiting for PENDING state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'waiting', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'pending', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'concluded', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'concluded', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'null', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': []}
+{'data': {'id': 'job0', 'status': 'waiting'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'pending'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'concluded'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'concluded', 'total-progress': 4194304, 'type': 'backup'}]}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'null'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': []}
 
 
 Starting block job: drive-backup (auto-finalize: False; auto-dismiss: True)
-{u'return': {}}
-{u'return': [{u'status': u'running', u'current-progress': 'FILTERED', u'total-progress': 'FILTERED', u'id': u'job0', u'type': u'backup'}]}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'created', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
+{'return': {}}
+{'return': [{'current-progress': 'FILTERED', 'id': 'job0', 'status': 'running', 'total-progress': 'FILTERED', 'type': 'backup'}]}
+{'data': {'id': 'job0', 'status': 'created'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 Pause/resume in RUNNING
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 65536, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 65536, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 131072, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 131072, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 196608, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 196608, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 262144, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 327680, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 262144, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 327680, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{'return': {}}
 
 Waiting for PENDING state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'waiting', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'pending', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'pending', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'concluded', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'null', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': []}
+{'data': {'id': 'job0', 'status': 'waiting'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'pending'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'pending', 'total-progress': 4194304, 'type': 'backup'}]}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'concluded'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'null'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': []}
 
 
 Starting block job: drive-backup (auto-finalize: False; auto-dismiss: False)
-{u'return': {}}
-{u'return': [{u'status': u'running', u'current-progress': 'FILTERED', u'total-progress': 'FILTERED', u'id': u'job0', u'type': u'backup'}]}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'created', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
+{'return': {}}
+{'return': [{'current-progress': 'FILTERED', 'id': 'job0', 'status': 'running', 'total-progress': 'FILTERED', 'type': 'backup'}]}
+{'data': {'id': 'job0', 'status': 'created'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
 
 Pause/resume in RUNNING
 === Testing block-job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 65536, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 65536, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 131072, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing block-job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 131072, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 131072, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 196608, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing job-pause/block-job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 196608, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 196608, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 262144, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
 === Testing job-pause/job-resume ===
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'paused', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'paused', u'current-progress': 262144, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'running', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'running', u'current-progress': 327680, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'paused'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 262144, 'id': 'job0', 'status': 'paused', 'total-progress': 4194304, 'type': 'backup'}]}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'running'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 327680, 'id': 'job0', 'status': 'running', 'total-progress': 4194304, 'type': 'backup'}]}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'running' cannot accept command verb 'dismiss'"}}
+{'return': {}}
 
 Waiting for PENDING state...
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'waiting', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'pending', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'pending', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'concluded', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': [{u'status': u'concluded', u'current-progress': 4194304, u'total-progress': 4194304, u'id': u'job0', u'type': u'backup'}]}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
-{u'error': {u'class': u'GenericError', u'desc': u"Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'status': u'null', u'id': u'job0'}, u'event': u'JOB_STATUS_CHANGE'}
-{u'return': []}
+{'data': {'id': 'job0', 'status': 'waiting'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'data': {'id': 'job0', 'status': 'pending'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'pending', 'total-progress': 4194304, 'type': 'backup'}]}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'pause'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'pending' cannot accept command verb 'dismiss'"}}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'concluded'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': [{'current-progress': 4194304, 'id': 'job0', 'status': 'concluded', 'total-progress': 4194304, 'type': 'backup'}]}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'pause'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'complete'"}}
+{'error': {'class': 'GenericError', 'desc': "Job 'job0' in state 'concluded' cannot accept command verb 'finalize'"}}
+{'return': {}}
+{'data': {'id': 'job0', 'status': 'null'}, 'event': 'JOB_STATUS_CHANGE', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': []}
diff --git a/tests/qemu-iotests/222.out b/tests/qemu-iotests/222.out
index 48f336a02b..37455f6c2d 100644
--- a/tests/qemu-iotests/222.out
+++ b/tests/qemu-iotests/222.out
@@ -8,13 +8,13 @@ Done
 
 --- Setting up Fleecing Graph ---
 
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 
 --- Setting up NBD Export ---
 
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'return': {}}
 
 --- Sanity Check ---
 
@@ -29,13 +29,13 @@ read -P0 0x3fe0000 64k
 --- Testing COW ---
 
 write -P0xab 0 64k
-{u'return': u''}
+{'return': ''}
 write -P0xad 0x00f8000 64k
-{u'return': u''}
+{'return': ''}
 write -P0x1d 0x2008000 64k
-{u'return': u''}
+{'return': ''}
 write -P0xea 0x3fe0000 64k
-{u'return': u''}
+{'return': ''}
 
 --- Verifying Data ---
 
@@ -49,10 +49,10 @@ read -P0 0x3fe0000 64k
 
 --- Cleanup ---
 
-{u'return': {}}
-{u'timestamp': {u'seconds': 'SECS', u'microseconds': 'USECS'}, u'data': {u'device': u'drive0', u'type': u'backup', u'speed': 0, u'len': 67108864, u'offset': 393216}, u'event': u'BLOCK_JOB_CANCELLED'}
-{u'return': {}}
-{u'return': {}}
+{'return': {}}
+{'data': {'device': 'drive0', 'len': 67108864, 'offset': 393216, 'speed': 0, 'type': 'backup'}, 'event': 'BLOCK_JOB_CANCELLED', 'timestamp': {'microseconds': 'USECS', 'seconds': 'SECS'}}
+{'return': {}}
+{'return': {}}
 
 --- Confirming writes ---
 
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index a64ea90fb4..f8dbc8cc71 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -250,10 +250,45 @@ def filter_img_info(output, filename):
         lines.append(line)
     return '\n'.join(lines)
 
+def log_to_string_repr(obj):
+    # Normal Python 3 strings are the unicode strings from Python 2;
+    # and normal Python 2 strings are byte strings in Python 3.  Thus,
+    # we convert bytes -> str in py3 and unicode -> str in py2.
+    if sys.version_info.major >= 3:
+        if type(obj) is bytes:
+            return repr(obj.decode('utf-8'))
+    else:
+        if type(obj) is unicode:
+            return repr(obj.encode('utf-8'))
+        elif type(obj) is long:
+            return str(obj) # repr() would include an 'L' suffix
+
+    if type(obj) is list:
+        return '[' + ', '.join(map(log_to_string_repr, obj)) + ']'
+    elif type(obj) is dict:
+        return '{' + ', '.join(map(lambda k: log_to_string_repr(k) + ': ' +
+                                             log_to_string_repr(obj[k]),
+                                   sorted(obj.keys()))) + '}'
+    else:
+        return repr(obj)
+
+def log_to_string(obj):
+    if type(obj) is str:
+        return obj
+
+    if sys.version_info.major >= 3:
+        if type(obj) is bytes:
+            return obj.decode('utf-8')
+    else:
+        if type(obj) is unicode:
+            return obj.encode('utf-8')
+
+    return log_to_string_repr(obj)
+
 def log(msg, filters=[]):
     for flt in filters:
         msg = flt(msg)
-    print(msg)
+    print(log_to_string(msg))
 
 class Timeout:
     def __init__(self, seconds, errmsg = "Timeout"):
@@ -441,10 +476,11 @@ class VM(qtest.QEMUQtestMachine):
         return result
 
     def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs):
-        logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs)
+        logmsg = "{'execute': '%s', 'arguments': %s}" % \
+            (cmd, log_to_string(kwargs))
         log(logmsg, filters)
         result = self.qmp(cmd, **kwargs)
-        log(str(result), filters)
+        log(log_to_string(result), filters)
         return result
 
     def run_job(self, job, auto_finalize=True, auto_dismiss=False):
-- 
2.17.1

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

[Cleber Rosa]

On 10/15/18 10:14 AM, Max Reitz wrote:
> There are two imports that need to be modified when running the iotests
> under Python 3: One is StringIO, which no longer exists; instead, the
> StringIO class comes from the io module, so import it from there.  The
> other is the ConfigParser, which has just been renamed to configparser.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/iotests.py            | 8 ++++++--
>  tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
>  2 files changed, 11 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 7ca94e9278..a64ea90fb4 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
>  
>      # We need to filter out the time taken from the output so that qemu-iotest
>      # can reliably diff the results against master output.
> -    import StringIO
> +    if sys.version_info.major >= 3:
> +        from io import StringIO
> +    else:
> +        from StringIO import StringIO
> +
>      if debug:
>          output = sys.stdout
>          verbosity = 2
>          sys.argv.remove('-d')
>      else:
> -        output = StringIO.StringIO()
> +        output = StringIO()
>  
>      logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
>  
> diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
> index d45e2e0a6a..6b2d659dee 100755
> --- a/tests/qemu-iotests/nbd-fault-injector.py
> +++ b/tests/qemu-iotests/nbd-fault-injector.py
> @@ -48,7 +48,10 @@ import sys
>  import socket
>  import struct
>  import collections
> -import ConfigParser
> +if sys.version_info.major >= 3:
> +    import configparser
> +else:
> +    import ConfigParser as configparser
>  
>  FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
>  
> @@ -225,7 +228,7 @@ def parse_config(config):
>      return rules
>  
>  def load_rules(filename):
> -    config = ConfigParser.RawConfigParser()
> +    config = configparser.RawConfigParser()
>      with open(filename, 'rt') as f:
>          config.readfp(f, filename)
>      return parse_config(config)
> 

This may be a type of culture clash (on my side, due to not enough QEMU
culture), but shouldn't this be applied before anything else on this series?

I mean, PATCH 1/9 is supposed to fix the reliability aspects of
nbd-fault-injector under Python 3, but without this patch, it won't
actually run on Python 3.

Regards,
- Cleber.

Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 04:14:45PM +0200, Max Reitz wrote:
> When closing a connection, make the nbd-fault-injector flush the socket.
> Without this, the output is a bit unreliable with Python 3.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [Qemu-devel] [PATCH 2/9] iotests: Flush in iotests.py's QemuIoInteractive

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 04:14:46PM +0200, Max Reitz wrote:
> After issuing a command, flush the pipe.  This does not change anything
> in Python 2, but it makes a difference in Python 3.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [Qemu-devel] [PATCH 3/9] iotests: Use Python byte strings where appropriate

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 04:14:47PM +0200, Max Reitz wrote:
> Since byte strings are no longer the default in Python 3, we have to
> explicitly use them where we need to, which is mostly when working with
> structures.  It also means that we need to open a file in binary mode
> when we want to use structures.
> 
> On the other hand, we have to accomodate for the fact that some
> functions (still) work with byte strings but we want to use unicode
> strings (in Python 3 at least, and it does not matter in Python 2).
> This includes base64 encoding, but it is most notable when working with
> the subprocess module: Either we set univeral_newlines to True so that
> the default streams are opened in text mode (hence this parameter is
> aliased as "text" as of 3.7), or, if that is not possible, we have to
> decode the output to a normal string.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
[...]
> diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
> index 9e0cad76f9..1225334cb8 100755
> --- a/tests/qemu-iotests/149
> +++ b/tests/qemu-iotests/149
> @@ -79,7 +79,7 @@ class LUKSConfig(object):
>  
>      def first_password_base64(self):
>          (pw, slot) = self.first_password()
> -        return base64.b64encode(pw)
> +        return base64.b64encode(pw.encode('ascii')).decode('ascii')

Would we want to have a test case for non-ascii passwords in the
future?  In that case you would probably need to make
self.passwords[] contain byte strings instead of text.

In either case, using 'ascii' as the encoding everywhere will
ensure the code will not try to be too smart about string
encodings if that happens.  I like that.

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [Qemu-devel] [PATCH 4/9] iotests: Use // for Python integer division

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 04:14:48PM +0200, Max Reitz wrote:
> In Python 3, / is always a floating-point division.  We usually do not
> want this, and as Python 2.7 understands // as well, change all integer
> divisions to use that.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

-- 
Eduardo

Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 04:14:49PM +0200, Max Reitz wrote:
> In Python 3, several functions now return iterators instead of lists.
> This includes range(), items(), map(), and filter().  This means that if
> we really want a list, we have to wrap those instances with list().  On
> the other hand, sometimes we do just want an iterator, in which case we
> have sometimes used xrange() and iteritems() which no longer exist in
> Python 3.  Just change these calls to be range() and items(), which
> costs a bit of performance in Python 2, but will do the right thing in
> Python 3 (which is what is important).
> 
> In one instance, we only wanted the first instance of the result of a
> filter() call.  Instead of using next(filter()) which would work only in
> Python 3, or list(filter())[0] which would work everywhere but is a bit
> weird, this instance is changed to a single-line for with next() wrapped
> around, which works both in 2.7 and 3.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

Minor comments below:

> ---
>  tests/qemu-iotests/044 | 12 ++++++------
>  tests/qemu-iotests/056 |  2 +-
>  tests/qemu-iotests/065 |  4 ++--
>  tests/qemu-iotests/124 |  4 ++--
>  tests/qemu-iotests/139 |  2 +-
>  tests/qemu-iotests/163 |  6 +++---
>  6 files changed, 15 insertions(+), 15 deletions(-)
> 
> diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
> index 7ef5e46fe9..e2d6c9b189 100755
> --- a/tests/qemu-iotests/044
> +++ b/tests/qemu-iotests/044
> @@ -52,23 +52,23 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
>              # Write a refcount table
>              fd.seek(off_reftable)
>  
> -            for i in xrange(0, h.refcount_table_clusters):
> +            for i in range(0, h.refcount_table_clusters):
>                  sector = b''.join(struct.pack('>Q',
>                      off_refblock + i * 64 * 512 + j * 512)
> -                    for j in xrange(0, 64))
> +                    for j in range(0, 64))
>                  fd.write(sector)
>  
>              # Write the refcount blocks
>              assert(fd.tell() == off_refblock)
>              sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256))
> -            for block in xrange(0, h.refcount_table_clusters):
> +            for block in range(0, h.refcount_table_clusters):
>                  fd.write(sector)
>  
>              # Write the L1 table
>              assert(fd.tell() == off_l1)
>              assert(off_l2 + 512 * h.l1_size == off_data)
>              table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
> -                for j in xrange(0, h.l1_size))
> +                for j in range(0, h.l1_size))
>              fd.write(table)
>  
>              # Write the L2 tables
> @@ -79,14 +79,14 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
>              off = off_data
>              while remaining > 1024 * 512:
>                  pytable = list((1 << 63) | off + 512 * j
> -                    for j in xrange(0, 1024))
> +                    for j in range(0, 1024))
>                  table = struct.pack('>1024Q', *pytable)
>                  fd.write(table)
>                  remaining = remaining - 1024 * 512
>                  off = off + 1024 * 512
>  
>              table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
> -                for j in xrange(0, remaining // 512))
> +                for j in range(0, remaining // 512))
>              fd.write(table)
>  
>  
> diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
> index 223292175a..3df323984d 100755
> --- a/tests/qemu-iotests/056
> +++ b/tests/qemu-iotests/056
> @@ -32,7 +32,7 @@ target_img = os.path.join(iotests.test_dir, 'target.img')
>  def img_create(img, fmt=iotests.imgfmt, size='64M', **kwargs):
>      fullname = os.path.join(iotests.test_dir, '%s.%s' % (img, fmt))
>      optargs = []
> -    for k,v in kwargs.iteritems():
> +    for k,v in kwargs.items():
>          optargs = optargs + ['-o', '%s=%s' % (k,v)]
>      args = ['create', '-f', fmt] + optargs + [fullname, size]
>      iotests.qemu_img(*args)
> diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
> index 72aa9707c7..a339bf6069 100755
> --- a/tests/qemu-iotests/065
> +++ b/tests/qemu-iotests/065
> @@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific):
>                      :data.index('')]
>          for field in data:
>              self.assertTrue(re.match('^ {4}[^ ]', field) is not None)
> -        data = map(lambda line: line.strip(), data)
> +        data = list(map(lambda line: line.strip(), data))

I would find this more readable:

        data = [line.strip() for line in data]

Not a blocker, though.

>          self.assertEqual(data, self.human_compare)
>  
>  class TestQMP(TestImageInfoSpecific):
> @@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific):
>  
>      def test_qmp(self):
>          result = self.vm.qmp('query-block')['return']
> -        drive = filter(lambda drive: drive['device'] == 'drive0', result)[0]
> +        drive = next(drive for drive in result if drive['device'] == 'drive0')

I didn't understand what you meant by "single-line for" on the
commit message, until I saw the generator expression here.  :)

This will raise an exception if there's no drive0 in the list,
but that was already true on the original code.


>          data = drive['inserted']['image']['format-specific']
>          self.assertEqual(data['type'], iotests.imgfmt)
>          self.assertEqual(data['data'], self.compare)
> diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
> index 3ea4ac53f5..9f189e3b54 100755
> --- a/tests/qemu-iotests/124
> +++ b/tests/qemu-iotests/124
> @@ -39,7 +39,7 @@ def try_remove(img):
>  def transaction_action(action, **kwargs):
>      return {
>          'type': action,
> -        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems())
> +        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items())
>      }
>  
>  
> @@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase):
>      def img_create(self, img, fmt=iotests.imgfmt, size='64M',
>                     parent=None, parentFormat=None, **kwargs):
>          optargs = []
> -        for k,v in kwargs.iteritems():
> +        for k,v in kwargs.items():
>              optargs = optargs + ['-o', '%s=%s' % (k,v)]
>          args = ['create', '-f', fmt] + optargs + [img, size]
>          if parent:
> diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
> index cc7fe337f3..e00f10b8c8 100755
> --- a/tests/qemu-iotests/139
> +++ b/tests/qemu-iotests/139
> @@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
>      # Check whether a BlockDriverState exists
>      def checkBlockDriverState(self, node, must_exist = True):
>          result = self.vm.qmp('query-named-block-nodes')
> -        nodes = filter(lambda x: x['node-name'] == node, result['return'])
> +        nodes = list(filter(lambda x: x['node-name'] == node, result['return']))

I'd prefer a list expression:

        nodes = [x for x in result['return'] if x['node-name'] == node]

Also not a blocker.

>          self.assertLessEqual(len(nodes), 1)
>          self.assertEqual(must_exist, len(nodes) == 1)
>  
> diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
> index 5fd424761b..35c1a2bafc 100755
> --- a/tests/qemu-iotests/163
> +++ b/tests/qemu-iotests/163
> @@ -41,7 +41,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
>          div_roundup = lambda n, d: (n + d - 1) // d
>  
>          def split_by_n(data, n):
> -            for x in xrange(0, len(data), n):
> +            for x in range(0, len(data), n):
>                  yield struct.unpack('>Q', data[x:x + n])[0] & l1_mask
>  
>          def check_l1_table(h, l1_data):
> @@ -135,8 +135,8 @@ class ShrinkBaseClass(iotests.QMPTestCase):
>          self.image_verify()
>  
>      def test_random_write(self):
> -        offs_list = range(0, size_to_int(self.image_len),
> -                          size_to_int(self.chunk_size))
> +        offs_list = list(range(0, size_to_int(self.image_len),
> +                               size_to_int(self.chunk_size)))
>          random.shuffle(offs_list)
>          for offs in offs_list:
>              qemu_io('-c', 'write -P 0xff %d %s' % (offs, self.chunk_size),
> -- 
> 2.17.1
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 02:59:28PM -0400, Cleber Rosa wrote:
> 
> 
> On 10/15/18 10:14 AM, Max Reitz wrote:
> > There are two imports that need to be modified when running the iotests
> > under Python 3: One is StringIO, which no longer exists; instead, the
> > StringIO class comes from the io module, so import it from there.  The
> > other is the ConfigParser, which has just been renamed to configparser.
> > 
> > Signed-off-by: Max Reitz <mreitz@redhat.com>
> > ---
> >  tests/qemu-iotests/iotests.py            | 8 ++++++--
> >  tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
> >  2 files changed, 11 insertions(+), 4 deletions(-)
> > 
> > diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> > index 7ca94e9278..a64ea90fb4 100644
> > --- a/tests/qemu-iotests/iotests.py
> > +++ b/tests/qemu-iotests/iotests.py
> > @@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
> >  
> >      # We need to filter out the time taken from the output so that qemu-iotest
> >      # can reliably diff the results against master output.
> > -    import StringIO
> > +    if sys.version_info.major >= 3:
> > +        from io import StringIO
> > +    else:
> > +        from StringIO import StringIO
> > +
> >      if debug:
> >          output = sys.stdout
> >          verbosity = 2
> >          sys.argv.remove('-d')
> >      else:
> > -        output = StringIO.StringIO()
> > +        output = StringIO()
> >  
> >      logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
> >  
> > diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
> > index d45e2e0a6a..6b2d659dee 100755
> > --- a/tests/qemu-iotests/nbd-fault-injector.py
> > +++ b/tests/qemu-iotests/nbd-fault-injector.py
> > @@ -48,7 +48,10 @@ import sys
> >  import socket
> >  import struct
> >  import collections
> > -import ConfigParser
> > +if sys.version_info.major >= 3:
> > +    import configparser
> > +else:
> > +    import ConfigParser as configparser
> >  
> >  FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
> >  
> > @@ -225,7 +228,7 @@ def parse_config(config):
> >      return rules
> >  
> >  def load_rules(filename):
> > -    config = ConfigParser.RawConfigParser()
> > +    config = configparser.RawConfigParser()
> >      with open(filename, 'rt') as f:
> >          config.readfp(f, filename)
> >      return parse_config(config)
> > 
> 
> This may be a type of culture clash (on my side, due to not enough QEMU
> culture), but shouldn't this be applied before anything else on this series?
> 
> I mean, PATCH 1/9 is supposed to fix the reliability aspects of
> nbd-fault-injector under Python 3, but without this patch, it won't
> actually run on Python 3.

Both patches are required to make the code run on Python 3, and
they don't depend on each other.  So I think the order doesn't
matter.

But I think the order chosen by Max is slightly more intuitive:
having explicit mentions of Python 3 in the code would be
confusing if we didn't fix the compatibility issues on patches
1-7 first.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush

[Cleber Rosa]

On 10/15/18 10:14 AM, Max Reitz wrote:
> When closing a connection, make the nbd-fault-injector flush the socket.
> Without this, the output is a bit unreliable with Python 3.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>

Reviewed-by: Cleber Rosa <crosa@redhat.com>

Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 04:14:50PM +0200, Max Reitz wrote:
> Python 3.2 introduced the inheritable attribute for FDs.  At the same
> time, it changed the default so that all FDs are not inheritable by
> default, that only inheritable FDs are inherited to subprocesses, and
> only if close_fds is explicitly set to False.
> 
> Adhere to this by setting close_fds to False when working with
> subprocesses that may want to inherit FDs, and by trying to
> set_inheritable() on FDs that we do want to bequeath to them.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  scripts/qemu.py        | 13 +++++++++++--
>  scripts/qmp/qmp.py     |  7 +++++++
>  tests/qemu-iotests/147 |  7 +++++++
>  3 files changed, 25 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/qemu.py b/scripts/qemu.py
> index f099ce7278..28366c4a67 100644
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
> @@ -142,10 +142,18 @@ class QEMUMachine(object):
>          if opts:
>              options.append(opts)
>  
> +        # This did not exist before 3.2, but since then it is
> +        # mandatory for our purpose
> +        try:
> +            os.set_inheritable(fd, True)
> +        except AttributeError:
> +            pass
> +

This is add_fd(), so calling set_inheritable() automatically here
makes sense.

>          self._args.append('-add-fd')
>          self._args.append(','.join(options))
>          return self
>  
> +    # The caller needs to make sure the FD is inheritable
>      def send_fd_scm(self, fd_file_path):
>          # In iotest.py, the qmp should always use unix socket.
>          assert self._qmp.is_scm_available()
> @@ -159,7 +167,7 @@ class QEMUMachine(object):
>                      "%s" % fd_file_path]
>          devnull = open(os.path.devnull, 'rb')
>          proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
> -                                stderr=subprocess.STDOUT)
> +                                stderr=subprocess.STDOUT, close_fds=False)
>          output = proc.communicate()[0]
>          if output:
>              LOG.debug(output)
> @@ -280,7 +288,8 @@ class QEMUMachine(object):
>                                         stdin=devnull,
>                                         stdout=self._qemu_log_file,
>                                         stderr=subprocess.STDOUT,
> -                                       shell=False)
> +                                       shell=False,
> +                                       close_fds=False)
>          self._post_launch()
>  
>      def wait(self):
> diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
> index 5c8cf6a056..009be8345b 100644
> --- a/scripts/qmp/qmp.py
> +++ b/scripts/qmp/qmp.py
> @@ -10,6 +10,7 @@
>  
>  import json
>  import errno
> +import os
>  import socket
>  import logging
>  
> @@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object):
>          return self.__sock.fileno()
>  
>      def is_scm_available(self):
> +        # This did not exist before 3.2, but since then it is
> +        # mandatory for our purpose
> +        try:
> +            os.set_inheritable(self.get_sock_fd(), True)
> +        except AttributeError:
> +            pass

Why did you decide to place this code inside is_scm_available()?

For reference, this is the only caller of is_scm_available():

    def send_fd_scm(self, fd_file_path):
        # In iotest.py, the qmp should always use unix socket.
        assert self._qmp.is_scm_available()
        ...

In addition to making a method called is_*() have an unexpected
side-effect, the method won't be called at all if running with
debugging disabled.

I suggest simply placing the os.set_inheritable() call inside
send_fd_scm(), as close as possible to the subprocess.Popen()
call.


>          return self.__sock.family == socket.AF_UNIX
> diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
> index d2081df84b..b58455645b 100755
> --- a/tests/qemu-iotests/147
> +++ b/tests/qemu-iotests/147
> @@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase):
>          sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>          sockfd.connect(unix_socket)
>  
> +        # This did not exist before 3.2, but since then it is
> +        # mandatory for our purpose
> +        try:
> +            os.set_inheritable(sockfd.fileno(), True)
> +        except AttributeError:
> +            pass
> +

Why not make send_fd_scm() responsible for calling
os.set_inheritable(), making this hunk unnecessary?


>          result = self.vm.send_fd_scm(str(sockfd.fileno()))
>          self.assertEqual(result, 0, 'Failed to send socket FD')
>  
> -- 
> 2.17.1
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 2/9] iotests: Flush in iotests.py's QemuIoInteractive

[Cleber Rosa]

On 10/15/18 10:14 AM, Max Reitz wrote:
> After issuing a command, flush the pipe.  This does not change anything
> in Python 2, but it makes a difference in Python 3.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>

Reviewed-by: Cleber Rosa <crosa@redhat.com>

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 04:14:51PM +0200, Max Reitz wrote:
> iotest 169 uses the 'new' module to add methods to a class.  This module
> no longer exists in Python 3.  Instead, we can use a lambda.  Best of
> all, this works in 2.7 just as well.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/169 | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
> index f243db9955..e5614b159d 100755
> --- a/tests/qemu-iotests/169
> +++ b/tests/qemu-iotests/169
> @@ -23,7 +23,6 @@ import iotests
>  import time
>  import itertools
>  import operator
> -import new
>  from iotests import qemu_img
>  
>  
> @@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
>  
>  def inject_test_case(klass, name, method, *args, **kwargs):
>      mc = operator.methodcaller(method, *args, **kwargs)
> -    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
> +    setattr(klass, 'test_' + name, lambda self: mc(self))

The "lambda self: mc(self)" expression looks weird and
unnecessary at first look, but I have just confirmed that this
doesn't work:

    setattr(klass, 'test_' + name, mc)

Probably because the methodcaller object won't automatically
become a instance method after the TestCase class is
instantiated.

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

>  
>  for cmb in list(itertools.product((True, False), repeat=4)):
>      name = ('_' if cmb[0] else '_not_') + 'persistent_'
> -- 
> 2.17.1
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 4/9] iotests: Use // for Python integer division

[Cleber Rosa]

On 10/15/18 10:14 AM, Max Reitz wrote:
> In Python 3, / is always a floating-point division.  We usually do not
> want this, and as Python 2.7 understands // as well, change all integer
> divisions to use that.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/040        |  4 ++--
>  tests/qemu-iotests/044        |  2 +-
>  tests/qemu-iotests/093        | 18 +++++++++---------
>  tests/qemu-iotests/149        |  6 +++---
>  tests/qemu-iotests/151        | 12 ++++++------
>  tests/qemu-iotests/163        |  2 +-
>  tests/qemu-iotests/iotests.py |  2 +-
>  7 files changed, 23 insertions(+), 23 deletions(-)
> 
> diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
> index 1cb1ceeb33..b81133a474 100755
> --- a/tests/qemu-iotests/040
> +++ b/tests/qemu-iotests/040
> @@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase):
>  
>          self.assert_no_active_block_jobs()
>          result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
> -                             base=backing_img, speed=(self.image_len / 4))
> +                             base=backing_img, speed=(self.image_len // 4))
>          self.assert_qmp(result, 'return', {})
>          result = self.vm.qmp('device_del', id='scsi0')
>          self.assert_qmp(result, 'return', {})
> @@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase):
>  
>          self.assert_no_active_block_jobs()
>          result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
> -                             base=backing_img, speed=(self.image_len / 4))
> +                             base=backing_img, speed=(self.image_len // 4))
>          self.assert_qmp(result, 'return', {})
>  
>          result = self.vm.qmp('query-block')
> diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
> index 69e736f687..7ef5e46fe9 100755
> --- a/tests/qemu-iotests/044
> +++ b/tests/qemu-iotests/044
> @@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
>                  off = off + 1024 * 512
>  
>              table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
> -                for j in xrange(0, remaining / 512))
> +                for j in xrange(0, remaining // 512))
>              fd.write(table)
>  
>  
> diff --git a/tests/qemu-iotests/093 b/tests/qemu-iotests/093
> index 9d1971a56c..d88fbc182e 100755
> --- a/tests/qemu-iotests/093
> +++ b/tests/qemu-iotests/093
> @@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase):
>          # in. The throttled requests won't be executed until we
>          # advance the virtual clock.
>          rq_size = 512
> -        rd_nr = max(params['bps'] / rq_size / 2,
> -                    params['bps_rd'] / rq_size,
> -                    params['iops'] / 2,
> +        rd_nr = max(params['bps'] // rq_size // 2,
> +                    params['bps_rd'] // rq_size,
> +                    params['iops'] // 2,
>                      params['iops_rd'])
>          rd_nr *= seconds * 2
> -        rd_nr /= ndrives
> -        wr_nr = max(params['bps'] / rq_size / 2,
> -                    params['bps_wr'] / rq_size,
> -                    params['iops'] / 2,
> +        rd_nr //= ndrives
> +        wr_nr = max(params['bps'] // rq_size // 2,
> +                    params['bps_wr'] // rq_size,
> +                    params['iops'] // 2,
>                      params['iops_wr'])
>          wr_nr *= seconds * 2
> -        wr_nr /= ndrives
> +        wr_nr //= ndrives
>  
>          # Send I/O requests to all drives
>          for i in range(rd_nr):
> @@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
>              self.configure_throttle(ndrives, settings)
>  
>              # Wait for the bucket to empty so we can do bursts
> -            wait_ns = nsec_per_sec * burst_length * burst_rate / rate
> +            wait_ns = nsec_per_sec * burst_length * burst_rate // rate
>              self.vm.qtest("clock_step %d" % wait_ns)
>  
>              # Test I/O at the max burst rate
> diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
> index 1225334cb8..4f363f295f 100755
> --- a/tests/qemu-iotests/149
> +++ b/tests/qemu-iotests/149
> @@ -314,13 +314,13 @@ def test_once(config, qemu_img=False):
>      image_size = 4 * oneTB
>      if qemu_img:
>          iotests.log("# Create image")
> -        qemu_img_create(config, image_size / oneMB)
> +        qemu_img_create(config, image_size // oneMB)
>      else:
>          iotests.log("# Create image")
> -        create_image(config, image_size / oneMB)
> +        create_image(config, image_size // oneMB)
>  
>      lowOffsetMB = 100
> -    highOffsetMB = 3 * oneTB / oneMB
> +    highOffsetMB = 3 * oneTB // oneMB
>  
>      try:
>          if not qemu_img:
> diff --git a/tests/qemu-iotests/151 b/tests/qemu-iotests/151
> index fe53b9f446..1bb74d67c4 100755
> --- a/tests/qemu-iotests/151
> +++ b/tests/qemu-iotests/151
> @@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase):
>                              'write -P 1 0 %i' % self.image_len);
>  
>          # Start some background requests
> -        for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
> +        for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
>              self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset)
> -        for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
> +        for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
>              self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
>  
>          # Start the block job
> @@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase):
>          self.assert_qmp(result, 'return', {})
>  
>          # Start some more requests
> -        for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
> +        for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
>              self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
> -        for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
> +        for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
>              self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
>  
>          # Wait for the READY event
> @@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase):
>          # the source) should be settled using the active mechanism.
>          # The mirror code itself asserts that the source BDS's dirty
>          # bitmap will stay clean between READY and COMPLETED.
> -        for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
> +        for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
>              self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
> -        for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
> +        for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
>              self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
>  
>          if sync_source_and_target:
> diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
> index 403842354e..5fd424761b 100755
> --- a/tests/qemu-iotests/163
> +++ b/tests/qemu-iotests/163
> @@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
>          entry_bits = 3
>          entry_size = 1 << entry_bits
>          l1_mask = 0x00fffffffffffe00
> -        div_roundup = lambda n, d: (n + d - 1) / d
> +        div_roundup = lambda n, d: (n + d - 1) // d
>  
>          def split_by_n(data, n):
>              for x in xrange(0, len(data), n):
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 7290c0b159..7ca94e9278 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -199,7 +199,7 @@ def create_image(name, size):
>      file = open(name, 'wb')
>      i = 0
>      while i < size:
> -        sector = struct.pack('>l504xl', i / 512, i / 512)
> +        sector = struct.pack('>l504xl', i // 512, i // 512)
>          file.write(sector)
>          i = i + 512
>      file.close()
> 

Found a few other occurrences:

030:''' % (event, errno, self.STREAM_BUFFER_SIZE / 512, event, event))
041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event))
041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event))

- Cleber.

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 04:14:52PM +0200, Max Reitz wrote:
> There are two imports that need to be modified when running the iotests
> under Python 3: One is StringIO, which no longer exists; instead, the
> StringIO class comes from the io module, so import it from there.  The
> other is the ConfigParser, which has just been renamed to configparser.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/iotests.py            | 8 ++++++--
>  tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
>  2 files changed, 11 insertions(+), 4 deletions(-)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 7ca94e9278..a64ea90fb4 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
>  
>      # We need to filter out the time taken from the output so that qemu-iotest
>      # can reliably diff the results against master output.
> -    import StringIO
> +    if sys.version_info.major >= 3:
> +        from io import StringIO
> +    else:
> +        from StringIO import StringIO

Considering that io.StringIO exists on Python 2.7, a comment
explaining why exactly it doesn't work would be nice.

But this shouldn't block this workaround, so:

Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>


> +
>      if debug:
>          output = sys.stdout
>          verbosity = 2
>          sys.argv.remove('-d')
>      else:
> -        output = StringIO.StringIO()
> +        output = StringIO()
>  
>      logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
>  
> diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
> index d45e2e0a6a..6b2d659dee 100755
> --- a/tests/qemu-iotests/nbd-fault-injector.py
> +++ b/tests/qemu-iotests/nbd-fault-injector.py
> @@ -48,7 +48,10 @@ import sys
>  import socket
>  import struct
>  import collections
> -import ConfigParser
> +if sys.version_info.major >= 3:
> +    import configparser
> +else:
> +    import ConfigParser as configparser
>  
>  FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
>  
> @@ -225,7 +228,7 @@ def parse_config(config):
>      return rules
>  
>  def load_rules(filename):
> -    config = ConfigParser.RawConfigParser()
> +    config = configparser.RawConfigParser()
>      with open(filename, 'rt') as f:
>          config.readfp(f, filename)
>      return parse_config(config)
> -- 
> 2.17.1
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 3/9] iotests: Use Python byte strings where appropriate

[Philippe Mathieu-Daudé]

On 15/10/2018 16:14, Max Reitz wrote:
> Since byte strings are no longer the default in Python 3, we have to
> explicitly use them where we need to, which is mostly when working with
> structures.  It also means that we need to open a file in binary mode
> when we want to use structures.
> 
> On the other hand, we have to accomodate for the fact that some
> functions (still) work with byte strings but we want to use unicode
> strings (in Python 3 at least, and it does not matter in Python 2).
> This includes base64 encoding, but it is most notable when working with
> the subprocess module: Either we set univeral_newlines to True so that

'universal_newlines'

> the default streams are opened in text mode (hence this parameter is
> aliased as "text" as of 3.7), or, if that is not possible, we have to
> decode the output to a normal string.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  scripts/qtest.py                         |  2 +-
>  tests/qemu-iotests/044                   |  8 ++++----
>  tests/qemu-iotests/149                   |  8 +++++---
>  tests/qemu-iotests/207                   |  4 ++--
>  tests/qemu-iotests/iotests.py            | 11 +++++++----
>  tests/qemu-iotests/nbd-fault-injector.py |  4 ++--
>  tests/qemu-iotests/qcow2.py              | 10 +++++-----
>  7 files changed, 26 insertions(+), 21 deletions(-)
> 
> diff --git a/scripts/qtest.py b/scripts/qtest.py
> index df0daf26ca..adf1fe3f26 100644
> --- a/scripts/qtest.py
> +++ b/scripts/qtest.py
> @@ -64,7 +64,7 @@ class QEMUQtestProtocol(object):
>  
>          @param qtest_cmd: qtest command text to be sent
>          """
> -        self._sock.sendall(qtest_cmd + "\n")
> +        self._sock.sendall((qtest_cmd + "\n").encode('utf-8'))
>  
>      def close(self):
>          self._sock.close()
> diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
> index 11ea0f4d35..69e736f687 100755
> --- a/tests/qemu-iotests/044
> +++ b/tests/qemu-iotests/044
> @@ -53,21 +53,21 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
>              fd.seek(off_reftable)
>  
>              for i in xrange(0, h.refcount_table_clusters):
> -                sector = ''.join(struct.pack('>Q',
> +                sector = b''.join(struct.pack('>Q',
>                      off_refblock + i * 64 * 512 + j * 512)
>                      for j in xrange(0, 64))
>                  fd.write(sector)
>  
>              # Write the refcount blocks
>              assert(fd.tell() == off_refblock)
> -            sector = ''.join(struct.pack('>H', 1) for j in xrange(0, 64 * 256))
> +            sector = b''.join(struct.pack('>H', 1) for j in range(0, 64 * 256))
>              for block in xrange(0, h.refcount_table_clusters):
>                  fd.write(sector)
>  
>              # Write the L1 table
>              assert(fd.tell() == off_l1)
>              assert(off_l2 + 512 * h.l1_size == off_data)
> -            table = ''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
> +            table = b''.join(struct.pack('>Q', (1 << 63) | off_l2 + 512 * j)
>                  for j in xrange(0, h.l1_size))
>              fd.write(table)
>  
> @@ -85,7 +85,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
>                  remaining = remaining - 1024 * 512
>                  off = off + 1024 * 512
>  
> -            table = ''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
> +            table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
>                  for j in xrange(0, remaining / 512))
>              fd.write(table)
>  
> diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
> index 9e0cad76f9..1225334cb8 100755
> --- a/tests/qemu-iotests/149
> +++ b/tests/qemu-iotests/149
> @@ -79,7 +79,7 @@ class LUKSConfig(object):
>  
>      def first_password_base64(self):
>          (pw, slot) = self.first_password()
> -        return base64.b64encode(pw)
> +        return base64.b64encode(pw.encode('ascii')).decode('ascii')
>  
>      def active_slots(self):
>          slots = []
> @@ -98,7 +98,8 @@ def verify_passwordless_sudo():
>      proc = subprocess.Popen(args,
>                              stdin=subprocess.PIPE,
>                              stdout=subprocess.PIPE,
> -                            stderr=subprocess.STDOUT)
> +                            stderr=subprocess.STDOUT,
> +                            universal_newlines=True)
>  
>      msg = proc.communicate()[0]
>  
> @@ -116,7 +117,8 @@ def cryptsetup(args, password=None):
>      proc = subprocess.Popen(fullargs,
>                              stdin=subprocess.PIPE,
>                              stdout=subprocess.PIPE,
> -                            stderr=subprocess.STDOUT)
> +                            stderr=subprocess.STDOUT,
> +                            universal_newlines=True)
>  
>      msg = proc.communicate(password)[0]
>  
> diff --git a/tests/qemu-iotests/207 b/tests/qemu-iotests/207
> index 444ae233ae..2d86a3da37 100755
> --- a/tests/qemu-iotests/207
> +++ b/tests/qemu-iotests/207
> @@ -109,7 +109,7 @@ with iotests.FilePath('t.img') as disk_path, \
>      md5_key = subprocess.check_output(
>          'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
>          'cut -d" " -f3 | base64 -d | md5sum -b | cut -d" " -f1',
> -        shell=True).rstrip()
> +        shell=True).rstrip().decode('ascii')
>  
>      vm.launch()
>      blockdev_create(vm, { 'driver': 'ssh',
> @@ -147,7 +147,7 @@ with iotests.FilePath('t.img') as disk_path, \
>      sha1_key = subprocess.check_output(
>          'ssh-keyscan -t rsa 127.0.0.1 2>/dev/null | grep -v "\\^#" | ' +
>          'cut -d" " -f3 | base64 -d | sha1sum -b | cut -d" " -f1',
> -        shell=True).rstrip()
> +        shell=True).rstrip().decode('ascii')
>  
>      vm.launch()
>      blockdev_create(vm, { 'driver': 'ssh',
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 10f2d17419..7290c0b159 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -104,7 +104,8 @@ def qemu_img_pipe(*args):
>      '''Run qemu-img and return its output'''
>      subp = subprocess.Popen(qemu_img_args + list(args),
>                              stdout=subprocess.PIPE,
> -                            stderr=subprocess.STDOUT)
> +                            stderr=subprocess.STDOUT,
> +                            universal_newlines=True)
>      exitcode = subp.wait()
>      if exitcode < 0:
>          sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
> @@ -128,7 +129,8 @@ def qemu_io(*args):
>      '''Run qemu-io and return the stdout data'''
>      args = qemu_io_args + list(args)
>      subp = subprocess.Popen(args, stdout=subprocess.PIPE,
> -                            stderr=subprocess.STDOUT)
> +                            stderr=subprocess.STDOUT,
> +                            universal_newlines=True)
>      exitcode = subp.wait()
>      if exitcode < 0:
>          sys.stderr.write('qemu-io received signal %i: %s\n' % (-exitcode, ' '.join(args)))
> @@ -149,7 +151,8 @@ class QemuIoInteractive:
>          self.args = qemu_io_args + list(args)
>          self._p = subprocess.Popen(self.args, stdin=subprocess.PIPE,
>                                     stdout=subprocess.PIPE,
> -                                   stderr=subprocess.STDOUT)
> +                                   stderr=subprocess.STDOUT,
> +                                   universal_newlines=True)
>          assert self._p.stdout.read(9) == 'qemu-io> '
>  
>      def close(self):
> @@ -193,7 +196,7 @@ def compare_images(img1, img2, fmt1=imgfmt, fmt2=imgfmt):
>  
>  def create_image(name, size):
>      '''Create a fully-allocated raw image with sector markers'''
> -    file = open(name, 'w')
> +    file = open(name, 'wb')
>      i = 0
>      while i < size:
>          sector = struct.pack('>l504xl', i / 512, i / 512)
> diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
> index 439a090eb6..d45e2e0a6a 100755
> --- a/tests/qemu-iotests/nbd-fault-injector.py
> +++ b/tests/qemu-iotests/nbd-fault-injector.py
> @@ -86,7 +86,7 @@ def recvall(sock, bufsize):
>              raise Exception('unexpected disconnect')
>          chunks.append(chunk)
>          received += len(chunk)
> -    return ''.join(chunks)
> +    return b''.join(chunks)
>  
>  class Rule(object):
>      def __init__(self, name, event, io, when):
> @@ -177,7 +177,7 @@ def handle_connection(conn, use_export):
>          req = read_request(conn)
>          if req.type == NBD_CMD_READ:
>              write_reply(conn, 0, req.handle)
> -            conn.send('\0' * req.len, event='data')
> +            conn.send(b'\0' * req.len, event='data')
>          elif req.type == NBD_CMD_WRITE:
>              _ = conn.recv(req.len, event='data')
>              write_reply(conn, 0, req.handle)
> diff --git a/tests/qemu-iotests/qcow2.py b/tests/qemu-iotests/qcow2.py
> index b95a837759..b392972d1b 100755
> --- a/tests/qemu-iotests/qcow2.py
> +++ b/tests/qemu-iotests/qcow2.py
> @@ -10,7 +10,7 @@ class QcowHeaderExtension:
>      def __init__(self, magic, length, data):
>          if length % 8 != 0:
>              padding = 8 - (length % 8)
> -            data += "\0" * padding
> +            data += b"\0" * padding
>  
>          self.magic  = magic
>          self.length = length
> @@ -103,7 +103,7 @@ class QcowHeader:
>  
>          fd.seek(self.header_length)
>          extensions = self.extensions
> -        extensions.append(QcowHeaderExtension(0, 0, ""))
> +        extensions.append(QcowHeaderExtension(0, 0, b""))
>          for ex in extensions:
>              buf = struct.pack('>II', ex.magic, ex.length)
>              fd.write(buf)
> @@ -137,8 +137,8 @@ class QcowHeader:
>          for ex in self.extensions:
>  
>              data = ex.data[:ex.length]
> -            if all(c in string.printable for c in data):
> -                data = "'%s'" % data
> +            if all(c in string.printable.encode('ascii') for c in data):
> +                data = "'%s'" % data.decode('ascii')
>              else:
>                  data = "<binary>"
>  
> @@ -178,7 +178,7 @@ def cmd_add_header_ext(fd, magic, data):
>          sys.exit(1)
>  
>      h = QcowHeader(fd)
> -    h.extensions.append(QcowHeaderExtension.create(magic, data))
> +    h.extensions.append(QcowHeaderExtension.create(magic, data.encode('ascii')))
>      h.update(fd)
>  
>  def cmd_add_header_ext_stdio(fd, magic):
> 

Re: [Qemu-devel] [PATCH 0/9] iotests: Make them work for both Python 2 and 3

[Philippe Mathieu-Daudé]

Hi Max,

On 15/10/2018 16:14, Max Reitz wrote:
> This series prepares the iotests to work with both Python 2 and 3.  In
> some places, it adds version-specific code and decides what to do based
> on the version (for instance, whether to import the StringIO class from
> the 'io' or the 'StringIO' module), but most of the time, it just makes
> code work for both versions in general.
> 
> And when we make the switch to make Python 3 mandatory, we can simply
> drop the Python 2 branches.
> 
> 
> Max Reitz (9):
>   iotests: Make nbd-fault-injector flush
>   iotests: Flush in iotests.py's QemuIoInteractive
>   iotests: Use Python byte strings where appropriate
>   iotests: Use // for Python integer division
>   iotests: Different iterator behavior in Python 3
>   iotests: Explicitly inherit FDs in Python
>   iotests: 'new' module replacement in 169
>   iotests: Modify imports for Python 3
>   iotests: Unify log outputs between Python 2 and 3

You forgot:

    MAINTAINERS: Add myself in the Python scripts section

;)

> 
>  scripts/qemu.py                          |  13 +-
>  scripts/qmp/qmp.py                       |   7 +
>  scripts/qtest.py                         |   2 +-
>  tests/qemu-iotests/040                   |   4 +-
>  tests/qemu-iotests/044                   |  20 +-
>  tests/qemu-iotests/056                   |   2 +-
>  tests/qemu-iotests/065                   |   4 +-
>  tests/qemu-iotests/083.out               |   9 +
>  tests/qemu-iotests/093                   |  18 +-
>  tests/qemu-iotests/124                   |   4 +-
>  tests/qemu-iotests/139                   |   2 +-
>  tests/qemu-iotests/147                   |   7 +
>  tests/qemu-iotests/149                   |  14 +-
>  tests/qemu-iotests/151                   |  12 +-
>  tests/qemu-iotests/163                   |   8 +-
>  tests/qemu-iotests/169                   |   3 +-
>  tests/qemu-iotests/194.out               |  22 +-
>  tests/qemu-iotests/202.out               |  12 +-
>  tests/qemu-iotests/203.out               |  14 +-
>  tests/qemu-iotests/206.out               | 144 +++----
>  tests/qemu-iotests/207                   |   4 +-
>  tests/qemu-iotests/207.out               |  52 +--
>  tests/qemu-iotests/208.out               |   8 +-
>  tests/qemu-iotests/210.out               |  72 ++--
>  tests/qemu-iotests/211.out               |  66 +--
>  tests/qemu-iotests/212.out               | 102 ++---
>  tests/qemu-iotests/213.out               | 124 +++---
>  tests/qemu-iotests/216.out               |   4 +-
>  tests/qemu-iotests/218.out               |  20 +-
>  tests/qemu-iotests/219.out               | 526 +++++++++++------------
>  tests/qemu-iotests/222.out               |  24 +-
>  tests/qemu-iotests/iotests.py            |  64 ++-
>  tests/qemu-iotests/nbd-fault-injector.py |  12 +-
>  tests/qemu-iotests/qcow2.py              |  10 +-
>  34 files changed, 745 insertions(+), 664 deletions(-)
> 

Re: [Qemu-devel] [PATCH 9/9] iotests: Unify log outputs between Python 2 and 3

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 04:14:53PM +0200, Max Reitz wrote:
> When dumping an object into the log, there are differences between
> Python 2 and 3.  First, unicode strings are prefixed by 'u' in Python 2
> (they are no longer in 3, because unicode strings are the default
> there).  [...]

This could be addressed using JSON.  See below[1].

> [...] Second, the order of keys in dicts may differ.  [...]

Can be addressed using json.dumps(..., sort_keys=True).

> [...] Third,
> especially long numbers are longs in Python 2 and thus get an 'L'
> suffix, which does not happen in Python 3.

print() doesn't add a L suffix on Python 2.7 on my system:

Python 2.7.15 (default, May 15 2018, 15:37:31)
[GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> print(99999999999999999999999999999999999999999999999999999999999999999999999999999999999L)
99999999999999999999999999999999999999999999999999999999999999999999999999999999999

So I assume this happens only for QMP commands.  It would be
addressed if using JSON, too.


> 
> To get around these differences, this patch introduces functions to
> convert an object to a string that looks the same regardless of the
> Python version: In Python 2, they decode unicode strings to byte strings
> (normal str); in Python 3, they encode byte strings to unicode strings
> (normal str).  They also manually convert lists and dicts to strings,
> which allows sorting the dicts by key, so we are no longer at the mercy
> of the internal implementation when it comes to how the keys appear in
> the output.
> 
> This changes the output of all tests that use these logging functions.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/194.out    |  22 +-
>  tests/qemu-iotests/202.out    |  12 +-
>  tests/qemu-iotests/203.out    |  14 +-
>  tests/qemu-iotests/206.out    | 144 +++++-----
>  tests/qemu-iotests/207.out    |  52 ++--
>  tests/qemu-iotests/208.out    |   8 +-
>  tests/qemu-iotests/210.out    |  72 ++---
>  tests/qemu-iotests/211.out    |  66 ++---
>  tests/qemu-iotests/212.out    | 102 +++----
>  tests/qemu-iotests/213.out    | 124 ++++----
>  tests/qemu-iotests/216.out    |   4 +-
>  tests/qemu-iotests/218.out    |  20 +-
>  tests/qemu-iotests/219.out    | 526 +++++++++++++++++-----------------
>  tests/qemu-iotests/222.out    |  24 +-
>  tests/qemu-iotests/iotests.py |  42 ++-
>  15 files changed, 634 insertions(+), 598 deletions(-)
[...]
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index a64ea90fb4..f8dbc8cc71 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -250,10 +250,45 @@ def filter_img_info(output, filename):
>          lines.append(line)
>      return '\n'.join(lines)
>  
> +def log_to_string_repr(obj):
> +    # Normal Python 3 strings are the unicode strings from Python 2;
> +    # and normal Python 2 strings are byte strings in Python 3.  Thus,
> +    # we convert bytes -> str in py3 and unicode -> str in py2.
> +    if sys.version_info.major >= 3:
> +        if type(obj) is bytes:
> +            return repr(obj.decode('utf-8'))
> +    else:
> +        if type(obj) is unicode:
> +            return repr(obj.encode('utf-8'))
> +        elif type(obj) is long:
> +            return str(obj) # repr() would include an 'L' suffix
> +
> +    if type(obj) is list:
> +        return '[' + ', '.join(map(log_to_string_repr, obj)) + ']'
> +    elif type(obj) is dict:
> +        return '{' + ', '.join(map(lambda k: log_to_string_repr(k) + ': ' +
> +                                             log_to_string_repr(obj[k]),
> +                                   sorted(obj.keys()))) + '}'
> +    else:
> +        return repr(obj)

I assume this function exists only because of QMP logging,
correct?

I would just use json.dumps() at qmp_log(), see below[1].


> +
> +def log_to_string(obj):
> +    if type(obj) is str:
> +        return obj
> +
> +    if sys.version_info.major >= 3:
> +        if type(obj) is bytes:
> +            return obj.decode('utf-8')

Do you know how many of existing log() calls actually pass a byte
string as argument?

> +    else:
> +        if type(obj) is unicode:
> +            return obj.encode('utf-8')

Is this really necessary?  The existing code just calls
print(msg) and it works, doesn't it?



> +
> +    return log_to_string_repr(obj)
> +
>  def log(msg, filters=[]):
>      for flt in filters:
>          msg = flt(msg)
> -    print(msg)
> +    print(log_to_string(msg))
>  
>  class Timeout:
>      def __init__(self, seconds, errmsg = "Timeout"):
> @@ -441,10 +476,11 @@ class VM(qtest.QEMUQtestMachine):
>          return result
>  
>      def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs):
> -        logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs)
> +        logmsg = "{'execute': '%s', 'arguments': %s}" % \
> +            (cmd, log_to_string(kwargs))
>          log(logmsg, filters)
>          result = self.qmp(cmd, **kwargs)
> -        log(str(result), filters)
> +        log(log_to_string(result), filters)

[1]

If we're being forced to regenerate all the QMP output in the
.out files, what about always using JSON instead of the hack at
log_to_string_repr()?  i.e.:

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index a64ea90fb4..0b28dc2a65 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -441,10 +441,12 @@ class VM(qtest.QEMUQtestMachine):
         return result
 
     def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs):
-        logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs)
+        logmsg = '{"execute": %s, "arguments": %s}' % \
+                 (json.dumps(cmd, sort_keys=True),
+                  json.dumps(kwargs, sort_keys=True))
         log(logmsg, filters)
         result = self.qmp(cmd, **kwargs)
-        log(str(result), filters)
+        log(json.dumps(result, sort_keys=True), filters)
         return result
 
     def run_job(self, job, auto_finalize=True, auto_dismiss=False):

>          return result
>  
>      def run_job(self, job, auto_finalize=True, auto_dismiss=False):
> -- 
> 2.17.1
> 
> 

-- 
Eduardo

Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3

[Cleber Rosa]

On 10/15/18 10:14 AM, Max Reitz wrote:
> In Python 3, several functions now return iterators instead of lists.
> This includes range(), items(), map(), and filter().  This means that if
> we really want a list, we have to wrap those instances with list().  On
> the other hand, sometimes we do just want an iterator, in which case we
> have sometimes used xrange() and iteritems() which no longer exist in
> Python 3.  Just change these calls to be range() and items(), which
> costs a bit of performance in Python 2, but will do the right thing in
> Python 3 (which is what is important).
> 
> In one instance, we only wanted the first instance of the result of a
> filter() call.  Instead of using next(filter()) which would work only in
> Python 3, or list(filter())[0] which would work everywhere but is a bit
> weird, this instance is changed to a single-line for with next() wrapped
> around, which works both in 2.7 and 3.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/044 | 12 ++++++------
>  tests/qemu-iotests/056 |  2 +-
>  tests/qemu-iotests/065 |  4 ++--
>  tests/qemu-iotests/124 |  4 ++--
>  tests/qemu-iotests/139 |  2 +-
>  tests/qemu-iotests/163 |  6 +++---
>  6 files changed, 15 insertions(+), 15 deletions(-)
> 

You have 2 files here which use xrange (which is a manageable size, and
whose occurrences involve a moderate size of items) to also consider:

if sys.version_info.major == 2:
   range = xrange

Defaulting to the Python 3 names, but behaving the same across Python 2
and 3.

To do the same for dict.iteritems() => dict.items() requires a lot more
code, so I'd stay away from it for now.  Also, it looks like most of
those dicts are small in size (**kwargs and the like).

Other than that suggestion,

Reviewed-by: Cleber Rosa <crosa@redhat.com>

Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python

[Cleber Rosa]

On 10/15/18 10:14 AM, Max Reitz wrote:
> Python 3.2 introduced the inheritable attribute for FDs.  At the same
> time, it changed the default so that all FDs are not inheritable by
> default, that only inheritable FDs are inherited to subprocesses, and
> only if close_fds is explicitly set to False.
> 

It's actually a change that was introduced in 3.4:

https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors

> Adhere to this by setting close_fds to False when working with
> subprocesses that may want to inherit FDs, and by trying to
> set_inheritable() on FDs that we do want to bequeath to them.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  scripts/qemu.py        | 13 +++++++++++--
>  scripts/qmp/qmp.py     |  7 +++++++
>  tests/qemu-iotests/147 |  7 +++++++
>  3 files changed, 25 insertions(+), 2 deletions(-)
> 
> diff --git a/scripts/qemu.py b/scripts/qemu.py
> index f099ce7278..28366c4a67 100644
> --- a/scripts/qemu.py
> +++ b/scripts/qemu.py
> @@ -142,10 +142,18 @@ class QEMUMachine(object):
>          if opts:
>              options.append(opts)
>  
> +        # This did not exist before 3.2, but since then it is
> +        # mandatory for our purpose
> +        try:

Version should be 3.4 here as well.

> +            os.set_inheritable(fd, True)
> +        except AttributeError:
> +            pass
> +

Doing hasattr(os, "set_inheritable") is cheaper.

- Cleber.

>          self._args.append('-add-fd')
>          self._args.append(','.join(options))
>          return self
>  
> +    # The caller needs to make sure the FD is inheritable
>      def send_fd_scm(self, fd_file_path):
>          # In iotest.py, the qmp should always use unix socket.
>          assert self._qmp.is_scm_available()
> @@ -159,7 +167,7 @@ class QEMUMachine(object):
>                      "%s" % fd_file_path]
>          devnull = open(os.path.devnull, 'rb')
>          proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
> -                                stderr=subprocess.STDOUT)
> +                                stderr=subprocess.STDOUT, close_fds=False)
>          output = proc.communicate()[0]
>          if output:
>              LOG.debug(output)
> @@ -280,7 +288,8 @@ class QEMUMachine(object):
>                                         stdin=devnull,
>                                         stdout=self._qemu_log_file,
>                                         stderr=subprocess.STDOUT,
> -                                       shell=False)
> +                                       shell=False,
> +                                       close_fds=False)
>          self._post_launch()
>  
>      def wait(self):
> diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
> index 5c8cf6a056..009be8345b 100644
> --- a/scripts/qmp/qmp.py
> +++ b/scripts/qmp/qmp.py
> @@ -10,6 +10,7 @@
>  
>  import json
>  import errno
> +import os
>  import socket
>  import logging
>  
> @@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object):
>          return self.__sock.fileno()
>  
>      def is_scm_available(self):
> +        # This did not exist before 3.2, but since then it is
> +        # mandatory for our purpose
> +        try:
> +            os.set_inheritable(self.get_sock_fd(), True)
> +        except AttributeError:
> +            pass
>          return self.__sock.family == socket.AF_UNIX
> diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
> index d2081df84b..b58455645b 100755
> --- a/tests/qemu-iotests/147
> +++ b/tests/qemu-iotests/147
> @@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase):
>          sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>          sockfd.connect(unix_socket)
>  
> +        # This did not exist before 3.2, but since then it is
> +        # mandatory for our purpose
> +        try:
> +            os.set_inheritable(sockfd.fileno(), True)
> +        except AttributeError:
> +            pass
> +
>          result = self.vm.send_fd_scm(str(sockfd.fileno()))
>          self.assertEqual(result, 0, 'Failed to send socket FD')
>  
> 

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

[Cleber Rosa]

On 10/15/18 10:14 AM, Max Reitz wrote:
> iotest 169 uses the 'new' module to add methods to a class.  This module
> no longer exists in Python 3.  Instead, we can use a lambda.  Best of
> all, this works in 2.7 just as well.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/169 | 3 +--
>  1 file changed, 1 insertion(+), 2 deletions(-)
> 
> diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
> index f243db9955..e5614b159d 100755
> --- a/tests/qemu-iotests/169
> +++ b/tests/qemu-iotests/169
> @@ -23,7 +23,6 @@ import iotests
>  import time
>  import itertools
>  import operator
> -import new
>  from iotests import qemu_img
>  
>  
> @@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
>  
>  def inject_test_case(klass, name, method, *args, **kwargs):
>      mc = operator.methodcaller(method, *args, **kwargs)
> -    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
> +    setattr(klass, 'test_' + name, lambda self: mc(self))
>  
>  for cmb in list(itertools.product((True, False), repeat=4)):
>      name = ('_' if cmb[0] else '_not_') + 'persistent_'
> 

This can be simplified with:

diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
index e5614b159d..2199f14ae7 100755
--- a/tests/qemu-iotests/169
+++ b/tests/qemu-iotests/169
@@ -22,7 +22,6 @@ import os
 import iotests
 import time
 import itertools
-import operator
 from iotests import qemu_img


@@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
             self.check_bitmap(self.vm_b, sha256 if persistent else False)


-def inject_test_case(klass, name, method, *args, **kwargs):
-    mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, lambda self: mc(self))
-
 for cmb in list(itertools.product((True, False), repeat=4)):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
     name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
     name += '_online' if cmb[2] else '_offline'
     name += '_shared' if cmb[3] else '_nonshared'

-    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
-                     *list(cmb))
+    setattr(TestDirtyBitmapMigration, 'test_' + name,
+            lambda self: TestDirtyBitmapMigration.do_test_migration(
+                self, *list(cmb)))


 if __name__ == '__main__':

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 07:38:45PM -0400, Cleber Rosa wrote:
> 
> 
> On 10/15/18 10:14 AM, Max Reitz wrote:
> > iotest 169 uses the 'new' module to add methods to a class.  This module
> > no longer exists in Python 3.  Instead, we can use a lambda.  Best of
> > all, this works in 2.7 just as well.
> > 
> > Signed-off-by: Max Reitz <mreitz@redhat.com>
> > ---
> >  tests/qemu-iotests/169 | 3 +--
> >  1 file changed, 1 insertion(+), 2 deletions(-)
> > 
> > diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
> > index f243db9955..e5614b159d 100755
> > --- a/tests/qemu-iotests/169
> > +++ b/tests/qemu-iotests/169
> > @@ -23,7 +23,6 @@ import iotests
> >  import time
> >  import itertools
> >  import operator
> > -import new
> >  from iotests import qemu_img
> >  
> >  
> > @@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
> >  
> >  def inject_test_case(klass, name, method, *args, **kwargs):
> >      mc = operator.methodcaller(method, *args, **kwargs)
> > -    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
> > +    setattr(klass, 'test_' + name, lambda self: mc(self))
> >  
> >  for cmb in list(itertools.product((True, False), repeat=4)):
> >      name = ('_' if cmb[0] else '_not_') + 'persistent_'
> > 
> 
> This can be simplified with:
> 
> diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
> index e5614b159d..2199f14ae7 100755
> --- a/tests/qemu-iotests/169
> +++ b/tests/qemu-iotests/169
> @@ -22,7 +22,6 @@ import os
>  import iotests
>  import time
>  import itertools
> -import operator
>  from iotests import qemu_img
> 
> 
> @@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
>              self.check_bitmap(self.vm_b, sha256 if persistent else False)
> 
> 
> -def inject_test_case(klass, name, method, *args, **kwargs):
> -    mc = operator.methodcaller(method, *args, **kwargs)
> -    setattr(klass, 'test_' + name, lambda self: mc(self))
> -
>  for cmb in list(itertools.product((True, False), repeat=4)):
>      name = ('_' if cmb[0] else '_not_') + 'persistent_'
>      name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
>      name += '_online' if cmb[2] else '_offline'
>      name += '_shared' if cmb[3] else '_nonshared'
> 
> -    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
> -                     *list(cmb))
> +    setattr(TestDirtyBitmapMigration, 'test_' + name,
> +            lambda self: TestDirtyBitmapMigration.do_test_migration(
> +                self, *list(cmb)))

I'm not fond of the long multi-line lambda expression, but I love
that you got rid of operator.methodcaller().  :)

However, this doesn't seem to work: `*list(cmb)` will be
evaluated only when the lambda is called, and `cmb` will be
always `(False, False, False, False)`.

I was going to suggest defining a nested function inside
inject_test_case() to replace operator.methodcaller(), but then I
thought it was not worth it.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

[Cleber Rosa]

On 10/15/18 5:17 PM, Eduardo Habkost wrote:
> On Mon, Oct 15, 2018 at 04:14:52PM +0200, Max Reitz wrote:
>> There are two imports that need to be modified when running the iotests
>> under Python 3: One is StringIO, which no longer exists; instead, the
>> StringIO class comes from the io module, so import it from there.  The
>> other is the ConfigParser, which has just been renamed to configparser.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>  tests/qemu-iotests/iotests.py            | 8 ++++++--
>>  tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
>>  2 files changed, 11 insertions(+), 4 deletions(-)
>>
>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>> index 7ca94e9278..a64ea90fb4 100644
>> --- a/tests/qemu-iotests/iotests.py
>> +++ b/tests/qemu-iotests/iotests.py
>> @@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
>>  
>>      # We need to filter out the time taken from the output so that qemu-iotest
>>      # can reliably diff the results against master output.
>> -    import StringIO
>> +    if sys.version_info.major >= 3:
>> +        from io import StringIO
>> +    else:
>> +        from StringIO import StringIO
> 
> Considering that io.StringIO exists on Python 2.7, a comment
> explaining why exactly it doesn't work would be nice.
> 

Another possibility, that I find self explanatory:

import io

if sys.version_info.major >= 3:
   output = io.StringIO()
else:
   output = io.BytesIO()

- Cleber.

> But this shouldn't block this workaround, so:
> 
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> 
> 
>> +
>>      if debug:
>>          output = sys.stdout
>>          verbosity = 2
>>          sys.argv.remove('-d')
>>      else:
>> -        output = StringIO.StringIO()
>> +        output = StringIO()
>>  
>>      logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
>>  
>> diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
>> index d45e2e0a6a..6b2d659dee 100755
>> --- a/tests/qemu-iotests/nbd-fault-injector.py
>> +++ b/tests/qemu-iotests/nbd-fault-injector.py
>> @@ -48,7 +48,10 @@ import sys
>>  import socket
>>  import struct
>>  import collections
>> -import ConfigParser
>> +if sys.version_info.major >= 3:
>> +    import configparser
>> +else:
>> +    import ConfigParser as configparser
>>  
>>  FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
>>  
>> @@ -225,7 +228,7 @@ def parse_config(config):
>>      return rules
>>  
>>  def load_rules(filename):
>> -    config = ConfigParser.RawConfigParser()
>> +    config = configparser.RawConfigParser()
>>      with open(filename, 'rt') as f:
>>          config.readfp(f, filename)
>>      return parse_config(config)
>> -- 
>> 2.17.1
>>
> 

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

[Eduardo Habkost]

On Mon, Oct 15, 2018 at 08:05:02PM -0400, Cleber Rosa wrote:
> 
> 
> On 10/15/18 5:17 PM, Eduardo Habkost wrote:
> > On Mon, Oct 15, 2018 at 04:14:52PM +0200, Max Reitz wrote:
> >> There are two imports that need to be modified when running the iotests
> >> under Python 3: One is StringIO, which no longer exists; instead, the
> >> StringIO class comes from the io module, so import it from there.  The
> >> other is the ConfigParser, which has just been renamed to configparser.
> >>
> >> Signed-off-by: Max Reitz <mreitz@redhat.com>
> >> ---
> >>  tests/qemu-iotests/iotests.py            | 8 ++++++--
> >>  tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
> >>  2 files changed, 11 insertions(+), 4 deletions(-)
> >>
> >> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> >> index 7ca94e9278..a64ea90fb4 100644
> >> --- a/tests/qemu-iotests/iotests.py
> >> +++ b/tests/qemu-iotests/iotests.py
> >> @@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
> >>  
> >>      # We need to filter out the time taken from the output so that qemu-iotest
> >>      # can reliably diff the results against master output.
> >> -    import StringIO
> >> +    if sys.version_info.major >= 3:
> >> +        from io import StringIO
> >> +    else:
> >> +        from StringIO import StringIO
> > 
> > Considering that io.StringIO exists on Python 2.7, a comment
> > explaining why exactly it doesn't work would be nice.
> > 
> 
> Another possibility, that I find self explanatory:
> 
> import io
> 
> if sys.version_info.major >= 3:
>    output = io.StringIO()
> else:
>    output = io.BytesIO()

Looks nice and clean.

But I'm not sure all output sent to `output` when running with
Python 2 will be byte strings.  What if `unittest.TextTestRunner`
tries to write unicode strings to `output`?

-- 
Eduardo

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

[Cleber Rosa]

On 10/15/18 7:57 PM, Eduardo Habkost wrote:
> On Mon, Oct 15, 2018 at 07:38:45PM -0400, Cleber Rosa wrote:
>>
>>
>> On 10/15/18 10:14 AM, Max Reitz wrote:
>>> iotest 169 uses the 'new' module to add methods to a class.  This module
>>> no longer exists in Python 3.  Instead, we can use a lambda.  Best of
>>> all, this works in 2.7 just as well.
>>>
>>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>>> ---
>>>  tests/qemu-iotests/169 | 3 +--
>>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>>
>>> diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
>>> index f243db9955..e5614b159d 100755
>>> --- a/tests/qemu-iotests/169
>>> +++ b/tests/qemu-iotests/169
>>> @@ -23,7 +23,6 @@ import iotests
>>>  import time
>>>  import itertools
>>>  import operator
>>> -import new
>>>  from iotests import qemu_img
>>>  
>>>  
>>> @@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
>>>  
>>>  def inject_test_case(klass, name, method, *args, **kwargs):
>>>      mc = operator.methodcaller(method, *args, **kwargs)
>>> -    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
>>> +    setattr(klass, 'test_' + name, lambda self: mc(self))
>>>  
>>>  for cmb in list(itertools.product((True, False), repeat=4)):
>>>      name = ('_' if cmb[0] else '_not_') + 'persistent_'
>>>
>>
>> This can be simplified with:
>>
>> diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
>> index e5614b159d..2199f14ae7 100755
>> --- a/tests/qemu-iotests/169
>> +++ b/tests/qemu-iotests/169
>> @@ -22,7 +22,6 @@ import os
>>  import iotests
>>  import time
>>  import itertools
>> -import operator
>>  from iotests import qemu_img
>>
>>
>> @@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
>>              self.check_bitmap(self.vm_b, sha256 if persistent else False)
>>
>>
>> -def inject_test_case(klass, name, method, *args, **kwargs):
>> -    mc = operator.methodcaller(method, *args, **kwargs)
>> -    setattr(klass, 'test_' + name, lambda self: mc(self))
>> -
>>  for cmb in list(itertools.product((True, False), repeat=4)):
>>      name = ('_' if cmb[0] else '_not_') + 'persistent_'
>>      name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
>>      name += '_online' if cmb[2] else '_offline'
>>      name += '_shared' if cmb[3] else '_nonshared'
>>
>> -    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
>> -                     *list(cmb))
>> +    setattr(TestDirtyBitmapMigration, 'test_' + name,
>> +            lambda self: TestDirtyBitmapMigration.do_test_migration(
>> +                self, *list(cmb)))
> 
> I'm not fond of the long multi-line lambda expression, but I love
> that you got rid of operator.methodcaller().  :)
> 
> However, this doesn't seem to work: `*list(cmb)` will be
> evaluated only when the lambda is called, and `cmb` will be
> always `(False, False, False, False)`.
> 
> I was going to suggest defining a nested function inside
> inject_test_case() to replace operator.methodcaller(), but then I
> thought it was not worth it.
> 

You're right! I missed that.  Anyway, another possibility:

diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
index e5614b159d..cd0d9c289c 100755
--- a/tests/qemu-iotests/169
+++ b/tests/qemu-iotests/169
@@ -22,7 +22,7 @@ import os
 import iotests
 import time
 import itertools
-import operator
+import functools
 from iotests import qemu_img


@@ -140,20 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
             self.vm_b.launch()
             self.check_bitmap(self.vm_b, sha256 if persistent else False)

-
-def inject_test_case(klass, name, method, *args, **kwargs):
-    mc = operator.methodcaller(method, *args, **kwargs)
-    setattr(klass, 'test_' + name, lambda self: mc(self))
-
-for cmb in list(itertools.product((True, False), repeat=4)):
+for cmb in itertools.product((True, False), repeat=4):
     name = ('_' if cmb[0] else '_not_') + 'persistent_'
     name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
     name += '_online' if cmb[2] else '_offline'
     name += '_shared' if cmb[3] else '_nonshared'

-    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
-                     *list(cmb))
-
+    test =
functools.partialmethod(TestDirtyBitmapMigration.do_test_migration,
+                                   *cmb)
+    setattr(TestDirtyBitmapMigration, 'test_' + name, test)

 if __name__ == '__main__':
     iotests.main(supported_fmts=['qcow2'])

Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush

[Eric Blake]

On 10/15/18 9:14 AM, Max Reitz wrote:
> When closing a connection, make the nbd-fault-injector flush the socket.
> Without this, the output is a bit unreliable with Python 3.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>   tests/qemu-iotests/083.out               | 9 +++++++++
>   tests/qemu-iotests/nbd-fault-injector.py | 1 +
>   2 files changed, 10 insertions(+)

I already had a complaint that the error message in 083.out should NOT 
be printing a message (whether the server is python 2 and auto-flushes, 
or python 3 and needs an explicit flush, the message itself is 
pointless, and the test is racy as a result).  We may need to revisit 
this patch when that thread is resolved.

https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg01041.html

That said, I'm not opposed to this patch, if it gets iotests to be more 
useful in the meantime.

> 
> diff --git a/tests/qemu-iotests/083.out b/tests/qemu-iotests/083.out
> index be6079d27e..f9af8bb691 100644
> --- a/tests/qemu-iotests/083.out
> +++ b/tests/qemu-iotests/083.out
> @@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
>   
>   === Check disconnect after neg2 ===
>   
> +Unable to read from socket: Connection reset by peer
>   Connection closed
>   read failed: Input/output error
>   
> @@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
>   
>   === Check disconnect before request ===
>   
> +Unable to read from socket: Connection reset by peer
>   Connection closed
>   read failed: Input/output error
>   
> @@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/
>   
>   === Check disconnect after neg-classic ===
>   
> +Unable to read from socket: Connection reset by peer
>   Connection closed
>   read failed: Input/output error
>   
> @@ -161,6 +164,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
>   
>   === Check disconnect after neg2 ===
>   
> +Unable to read from socket: Connection reset by peer
> +Connection closed
>   read failed: Input/output error
>   
>   === Check disconnect 8 neg2 ===
> @@ -173,6 +178,8 @@ can't open device nbd+unix:///foo?socket=TEST_DIR/nbd.sock
>   
>   === Check disconnect before request ===
>   
> +Unable to read from socket: Connection reset by peer
> +Connection closed
>   read failed: Input/output error
>   
>   === Check disconnect after request ===
> @@ -234,6 +241,8 @@ can't open device nbd+unix:///?socket=TEST_DIR/nbd.sock
>   
>   === Check disconnect after neg-classic ===
>   
> +Unable to read from socket: Connection reset by peer
> +Connection closed
>   read failed: Input/output error
>   
>   *** done
> diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
> index f9193c0fae..439a090eb6 100755
> --- a/tests/qemu-iotests/nbd-fault-injector.py
> +++ b/tests/qemu-iotests/nbd-fault-injector.py
> @@ -112,6 +112,7 @@ class FaultInjectionSocket(object):
>               if rule.match(event, io):
>                   if rule.when == 0 or bufsize is None:
>                       print('Closing connection on rule match %s' % rule.name)
> +                    self.sock.flush()
>                       sys.exit(0)
>                   if rule.when != -1:
>                       return rule.when
> 

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 2944 bytes --]

On 15.10.18 20:59, Cleber Rosa wrote:
> 
> 
> On 10/15/18 10:14 AM, Max Reitz wrote:
>> There are two imports that need to be modified when running the iotests
>> under Python 3: One is StringIO, which no longer exists; instead, the
>> StringIO class comes from the io module, so import it from there.  The
>> other is the ConfigParser, which has just been renamed to configparser.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>  tests/qemu-iotests/iotests.py            | 8 ++++++--
>>  tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
>>  2 files changed, 11 insertions(+), 4 deletions(-)
>>
>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>> index 7ca94e9278..a64ea90fb4 100644
>> --- a/tests/qemu-iotests/iotests.py
>> +++ b/tests/qemu-iotests/iotests.py
>> @@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
>>  
>>      # We need to filter out the time taken from the output so that qemu-iotest
>>      # can reliably diff the results against master output.
>> -    import StringIO
>> +    if sys.version_info.major >= 3:
>> +        from io import StringIO
>> +    else:
>> +        from StringIO import StringIO
>> +
>>      if debug:
>>          output = sys.stdout
>>          verbosity = 2
>>          sys.argv.remove('-d')
>>      else:
>> -        output = StringIO.StringIO()
>> +        output = StringIO()
>>  
>>      logging.basicConfig(level=(logging.DEBUG if debug else logging.WARN))
>>  
>> diff --git a/tests/qemu-iotests/nbd-fault-injector.py b/tests/qemu-iotests/nbd-fault-injector.py
>> index d45e2e0a6a..6b2d659dee 100755
>> --- a/tests/qemu-iotests/nbd-fault-injector.py
>> +++ b/tests/qemu-iotests/nbd-fault-injector.py
>> @@ -48,7 +48,10 @@ import sys
>>  import socket
>>  import struct
>>  import collections
>> -import ConfigParser
>> +if sys.version_info.major >= 3:
>> +    import configparser
>> +else:
>> +    import ConfigParser as configparser
>>  
>>  FAKE_DISK_SIZE = 8 * 1024 * 1024 * 1024 # 8 GB
>>  
>> @@ -225,7 +228,7 @@ def parse_config(config):
>>      return rules
>>  
>>  def load_rules(filename):
>> -    config = ConfigParser.RawConfigParser()
>> +    config = configparser.RawConfigParser()
>>      with open(filename, 'rt') as f:
>>          config.readfp(f, filename)
>>      return parse_config(config)
>>
> 
> This may be a type of culture clash (on my side, due to not enough QEMU
> culture), but shouldn't this be applied before anything else on this series?
> 
> I mean, PATCH 1/9 is supposed to fix the reliability aspects of
> nbd-fault-injector under Python 3, but without this patch, it won't
> actually run on Python 3.

I don't mind, I followed no specific order.  Well, patch 9 is patch 9
because it is the largest one, so I didn't want to discourage people
before the end of the series. :-)

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 3/9] iotests: Use Python byte strings where appropriate

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 2284 bytes --]

On 15.10.18 21:53, Eduardo Habkost wrote:
> On Mon, Oct 15, 2018 at 04:14:47PM +0200, Max Reitz wrote:
>> Since byte strings are no longer the default in Python 3, we have to
>> explicitly use them where we need to, which is mostly when working with
>> structures.  It also means that we need to open a file in binary mode
>> when we want to use structures.
>>
>> On the other hand, we have to accomodate for the fact that some
>> functions (still) work with byte strings but we want to use unicode
>> strings (in Python 3 at least, and it does not matter in Python 2).
>> This includes base64 encoding, but it is most notable when working with
>> the subprocess module: Either we set univeral_newlines to True so that
>> the default streams are opened in text mode (hence this parameter is
>> aliased as "text" as of 3.7), or, if that is not possible, we have to
>> decode the output to a normal string.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
> [...]
>> diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
>> index 9e0cad76f9..1225334cb8 100755
>> --- a/tests/qemu-iotests/149
>> +++ b/tests/qemu-iotests/149
>> @@ -79,7 +79,7 @@ class LUKSConfig(object):
>>  
>>      def first_password_base64(self):
>>          (pw, slot) = self.first_password()
>> -        return base64.b64encode(pw)
>> +        return base64.b64encode(pw.encode('ascii')).decode('ascii')
> 
> Would we want to have a test case for non-ascii passwords in the
> future?  In that case you would probably need to make
> self.passwords[] contain byte strings instead of text.

I remember someone once providing a non-ASCII initial password to some
system for me.  I remember the system was running Windows, and I was
running Linux, so the system expected ISO-8859-1, while I was sending UTF-8.

The moral of the story is that you probably don't want non-ASCII
passwords.  And if we do want to test them, well, we'll need to decide
on an encoding then (or use byte strings, as you suggest).

> In either case, using 'ascii' as the encoding everywhere will
> ensure the code will not try to be too smart about string
> encodings if that happens.  I like that.
> 
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>

Thanks for reviewing!

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 4588 bytes --]

On 15.10.18 22:07, Eduardo Habkost wrote:
> On Mon, Oct 15, 2018 at 04:14:49PM +0200, Max Reitz wrote:
>> In Python 3, several functions now return iterators instead of lists.
>> This includes range(), items(), map(), and filter().  This means that if
>> we really want a list, we have to wrap those instances with list().  On
>> the other hand, sometimes we do just want an iterator, in which case we
>> have sometimes used xrange() and iteritems() which no longer exist in
>> Python 3.  Just change these calls to be range() and items(), which
>> costs a bit of performance in Python 2, but will do the right thing in
>> Python 3 (which is what is important).
>>
>> In one instance, we only wanted the first instance of the result of a
>> filter() call.  Instead of using next(filter()) which would work only in
>> Python 3, or list(filter())[0] which would work everywhere but is a bit
>> weird, this instance is changed to a single-line for with next() wrapped
>> around, which works both in 2.7 and 3.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
> 
> Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
> 
> Minor comments below:

[...]

>> diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
>> index 72aa9707c7..a339bf6069 100755
>> --- a/tests/qemu-iotests/065
>> +++ b/tests/qemu-iotests/065
>> @@ -59,7 +59,7 @@ class TestQemuImgInfo(TestImageInfoSpecific):
>>                      :data.index('')]
>>          for field in data:
>>              self.assertTrue(re.match('^ {4}[^ ]', field) is not None)
>> -        data = map(lambda line: line.strip(), data)
>> +        data = list(map(lambda line: line.strip(), data))
> 
> I would find this more readable:
> 
>         data = [line.strip() for line in data]
> 
> Not a blocker, though.

Yes, I agree, that looks better.

>>          self.assertEqual(data, self.human_compare)
>>  
>>  class TestQMP(TestImageInfoSpecific):
>> @@ -80,7 +80,7 @@ class TestQMP(TestImageInfoSpecific):
>>  
>>      def test_qmp(self):
>>          result = self.vm.qmp('query-block')['return']
>> -        drive = filter(lambda drive: drive['device'] == 'drive0', result)[0]
>> +        drive = next(drive for drive in result if drive['device'] == 'drive0')
> 
> I didn't understand what you meant by "single-line for" on the
> commit message, until I saw the generator expression here.  :)

Maybe I should at least put quotes around the "for", because the
combination of "for with" really makes it difficult to read...

> This will raise an exception if there's no drive0 in the list,
> but that was already true on the original code.
> 
> 
>>          data = drive['inserted']['image']['format-specific']
>>          self.assertEqual(data['type'], iotests.imgfmt)
>>          self.assertEqual(data['data'], self.compare)
>> diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
>> index 3ea4ac53f5..9f189e3b54 100755
>> --- a/tests/qemu-iotests/124
>> +++ b/tests/qemu-iotests/124
>> @@ -39,7 +39,7 @@ def try_remove(img):
>>  def transaction_action(action, **kwargs):
>>      return {
>>          'type': action,
>> -        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.iteritems())
>> +        'data': dict((k.replace('_', '-'), v) for k, v in kwargs.items())
>>      }
>>  
>>  
>> @@ -134,7 +134,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase):
>>      def img_create(self, img, fmt=iotests.imgfmt, size='64M',
>>                     parent=None, parentFormat=None, **kwargs):
>>          optargs = []
>> -        for k,v in kwargs.iteritems():
>> +        for k,v in kwargs.items():
>>              optargs = optargs + ['-o', '%s=%s' % (k,v)]
>>          args = ['create', '-f', fmt] + optargs + [img, size]
>>          if parent:
>> diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
>> index cc7fe337f3..e00f10b8c8 100755
>> --- a/tests/qemu-iotests/139
>> +++ b/tests/qemu-iotests/139
>> @@ -51,7 +51,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
>>      # Check whether a BlockDriverState exists
>>      def checkBlockDriverState(self, node, must_exist = True):
>>          result = self.vm.qmp('query-named-block-nodes')
>> -        nodes = filter(lambda x: x['node-name'] == node, result['return'])
>> +        nodes = list(filter(lambda x: x['node-name'] == node, result['return']))
> 
> I'd prefer a list expression:
> 
>         nodes = [x for x in result['return'] if x['node-name'] == node]
> 
> Also not a blocker.

I don't mind either way, so why not.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 6369 bytes --]

On 15.10.18 22:30, Eduardo Habkost wrote:
> On Mon, Oct 15, 2018 at 04:14:50PM +0200, Max Reitz wrote:
>> Python 3.2 introduced the inheritable attribute for FDs.  At the same
>> time, it changed the default so that all FDs are not inheritable by
>> default, that only inheritable FDs are inherited to subprocesses, and
>> only if close_fds is explicitly set to False.
>>
>> Adhere to this by setting close_fds to False when working with
>> subprocesses that may want to inherit FDs, and by trying to
>> set_inheritable() on FDs that we do want to bequeath to them.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>  scripts/qemu.py        | 13 +++++++++++--
>>  scripts/qmp/qmp.py     |  7 +++++++
>>  tests/qemu-iotests/147 |  7 +++++++
>>  3 files changed, 25 insertions(+), 2 deletions(-)
>>
>> diff --git a/scripts/qemu.py b/scripts/qemu.py
>> index f099ce7278..28366c4a67 100644
>> --- a/scripts/qemu.py
>> +++ b/scripts/qemu.py
>> @@ -142,10 +142,18 @@ class QEMUMachine(object):
>>          if opts:
>>              options.append(opts)
>>  
>> +        # This did not exist before 3.2, but since then it is
>> +        # mandatory for our purpose
>> +        try:
>> +            os.set_inheritable(fd, True)
>> +        except AttributeError:
>> +            pass
>> +
> 
> This is add_fd(), so calling set_inheritable() automatically here
> makes sense.
> 
>>          self._args.append('-add-fd')
>>          self._args.append(','.join(options))
>>          return self
>>  
>> +    # The caller needs to make sure the FD is inheritable
>>      def send_fd_scm(self, fd_file_path):
>>          # In iotest.py, the qmp should always use unix socket.
>>          assert self._qmp.is_scm_available()
>> @@ -159,7 +167,7 @@ class QEMUMachine(object):
>>                      "%s" % fd_file_path]
>>          devnull = open(os.path.devnull, 'rb')
>>          proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
>> -                                stderr=subprocess.STDOUT)
>> +                                stderr=subprocess.STDOUT, close_fds=False)
>>          output = proc.communicate()[0]
>>          if output:
>>              LOG.debug(output)
>> @@ -280,7 +288,8 @@ class QEMUMachine(object):
>>                                         stdin=devnull,
>>                                         stdout=self._qemu_log_file,
>>                                         stderr=subprocess.STDOUT,
>> -                                       shell=False)
>> +                                       shell=False,
>> +                                       close_fds=False)
>>          self._post_launch()
>>  
>>      def wait(self):
>> diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
>> index 5c8cf6a056..009be8345b 100644
>> --- a/scripts/qmp/qmp.py
>> +++ b/scripts/qmp/qmp.py
>> @@ -10,6 +10,7 @@
>>  
>>  import json
>>  import errno
>> +import os
>>  import socket
>>  import logging
>>  
>> @@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object):
>>          return self.__sock.fileno()
>>  
>>      def is_scm_available(self):
>> +        # This did not exist before 3.2, but since then it is
>> +        # mandatory for our purpose
>> +        try:
>> +            os.set_inheritable(self.get_sock_fd(), True)
>> +        except AttributeError:
>> +            pass
> 
> Why did you decide to place this code inside is_scm_available()?
> 
> For reference, this is the only caller of is_scm_available():
> 
>     def send_fd_scm(self, fd_file_path):
>         # In iotest.py, the qmp should always use unix socket.
>         assert self._qmp.is_scm_available()
>         ...
> 
> In addition to making a method called is_*() have an unexpected
> side-effect,

True.  My idea was that a function that asks for SCM to be available
might as well make it available.

>              the method won't be called at all if running with
> debugging disabled.

Well, I sure hope we don't disable debugging in the iotests.  We use
assert quite a number of times there.

On the other hand, someone might want to use this outside of the iotests
but I don't even know whether that works, considering the SCM helper
program is part of the iotests.

> I suggest simply placing the os.set_inheritable() call inside
> send_fd_scm(), as close as possible to the subprocess.Popen()
> call.

Yes, I think you're right.  I didn't want to put it into send_fd_scm(),
because that method is only there to send some FD over QMP/SCM; it isn't
really supposed to send the QMP socket FD somewhere.  But sending
something over QMP/SCM is different from bequeathing something to a
child process (the socket_scm_helper).  And since send_fd_scm() needs to
bequeath the QMP socket FD to that helper, it should be responsible for
making it inheritable.

And maybe even more importantly, whether the socket allows for SCM
really has nothing to do with whether it's inheritable.  So it's
actually just wrong to put it here.

>>          return self.__sock.family == socket.AF_UNIX
>> diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
>> index d2081df84b..b58455645b 100755
>> --- a/tests/qemu-iotests/147
>> +++ b/tests/qemu-iotests/147
>> @@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase):
>>          sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>          sockfd.connect(unix_socket)
>>  
>> +        # This did not exist before 3.2, but since then it is
>> +        # mandatory for our purpose
>> +        try:
>> +            os.set_inheritable(sockfd.fileno(), True)
>> +        except AttributeError:
>> +            pass
>> +
> 
> Why not make send_fd_scm() responsible for calling
> os.set_inheritable(), making this hunk unnecessary?

My idea was: Because send_fd_scm() takes a string and not an integer.
The socket_scm_helper takes either an FD or a path, so send_fd_scm()
accepts either.

But now I realize that if we pass a path, we don't need to make the FD
inheritable.  So send_fd_scm() can check whether it's supposed to pass
an FD, and if so, make it inheritable, yes.

Max

>>          result = self.vm.send_fd_scm(str(sockfd.fileno()))
>>          self.assertEqual(result, 0, 'Failed to send socket FD')
>>  
>> -- 
>> 2.17.1
>>
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 4/9] iotests: Use // for Python integer division

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 9127 bytes --]

On 15.10.18 23:13, Cleber Rosa wrote:
> 
> 
> On 10/15/18 10:14 AM, Max Reitz wrote:
>> In Python 3, / is always a floating-point division.  We usually do not
>> want this, and as Python 2.7 understands // as well, change all integer
>> divisions to use that.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>  tests/qemu-iotests/040        |  4 ++--
>>  tests/qemu-iotests/044        |  2 +-
>>  tests/qemu-iotests/093        | 18 +++++++++---------
>>  tests/qemu-iotests/149        |  6 +++---
>>  tests/qemu-iotests/151        | 12 ++++++------
>>  tests/qemu-iotests/163        |  2 +-
>>  tests/qemu-iotests/iotests.py |  2 +-
>>  7 files changed, 23 insertions(+), 23 deletions(-)
>>
>> diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
>> index 1cb1ceeb33..b81133a474 100755
>> --- a/tests/qemu-iotests/040
>> +++ b/tests/qemu-iotests/040
>> @@ -195,7 +195,7 @@ class TestSingleDrive(ImageCommitTestCase):
>>  
>>          self.assert_no_active_block_jobs()
>>          result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
>> -                             base=backing_img, speed=(self.image_len / 4))
>> +                             base=backing_img, speed=(self.image_len // 4))
>>          self.assert_qmp(result, 'return', {})
>>          result = self.vm.qmp('device_del', id='scsi0')
>>          self.assert_qmp(result, 'return', {})
>> @@ -225,7 +225,7 @@ class TestSingleDrive(ImageCommitTestCase):
>>  
>>          self.assert_no_active_block_jobs()
>>          result = self.vm.qmp('block-commit', device='drive0', top=mid_img,
>> -                             base=backing_img, speed=(self.image_len / 4))
>> +                             base=backing_img, speed=(self.image_len // 4))
>>          self.assert_qmp(result, 'return', {})
>>  
>>          result = self.vm.qmp('query-block')
>> diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
>> index 69e736f687..7ef5e46fe9 100755
>> --- a/tests/qemu-iotests/044
>> +++ b/tests/qemu-iotests/044
>> @@ -86,7 +86,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
>>                  off = off + 1024 * 512
>>  
>>              table = b''.join(struct.pack('>Q', (1 << 63) | off + 512 * j)
>> -                for j in xrange(0, remaining / 512))
>> +                for j in xrange(0, remaining // 512))
>>              fd.write(table)
>>  
>>  
>> diff --git a/tests/qemu-iotests/093 b/tests/qemu-iotests/093
>> index 9d1971a56c..d88fbc182e 100755
>> --- a/tests/qemu-iotests/093
>> +++ b/tests/qemu-iotests/093
>> @@ -69,18 +69,18 @@ class ThrottleTestCase(iotests.QMPTestCase):
>>          # in. The throttled requests won't be executed until we
>>          # advance the virtual clock.
>>          rq_size = 512
>> -        rd_nr = max(params['bps'] / rq_size / 2,
>> -                    params['bps_rd'] / rq_size,
>> -                    params['iops'] / 2,
>> +        rd_nr = max(params['bps'] // rq_size // 2,
>> +                    params['bps_rd'] // rq_size,
>> +                    params['iops'] // 2,
>>                      params['iops_rd'])
>>          rd_nr *= seconds * 2
>> -        rd_nr /= ndrives
>> -        wr_nr = max(params['bps'] / rq_size / 2,
>> -                    params['bps_wr'] / rq_size,
>> -                    params['iops'] / 2,
>> +        rd_nr //= ndrives
>> +        wr_nr = max(params['bps'] // rq_size // 2,
>> +                    params['bps_wr'] // rq_size,
>> +                    params['iops'] // 2,
>>                      params['iops_wr'])
>>          wr_nr *= seconds * 2
>> -        wr_nr /= ndrives
>> +        wr_nr //= ndrives
>>  
>>          # Send I/O requests to all drives
>>          for i in range(rd_nr):
>> @@ -196,7 +196,7 @@ class ThrottleTestCase(iotests.QMPTestCase):
>>              self.configure_throttle(ndrives, settings)
>>  
>>              # Wait for the bucket to empty so we can do bursts
>> -            wait_ns = nsec_per_sec * burst_length * burst_rate / rate
>> +            wait_ns = nsec_per_sec * burst_length * burst_rate // rate
>>              self.vm.qtest("clock_step %d" % wait_ns)
>>  
>>              # Test I/O at the max burst rate
>> diff --git a/tests/qemu-iotests/149 b/tests/qemu-iotests/149
>> index 1225334cb8..4f363f295f 100755
>> --- a/tests/qemu-iotests/149
>> +++ b/tests/qemu-iotests/149
>> @@ -314,13 +314,13 @@ def test_once(config, qemu_img=False):
>>      image_size = 4 * oneTB
>>      if qemu_img:
>>          iotests.log("# Create image")
>> -        qemu_img_create(config, image_size / oneMB)
>> +        qemu_img_create(config, image_size // oneMB)
>>      else:
>>          iotests.log("# Create image")
>> -        create_image(config, image_size / oneMB)
>> +        create_image(config, image_size // oneMB)
>>  
>>      lowOffsetMB = 100
>> -    highOffsetMB = 3 * oneTB / oneMB
>> +    highOffsetMB = 3 * oneTB // oneMB
>>  
>>      try:
>>          if not qemu_img:
>> diff --git a/tests/qemu-iotests/151 b/tests/qemu-iotests/151
>> index fe53b9f446..1bb74d67c4 100755
>> --- a/tests/qemu-iotests/151
>> +++ b/tests/qemu-iotests/151
>> @@ -67,9 +67,9 @@ class TestActiveMirror(iotests.QMPTestCase):
>>                              'write -P 1 0 %i' % self.image_len);
>>  
>>          # Start some background requests
>> -        for offset in range(1 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
>> +        for offset in range(1 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
>>              self.vm.hmp_qemu_io('source', 'aio_write -P 2 %i 1M' % offset)
>> -        for offset in range(2 * self.image_len / 8, 3 * self.image_len / 8, 1024 * 1024):
>> +        for offset in range(2 * self.image_len // 8, 3 * self.image_len // 8, 1024 * 1024):
>>              self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
>>  
>>          # Start the block job
>> @@ -83,9 +83,9 @@ class TestActiveMirror(iotests.QMPTestCase):
>>          self.assert_qmp(result, 'return', {})
>>  
>>          # Start some more requests
>> -        for offset in range(3 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
>> +        for offset in range(3 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
>>              self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
>> -        for offset in range(4 * self.image_len / 8, 5 * self.image_len / 8, 1024 * 1024):
>> +        for offset in range(4 * self.image_len // 8, 5 * self.image_len // 8, 1024 * 1024):
>>              self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
>>  
>>          # Wait for the READY event
>> @@ -95,9 +95,9 @@ class TestActiveMirror(iotests.QMPTestCase):
>>          # the source) should be settled using the active mechanism.
>>          # The mirror code itself asserts that the source BDS's dirty
>>          # bitmap will stay clean between READY and COMPLETED.
>> -        for offset in range(5 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
>> +        for offset in range(5 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
>>              self.vm.hmp_qemu_io('source', 'aio_write -P 3 %i 1M' % offset)
>> -        for offset in range(6 * self.image_len / 8, 7 * self.image_len / 8, 1024 * 1024):
>> +        for offset in range(6 * self.image_len // 8, 7 * self.image_len // 8, 1024 * 1024):
>>              self.vm.hmp_qemu_io('source', 'aio_write -z %i 1M' % offset)
>>  
>>          if sync_source_and_target:
>> diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
>> index 403842354e..5fd424761b 100755
>> --- a/tests/qemu-iotests/163
>> +++ b/tests/qemu-iotests/163
>> @@ -38,7 +38,7 @@ class ShrinkBaseClass(iotests.QMPTestCase):
>>          entry_bits = 3
>>          entry_size = 1 << entry_bits
>>          l1_mask = 0x00fffffffffffe00
>> -        div_roundup = lambda n, d: (n + d - 1) / d
>> +        div_roundup = lambda n, d: (n + d - 1) // d
>>  
>>          def split_by_n(data, n):
>>              for x in xrange(0, len(data), n):
>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>> index 7290c0b159..7ca94e9278 100644
>> --- a/tests/qemu-iotests/iotests.py
>> +++ b/tests/qemu-iotests/iotests.py
>> @@ -199,7 +199,7 @@ def create_image(name, size):
>>      file = open(name, 'wb')
>>      i = 0
>>      while i < size:
>> -        sector = struct.pack('>l504xl', i / 512, i / 512)
>> +        sector = struct.pack('>l504xl', i // 512, i // 512)
>>          file.write(sector)
>>          i = i + 512
>>      file.close()
>>
> 
> Found a few other occurrences:
> 
> 030:''' % (event, errno, self.STREAM_BUFFER_SIZE / 512, event, event))
> 041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event))
> 041:''' % (event, errno, self.MIRROR_GRANULARITY / 512, event, event))

Oh, thanks.  Right, those just worked because they were converted back
to integers thanks to the %d in the format string.

Will fix.  Thanks for reviewing!

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 0/9] iotests: Make them work for both Python 2 and 3

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 1214 bytes --]

On 16.10.18 00:19, Philippe Mathieu-Daudé wrote:
> Hi Max,
> 
> On 15/10/2018 16:14, Max Reitz wrote:
>> This series prepares the iotests to work with both Python 2 and 3.  In
>> some places, it adds version-specific code and decides what to do based
>> on the version (for instance, whether to import the StringIO class from
>> the 'io' or the 'StringIO' module), but most of the time, it just makes
>> code work for both versions in general.
>>
>> And when we make the switch to make Python 3 mandatory, we can simply
>> drop the Python 2 branches.
>>
>>
>> Max Reitz (9):
>>   iotests: Make nbd-fault-injector flush
>>   iotests: Flush in iotests.py's QemuIoInteractive
>>   iotests: Use Python byte strings where appropriate
>>   iotests: Use // for Python integer division
>>   iotests: Different iterator behavior in Python 3
>>   iotests: Explicitly inherit FDs in Python
>>   iotests: 'new' module replacement in 169
>>   iotests: Modify imports for Python 3
>>   iotests: Unify log outputs between Python 2 and 3
> 
> You forgot:
> 
>     MAINTAINERS: Add myself in the Python scripts section
> 
> ;)

Aw, please, no.  You do know I don't really know Python? ;-)

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 8/9] iotests: Modify imports for Python 3

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 2496 bytes --]

On 16.10.18 02:12, Eduardo Habkost wrote:
> On Mon, Oct 15, 2018 at 08:05:02PM -0400, Cleber Rosa wrote:
>>
>>
>> On 10/15/18 5:17 PM, Eduardo Habkost wrote:
>>> On Mon, Oct 15, 2018 at 04:14:52PM +0200, Max Reitz wrote:
>>>> There are two imports that need to be modified when running the iotests
>>>> under Python 3: One is StringIO, which no longer exists; instead, the
>>>> StringIO class comes from the io module, so import it from there.  The
>>>> other is the ConfigParser, which has just been renamed to configparser.
>>>>
>>>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>>>> ---
>>>>  tests/qemu-iotests/iotests.py            | 8 ++++++--
>>>>  tests/qemu-iotests/nbd-fault-injector.py | 7 +++++--
>>>>  2 files changed, 11 insertions(+), 4 deletions(-)
>>>>
>>>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>>>> index 7ca94e9278..a64ea90fb4 100644
>>>> --- a/tests/qemu-iotests/iotests.py
>>>> +++ b/tests/qemu-iotests/iotests.py
>>>> @@ -683,13 +683,17 @@ def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[],
>>>>  
>>>>      # We need to filter out the time taken from the output so that qemu-iotest
>>>>      # can reliably diff the results against master output.
>>>> -    import StringIO
>>>> +    if sys.version_info.major >= 3:
>>>> +        from io import StringIO
>>>> +    else:
>>>> +        from StringIO import StringIO
>>>
>>> Considering that io.StringIO exists on Python 2.7, a comment
>>> explaining why exactly it doesn't work would be nice.

Oh, it does exist?  I didn't know. O:-)

So I suppose it's because the test runner emits just normal strings,
which in 2.x are byte strings; but io's StringIO always expects Unicode
strings.  StringIO, OTOH, accepts both (and returns Unicode strings once
you put a Unicode string into it).

>> Another possibility, that I find self explanatory:
>>
>> import io
>>
>> if sys.version_info.major >= 3:
>>    output = io.StringIO()
>> else:
>>    output = io.BytesIO()
> 
> Looks nice and clean.
> 
> But I'm not sure all output sent to `output` when running with
> Python 2 will be byte strings.  What if `unittest.TextTestRunner`
> tries to write unicode strings to `output`?

Hm.  It doesn't look this way to me in practice.  And considering that
it only really prints a test summary, I don't think there are edge cases
where it behaves differently.

So I think using BytesIO() in 2.x is better.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 9/9] iotests: Unify log outputs between Python 2 and 3

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 7620 bytes --]

On 16.10.18 00:26, Eduardo Habkost wrote:
> On Mon, Oct 15, 2018 at 04:14:53PM +0200, Max Reitz wrote:
>> When dumping an object into the log, there are differences between
>> Python 2 and 3.  First, unicode strings are prefixed by 'u' in Python 2
>> (they are no longer in 3, because unicode strings are the default
>> there).  [...]
> 
> This could be addressed using JSON.  See below[1].
> 
>> [...] Second, the order of keys in dicts may differ.  [...]
> 
> Can be addressed using json.dumps(..., sort_keys=True).

Ah.  Nice. :-)

>> [...] Third,
>> especially long numbers are longs in Python 2 and thus get an 'L'
>> suffix, which does not happen in Python 3.
> 
> print() doesn't add a L suffix on Python 2.7 on my system:
> 
> Python 2.7.15 (default, May 15 2018, 15:37:31)
> [GCC 7.3.1 20180303 (Red Hat 7.3.1-5)] on linux2
> Type "help", "copyright", "credits" or "license" for more information.
>>>> print(99999999999999999999999999999999999999999999999999999999999999999999999999999999999L)
> 99999999999999999999999999999999999999999999999999999999999999999999999999999999999
> 
> So I assume this happens only for QMP commands.

It happens for dicts:

>>> print 99999999999999999999L
99999999999999999999
>>> print {'foo':99999999999999999999L}
{'foo': 99999999999999999999L}

> It would be addressed if using JSON, too.

OK.

>> To get around these differences, this patch introduces functions to
>> convert an object to a string that looks the same regardless of the
>> Python version: In Python 2, they decode unicode strings to byte strings
>> (normal str); in Python 3, they encode byte strings to unicode strings
>> (normal str).  They also manually convert lists and dicts to strings,
>> which allows sorting the dicts by key, so we are no longer at the mercy
>> of the internal implementation when it comes to how the keys appear in
>> the output.
>>
>> This changes the output of all tests that use these logging functions.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>  tests/qemu-iotests/194.out    |  22 +-
>>  tests/qemu-iotests/202.out    |  12 +-
>>  tests/qemu-iotests/203.out    |  14 +-
>>  tests/qemu-iotests/206.out    | 144 +++++-----
>>  tests/qemu-iotests/207.out    |  52 ++--
>>  tests/qemu-iotests/208.out    |   8 +-
>>  tests/qemu-iotests/210.out    |  72 ++---
>>  tests/qemu-iotests/211.out    |  66 ++---
>>  tests/qemu-iotests/212.out    | 102 +++----
>>  tests/qemu-iotests/213.out    | 124 ++++----
>>  tests/qemu-iotests/216.out    |   4 +-
>>  tests/qemu-iotests/218.out    |  20 +-
>>  tests/qemu-iotests/219.out    | 526 +++++++++++++++++-----------------
>>  tests/qemu-iotests/222.out    |  24 +-
>>  tests/qemu-iotests/iotests.py |  42 ++-
>>  15 files changed, 634 insertions(+), 598 deletions(-)
> [...]
>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>> index a64ea90fb4..f8dbc8cc71 100644
>> --- a/tests/qemu-iotests/iotests.py
>> +++ b/tests/qemu-iotests/iotests.py
>> @@ -250,10 +250,45 @@ def filter_img_info(output, filename):
>>          lines.append(line)
>>      return '\n'.join(lines)
>>  
>> +def log_to_string_repr(obj):
>> +    # Normal Python 3 strings are the unicode strings from Python 2;
>> +    # and normal Python 2 strings are byte strings in Python 3.  Thus,
>> +    # we convert bytes -> str in py3 and unicode -> str in py2.
>> +    if sys.version_info.major >= 3:
>> +        if type(obj) is bytes:
>> +            return repr(obj.decode('utf-8'))
>> +    else:
>> +        if type(obj) is unicode:
>> +            return repr(obj.encode('utf-8'))
>> +        elif type(obj) is long:
>> +            return str(obj) # repr() would include an 'L' suffix
>> +
>> +    if type(obj) is list:
>> +        return '[' + ', '.join(map(log_to_string_repr, obj)) + ']'
>> +    elif type(obj) is dict:
>> +        return '{' + ', '.join(map(lambda k: log_to_string_repr(k) + ': ' +
>> +                                             log_to_string_repr(obj[k]),
>> +                                   sorted(obj.keys()))) + '}'
>> +    else:
>> +        return repr(obj)
> 
> I assume this function exists only because of QMP logging,
> correct?

In practice probably yes.

> I would just use json.dumps() at qmp_log(), see below[1].

However, there is not just qmp_log(), there are places that dump objects
directly into log().

>> +
>> +def log_to_string(obj):
>> +    if type(obj) is str:
>> +        return obj
>> +
>> +    if sys.version_info.major >= 3:
>> +        if type(obj) is bytes:
>> +            return obj.decode('utf-8')
> 
> Do you know how many of existing log() calls actually pass a byte
> string as argument?

Frankly I hope none.

>> +    else:
>> +        if type(obj) is unicode:
>> +            return obj.encode('utf-8')
> 
> Is this really necessary?  The existing code just calls
> print(msg) and it works, doesn't it?

True.  But the main issue is that we still need to return immediately
for byte strings and Unicode strings, but the type "bytes" only exists
in 3.x and "unicode" only exists in 2.x, i.e. "if type(obj) is unicode"
throws an exception in 3.x.

So I have to distinguish between 2.x and 3.x anyway, and I thought to
myself that if I distinguished anyway, I might do the conversion to the
native type at the same time.

>> +
>> +    return log_to_string_repr(obj)
>> +
>>  def log(msg, filters=[]):
>>      for flt in filters:
>>          msg = flt(msg)
>> -    print(msg)
>> +    print(log_to_string(msg))
>>  
>>  class Timeout:
>>      def __init__(self, seconds, errmsg = "Timeout"):
>> @@ -441,10 +476,11 @@ class VM(qtest.QEMUQtestMachine):
>>          return result
>>  
>>      def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs):
>> -        logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs)
>> +        logmsg = "{'execute': '%s', 'arguments': %s}" % \
>> +            (cmd, log_to_string(kwargs))
>>          log(logmsg, filters)
>>          result = self.qmp(cmd, **kwargs)
>> -        log(str(result), filters)
>> +        log(log_to_string(result), filters)
> 
> [1]
> 
> If we're being forced to regenerate all the QMP output in the
> .out files, what about always using JSON instead of the hack at
> log_to_string_repr()?  i.e.:
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index a64ea90fb4..0b28dc2a65 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -441,10 +441,12 @@ class VM(qtest.QEMUQtestMachine):
>          return result
>  
>      def qmp_log(self, cmd, filters=[filter_testfiles], **kwargs):
> -        logmsg = "{'execute': '%s', 'arguments': %s}" % (cmd, kwargs)
> +        logmsg = '{"execute": %s, "arguments": %s}' % \
> +                 (json.dumps(cmd, sort_keys=True),
> +                  json.dumps(kwargs, sort_keys=True))
>          log(logmsg, filters)
>          result = self.qmp(cmd, **kwargs)
> -        log(str(result), filters)
> +        log(json.dumps(result, sort_keys=True), filters)
>          return result

Sure, sounds good, thanks. :-)

The only thing is I would allow log() to still accept objects, it should
do the conversion to JSON itself (if the object to be logged is a list
or dict).

Max

>      def run_job(self, job, auto_finalize=True, auto_dismiss=False):
> 
>>          return result
>>  
>>      def run_job(self, job, auto_finalize=True, auto_dismiss=False):
>> -- 
>> 2.17.1
>>
>>
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 5/9] iotests: Different iterator behavior in Python 3

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 2150 bytes --]

On 16.10.18 00:39, Cleber Rosa wrote:
> 
> 
> On 10/15/18 10:14 AM, Max Reitz wrote:
>> In Python 3, several functions now return iterators instead of lists.
>> This includes range(), items(), map(), and filter().  This means that if
>> we really want a list, we have to wrap those instances with list().  On
>> the other hand, sometimes we do just want an iterator, in which case we
>> have sometimes used xrange() and iteritems() which no longer exist in
>> Python 3.  Just change these calls to be range() and items(), which
>> costs a bit of performance in Python 2, but will do the right thing in
>> Python 3 (which is what is important).
>>
>> In one instance, we only wanted the first instance of the result of a
>> filter() call.  Instead of using next(filter()) which would work only in
>> Python 3, or list(filter())[0] which would work everywhere but is a bit
>> weird, this instance is changed to a single-line for with next() wrapped
>> around, which works both in 2.7 and 3.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>  tests/qemu-iotests/044 | 12 ++++++------
>>  tests/qemu-iotests/056 |  2 +-
>>  tests/qemu-iotests/065 |  4 ++--
>>  tests/qemu-iotests/124 |  4 ++--
>>  tests/qemu-iotests/139 |  2 +-
>>  tests/qemu-iotests/163 |  6 +++---
>>  6 files changed, 15 insertions(+), 15 deletions(-)
>>
> 
> You have 2 files here which use xrange (which is a manageable size, and
> whose occurrences involve a moderate size of items) to also consider:
> 
> if sys.version_info.major == 2:
>    range = xrange

I don't think it's necessary, but it's so easy to grep for
"sys.version_info" once we want to completely switch to Python 3 that I
can't find good arguments against it. O:-)

Max

> Defaulting to the Python 3 names, but behaving the same across Python 2
> and 3.
> 
> To do the same for dict.iteritems() => dict.items() requires a lot more
> code, so I'd stay away from it for now.  Also, it looks like most of
> those dicts are small in size (**kwargs and the like).
> 
> Other than that suggestion,
> 
> Reviewed-by: Cleber Rosa <crosa@redhat.com>
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 6/9] iotests: Explicitly inherit FDs in Python

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 4473 bytes --]

On 16.10.18 01:18, Cleber Rosa wrote:
> 
> 
> On 10/15/18 10:14 AM, Max Reitz wrote:
>> Python 3.2 introduced the inheritable attribute for FDs.  At the same
>> time, it changed the default so that all FDs are not inheritable by
>> default, that only inheritable FDs are inherited to subprocesses, and
>> only if close_fds is explicitly set to False.
>>
> 
> It's actually a change that was introduced in 3.4:
> 
> https://docs.python.org/3/library/os.html#inheritance-of-file-descriptors

Oops, don't know how I got that wrong.

>> Adhere to this by setting close_fds to False when working with
>> subprocesses that may want to inherit FDs, and by trying to
>> set_inheritable() on FDs that we do want to bequeath to them.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>  scripts/qemu.py        | 13 +++++++++++--
>>  scripts/qmp/qmp.py     |  7 +++++++
>>  tests/qemu-iotests/147 |  7 +++++++
>>  3 files changed, 25 insertions(+), 2 deletions(-)
>>
>> diff --git a/scripts/qemu.py b/scripts/qemu.py
>> index f099ce7278..28366c4a67 100644
>> --- a/scripts/qemu.py
>> +++ b/scripts/qemu.py
>> @@ -142,10 +142,18 @@ class QEMUMachine(object):
>>          if opts:
>>              options.append(opts)
>>  
>> +        # This did not exist before 3.2, but since then it is
>> +        # mandatory for our purpose
>> +        try:
> 
> Version should be 3.4 here as well.

Yes, will fix everywhere.

>> +            os.set_inheritable(fd, True)
>> +        except AttributeError:
>> +            pass
>> +
> 
> Doing hasattr(os, "set_inheritable") is cheaper.
> 
> - Cleber.

Nice, I'll use that then.

Max

>>          self._args.append('-add-fd')
>>          self._args.append(','.join(options))
>>          return self
>>  
>> +    # The caller needs to make sure the FD is inheritable
>>      def send_fd_scm(self, fd_file_path):
>>          # In iotest.py, the qmp should always use unix socket.
>>          assert self._qmp.is_scm_available()
>> @@ -159,7 +167,7 @@ class QEMUMachine(object):
>>                      "%s" % fd_file_path]
>>          devnull = open(os.path.devnull, 'rb')
>>          proc = subprocess.Popen(fd_param, stdin=devnull, stdout=subprocess.PIPE,
>> -                                stderr=subprocess.STDOUT)
>> +                                stderr=subprocess.STDOUT, close_fds=False)
>>          output = proc.communicate()[0]
>>          if output:
>>              LOG.debug(output)
>> @@ -280,7 +288,8 @@ class QEMUMachine(object):
>>                                         stdin=devnull,
>>                                         stdout=self._qemu_log_file,
>>                                         stderr=subprocess.STDOUT,
>> -                                       shell=False)
>> +                                       shell=False,
>> +                                       close_fds=False)
>>          self._post_launch()
>>  
>>      def wait(self):
>> diff --git a/scripts/qmp/qmp.py b/scripts/qmp/qmp.py
>> index 5c8cf6a056..009be8345b 100644
>> --- a/scripts/qmp/qmp.py
>> +++ b/scripts/qmp/qmp.py
>> @@ -10,6 +10,7 @@
>>  
>>  import json
>>  import errno
>> +import os
>>  import socket
>>  import logging
>>  
>> @@ -253,4 +254,10 @@ class QEMUMonitorProtocol(object):
>>          return self.__sock.fileno()
>>  
>>      def is_scm_available(self):
>> +        # This did not exist before 3.2, but since then it is
>> +        # mandatory for our purpose
>> +        try:
>> +            os.set_inheritable(self.get_sock_fd(), True)
>> +        except AttributeError:
>> +            pass
>>          return self.__sock.family == socket.AF_UNIX
>> diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
>> index d2081df84b..b58455645b 100755
>> --- a/tests/qemu-iotests/147
>> +++ b/tests/qemu-iotests/147
>> @@ -229,6 +229,13 @@ class BuiltinNBD(NBDBlockdevAddBase):
>>          sockfd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
>>          sockfd.connect(unix_socket)
>>  
>> +        # This did not exist before 3.2, but since then it is
>> +        # mandatory for our purpose
>> +        try:
>> +            os.set_inheritable(sockfd.fileno(), True)
>> +        except AttributeError:
>> +            pass
>> +
>>          result = self.vm.send_fd_scm(str(sockfd.fileno()))
>>          self.assertEqual(result, 0, 'Failed to send socket FD')
>>  
>>
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 4953 bytes --]

On 16.10.18 03:01, Cleber Rosa wrote:
> 
> 
> On 10/15/18 7:57 PM, Eduardo Habkost wrote:
>> On Mon, Oct 15, 2018 at 07:38:45PM -0400, Cleber Rosa wrote:
>>>
>>>
>>> On 10/15/18 10:14 AM, Max Reitz wrote:
>>>> iotest 169 uses the 'new' module to add methods to a class.  This module
>>>> no longer exists in Python 3.  Instead, we can use a lambda.  Best of
>>>> all, this works in 2.7 just as well.
>>>>
>>>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>>>> ---
>>>>  tests/qemu-iotests/169 | 3 +--
>>>>  1 file changed, 1 insertion(+), 2 deletions(-)
>>>>
>>>> diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
>>>> index f243db9955..e5614b159d 100755
>>>> --- a/tests/qemu-iotests/169
>>>> +++ b/tests/qemu-iotests/169
>>>> @@ -23,7 +23,6 @@ import iotests
>>>>  import time
>>>>  import itertools
>>>>  import operator
>>>> -import new
>>>>  from iotests import qemu_img
>>>>  
>>>>  
>>>> @@ -144,7 +143,7 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
>>>>  
>>>>  def inject_test_case(klass, name, method, *args, **kwargs):
>>>>      mc = operator.methodcaller(method, *args, **kwargs)
>>>> -    setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
>>>> +    setattr(klass, 'test_' + name, lambda self: mc(self))
>>>>  
>>>>  for cmb in list(itertools.product((True, False), repeat=4)):
>>>>      name = ('_' if cmb[0] else '_not_') + 'persistent_'
>>>>
>>>
>>> This can be simplified with:
>>>
>>> diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
>>> index e5614b159d..2199f14ae7 100755
>>> --- a/tests/qemu-iotests/169
>>> +++ b/tests/qemu-iotests/169
>>> @@ -22,7 +22,6 @@ import os
>>>  import iotests
>>>  import time
>>>  import itertools
>>> -import operator
>>>  from iotests import qemu_img
>>>
>>>
>>> @@ -141,18 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
>>>              self.check_bitmap(self.vm_b, sha256 if persistent else False)
>>>
>>>
>>> -def inject_test_case(klass, name, method, *args, **kwargs):
>>> -    mc = operator.methodcaller(method, *args, **kwargs)
>>> -    setattr(klass, 'test_' + name, lambda self: mc(self))
>>> -
>>>  for cmb in list(itertools.product((True, False), repeat=4)):
>>>      name = ('_' if cmb[0] else '_not_') + 'persistent_'
>>>      name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
>>>      name += '_online' if cmb[2] else '_offline'
>>>      name += '_shared' if cmb[3] else '_nonshared'
>>>
>>> -    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
>>> -                     *list(cmb))
>>> +    setattr(TestDirtyBitmapMigration, 'test_' + name,
>>> +            lambda self: TestDirtyBitmapMigration.do_test_migration(
>>> +                self, *list(cmb)))
>>
>> I'm not fond of the long multi-line lambda expression, but I love
>> that you got rid of operator.methodcaller().  :)
>>
>> However, this doesn't seem to work: `*list(cmb)` will be
>> evaluated only when the lambda is called, and `cmb` will be
>> always `(False, False, False, False)`.
>>
>> I was going to suggest defining a nested function inside
>> inject_test_case() to replace operator.methodcaller(), but then I
>> thought it was not worth it.
>>
> 
> You're right! I missed that.  Anyway, another possibility:
> 
> diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
> index e5614b159d..cd0d9c289c 100755
> --- a/tests/qemu-iotests/169
> +++ b/tests/qemu-iotests/169
> @@ -22,7 +22,7 @@ import os
>  import iotests
>  import time
>  import itertools
> -import operator
> +import functools
>  from iotests import qemu_img
> 
> 
> @@ -140,20 +140,15 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
>              self.vm_b.launch()
>              self.check_bitmap(self.vm_b, sha256 if persistent else False)
> 
> -
> -def inject_test_case(klass, name, method, *args, **kwargs):
> -    mc = operator.methodcaller(method, *args, **kwargs)
> -    setattr(klass, 'test_' + name, lambda self: mc(self))
> -
> -for cmb in list(itertools.product((True, False), repeat=4)):
> +for cmb in itertools.product((True, False), repeat=4):
>      name = ('_' if cmb[0] else '_not_') + 'persistent_'
>      name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
>      name += '_online' if cmb[2] else '_offline'
>      name += '_shared' if cmb[3] else '_nonshared'
> 
> -    inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
> -                     *list(cmb))
> -
> +    test =
> functools.partialmethod(TestDirtyBitmapMigration.do_test_migration,
> +                                   *cmb)
> +    setattr(TestDirtyBitmapMigration, 'test_' + name, test)

I'm not sure how that is any simpler, though (apart from the inlining). :-)

I mean, my personal goal is not to touch this beyond what I need to
because it's black magic to me anyway, so I'm happy with what works.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush

[Max Reitz]

[-- Attachment #1: Type: text/plain, Size: 3724 bytes --]

On 16.10.18 20:07, Eric Blake wrote:
> On 10/15/18 9:14 AM, Max Reitz wrote:
>> When closing a connection, make the nbd-fault-injector flush the socket.
>> Without this, the output is a bit unreliable with Python 3.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>   tests/qemu-iotests/083.out               | 9 +++++++++
>>   tests/qemu-iotests/nbd-fault-injector.py | 1 +
>>   2 files changed, 10 insertions(+)
> 
> I already had a complaint that the error message in 083.out should NOT
> be printing a message

You mean the NBD server itself, right?

>                       (whether the server is python 2 and auto-flushes,
> or python 3 and needs an explicit flush, the message itself is
> pointless, and the test is racy as a result).  We may need to revisit
> this patch when that thread is resolved.
> 
> https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg01041.html

Well, it's not like the flush hurts either way. :-)

Max

> That said, I'm not opposed to this patch, if it gets iotests to be more
> useful in the meantime.
> 
>>
>> diff --git a/tests/qemu-iotests/083.out b/tests/qemu-iotests/083.out
>> index be6079d27e..f9af8bb691 100644
>> --- a/tests/qemu-iotests/083.out
>> +++ b/tests/qemu-iotests/083.out
>> @@ -41,6 +41,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
>>     === Check disconnect after neg2 ===
>>   +Unable to read from socket: Connection reset by peer
>>   Connection closed
>>   read failed: Input/output error
>>   @@ -54,6 +55,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/foo
>>     === Check disconnect before request ===
>>   +Unable to read from socket: Connection reset by peer
>>   Connection closed
>>   read failed: Input/output error
>>   @@ -116,6 +118,7 @@ can't open device nbd+tcp://127.0.0.1:PORT/
>>     === Check disconnect after neg-classic ===
>>   +Unable to read from socket: Connection reset by peer
>>   Connection closed
>>   read failed: Input/output error
>>   @@ -161,6 +164,8 @@ can't open device
>> nbd+unix:///foo?socket=TEST_DIR/nbd.sock
>>     === Check disconnect after neg2 ===
>>   +Unable to read from socket: Connection reset by peer
>> +Connection closed
>>   read failed: Input/output error
>>     === Check disconnect 8 neg2 ===
>> @@ -173,6 +178,8 @@ can't open device
>> nbd+unix:///foo?socket=TEST_DIR/nbd.sock
>>     === Check disconnect before request ===
>>   +Unable to read from socket: Connection reset by peer
>> +Connection closed
>>   read failed: Input/output error
>>     === Check disconnect after request ===
>> @@ -234,6 +241,8 @@ can't open device
>> nbd+unix:///?socket=TEST_DIR/nbd.sock
>>     === Check disconnect after neg-classic ===
>>   +Unable to read from socket: Connection reset by peer
>> +Connection closed
>>   read failed: Input/output error
>>     *** done
>> diff --git a/tests/qemu-iotests/nbd-fault-injector.py
>> b/tests/qemu-iotests/nbd-fault-injector.py
>> index f9193c0fae..439a090eb6 100755
>> --- a/tests/qemu-iotests/nbd-fault-injector.py
>> +++ b/tests/qemu-iotests/nbd-fault-injector.py
>> @@ -112,6 +112,7 @@ class FaultInjectionSocket(object):
>>               if rule.match(event, io):
>>                   if rule.when == 0 or bufsize is None:
>>                       print('Closing connection on rule match %s' %
>> rule.name)
>> +                    self.sock.flush()
>>                       sys.exit(0)
>>                   if rule.when != -1:
>>                       return rule.when
>>
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [Qemu-devel] [PATCH 7/9] iotests: 'new' module replacement in 169

[Eduardo Habkost]

On Fri, Oct 19, 2018 at 11:46:59AM +0200, Max Reitz wrote:
[...]
> I mean, my personal goal is not to touch this beyond what I need to
> because it's black magic to me anyway, so I'm happy with what works.

I agree with this approach, and your original patch looks good
enough to me.

-- 
Eduardo

Re: [Qemu-devel] [PATCH 1/9] iotests: Make nbd-fault-injector flush

[Eric Blake]

On 10/19/18 4:48 AM, Max Reitz wrote:
> On 16.10.18 20:07, Eric Blake wrote:
>> On 10/15/18 9:14 AM, Max Reitz wrote:
>>> When closing a connection, make the nbd-fault-injector flush the socket.
>>> Without this, the output is a bit unreliable with Python 3.
>>>
>>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>>> ---
>>>    tests/qemu-iotests/083.out               | 9 +++++++++
>>>    tests/qemu-iotests/nbd-fault-injector.py | 1 +
>>>    2 files changed, 10 insertions(+)
>>
>> I already had a complaint that the error message in 083.out should NOT
>> be printing a message
> 
> You mean the NBD server itself, right?

Yes, the NBD server should not be printing the redundant error messages 
that 083.out exposes.

> 
>>                        (whether the server is python 2 and auto-flushes,
>> or python 3 and needs an explicit flush, the message itself is
>> pointless, and the test is racy as a result).  We may need to revisit
>> this patch when that thread is resolved.
>>
>> https://lists.gnu.org/archive/html/qemu-devel/2018-08/msg01041.html
> 
> Well, it's not like the flush hurts either way. :-)

True. On that grounds:

Reviewed-by: Eric Blake <eblake@redhat.com>

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3266
Virtualization:  qemu.org | libvirt.org