[PATCH 00/67] iotests: Honor $IMGOPTS in Python tests

[PATCH 00/67] iotests: Honor $IMGOPTS in Python tests

[Max Reitz]

First of all: Sorry.


Second:

Based-on: My block branch
          (https://github.com/XanClic/qemu.git block)

Based-on: 20190917234549.22910-1-jsnow@redhat.com
          (“iotests: use python logging”)

Based-on: 20190927094242.11152-1-mreitz@redhat.com
          (“iotests: Allow ./check -o data_file”)

Based-on: 20190917092004.999-1-mreitz@redhat.com
          (“iotests: Selfish patches”)

Based-on: 20191001174827.11081-1-mreitz@redhat.com
          (“block: Skip COR for inactive nodes”)


OK, now:

Hi,

My recent series “iotests: Allow ./check -o data_file” enabled our bash
tests to interpret the data_file qcow2 option.  It didn’t do anything
for Python tests because those currently completely ignore all image
format options.

This is where it gets hairy.  To do so, we need two things: First of
all, whatever way Python tests use to create images needs to interpret
$IMGOPTS.  Second, when deleting image files, they must not use a plain
os.remove(), but a special function that will clean up data files, too.

The heap of patches in this series comes from making the Python tests
use these new functions.

Most Python tests just run qemu-img through a helper function that does
not care about the exact subcommand to create images.  I could add
$IMGOPTS support to it, but that doesn’t feel quite right to me, and it
wouldn’t reduce the patch count because we still need a special removal
function.


This series is structured as follows:
- Patches 1 through 7 add support to handle image files differently from
  other files (consider $IMGOPTS when creating them, consider data files
  when deleting them, separate ImagePaths from FilePaths, and so on)

- Patches 8 and 9 add two filters we’ll need in the next range:

- Patches 10 through 13 address some issues in a handful of tests that
  just need to be changed a little so they can overall work with some
  format options

- Patch 14 makes all tests pass unsupported_imgopts where there are
  options that they cannot support.

- Patches 15 through 65 make all Python tests use the new functions
  introduced in the first 7 patches so they no longer ignore $IMGOPTS.

  I felt like this is much better than munching everything together into
  a single big commit (better to rebase, better to review), and I don’t
  really like ideas like “Just do five patches that each address ten
  iotests”.

  But I’m still very much open to suggestions on how to combine these
  many small patches to reduce the overall patch count.

- Patch 66 ensures that Python tests always use the new function to
  create test images so they won’t bypass $IMGOPTS.

- Patch 67 cleans up.  qemu_img_log() is only used for image creation,
  and I don’t see the point in that.  The output is predictable and it
  is very unlikely to fail.  We can see in the bash tests that regularly
  we basically just filter everything from it anyway.
  (So this series replaces log(qemu_img_pipe()) instances by asserting
  that image creation did not fail.)
  ((qemu_img_create() obviously no longer has any use after this
  series.))


After this series, running the iotests with -o compat=0.10,
-o refcount_bits=1, and -o 'data_file=$TEST_IMG.data_file' does
something sensible even for the Python tests, and it passes.


Max Reitz (67):
  iotests.py: Read $IMGOPTS
  iotests.py: Add @skip_for_imgopts()
  iotests.py: Add unsupported_imgopts
  iotests.py: create_test_image, remove_test_image
  iotests.py: Add ImagePaths
  iotests.py: Add image_path()
  iotests.py: Filter data_file in filter_img_info
  iotests.py: Add filter_json_filename()
  iotests.py: Add @hide_fields to img_info_log
  iotests/169: Skip persistent cases for compat=0.10
  iotests/224: Filter json:{} from commit command
  iotests/228: Filter json:{} filenames
  iotests/242: Hide refcount bit information
  iotests: Use unsupported_imgopts in Python tests
  iotests/030: Honor $IMGOPTS
  iotests/040: Honor $IMGOPTS
  iotests/041: Honor $IMGOPTS
  iotests/044: Honor $IMGOPTS
  iotests/045: Honor $IMGOPTS
  iotests/055: Honor $IMGOPTS
  iotests/056: Honor $IMGOPTS
  iotests/057: Honor $IMGOPTS
  iotests/065: Honor $IMGOPTS
  iotests/096: Honor $IMGOPTS
  iotests/118: Honor $IMGOPTS
  iotests/124: Honor $IMGOPTS
  iotests/129: Honor $IMGOPTS
  iotests/132: Honor $IMGOPTS
  iotests/139: Honor $IMGOPTS
  iotests/147: Honor $IMGOPTS
  iotests/148: Honor $IMGOPTS
  iotests/151: Honor $IMGOPTS
  iotests/152: Honor $IMGOPTS
  iotests/155: Honor $IMGOPTS
  iotests/163: Honor $IMGOPTS
  iotests/165: Honor $IMGOPTS
  iotests/169: Honor $IMGOPTS
  iotests/194: Honor $IMGOPTS
  iotests/196: Honor $IMGOPTS
  iotests/199: Honor $IMGOPTS
  iotests/202: Honor $IMGOPTS
  iotests/203: Honor $IMGOPTS
  iotests/205: Honor $IMGOPTS
  iotests/208: Honor $IMGOPTS
  iotests/208: Honor $IMGOPTS
  iotests/216: Honor $IMGOPTS
  iotests/218: Honor $IMGOPTS
  iotests/219: Honor $IMGOPTS
  iotests/222: Honor $IMGOPTS
  iotests/224: Honor $IMGOPTS
  iotests/228: Honor $IMGOPTS
  iotests/234: Honor $IMGOPTS
  iotests/235: Honor $IMGOPTS
  iotests/236: Honor $IMGOPTS
  iotests/237: Honor $IMGOPTS
  iotests/242: Honor $IMGOPTS
  iotests/245: Honor $IMGOPTS
  iotests/246: Honor $IMGOPTS
  iotests/248: Honor $IMGOPTS
  iotests/254: Honor $IMGOPTS
  iotests/255: Honor $IMGOPTS
  iotests/256: Honor $IMGOPTS
  iotests/257: Honor $IMGOPTS
  iotests/258: Honor $IMGOPTS
  iotests/262: Honor $IMGOPTS
  iotests.py: Forbid qemu_img*('create', ...)
  iotests.py: Drop qemu_img_log(), qemu_img_create()

 tests/qemu-iotests/030        |  69 ++++++------
 tests/qemu-iotests/040        |  42 ++++----
 tests/qemu-iotests/041        | 108 +++++++++----------
 tests/qemu-iotests/044        |  11 +-
 tests/qemu-iotests/045        |  26 ++---
 tests/qemu-iotests/055        |  41 +++----
 tests/qemu-iotests/056        |  30 +++---
 tests/qemu-iotests/057        |  10 +-
 tests/qemu-iotests/065        |  21 ++--
 tests/qemu-iotests/096        |   5 +-
 tests/qemu-iotests/118        |  26 ++---
 tests/qemu-iotests/124        |  29 +++--
 tests/qemu-iotests/129        |  11 +-
 tests/qemu-iotests/132        |   6 +-
 tests/qemu-iotests/139        |  15 ++-
 tests/qemu-iotests/147        |  11 +-
 tests/qemu-iotests/148        |   5 +-
 tests/qemu-iotests/151        |  10 +-
 tests/qemu-iotests/152        |   6 +-
 tests/qemu-iotests/155        |  29 +++--
 tests/qemu-iotests/163        |  29 ++---
 tests/qemu-iotests/165        |  10 +-
 tests/qemu-iotests/169        |  23 ++--
 tests/qemu-iotests/194        |   9 +-
 tests/qemu-iotests/196        |  10 +-
 tests/qemu-iotests/199        |  10 +-
 tests/qemu-iotests/202        |   9 +-
 tests/qemu-iotests/203        |   9 +-
 tests/qemu-iotests/205        |   7 +-
 tests/qemu-iotests/206        |   5 +-
 tests/qemu-iotests/208        |   5 +-
 tests/qemu-iotests/209        |   9 +-
 tests/qemu-iotests/216        |  11 +-
 tests/qemu-iotests/218        |   6 +-
 tests/qemu-iotests/219        |   5 +-
 tests/qemu-iotests/222        |  13 +--
 tests/qemu-iotests/224        |  33 +++---
 tests/qemu-iotests/224.out    |   4 +-
 tests/qemu-iotests/228        |  25 ++---
 tests/qemu-iotests/228.out    |   8 +-
 tests/qemu-iotests/234        |   9 +-
 tests/qemu-iotests/235        |   7 +-
 tests/qemu-iotests/236        |   6 +-
 tests/qemu-iotests/237        |  15 +--
 tests/qemu-iotests/237.out    |   6 --
 tests/qemu-iotests/242        |  21 ++--
 tests/qemu-iotests/242.out    |   5 -
 tests/qemu-iotests/245        |  21 ++--
 tests/qemu-iotests/246        |  11 +-
 tests/qemu-iotests/248        |  14 ++-
 tests/qemu-iotests/254        |  10 +-
 tests/qemu-iotests/255        |  20 ++--
 tests/qemu-iotests/255.out    |   8 --
 tests/qemu-iotests/256        |  10 +-
 tests/qemu-iotests/257        |  18 ++--
 tests/qemu-iotests/258        |  16 +--
 tests/qemu-iotests/262        |   5 +-
 tests/qemu-iotests/iotests.py | 197 +++++++++++++++++++++++++++-------
 58 files changed, 654 insertions(+), 496 deletions(-)

-- 
2.21.0

[PATCH 01/67] iotests.py: Read $IMGOPTS

[Max Reitz]

We do not do anything with yet, but this is the first step.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 1c5fce3e9e..7030900807 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -69,6 +69,12 @@ output_dir = os.environ.get('OUTPUT_DIR', '.')
 cachemode = os.environ.get('CACHEMODE')
 qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
 
+imgopts = os.environ.get('IMGOPTS', '')
+if len(imgopts) == 0:
+    imgopts = []
+else:
+    imgopts = imgopts.split(',')
+
 socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
 
 luks_default_secret_object = 'secret,id=keysec0,data=' + \
-- 
2.21.0

[PATCH 02/67] iotests.py: Add @skip_for_imgopts()

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 13 +++++++++++++
 1 file changed, 13 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 7030900807..cdcb62c4ac 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -950,6 +950,19 @@ def skip_if_unsupported(required_formats=[], read_only=False):
         return func_wrapper
     return skip_test_decorator
 
+def skip_for_imgopts(unsupported_opts=[]):
+    '''Skip Test Decorator
+       Skips the test if imgopts contains any of the given options'''
+    def skip_test_decorator(func):
+        def func_wrapper(test_case: QMPTestCase, *args, **kwargs):
+            for opt in imgopts:
+                if any(unsupported in opt for unsupported in unsupported_opts):
+                    test_case.case_skip('{}: Option {} is unsupported'.format(
+                                        test_case, opt))
+            return func(test_case, *args, **kwargs)
+        return func_wrapper
+    return skip_test_decorator
+
 def execute_unittest(debug=False):
     """Executes unittests within the calling module."""
 
-- 
2.21.0

[PATCH 03/67] iotests.py: Add unsupported_imgopts

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 7 ++++++-
 1 file changed, 6 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index cdcb62c4ac..b5ea424de4 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -998,7 +998,8 @@ def execute_setup_common(supported_fmts=[],
                          supported_cache_modes=[],
                          unsupported_fmts=[],
                          supported_protocols=[],
-                         unsupported_protocols=[]):
+                         unsupported_protocols=[],
+                         unsupported_imgopts=[]):
     """
     Perform necessary setup for either script-style or unittest-style tests.
     """
@@ -1016,6 +1017,10 @@ def execute_setup_common(supported_fmts=[],
     verify_platform(supported=supported_platforms)
     verify_cache_mode(supported_cache_modes)
 
+    for opt in imgopts:
+        if any(unsupported in opt for unsupported in unsupported_imgopts):
+            notrun('not suitable for this option: %s' % opt)
+
     debug = '-d' in sys.argv
     if debug:
         sys.argv.remove('-d')
-- 
2.21.0

[PATCH 04/67] iotests.py: create_test_image, remove_test_image

[Max Reitz]

Python tests should use these two new functions instead of
qemu_img('create', ...) + os.remove(), so that user-supplied image
options are interpreted and handled correctly.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 56 +++++++++++++++++++++++++++++++++++
 1 file changed, 56 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index b5ea424de4..fce1ab04c9 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -122,6 +122,62 @@ def qemu_img_create(*args):
 
     return qemu_img(*args)
 
+def create_test_image(filename, size=None, fmt=imgfmt, opts=[],
+                      backing_file=None, backing_fmt=None,
+                      objects=[], unsafe=False):
+    if fmt == imgfmt:
+        # Only use imgopts for the default format
+        opts = imgopts + opts
+
+    for i, opt in enumerate(opts):
+        if '$TEST_IMG' in opt:
+            opts[i] = opt.replace('$TEST_IMG', filename)
+
+    # default luks support
+    if fmt == 'luks':
+        if not any('key-secret' in opt for opt in opts):
+            opts.append(luks_default_key_secret_opt)
+        objects.append(luks_default_secret_object)
+
+    args = ['create', '-f', fmt]
+
+    if len(opts) > 0:
+        args += ['-o', ','.join(opts)]
+
+    if backing_file is not None:
+        args += ['-b', backing_file]
+
+    if backing_fmt is not None:
+        args += ['-F', backing_fmt]
+
+    if len(objects) > 0:
+        # Generate a [['--object', $obj], [...], ...] list and flatten it
+        args += [arg for objarg in (['--object', obj] for obj in objects) \
+                     for arg in objarg]
+
+    if unsafe:
+        args.append('-u')
+
+    args.append(filename)
+    if size is not None:
+        args.append(str(size))
+
+    return qemu_img(*args)
+
+# Use this to remove images create with create_test_image in the
+# default image format (iotests.imgfmt)
+def remove_test_image(filename):
+    try:
+        os.remove(filename)
+
+        data_file = next(opt.replace('data_file=', '') \
+                            .replace('$TEST_IMG', filename) \
+                         for opt in imgopts if opt.startswith('data_file='))
+
+        os.remove(data_file)
+    except:
+        pass
+
 def qemu_img_verbose(*args):
     '''Run qemu-img without suppressing its output and return the exit code'''
     exitcode = subprocess.call(qemu_img_args + list(args))
-- 
2.21.0

[PATCH 05/67] iotests.py: Add ImagePaths

[Max Reitz]

create_test_image() must be paired with an ImagePath so that the image
is properly cleaned up with remove_test_image() instead of just
os.remove().

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 32 ++++++++++++++++++++++++++++++--
 1 file changed, 30 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index fce1ab04c9..5be6ca674c 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -440,8 +440,8 @@ class FilePaths(object):
     Use this context manager to generate filenames and ensure that the file
     gets deleted::
 
-        with FilePaths(['test.img']) as img_path:
-            qemu_img('create', img_path, '1G')
+        with FilePaths(['migration.sock']) as migration_sock_path:
+            # Set up and use UNIX socket on migration.sock
         # migration_sock_path is automatically deleted
     """
     def __init__(self, names):
@@ -460,6 +460,23 @@ class FilePaths(object):
             pass
         return False
 
+class ImagePaths(FilePaths):
+    """
+    Same as FilePaths, except it calls remove_test_image() to clean up
+    (which ensures that external data files are cleaned up, too).
+
+    Use this class for test images in the default format
+    (iotests.imgfmt):
+
+    with ImagePaths(['test.img']) as img_path:
+        create_test_image(img_path, '1G')
+    # The test image is automatically cleaned up
+    """
+    def __exit__(self, exc_type, exc_val, exc_tb):
+        for path in self.paths:
+            remove_test_image(path)
+        return False
+
 class FilePath(FilePaths):
     """
     FilePath is a specialization of FilePaths that takes a single filename.
@@ -470,6 +487,17 @@ class FilePath(FilePaths):
     def __enter__(self):
         return self.paths[0]
 
+class ImagePath(ImagePaths):
+    """
+    ImagePath is a specialization of ImagePaths that takes a single
+    filename.
+    """
+    def __init__(self, name):
+        super(ImagePath, self).__init__([name])
+
+    def __enter__(self):
+        return self.paths[0]
+
 def file_path_remover():
     for path in reversed(file_path_remover.paths):
         try:
-- 
2.21.0

[PATCH 06/67] iotests.py: Add image_path()

[Max Reitz]

Just like we have file_path() as an alternative to FilePath, this is an
alternative fo ImagePath.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 17 ++++++++++++-----
 1 file changed, 12 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 5be6ca674c..280e6c2ec2 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -505,8 +505,12 @@ def file_path_remover():
         except OSError:
             pass
 
+def image_path_remover():
+    for path in reversed(image_path_remover.paths):
+        remove_test_image(path)
 
-def file_path(*names):
+
+def file_path(*names, remover=file_path_remover):
     ''' Another way to get auto-generated filename that cleans itself up.
 
     Use is as simple as:
@@ -515,19 +519,22 @@ def file_path(*names):
     sock = file_path('socket')
     '''
 
-    if not hasattr(file_path_remover, 'paths'):
-        file_path_remover.paths = []
-        atexit.register(file_path_remover)
+    if not hasattr(remover, 'paths'):
+        remover.paths = []
+        atexit.register(remover)
 
     paths = []
     for name in names:
         filename = file_pattern(name)
         path = os.path.join(test_dir, filename)
-        file_path_remover.paths.append(path)
+        remover.paths.append(path)
         paths.append(path)
 
     return paths[0] if len(paths) == 1 else paths
 
+def image_path(*names):
+    return file_path(*names, remover=image_path_remover)
+
 def remote_filename(path):
     if imgproto == 'file':
         return path
-- 
2.21.0

[PATCH 07/67] iotests.py: Filter data_file in filter_img_info

[Max Reitz]

If $IMGOPTS contains a data_file reference, let filter_img_info filter
all data_file information.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 280e6c2ec2..fb0a49372d 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -380,9 +380,13 @@ def filter_generated_node_ids(msg):
 
 def filter_img_info(output, filename):
     lines = []
+    user_data_file = any('data_file' in opt for opt in imgopts)
     for line in output.split('\n'):
         if 'disk size' in line or 'actual-size' in line:
             continue
+        if user_data_file:
+            if 'data file' in line:
+                continue
         line = line.replace(filename, 'TEST_IMG') \
                    .replace(imgfmt, 'IMGFMT')
         line = re.sub('iters: [0-9]+', 'iters: XXX', line)
-- 
2.21.0

[PATCH 08/67] iotests.py: Add filter_json_filename()

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index fb0a49372d..9737dd881b 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -378,6 +378,9 @@ def filter_qmp_testfiles(qmsg):
 def filter_generated_node_ids(msg):
     return re.sub("#block[0-9]+", "NODE_NAME", msg)
 
+def filter_json_filename(msg):
+    return re.sub('json:{.*}', 'json:{ /* filtered */ }', msg)
+
 def filter_img_info(output, filename):
     lines = []
     user_data_file = any('data_file' in opt for opt in imgopts)
-- 
2.21.0

[PATCH 09/67] iotests.py: Add @hide_fields to img_info_log

[Max Reitz]

You can specify fields here that should be hidden from the output.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 9737dd881b..f3c80ba4e9 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -201,7 +201,8 @@ def qemu_img_log(*args):
     log(result, filters=[filter_testfiles])
     return result
 
-def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]):
+def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[],
+                 hide_fields=[]):
     args = [ 'info' ]
     if imgopts:
         args.append('--image-opts')
@@ -210,7 +211,13 @@ def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[]):
     args += extra_args
     args.append(filename)
 
-    output = qemu_img_pipe(*args)
+    output = qemu_img_pipe(*args).splitlines(keepends=True)
+
+    output = [line for line in output
+              if not any(line.strip().startswith(field)
+                         for field in hide_fields)]
+    output = ''.join(output)
+
     if not filter_path:
         filter_path = filename
     log(filter_img_info(output, filter_path))
-- 
2.21.0

[PATCH 10/67] iotests/169: Skip persistent cases for compat=0.10

[Max Reitz]

compat=0.10 images cannot store persistent dirty bitmaps; let 169 skip
all such test cases when compat=0.10 is in the $IMGOPTS.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/169 | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
index 8c204caf20..5e978b7b7d 100755
--- a/tests/qemu-iotests/169
+++ b/tests/qemu-iotests/169
@@ -41,7 +41,10 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
         self.vm_b.shutdown()
         os.remove(disk_a)
         os.remove(disk_b)
-        os.remove(mig_file)
+        try:
+            os.remove(mig_file)
+        except OSError:
+            pass
 
     def setUp(self):
         qemu_img('create', '-f', iotests.imgfmt, disk_a, size)
@@ -79,6 +82,10 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
     def do_test_migration_resume_source(self, persistent, migrate_bitmaps):
         granularity = 512
 
+        if persistent and 'compat=0.10' in iotests.imgopts:
+            self.case_skip('compat=0.10 does not support persistent dirty ' +
+                           'bitmaps')
+
         # regions = ((start, count), ...)
         regions = ((0, 0x10000),
                    (0xf0000, 0x10000),
@@ -137,6 +144,10 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
                           shared_storage):
         granularity = 512
 
+        if persistent and 'compat=0.10' in iotests.imgopts:
+            self.case_skip('compat=0.10 does not support persistent dirty ' +
+                           'bitmaps')
+
         # regions = ((start, count), ...)
         regions = ((0, 0x10000),
                    (0xf0000, 0x10000),
-- 
2.21.0

[PATCH 11/67] iotests/224: Filter json:{} from commit command

[Max Reitz]

We only care that block-commit works with json:{} filenames, we do not
need to see their exact values in the output.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/224     | 15 ++++++++++++---
 tests/qemu-iotests/224.out |  4 ++--
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/224 b/tests/qemu-iotests/224
index d0d0c44104..e227d3ab3c 100755
--- a/tests/qemu-iotests/224
+++ b/tests/qemu-iotests/224
@@ -22,12 +22,19 @@
 
 import iotests
 from iotests import log, qemu_img, qemu_io_silent, filter_qmp_testfiles, \
-                    filter_qmp_imgfmt
+                    filter_qmp_imgfmt, filter_json_filename
 import json
+import re
 
 # Need backing file support (for arbitrary backing formats)
 iotests.script_initialize(supported_fmts=['qcow2', 'qcow', 'qed'])
 
+def filter_json_top_or_base(qmsg):
+    def _filter(key, value):
+        if key == 'base' or key == 'top':
+            return filter_json_filename(value)
+        return value
+    return iotests.filter_qmp(qmsg, _filter)
 
 # There are two variations of this test:
 # (1) We do not set filter_node_name.  In that case, the commit_top
@@ -104,7 +111,8 @@ for filter_node_name in False, True:
                         top=mid_name,
                         base=base_name,
                         speed=1,
-                        filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+                        filters=[filter_qmp_testfiles, filter_qmp_imgfmt,
+                                 filter_json_top_or_base])
         else:
             vm.qmp_log('block-commit',
                         job_id='commit',
@@ -112,7 +120,8 @@ for filter_node_name in False, True:
                         top=mid_name,
                         base=base_name,
                         speed=1,
-                        filters=[filter_qmp_testfiles, filter_qmp_imgfmt])
+                        filters=[filter_qmp_testfiles, filter_qmp_imgfmt,
+                                 filter_json_top_or_base])
 
         vm.qmp_log('job-pause', id='commit')
 
diff --git a/tests/qemu-iotests/224.out b/tests/qemu-iotests/224.out
index 23374a1d29..287d2eac7a 100644
--- a/tests/qemu-iotests/224.out
+++ b/tests/qemu-iotests/224.out
@@ -3,7 +3,7 @@
 
 {"execute": "blockdev-add", "arguments": {"backing": {"backing": {"backing": {"driver": "null-co"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}, "node-name": "base"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-mid.img"}, "node-name": "mid"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "top"}}
 {"return": {}}
-{"execute": "block-commit", "arguments": {"base": "json:{\"backing\": {\"driver\": \"null-co\"}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-base.img\"}}", "device": "top", "job-id": "commit", "speed": 1, "top": "json:{\"backing\": {\"backing\": {\"driver\": \"null-co\"}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-base.img\"}}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-mid.img\"}}"}}
+{"execute": "block-commit", "arguments": {"base": "json:{ /* filtered */ }", "device": "top", "job-id": "commit", "speed": 1, "top": "json:{ /* filtered */ }"}}
 {"return": {}}
 {"execute": "job-pause", "arguments": {"id": "commit"}}
 {"return": {}}
@@ -12,7 +12,7 @@
 
 {"execute": "blockdev-add", "arguments": {"backing": {"backing": {"backing": {"driver": "null-co"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}, "node-name": "base"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-mid.img"}, "node-name": "mid"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "top"}}
 {"return": {}}
-{"execute": "block-commit", "arguments": {"base": "json:{\"backing\": {\"driver\": \"null-co\"}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-base.img\"}}", "device": "top", "filter-node-name": "filter_node", "job-id": "commit", "speed": 1, "top": "json:{\"backing\": {\"backing\": {\"driver\": \"null-co\"}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-base.img\"}}, \"driver\": \"IMGFMT\", \"file\": {\"driver\": \"file\", \"filename\": \"TEST_DIR/PID-mid.img\"}}"}}
+{"execute": "block-commit", "arguments": {"base": "json:{ /* filtered */ }", "device": "top", "filter-node-name": "filter_node", "job-id": "commit", "speed": 1, "top": "json:{ /* filtered */ }"}}
 {"return": {}}
 {"execute": "job-pause", "arguments": {"id": "commit"}}
 {"return": {}}
-- 
2.21.0

[PATCH 12/67] iotests/228: Filter json:{} filenames

[Max Reitz]

We only care about where we get json:{} filenames (and what the
non-json:{} filenames look like), but not about how the json:{}
filenames look.

Filter them so we will not get a different output for qcow2 images with
external data files.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/228     | 8 ++++----
 tests/qemu-iotests/228.out | 8 ++++----
 2 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/tests/qemu-iotests/228 b/tests/qemu-iotests/228
index 9785868ab3..e507db4f27 100755
--- a/tests/qemu-iotests/228
+++ b/tests/qemu-iotests/228
@@ -22,7 +22,7 @@
 
 import iotests
 from iotests import log, qemu_img, filter_testfiles, filter_imgfmt, \
-        filter_qmp_testfiles, filter_qmp_imgfmt
+        filter_qmp_testfiles, filter_qmp_imgfmt, filter_json_filename
 
 # Need backing file and change-backing-file support
 iotests.script_initialize(supported_fmts=['qcow2', 'qed'])
@@ -32,14 +32,14 @@ def log_node_info(node):
     log('')
 
     log('bs->filename: ' + node['image']['filename'],
-        filters=[filter_testfiles, filter_imgfmt])
+        filters=[filter_testfiles, filter_imgfmt, filter_json_filename])
     log('bs->backing_file: ' + node['backing_file'],
-        filters=[filter_testfiles, filter_imgfmt])
+        filters=[filter_testfiles, filter_imgfmt, filter_json_filename])
 
     if 'backing-image' in node['image']:
         log('bs->backing->bs->filename: ' +
             node['image']['backing-image']['filename'],
-            filters=[filter_testfiles, filter_imgfmt])
+            filters=[filter_testfiles, filter_imgfmt, filter_json_filename])
     else:
         log('bs->backing: (none)')
 
diff --git a/tests/qemu-iotests/228.out b/tests/qemu-iotests/228.out
index 4217df24fe..f8155019d5 100644
--- a/tests/qemu-iotests/228.out
+++ b/tests/qemu-iotests/228.out
@@ -17,7 +17,7 @@ bs->backing->bs->filename: TEST_DIR/PID-base.img
 {"execute": "change-backing-file", "arguments": {"backing-file": "null-co://", "device": "node0", "image-node-name": "node0"}}
 {"return": {}}
 
-bs->filename: json:{"backing": {"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->filename: json:{ /* filtered */ }
 bs->backing_file: null-co://
 bs->backing->bs->filename: TEST_DIR/PID-base.img
 
@@ -31,7 +31,7 @@ bs->backing->bs->filename: TEST_DIR/PID-base.img
 {"execute": "change-backing-file", "arguments": {"backing-file": "file:TEST_DIR/PID-base.img", "device": "node0", "image-node-name": "node0"}}
 {"return": {}}
 
-bs->filename: json:{"backing": {"driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-base.img"}}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->filename: json:{ /* filtered */ }
 bs->backing_file: file:TEST_DIR/PID-base.img
 bs->backing->bs->filename: TEST_DIR/PID-base.img
 
@@ -54,7 +54,7 @@ bs->backing->bs->filename: TEST_DIR/PID-base.img
 {"execute": "blockdev-add", "arguments": {"backing": "null", "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "node0"}}
 {"return": {}}
 
-bs->filename: json:{"backing": {"driver": "null-co"}, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->filename: json:{ /* filtered */ }
 bs->backing_file: null-co://
 bs->backing->bs->filename: null-co://
 
@@ -65,7 +65,7 @@ bs->backing->bs->filename: null-co://
 {"execute": "blockdev-add", "arguments": {"backing": null, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}, "node-name": "node0"}}
 {"return": {}}
 
-bs->filename: json:{"backing": null, "driver": "IMGFMT", "file": {"driver": "file", "filename": "TEST_DIR/PID-top.img"}}
+bs->filename: json:{ /* filtered */ }
 bs->backing_file: TEST_DIR/PID-base.img
 bs->backing: (none)
 
-- 
2.21.0

[PATCH 13/67] iotests/242: Hide refcount bit information

[Max Reitz]

It does not matter, and this lets the test work with different refcount
widths.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/242     | 6 +++---
 tests/qemu-iotests/242.out | 5 -----
 2 files changed, 3 insertions(+), 8 deletions(-)

diff --git a/tests/qemu-iotests/242 b/tests/qemu-iotests/242
index 7c2685b4cc..21f69a0d2c 100755
--- a/tests/qemu-iotests/242
+++ b/tests/qemu-iotests/242
@@ -35,7 +35,7 @@ flag_offset = 0x5000f
 
 def print_bitmap(extra_args):
     log('qemu-img info dump:\n')
-    img_info_log(disk, extra_args=extra_args)
+    img_info_log(disk, extra_args=extra_args, hide_fields=['refcount bits'])
     result = json.loads(qemu_img_pipe('info', '--force-share',
                                       '--output=json', disk))
     if 'bitmaps' in result['format-specific']['data']:
@@ -98,9 +98,9 @@ add_bitmap(1, True, False)
 log('Write an unknown bitmap flag \'{}\' into a new QCOW2 image at offset {}'
     .format(hex(bitmap_flag_unknown), flag_offset))
 toggle_flag(flag_offset)
-img_info_log(disk)
+img_info_log(disk, hide_fields=['refcount bits'])
 toggle_flag(flag_offset)
 log('Unset the unknown bitmap flag \'{}\' in the bitmap directory entry:\n'
     .format(hex(bitmap_flag_unknown)))
-img_info_log(disk)
+img_info_log(disk, hide_fields=['refcount bits'])
 log('Test complete')
diff --git a/tests/qemu-iotests/242.out b/tests/qemu-iotests/242.out
index 7ac8404d11..112b389241 100644
--- a/tests/qemu-iotests/242.out
+++ b/tests/qemu-iotests/242.out
@@ -13,7 +13,6 @@ cluster_size: 65536
 Format specific information:
     compat: 1.1
     lazy refcounts: false
-    refcount bits: 16
     corrupt: false
 
 No bitmap in JSON format output
@@ -38,7 +37,6 @@ Format specific information:
             flags:
             name: bitmap-1
             granularity: 32768
-    refcount bits: 16
     corrupt: false
 
 The same bitmaps in JSON format:
@@ -75,7 +73,6 @@ Format specific information:
                 [0]: auto
             name: bitmap-2
             granularity: 65536
-    refcount bits: 16
     corrupt: false
 
 The same bitmaps in JSON format:
@@ -117,7 +114,6 @@ Format specific information:
                 [1]: auto
             name: bitmap-2
             granularity: 65536
-    refcount bits: 16
     corrupt: false
 
 The same bitmaps in JSON format:
@@ -160,7 +156,6 @@ Format specific information:
                 [0]: auto
             name: bitmap-0
             granularity: 16384
-    refcount bits: 16
     corrupt: false
 
 Test complete
-- 
2.21.0

[PATCH 14/67] iotests: Use unsupported_imgopts in Python tests

[Max Reitz]

Set unsupported_imgopts as appropriate for all tests that should make
use of it.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/044 | 5 ++++-
 tests/qemu-iotests/057 | 4 +++-
 tests/qemu-iotests/065 | 4 +++-
 tests/qemu-iotests/163 | 5 ++++-
 tests/qemu-iotests/165 | 4 +++-
 tests/qemu-iotests/196 | 4 +++-
 tests/qemu-iotests/206 | 5 ++++-
 tests/qemu-iotests/222 | 3 ++-
 tests/qemu-iotests/237 | 4 +++-
 tests/qemu-iotests/242 | 4 +++-
 tests/qemu-iotests/245 | 4 +++-
 tests/qemu-iotests/246 | 5 ++++-
 tests/qemu-iotests/248 | 6 +++++-
 tests/qemu-iotests/254 | 4 +++-
 tests/qemu-iotests/257 | 4 +++-
 15 files changed, 50 insertions(+), 15 deletions(-)

diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 8b2afa2a11..97ba98e628 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -116,4 +116,7 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=['qcow2'],
-                 supported_protocols=['file'])
+                 supported_protocols=['file'],
+                 # These refcount calculations do not work with data_file or
+                 # with any refcount width but the default (of 16)
+                 unsupported_imgopts=['data_file', 'refcount_bits'])
diff --git a/tests/qemu-iotests/057 b/tests/qemu-iotests/057
index 9fbba759b6..f5aa5929dc 100755
--- a/tests/qemu-iotests/057
+++ b/tests/qemu-iotests/057
@@ -257,4 +257,6 @@ class TestSnapshotDelete(ImageSnapshotTestCase):
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=['qcow2'],
-                 supported_protocols=['file'])
+                 supported_protocols=['file'],
+                 # Snapshots do not work with data_file
+                 unsupported_imgopts=['data_file'])
diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 5b21eb96bd..9db9552784 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -130,4 +130,6 @@ TestQMP = None
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=['qcow2'],
-                 supported_protocols=['file'])
+                 supported_protocols=['file'],
+                 # Neither of these options would work with compat=0.10
+                 unsupported_imgopts=['refcount_bits', 'data_file'])
diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
index d94728e080..92633f1b21 100755
--- a/tests/qemu-iotests/163
+++ b/tests/qemu-iotests/163
@@ -168,4 +168,7 @@ ShrinkBaseClass = None
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=['raw', 'qcow2'],
-                 supported_protocols=['file'])
+                 supported_protocols=['file'],
+                 # We want to set our own refcount_bits, and that will
+                 # not work with compat=0.10
+                 unsupported_imgopts=['refcount_bits', 'compat=0.10'])
diff --git a/tests/qemu-iotests/165 b/tests/qemu-iotests/165
index 5650dc7c87..97dd7102c3 100755
--- a/tests/qemu-iotests/165
+++ b/tests/qemu-iotests/165
@@ -104,4 +104,6 @@ class TestPersistentDirtyBitmap(iotests.QMPTestCase):
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=['qcow2'],
-                 supported_protocols=['file'])
+                 supported_protocols=['file'],
+                 # compat=1.1 is needed for dirty bitmaps
+                 unsupported_imgopts=['compat=0.10'])
diff --git a/tests/qemu-iotests/196 b/tests/qemu-iotests/196
index 92fe9244f8..ec4852a19c 100755
--- a/tests/qemu-iotests/196
+++ b/tests/qemu-iotests/196
@@ -64,4 +64,6 @@ class TestInvalidateAutoclear(iotests.QMPTestCase):
 
 if __name__ == '__main__':
     iotests.main(supported_fmts=['qcow2'],
-                 supported_protocols=['file'])
+                 supported_protocols=['file'],
+                 # compat=1.1 is needed for autoclear flags
+                 unsupported_imgopts=['compat=0.10'])
diff --git a/tests/qemu-iotests/206 b/tests/qemu-iotests/206
index 23ff2f624b..91dd6ee176 100755
--- a/tests/qemu-iotests/206
+++ b/tests/qemu-iotests/206
@@ -23,7 +23,10 @@
 import iotests
 from iotests import imgfmt
 
-iotests.script_initialize(supported_fmts=['qcow2'])
+iotests.script_initialize(supported_fmts=['qcow2'],
+                          # All options are ignored
+                          unsupported_imgopts=['compat=0.10', 'refcount_bits',
+                                               'data_file'])
 
 def blockdev_create(vm, options):
     result = vm.qmp_log('blockdev-create',
diff --git a/tests/qemu-iotests/222 b/tests/qemu-iotests/222
index 6788979ed3..ffa88d4bb9 100644
--- a/tests/qemu-iotests/222
+++ b/tests/qemu-iotests/222
@@ -25,7 +25,8 @@ import iotests
 from iotests import log, qemu_img, qemu_io, qemu_io_silent
 
 iotests.script_initialize(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk',
-                                          'vhdx', 'raw'])
+                                          'vhdx', 'raw'],
+                          unsupported_imgopts=['streamOptimized'])
 
 patterns = [("0x5d", "0",         "64k"),
             ("0xd5", "1M",        "64k"),
diff --git a/tests/qemu-iotests/237 b/tests/qemu-iotests/237
index 3758ace0bc..34ff4b55db 100755
--- a/tests/qemu-iotests/237
+++ b/tests/qemu-iotests/237
@@ -24,7 +24,9 @@ import math
 import iotests
 from iotests import imgfmt
 
-iotests.script_initialize(supported_fmts=['vmdk'])
+iotests.script_initialize(supported_fmts=['vmdk'],
+                          # All options are ignored
+                          unsupported_imgopts=['subformat'])
 
 def blockdev_create(vm, options):
     result = vm.qmp_log('blockdev-create', job_id='job0', options=options,
diff --git a/tests/qemu-iotests/242 b/tests/qemu-iotests/242
index 21f69a0d2c..333fb8b56c 100755
--- a/tests/qemu-iotests/242
+++ b/tests/qemu-iotests/242
@@ -24,7 +24,9 @@ import struct
 from iotests import qemu_img_create, qemu_io, qemu_img_pipe, \
     file_path, img_info_log, log, filter_qemu_io
 
-iotests.script_initialize(supported_fmts=['qcow2'])
+iotests.script_initialize(supported_fmts=['qcow2'],
+                          # compat=1.1 is needed for dirty bitmaps
+                          unsupported_imgopts=['compat=0.10'])
 
 disk = file_path('disk')
 chunk = 256 * 1024
diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245
index 50544a7836..72ab2a248d 100644
--- a/tests/qemu-iotests/245
+++ b/tests/qemu-iotests/245
@@ -1002,4 +1002,6 @@ class TestBlockdevReopen(iotests.QMPTestCase):
 if __name__ == '__main__':
     iotests.activate_logging()
     iotests.main(supported_fmts=["qcow2"],
-                 supported_protocols=["file"])
+                 supported_protocols=["file"],
+                 # reopen options differ with an external data file
+                 unsupported_imgopts=['data_file'])
diff --git a/tests/qemu-iotests/246 b/tests/qemu-iotests/246
index 1d7747d62d..d1364d5901 100755
--- a/tests/qemu-iotests/246
+++ b/tests/qemu-iotests/246
@@ -22,7 +22,10 @@
 import iotests
 from iotests import log
 
-iotests.script_initialize(supported_fmts=['qcow2'])
+iotests.script_initialize(supported_fmts=['qcow2'],
+                          # compat=1.1 is needed for dirty bitmaps
+                          unsupported_imgopts=['compat=0.10'])
+
 size = 64 * 1024 * 1024 * 1024
 gran_small = 32 * 1024
 gran_large = 128 * 1024
diff --git a/tests/qemu-iotests/248 b/tests/qemu-iotests/248
index 781b21b227..0adc4da802 100755
--- a/tests/qemu-iotests/248
+++ b/tests/qemu-iotests/248
@@ -21,7 +21,11 @@
 import iotests
 from iotests import qemu_img_create, qemu_io, file_path, filter_qmp_testfiles
 
-iotests.script_initialize(supported_fmts=['qcow2'])
+iotests.script_initialize(supported_fmts=['qcow2'],
+                          # With an external data file, we would need
+                          # to impose the limit on @data-file instead
+                          # of @file
+                          unsupported_imgopts=['data_file'])
 
 source, target = file_path('source', 'target')
 size = 5 * 1024 * 1024
diff --git a/tests/qemu-iotests/254 b/tests/qemu-iotests/254
index 43b40f4f71..0404faf853 100755
--- a/tests/qemu-iotests/254
+++ b/tests/qemu-iotests/254
@@ -21,7 +21,9 @@
 import iotests
 from iotests import qemu_img_create, file_path, log
 
-iotests.script_initialize(supported_fmts=['qcow2'])
+iotests.script_initialize(supported_fmts=['qcow2'],
+                          # compat=1.1 is needed for dirty bitmaps
+                          unsupported_imgopts=['compat=0.10'])
 
 disk, top = file_path('disk', 'top')
 size = 1024 * 1024
diff --git a/tests/qemu-iotests/257 b/tests/qemu-iotests/257
index de8b45f094..82c10e9b52 100755
--- a/tests/qemu-iotests/257
+++ b/tests/qemu-iotests/257
@@ -526,4 +526,6 @@ def main():
 
 if __name__ == '__main__':
     iotests.script_main(main, supported_fmts=['qcow2'],
-                        supported_protocols=['file'])
+                        supported_protocols=['file'],
+                        # blkdebug does not work with an external data file
+                        unsupported_imgopts=['data_file'])
-- 
2.21.0

[PATCH 15/67] iotests/030: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/030 | 69 ++++++++++++++++++++----------------------
 1 file changed, 32 insertions(+), 37 deletions(-)

diff --git a/tests/qemu-iotests/030 b/tests/qemu-iotests/030
index 01aa96ed16..cddb017496 100755
--- a/tests/qemu-iotests/030
+++ b/tests/qemu-iotests/030
@@ -21,7 +21,7 @@
 import time
 import os
 import iotests
-from iotests import qemu_img, qemu_io
+from iotests import create_test_image, remove_test_image, qemu_io
 
 backing_img = os.path.join(iotests.test_dir, 'backing.img')
 mid_img = os.path.join(iotests.test_dir, 'mid.img')
@@ -32,8 +32,8 @@ class TestSingleDrive(iotests.QMPTestCase):
 
     def setUp(self):
         iotests.create_image(backing_img, TestSingleDrive.image_len)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, mid_img)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % mid_img, test_img)
+        create_test_image(mid_img, backing_file=backing_img)
+        create_test_image(test_img, backing_file=mid_img)
         qemu_io('-f', 'raw', '-c', 'write -P 0x1 0 512', backing_img)
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0x1 524288 512', mid_img)
         self.vm = iotests.VM().add_drive("blkdebug::" + test_img,
@@ -43,8 +43,8 @@ class TestSingleDrive(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
-        os.remove(mid_img)
+        remove_test_image(test_img)
+        remove_test_image(mid_img)
         os.remove(backing_img)
 
     def test_stream(self):
@@ -155,9 +155,8 @@ class TestSingleDrive(iotests.QMPTestCase):
 
     def test_read_only(self):
         # Create a new file that we can attach (we need a read-only top)
-        with iotests.FilePath('ro-top.img') as ro_top_path:
-            qemu_img('create', '-f', iotests.imgfmt, ro_top_path,
-                     str(self.image_len))
+        with iotests.ImagePath('ro-top.img') as ro_top_path:
+            create_test_image(ro_top_path, self.image_len)
 
             result = self.vm.qmp('blockdev-add',
                                  node_name='ro-top',
@@ -198,8 +197,7 @@ class TestParallelOps(iotests.QMPTestCase):
         # Create all images
         iotests.create_image(self.imgs[0], self.image_len)
         for i in range(1, self.num_imgs):
-            qemu_img('create', '-f', iotests.imgfmt,
-                     '-o', 'backing_file=%s' % self.imgs[i-1], self.imgs[i])
+            create_test_image(self.imgs[i], backing_file=self.imgs[i - 1])
 
         # Put data into the images we are copying data from
         odd_img_indexes = [x for x in reversed(range(self.num_imgs)) if x % 2 == 1]
@@ -219,7 +217,7 @@ class TestParallelOps(iotests.QMPTestCase):
     def tearDown(self):
         self.vm.shutdown()
         for img in self.imgs:
-            os.remove(img)
+            remove_test_image(img)
 
     # Test that it's possible to run several block-stream operations
     # in parallel in the same snapshot chain
@@ -539,11 +537,10 @@ class TestQuorum(iotests.QMPTestCase):
             backing_img = os.path.join(iotests.test_dir, 'backing-%d.img' % i)
             self.children.append(child_img)
             self.backing.append(backing_img)
-            qemu_img('create', '-f', iotests.imgfmt, backing_img, '1M')
+            create_test_image(backing_img, '1M')
             qemu_io('-f', iotests.imgfmt,
                     '-c', 'write -P 0x55 0 1024', backing_img)
-            qemu_img('create', '-f', iotests.imgfmt,
-                     '-o', 'backing_file=%s' % backing_img, child_img)
+            create_test_image(child_img, backing_file=backing_img)
             opts.append("children.%d.file.filename=%s" % (i, child_img))
             opts.append("children.%d.node-name=node%d" % (i, i))
 
@@ -555,9 +552,9 @@ class TestQuorum(iotests.QMPTestCase):
     def tearDown(self):
         self.vm.shutdown()
         for img in self.children:
-            os.remove(img)
+            remove_test_image(img)
         for img in self.backing:
-            os.remove(img)
+            remove_test_image(img)
 
     def test_stream_quorum(self):
         if not iotests.supports_quorum():
@@ -587,7 +584,7 @@ class TestSmallerBackingFile(iotests.QMPTestCase):
 
     def setUp(self):
         iotests.create_image(backing_img, self.backing_len)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, test_img, str(self.image_len))
+        create_test_image(test_img, self.image_len, backing_file=backing_img)
         self.vm = iotests.VM().add_drive(test_img)
         self.vm.launch()
 
@@ -638,16 +635,15 @@ class TestEIO(TestErrors):
         self.blkdebug_file = backing_img + ".blkdebug"
         iotests.create_image(backing_img, TestErrors.image_len)
         self.create_blkdebug_file(self.blkdebug_file, "read_aio", 5)
-        qemu_img('create', '-f', iotests.imgfmt,
-                 '-o', 'backing_file=blkdebug:%s:%s,backing_fmt=raw'
-                       % (self.blkdebug_file, backing_img),
-                 test_img)
+        create_test_image(test_img, backing_fmt='raw',
+                          backing_file=('blkdebug:%s:%s' % (self.blkdebug_file,
+                                                            backing_img)))
         self.vm = iotests.VM().add_drive(test_img)
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
+        remove_test_image(test_img)
         os.remove(backing_img)
         os.remove(self.blkdebug_file)
 
@@ -788,16 +784,15 @@ class TestENOSPC(TestErrors):
         self.blkdebug_file = backing_img + ".blkdebug"
         iotests.create_image(backing_img, TestErrors.image_len)
         self.create_blkdebug_file(self.blkdebug_file, "read_aio", 28)
-        qemu_img('create', '-f', iotests.imgfmt,
-                 '-o', 'backing_file=blkdebug:%s:%s,backing_fmt=raw'
-                       % (self.blkdebug_file, backing_img),
-                 test_img)
+        create_test_image(test_img, backing_fmt='raw',
+                          backing_file=('blkdebug:%s:%s' % (self.blkdebug_file,
+                                                            backing_img)))
         self.vm = iotests.VM().add_drive(test_img)
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
+        remove_test_image(test_img)
         os.remove(backing_img)
         os.remove(self.blkdebug_file)
 
@@ -848,17 +843,17 @@ class TestStreamStop(iotests.QMPTestCase):
     image_len = 8 * 1024 * 1024 * 1024 # GB
 
     def setUp(self):
-        qemu_img('create', backing_img, str(TestStreamStop.image_len))
-        qemu_io('-f', 'raw', '-c', 'write -P 0x1 0 32M', backing_img)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, test_img)
+        create_test_image(backing_img, TestStreamStop.image_len)
+        qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0x1 0 32M', backing_img)
+        create_test_image(test_img, backing_file=backing_img)
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0x1 32M 32M', test_img)
         self.vm = iotests.VM().add_drive("blkdebug::" + test_img)
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
-        os.remove(backing_img)
+        remove_test_image(test_img)
+        remove_test_image(backing_img)
 
     def test_stream_stop(self):
         self.assert_no_active_block_jobs()
@@ -879,17 +874,17 @@ class TestSetSpeed(iotests.QMPTestCase):
     image_len = 80 * 1024 * 1024 # MB
 
     def setUp(self):
-        qemu_img('create', backing_img, str(TestSetSpeed.image_len))
-        qemu_io('-f', 'raw', '-c', 'write -P 0x1 0 32M', backing_img)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, test_img)
+        create_test_image(backing_img, TestSetSpeed.image_len)
+        qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0x1 0 32M', backing_img)
+        create_test_image(test_img, backing_file=backing_img)
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0x1 32M 32M', test_img)
         self.vm = iotests.VM().add_drive('blkdebug::' + test_img)
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
-        os.remove(backing_img)
+        remove_test_image(test_img)
+        remove_test_image(backing_img)
 
     # This is a short performance test which is not run by default.
     # Invoke "IMGFMT=qed ./030 TestSetSpeed.perf_test_throughput"
-- 
2.21.0

[PATCH 16/67] iotests/040: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/040 | 42 +++++++++++++++++++++---------------------
 1 file changed, 21 insertions(+), 21 deletions(-)

diff --git a/tests/qemu-iotests/040 b/tests/qemu-iotests/040
index 762ad1ebcb..9584eeeb1f 100755
--- a/tests/qemu-iotests/040
+++ b/tests/qemu-iotests/040
@@ -24,7 +24,7 @@
 import time
 import os
 import iotests
-from iotests import qemu_img, qemu_io
+from iotests import create_test_image, remove_test_image, qemu_img, qemu_io
 import struct
 import errno
 
@@ -80,8 +80,8 @@ class TestSingleDrive(ImageCommitTestCase):
 
     def setUp(self):
         iotests.create_image(backing_img, self.image_len)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, mid_img)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % mid_img, test_img)
+        create_test_image(mid_img, backing_file=backing_img)
+        create_test_image(test_img, backing_file=mid_img)
         qemu_io('-f', 'raw', '-c', 'write -P 0xab 0 524288', backing_img)
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xef 524288 524288', mid_img)
         self.vm = iotests.VM().add_drive(test_img, "node-name=top,backing.node-name=mid,backing.backing.node-name=base", interface="none")
@@ -92,8 +92,8 @@ class TestSingleDrive(ImageCommitTestCase):
 
     def tearDown(self):
         self.vm.shutdown(has_quit=self.has_quit)
-        os.remove(test_img)
-        os.remove(mid_img)
+        remove_test_image(test_img)
+        remove_test_image(mid_img)
         os.remove(backing_img)
 
     def test_commit(self):
@@ -303,8 +303,8 @@ class TestRelativePaths(ImageCommitTestCase):
             if exception.errno != errno.EEXIST:
                 raise
         iotests.create_image(self.backing_img_abs, TestRelativePaths.image_len)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % self.backing_img_abs, self.mid_img_abs)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % self.mid_img_abs, self.test_img)
+        create_test_image(self.mid_img_abs, backing_file=self.backing_img_abs)
+        create_test_image(self.test_img, backing_file=self.mid_img_abs)
         qemu_img('rebase', '-u', '-b', self.backing_img, self.mid_img_abs)
         qemu_img('rebase', '-u', '-b', self.mid_img, self.test_img)
         qemu_io('-f', 'raw', '-c', 'write -P 0xab 0 524288', self.backing_img_abs)
@@ -314,8 +314,8 @@ class TestRelativePaths(ImageCommitTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(self.test_img)
-        os.remove(self.mid_img_abs)
+        remove_test_image(self.test_img)
+        remove_test_image(self.mid_img_abs)
         os.remove(self.backing_img_abs)
         try:
             os.rmdir(os.path.join(iotests.test_dir, self.dir1))
@@ -368,9 +368,9 @@ class TestSetSpeed(ImageCommitTestCase):
     image_len = 80 * 1024 * 1024 # MB
 
     def setUp(self):
-        qemu_img('create', backing_img, str(TestSetSpeed.image_len))
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, mid_img)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % mid_img, test_img)
+        create_test_image(backing_img, TestSetSpeed.image_len)
+        create_test_image(mid_img, backing_file=backing_img)
+        create_test_image(test_img, backing_file=mid_img)
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0x1 0 512', test_img)
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xef 524288 524288', mid_img)
         self.vm = iotests.VM().add_drive('blkdebug::' + test_img)
@@ -378,9 +378,9 @@ class TestSetSpeed(ImageCommitTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
-        os.remove(mid_img)
-        os.remove(backing_img)
+        remove_test_image(test_img)
+        remove_test_image(mid_img)
+        remove_test_image(backing_img)
 
     def test_set_speed(self):
         self.assert_no_active_block_jobs()
@@ -408,9 +408,9 @@ class TestReopenOverlay(ImageCommitTestCase):
 
     def setUp(self):
         iotests.create_image(self.img0, self.image_len)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % self.img0, self.img1)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % self.img1, self.img2)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % self.img2, self.img3)
+        create_test_image(self.img1, backing_file=self.img0)
+        create_test_image(self.img2, backing_file=self.img1)
+        create_test_image(self.img3, backing_file=self.img2)
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xab 0 128K', self.img1)
         self.vm = iotests.VM().add_drive(self.img3)
         self.vm.launch()
@@ -418,9 +418,9 @@ class TestReopenOverlay(ImageCommitTestCase):
     def tearDown(self):
         self.vm.shutdown()
         os.remove(self.img0)
-        os.remove(self.img1)
-        os.remove(self.img2)
-        os.remove(self.img3)
+        remove_test_image(self.img1)
+        remove_test_image(self.img2)
+        remove_test_image(self.img3)
 
     # This tests what happens when the overlay image of the 'top' node
     # needs to be reopened in read-write mode in order to update the
-- 
2.21.0

[PATCH 17/67] iotests/041: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/041 | 108 +++++++++++++++++++----------------------
 1 file changed, 51 insertions(+), 57 deletions(-)

diff --git a/tests/qemu-iotests/041 b/tests/qemu-iotests/041
index 8568426311..5cf2e2cb52 100755
--- a/tests/qemu-iotests/041
+++ b/tests/qemu-iotests/041
@@ -21,7 +21,7 @@
 import time
 import os
 import iotests
-from iotests import qemu_img, qemu_io
+from iotests import create_test_image, remove_test_image, qemu_img, qemu_io
 
 backing_img = os.path.join(iotests.test_dir, 'backing.img')
 target_backing_img = os.path.join(iotests.test_dir, 'target-backing.img')
@@ -41,7 +41,7 @@ class TestSingleDrive(iotests.QMPTestCase):
 
     def setUp(self):
         iotests.create_image(backing_img, self.image_len)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, test_img)
+        create_test_image(test_img, backing_file=backing_img)
         self.vm = iotests.VM().add_drive(test_img, "node-name=top,backing.node-name=base")
         if iotests.qemu_default_machine == 'pc':
             self.vm.add_drive(None, 'media=cdrom', 'ide')
@@ -49,12 +49,9 @@ class TestSingleDrive(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
+        remove_test_image(test_img)
         os.remove(backing_img)
-        try:
-            os.remove(target_img)
-        except OSError:
-            pass
+        remove_test_image(target_img)
 
     def test_complete(self):
         self.assert_no_active_block_jobs()
@@ -138,8 +135,8 @@ class TestSingleDrive(iotests.QMPTestCase):
     def test_small_buffer2(self):
         self.assert_no_active_block_jobs()
 
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'cluster_size=%d,size=%d'
-                        % (self.image_len, self.image_len), target_img)
+        create_test_image(target_img, self.image_len,
+                          opts=['cluster_size=%d' % self.image_len])
         result = self.vm.qmp(self.qmp_cmd, device='drive0', sync='full',
                              buf_size=65536, mode='existing', target=self.qmp_target)
         self.assert_qmp(result, 'return', {})
@@ -154,8 +151,8 @@ class TestSingleDrive(iotests.QMPTestCase):
     def test_large_cluster(self):
         self.assert_no_active_block_jobs()
 
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'cluster_size=%d,backing_file=%s'
-                        % (self.image_len, backing_img), target_img)
+        create_test_image(target_img, backing_file=backing_img,
+                          opts=['cluster_size=%d' % self.image_len])
         result = self.vm.qmp(self.qmp_cmd, device='drive0', sync='full',
                              mode='existing', target=self.qmp_target)
         self.assert_qmp(result, 'return', {})
@@ -227,7 +224,7 @@ class TestSingleBlockdev(TestSingleDrive):
 
     def setUp(self):
         TestSingleDrive.setUp(self)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, target_img)
+        create_test_image(target_img, backing_file=backing_img)
         args = {'driver': iotests.imgfmt,
                 'node-name': self.qmp_target,
                 'file': { 'filename': target_img, 'driver': 'file' } }
@@ -265,24 +262,21 @@ class TestMirrorNoBacking(iotests.QMPTestCase):
 
     def setUp(self):
         iotests.create_image(backing_img, TestMirrorNoBacking.image_len)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, test_img)
+        create_test_image(test_img, backing_file=backing_img)
         self.vm = iotests.VM().add_drive(test_img)
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
+        remove_test_image(test_img)
         os.remove(backing_img)
-        try:
-            os.remove(target_backing_img)
-        except:
-            pass
-        os.remove(target_img)
+        remove_test_image(target_backing_img)
+        remove_test_image(target_img)
 
     def test_complete(self):
         self.assert_no_active_block_jobs()
 
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, target_img)
+        create_test_image(target_img, backing_file=backing_img)
         result = self.vm.qmp('drive-mirror', device='drive0', sync='full',
                              mode='existing', target=target_img)
         self.assert_qmp(result, 'return', {})
@@ -297,7 +291,7 @@ class TestMirrorNoBacking(iotests.QMPTestCase):
     def test_cancel(self):
         self.assert_no_active_block_jobs()
 
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, target_img)
+        create_test_image(target_img, backing_file=backing_img)
         result = self.vm.qmp('drive-mirror', device='drive0', sync='full',
                              mode='existing', target=target_img)
         self.assert_qmp(result, 'return', {})
@@ -313,10 +307,9 @@ class TestMirrorNoBacking(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
 
         # qemu-img create fails if the image is not there
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'size=%d'
-                        %(TestMirrorNoBacking.image_len), target_backing_img)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'cluster_size=%d,backing_file=%s'
-                        % (TestMirrorNoBacking.image_len, target_backing_img), target_img)
+        create_test_image(target_backing_img, TestMirrorNoBacking.image_len)
+        create_test_image(target_img, backing_file=target_backing_img,
+                          opts=['cluster_size=%d' % TestMirrorNoBacking.image_len])
 
         result = self.vm.qmp('drive-mirror', device='drive0', sync='full',
                              mode='existing', target=target_img)
@@ -335,14 +328,14 @@ class TestMirrorResized(iotests.QMPTestCase):
 
     def setUp(self):
         iotests.create_image(backing_img, TestMirrorResized.backing_len)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, test_img)
+        create_test_image(test_img, backing_file=backing_img)
         qemu_img('resize', test_img, '2M')
         self.vm = iotests.VM().add_drive(test_img)
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
+        remove_test_image(test_img)
         os.remove(backing_img)
         try:
             os.remove(target_img)
@@ -407,14 +400,15 @@ new_state = "1"
 ''' % (event, errno, self.MIRROR_GRANULARITY // 512, event, event))
         file.close()
 
+    # blkdebug does not work with an external data file
+    @iotests.skip_for_imgopts(['data_file'])
     def setUp(self):
         self.blkdebug_file = backing_img + ".blkdebug"
         iotests.create_image(backing_img, TestReadErrors.image_len)
         self.create_blkdebug_file(self.blkdebug_file, "read_aio", 5)
-        qemu_img('create', '-f', iotests.imgfmt,
-                 '-o', 'backing_file=blkdebug:%s:%s,backing_fmt=raw'
-                       % (self.blkdebug_file, backing_img),
-                 test_img)
+        create_test_image(test_img, backing_fmt='raw',
+                          backing_file=('blkdebug:%s:%s' % (self.blkdebug_file,
+                                                            backing_img)))
         # Write something for tests that use sync='top'
         qemu_io('-c', 'write %d 512' % (self.MIRROR_GRANULARITY + 65536),
                         test_img)
@@ -423,8 +417,8 @@ new_state = "1"
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
-        os.remove(target_img)
+        remove_test_image(test_img)
+        remove_test_image(target_img)
         os.remove(backing_img)
         os.remove(self.blkdebug_file)
 
@@ -483,7 +477,8 @@ new_state = "1"
         # Test COW into the target image.  The first half of the
         # cluster at MIRROR_GRANULARITY has to be copied from
         # backing_img, even though sync='top'.
-        qemu_img('create', '-f', iotests.imgfmt, '-ocluster_size=131072,backing_file=%s' %(backing_img), target_img)
+        create_test_image(target_img, backing_file=backing_img,
+                          opts=['cluster_size=131072'])
         result = self.vm.qmp('drive-mirror', device='drive0', sync='top',
                              on_source_error='ignore',
                              mode='existing', target=target_img)
@@ -572,20 +567,22 @@ new_state = "1"
 ''' % (event, errno, self.MIRROR_GRANULARITY // 512, event, event))
         file.close()
 
+    # blkdebug does not work with an external data file
+    @iotests.skip_for_imgopts(['data_file'])
     def setUp(self):
         self.blkdebug_file = target_img + ".blkdebug"
         iotests.create_image(backing_img, TestWriteErrors.image_len)
         self.create_blkdebug_file(self.blkdebug_file, "write_aio", 5)
-        qemu_img('create', '-f', iotests.imgfmt, '-obacking_file=%s' %(backing_img), test_img)
+        create_test_image(test_img, backing_file=backing_img)
         self.vm = iotests.VM().add_drive(test_img)
         self.target_img = 'blkdebug:%s:%s' % (self.blkdebug_file, target_img)
-        qemu_img('create', '-f', iotests.imgfmt, '-osize=%d' %(TestWriteErrors.image_len), target_img)
+        create_test_image(target_img, TestWriteErrors.image_len)
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
-        os.remove(target_img)
+        remove_test_image(test_img)
+        remove_test_image(target_img)
         os.remove(backing_img)
         os.remove(self.blkdebug_file)
 
@@ -673,15 +670,15 @@ class TestSetSpeed(iotests.QMPTestCase):
     image_len = 80 * 1024 * 1024 # MB
 
     def setUp(self):
-        qemu_img('create', backing_img, str(TestSetSpeed.image_len))
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, test_img)
+        create_test_image(backing_img, TestSetSpeed.image_len)
+        create_test_image(test_img, backing_file=backing_img)
         self.vm = iotests.VM().add_drive(test_img)
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
-        os.remove(backing_img)
+        remove_test_image(test_img)
+        remove_test_image(backing_img)
         os.remove(target_img)
 
     def test_set_speed(self):
@@ -739,8 +736,7 @@ class TestUnbackedSource(iotests.QMPTestCase):
     image_len = 2 * 1024 * 1024 # MB
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, test_img,
-                 str(TestUnbackedSource.image_len))
+        create_test_image(test_img, TestUnbackedSource.image_len)
         self.vm = iotests.VM()
         self.vm.launch()
         result = self.vm.qmp('blockdev-add', node_name='drive0',
@@ -753,8 +749,8 @@ class TestUnbackedSource(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
-        os.remove(target_img)
+        remove_test_image(test_img)
+        remove_test_image(target_img)
 
     def test_absolute_paths_full(self):
         self.assert_no_active_block_jobs()
@@ -784,8 +780,7 @@ class TestUnbackedSource(iotests.QMPTestCase):
         self.assert_no_active_block_jobs()
 
     def test_existing_full(self):
-        qemu_img('create', '-f', iotests.imgfmt, target_img,
-                 str(self.image_len))
+        create_test_image(target_img, self.image_len)
         qemu_io('-c', 'write -P 42 0 64k', target_img)
 
         self.assert_no_active_block_jobs()
@@ -802,8 +797,7 @@ class TestUnbackedSource(iotests.QMPTestCase):
                         'target image does not match source after mirroring')
 
     def test_blockdev_full(self):
-        qemu_img('create', '-f', iotests.imgfmt, target_img,
-                 str(self.image_len))
+        create_test_image(target_img, self.image_len)
         qemu_io('-c', 'write -P 42 0 64k', target_img)
 
         result = self.vm.qmp('blockdev-add', node_name='target',
@@ -834,8 +828,7 @@ class TestGranularity(iotests.QMPTestCase):
     image_len = 10 * 1024 * 1024 # MB
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, test_img,
-                 str(TestGranularity.image_len))
+        create_test_image(test_img, TestGranularity.image_len)
         qemu_io('-c', 'write 0 %d' % (self.image_len),
                 test_img)
         self.vm = iotests.VM().add_drive(test_img)
@@ -845,7 +838,7 @@ class TestGranularity(iotests.QMPTestCase):
         self.vm.shutdown()
         self.assertTrue(iotests.compare_images(test_img, target_img),
                         'target image does not match source after mirroring')
-        os.remove(test_img)
+        remove_test_image(test_img)
         os.remove(target_img)
 
     def test_granularity(self):
@@ -879,8 +872,7 @@ class TestRepairQuorum(iotests.QMPTestCase):
 
         # Add each individual quorum images
         for i in self.IMAGES:
-            qemu_img('create', '-f', iotests.imgfmt, i,
-                     str(TestSingleDrive.image_len))
+            create_test_image(i, TestSingleDrive.image_len)
             # Assign a node name to each quorum image in order to manipulate
             # them
             opts = "node-name=img%i" % self.IMAGES.index(i)
@@ -898,8 +890,10 @@ class TestRepairQuorum(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        for i in self.IMAGES + [ quorum_repair_img, quorum_snapshot_file ]:
-            # Do a try/except because the test may have deleted some images
+        for i in self.IMAGES:
+            remove_test_image(i)
+
+        for i in [ quorum_repair_img, quorum_snapshot_file ]:
             try:
                 os.remove(i)
             except OSError:
-- 
2.21.0

[PATCH 18/67] iotests/044: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/044 | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/044 b/tests/qemu-iotests/044
index 97ba98e628..a5e13751e1 100755
--- a/tests/qemu-iotests/044
+++ b/tests/qemu-iotests/044
@@ -23,7 +23,7 @@ import os
 import qcow2
 from qcow2 import QcowHeader
 import iotests
-from iotests import qemu_img, qemu_img_verbose, qemu_io
+from iotests import create_test_image, remove_test_image, qemu_img_verbose, qemu_io
 import struct
 import subprocess
 import sys
@@ -100,13 +100,13 @@ class TestRefcountTableGrowth(iotests.QMPTestCase):
 
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'cluster_size=512', test_img, '16G')
+        create_test_image(test_img, '16G', opts=['cluster_size=512'])
         self.preallocate(test_img)
         pass
 
 
     def tearDown(self):
-        os.remove(test_img)
+        remove_test_image(test_img)
         pass
 
     def test_grow_refcount_table(self):
-- 
2.21.0

[PATCH 19/67] iotests/045: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/045 | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/tests/qemu-iotests/045 b/tests/qemu-iotests/045
index 01cc038884..781ceb8d19 100755
--- a/tests/qemu-iotests/045
+++ b/tests/qemu-iotests/045
@@ -20,7 +20,7 @@
 
 import os
 import iotests
-from iotests import qemu_img
+from iotests import create_test_image, remove_test_image
 
 image0 = os.path.join(iotests.test_dir, 'image0')
 image1 = os.path.join(iotests.test_dir, 'image1')
@@ -32,11 +32,11 @@ class TestFdSets(iotests.QMPTestCase):
 
     def setUp(self):
         self.vm = iotests.VM()
-        qemu_img('create', '-f', iotests.imgfmt, image0, '128K')
-        qemu_img('create', '-f', iotests.imgfmt, image1, '128K')
-        qemu_img('create', '-f', iotests.imgfmt, image2, '128K')
-        qemu_img('create', '-f', iotests.imgfmt, image3, '128K')
-        qemu_img('create', '-f', iotests.imgfmt, image4, '128K')
+        create_test_image(image0, '128K')
+        create_test_image(image1, '128K')
+        create_test_image(image2, '128K')
+        create_test_image(image3, '128K')
+        create_test_image(image4, '128K')
         self.file0 = open(image0, 'r')
         self.file1 = open(image1, 'w+')
         self.file2 = open(image2, 'r')
@@ -57,11 +57,11 @@ class TestFdSets(iotests.QMPTestCase):
         self.file2.close()
         self.file3.close()
         self.file4.close()
-        os.remove(image0)
-        os.remove(image1)
-        os.remove(image2)
-        os.remove(image3)
-        os.remove(image4)
+        remove_test_image(image0)
+        remove_test_image(image1)
+        remove_test_image(image2)
+        remove_test_image(image3)
+        remove_test_image(image4)
 
     def test_query_fdset(self):
         result = self.vm.qmp('query-fdsets')
@@ -129,7 +129,7 @@ class TestFdSets(iotests.QMPTestCase):
 class TestSCMFd(iotests.QMPTestCase):
     def setUp(self):
         self.vm = iotests.VM()
-        qemu_img('create', '-f', iotests.imgfmt, image0, '128K')
+        create_test_image(image0, '128K')
         # Add an unused monitor, to verify it works fine when two monitor
         # instances present
         self.vm.add_monitor_null()
@@ -137,7 +137,7 @@ class TestSCMFd(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(image0)
+        remove_test_image(image0)
 
     def _send_fd_by_SCM(self):
         ret = self.vm.send_fd_scm(file_path=image0)
-- 
2.21.0

[PATCH 20/67] iotests/055: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/055 | 41 +++++++++++++++++++++++------------------
 1 file changed, 23 insertions(+), 18 deletions(-)

diff --git a/tests/qemu-iotests/055 b/tests/qemu-iotests/055
index c732a112d6..e44f18570d 100755
--- a/tests/qemu-iotests/055
+++ b/tests/qemu-iotests/055
@@ -23,7 +23,7 @@
 import time
 import os
 import iotests
-from iotests import qemu_img, qemu_io
+from iotests import create_test_image, remove_test_image, qemu_io
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
 target_img = os.path.join(iotests.test_dir, 'target.img')
@@ -32,7 +32,7 @@ blockdev_target_img = os.path.join(iotests.test_dir, 'blockdev-target.img')
 image_len = 64 * 1024 * 1024 # MB
 
 def setUpModule():
-    qemu_img('create', '-f', iotests.imgfmt, test_img, str(image_len))
+    create_test_image(test_img, image_len)
     qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x11 0 64k', test_img)
     qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x00 64k 128k', test_img)
     qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x22 162k 32k', test_img)
@@ -41,12 +41,12 @@ def setUpModule():
     qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x33 67043328 64k', test_img)
 
 def tearDownModule():
-    os.remove(test_img)
+    remove_test_image(test_img)
 
 
 class TestSingleDrive(iotests.QMPTestCase):
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, blockdev_target_img, str(image_len))
+        create_test_image(blockdev_target_img, image_len)
 
         self.vm = iotests.VM().add_drive('blkdebug::' + test_img)
         self.vm.add_drive(blockdev_target_img, interface="none")
@@ -56,7 +56,7 @@ class TestSingleDrive(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(blockdev_target_img)
+        remove_test_image(blockdev_target_img)
         try:
             os.remove(target_img)
         except OSError:
@@ -163,7 +163,7 @@ class TestSingleDrive(iotests.QMPTestCase):
 
 class TestSetSpeed(iotests.QMPTestCase):
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, blockdev_target_img, str(image_len))
+        create_test_image(blockdev_target_img, image_len)
 
         self.vm = iotests.VM().add_drive('blkdebug::' + test_img)
         self.vm.add_drive(blockdev_target_img, interface="none")
@@ -171,7 +171,7 @@ class TestSetSpeed(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(blockdev_target_img)
+        remove_test_image(blockdev_target_img)
         try:
             os.remove(target_img)
         except OSError:
@@ -249,7 +249,7 @@ class TestSetSpeed(iotests.QMPTestCase):
 #       would stall.  Instead, we limit the block job speed here.
 class TestSingleTransaction(iotests.QMPTestCase):
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, blockdev_target_img, str(image_len))
+        create_test_image(blockdev_target_img, image_len)
 
         self.vm = iotests.VM().add_drive(test_img)
         self.vm.add_drive(blockdev_target_img, interface="none")
@@ -259,7 +259,7 @@ class TestSingleTransaction(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(blockdev_target_img)
+        remove_test_image(blockdev_target_img)
         try:
             os.remove(target_img)
         except OSError:
@@ -452,29 +452,34 @@ class TestSingleTransaction(iotests.QMPTestCase):
 
 class TestDriveCompression(iotests.QMPTestCase):
     image_len = 64 * 1024 * 1024 # MB
-    fmt_supports_compression = [{'type': 'qcow2', 'args': ()},
-                                {'type': 'vmdk', 'args': ('-o', 'subformat=streamOptimized')}]
+    fmt_supports_compression = [{'type': 'qcow2', 'opts': []},
+                                {'type': 'vmdk', 'opts': ['subformat=streamOptimized']}]
+
+    # Compression does not work with external data files
+    @iotests.skip_for_imgopts(['data_file'])
+    def setUp(self):
+        pass
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(blockdev_target_img)
+        remove_test_image(blockdev_target_img)
         try:
             os.remove(target_img)
         except OSError:
             pass
 
-    def do_prepare_drives(self, fmt, args, attach_target):
+    def do_prepare_drives(self, fmt, opts, attach_target):
         self.vm = iotests.VM().add_drive('blkdebug::' + test_img)
 
-        qemu_img('create', '-f', fmt, blockdev_target_img,
-                 str(TestDriveCompression.image_len), *args)
+        create_test_image(blockdev_target_img, TestDriveCompression.image_len,
+                          fmt=fmt, opts=opts)
         if attach_target:
             self.vm.add_drive(blockdev_target_img, format=fmt, interface="none")
 
         self.vm.launch()
 
     def do_test_compress_complete(self, cmd, format, attach_target, **args):
-        self.do_prepare_drives(format['type'], format['args'], attach_target)
+        self.do_prepare_drives(format['type'], format['opts'], attach_target)
 
         self.assert_no_active_block_jobs()
 
@@ -499,7 +504,7 @@ class TestDriveCompression(iotests.QMPTestCase):
                                            target='drive1')
 
     def do_test_compress_cancel(self, cmd, format, attach_target, **args):
-        self.do_prepare_drives(format['type'], format['args'], attach_target)
+        self.do_prepare_drives(format['type'], format['opts'], attach_target)
 
         self.assert_no_active_block_jobs()
 
@@ -523,7 +528,7 @@ class TestDriveCompression(iotests.QMPTestCase):
                                          target='drive1')
 
     def do_test_compress_pause(self, cmd, format, attach_target, **args):
-        self.do_prepare_drives(format['type'], format['args'], attach_target)
+        self.do_prepare_drives(format['type'], format['opts'], attach_target)
 
         self.assert_no_active_block_jobs()
 
-- 
2.21.0

[PATCH 21/67] iotests/056: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/056 | 30 ++++++++++++++----------------
 1 file changed, 14 insertions(+), 16 deletions(-)

diff --git a/tests/qemu-iotests/056 b/tests/qemu-iotests/056
index 98c55d8e5a..58b4103cd9 100755
--- a/tests/qemu-iotests/056
+++ b/tests/qemu-iotests/056
@@ -23,7 +23,7 @@
 import time
 import os
 import iotests
-from iotests import qemu_img, qemu_io, create_image
+from iotests import create_test_image, remove_test_image, qemu_io, create_image
 
 backing_img = os.path.join(iotests.test_dir, 'backing.img')
 test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -31,18 +31,12 @@ target_img = os.path.join(iotests.test_dir, 'target.img')
 
 def img_create(img, fmt=iotests.imgfmt, size='64M', **kwargs):
     fullname = os.path.join(iotests.test_dir, '%s.%s' % (img, fmt))
-    optargs = []
-    for k,v in kwargs.items():
-        optargs = optargs + ['-o', '%s=%s' % (k,v)]
-    args = ['create', '-f', fmt] + optargs + [fullname, size]
-    iotests.qemu_img(*args)
+    opts = []
+    for k, v in kwargs.items():
+        opts.append('%s=%s' % (k, v))
+    create_test_image(fullname, size, fmt=fmt, opts=opts)
     return fullname
 
-def try_remove(img):
-    try:
-        os.remove(img)
-    except OSError:
-        pass
 
 def io_write_patterns(img, patterns):
     for pattern in patterns:
@@ -54,7 +48,7 @@ class TestSyncModesNoneAndTop(iotests.QMPTestCase):
 
     def setUp(self):
         create_image(backing_img, TestSyncModesNoneAndTop.image_len)
-        qemu_img('create', '-f', iotests.imgfmt, '-o', 'backing_file=%s' % backing_img, test_img)
+        create_test_image(test_img, backing_file=backing_img)
         qemu_io('-c', 'write -P0x41 0 512', test_img)
         qemu_io('-c', 'write -P0xd5 1M 32k', test_img)
         qemu_io('-c', 'write -P0xdc 32M 124k', test_img)
@@ -64,7 +58,7 @@ class TestSyncModesNoneAndTop(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
+        remove_test_image(test_img)
         os.remove(backing_img)
         try:
             os.remove(target_img)
@@ -103,6 +97,8 @@ class TestSyncModesNoneAndTop(iotests.QMPTestCase):
         self.assertEqual(-1, qemu_io('-c', 'read -P0x41 0 512', target_img).find("verification failed"))
 
 class TestBeforeWriteNotifier(iotests.QMPTestCase):
+    # blkdebug does not work with an external data file
+    @iotests.skip_for_imgopts(['data_file'])
     def setUp(self):
         self.vm = iotests.VM().add_drive_raw("file=blkdebug::null-co://,id=drive0,align=65536,driver=blkdebug")
         self.vm.launch()
@@ -139,9 +135,9 @@ class BackupTest(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        try_remove(self.test_img)
-        try_remove(self.dest_img)
-        try_remove(self.ref_img)
+        remove_test_image(self.test_img)
+        remove_test_image(self.dest_img)
+        remove_test_image(self.ref_img)
 
     def hmp_io_writes(self, drive, patterns):
         for pattern in patterns:
@@ -268,6 +264,8 @@ class BackupTest(iotests.QMPTestCase):
                                  sync='full', target=self.dest_img,
                                  auto_dismiss=False)
 
+    # blkdebug does not work with an external data file
+    @iotests.skip_for_imgopts(['data_file'])
     def dismissal_failure(self, dismissal_opt):
         res = self.vm.qmp('query-block-jobs')
         self.assert_qmp(res, 'return', [])
-- 
2.21.0

[PATCH 22/67] iotests/057: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/057 | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/057 b/tests/qemu-iotests/057
index f5aa5929dc..c83f969b09 100755
--- a/tests/qemu-iotests/057
+++ b/tests/qemu-iotests/057
@@ -23,7 +23,7 @@
 import time
 import os
 import iotests
-from iotests import qemu_img, qemu_io
+from iotests import create_test_image, remove_test_image, qemu_io
 
 test_drv_base_name = 'drive'
 
@@ -40,7 +40,7 @@ class ImageSnapshotTestCase(iotests.QMPTestCase):
             filename = '%s%d' % (test_img_base_name, i)
             img = os.path.join(iotests.test_dir, filename)
             device = '%s%d' % (test_drv_base_name, i)
-            qemu_img('create', '-f', iotests.imgfmt, img, str(self.image_len))
+            create_test_image(img, self.image_len)
             self.vm.add_drive(img)
             self.expect.append({'image': img, 'device': device,
                                 'snapshots': [],
@@ -50,7 +50,7 @@ class ImageSnapshotTestCase(iotests.QMPTestCase):
     def tearDown(self):
         self.vm.shutdown()
         for dev_expect in self.expect:
-            os.remove(dev_expect['image'])
+            remove_test_image(dev_expect['image'])
 
     def createSnapshotInTransaction(self, snapshot_num, abort = False):
         actions = []
-- 
2.21.0

[PATCH 23/67] iotests/065: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/065 | 17 ++++++++---------
 1 file changed, 8 insertions(+), 9 deletions(-)

diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 9db9552784..221a87417d 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -23,7 +23,7 @@ import os
 import re
 import json
 import iotests
-from iotests import qemu_img, qemu_img_pipe
+from iotests import create_test_image, remove_test_image, qemu_img_pipe
 import unittest
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
@@ -34,11 +34,10 @@ class TestImageInfoSpecific(iotests.QMPTestCase):
     def setUp(self):
         if self.img_options is None:
             self.skipTest('Skipping abstract test class')
-        qemu_img('create', '-f', iotests.imgfmt, '-o', self.img_options,
-                 test_img, '128K')
+        create_test_image(test_img, '128K', opts=self.img_options)
 
     def tearDown(self):
-        os.remove(test_img)
+        remove_test_image(test_img)
 
 class TestQemuImgInfo(TestImageInfoSpecific):
     '''Abstract base class for qemu-img info tests'''
@@ -87,13 +86,13 @@ class TestQMP(TestImageInfoSpecific):
 
 class TestQCow2(TestQemuImgInfo):
     '''Testing a qcow2 version 2 image'''
-    img_options = 'compat=0.10'
+    img_options = ['compat=0.10']
     json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
     human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
 
 class TestQCow3NotLazy(TestQemuImgInfo):
     '''Testing a qcow2 version 3 image with lazy refcounts disabled'''
-    img_options = 'compat=1.1,lazy_refcounts=off'
+    img_options = ['compat=1.1', 'lazy_refcounts=off']
     json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
                      'refcount-bits': 16, 'corrupt': False }
     human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
@@ -101,7 +100,7 @@ class TestQCow3NotLazy(TestQemuImgInfo):
 
 class TestQCow3Lazy(TestQemuImgInfo):
     '''Testing a qcow2 version 3 image with lazy refcounts enabled'''
-    img_options = 'compat=1.1,lazy_refcounts=on'
+    img_options = ['compat=1.1', 'lazy_refcounts=on']
     json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
                      'refcount-bits': 16, 'corrupt': False }
     human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
@@ -110,7 +109,7 @@ class TestQCow3Lazy(TestQemuImgInfo):
 class TestQCow3NotLazyQMP(TestQMP):
     '''Testing a qcow2 version 3 image with lazy refcounts disabled, opening
        with lazy refcounts enabled'''
-    img_options = 'compat=1.1,lazy_refcounts=off'
+    img_options = ['compat=1.1', 'lazy_refcounts=off']
     qemu_options = 'lazy-refcounts=on'
     compare = { 'compat': '1.1', 'lazy-refcounts': False,
                 'refcount-bits': 16, 'corrupt': False }
@@ -119,7 +118,7 @@ class TestQCow3NotLazyQMP(TestQMP):
 class TestQCow3LazyQMP(TestQMP):
     '''Testing a qcow2 version 3 image with lazy refcounts enabled, opening
        with lazy refcounts disabled'''
-    img_options = 'compat=1.1,lazy_refcounts=on'
+    img_options = ['compat=1.1', 'lazy_refcounts=on']
     qemu_options = 'lazy-refcounts=off'
     compare = { 'compat': '1.1', 'lazy-refcounts': True,
                 'refcount-bits': 16, 'corrupt': False }
-- 
2.21.0

[PATCH 24/67] iotests/096: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/096 | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/096 b/tests/qemu-iotests/096
index ab9cb47822..ee0cfbffe8 100755
--- a/tests/qemu-iotests/096
+++ b/tests/qemu-iotests/096
@@ -20,6 +20,7 @@
 #
 
 import iotests
+from iotests import create_test_image, remove_test_image
 import os
 
 class TestLiveSnapshot(iotests.QMPTestCase):
@@ -35,13 +36,13 @@ class TestLiveSnapshot(iotests.QMPTestCase):
         opts.append('throttling.group=%s' % self.group)
         opts.append('throttling.iops-total=%d' % self.iops)
         opts.append('throttling.iops-size=%d' % self.iops_size)
-        iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, '100M')
+        create_test_image(self.base_img, '100M')
         self.vm = iotests.VM().add_drive(self.base_img, ','.join(opts))
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(self.base_img)
+        remove_test_image(self.base_img)
         os.remove(self.target_img)
 
     def checkConfig(self, active_layer):
-- 
2.21.0

[PATCH 25/67] iotests/118: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/118 | 26 +++++++++++++-------------
 1 file changed, 13 insertions(+), 13 deletions(-)

diff --git a/tests/qemu-iotests/118 b/tests/qemu-iotests/118
index ea0b326ae0..634a0b6445 100755
--- a/tests/qemu-iotests/118
+++ b/tests/qemu-iotests/118
@@ -23,7 +23,7 @@ import os
 import stat
 import time
 import iotests
-from iotests import qemu_img
+from iotests import create_test_image, remove_test_image
 
 old_img = os.path.join(iotests.test_dir, 'test0.img')
 new_img = os.path.join(iotests.test_dir, 'test1.img')
@@ -301,8 +301,8 @@ class TestInitiallyFilled(GeneralChangeTestsBaseClass):
     was_empty = False
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, old_img, '1440k')
-        qemu_img('create', '-f', iotests.imgfmt, new_img, '1440k')
+        create_test_image(old_img, '1440k')
+        create_test_image(new_img, '1440k')
         self.vm = iotests.VM()
         if self.use_drive:
             self.vm.add_drive(old_img, 'media=%s' % self.media, 'none')
@@ -320,8 +320,8 @@ class TestInitiallyFilled(GeneralChangeTestsBaseClass):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(old_img)
-        os.remove(new_img)
+        remove_test_image(old_img)
+        remove_test_image(new_img)
 
     def test_insert_on_filled(self):
         result = self.vm.qmp('blockdev-add',
@@ -344,7 +344,7 @@ class TestInitiallyEmpty(GeneralChangeTestsBaseClass):
     was_empty = True
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, new_img, '1440k')
+        create_test_image(new_img, '1440k')
         self.vm = iotests.VM()
         if self.use_drive:
             self.vm.add_drive(None, 'media=%s' % self.media, 'none')
@@ -358,7 +358,7 @@ class TestInitiallyEmpty(GeneralChangeTestsBaseClass):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(new_img)
+        remove_test_image(new_img)
 
     def test_remove_on_empty(self):
         result = self.vm.qmp('blockdev-open-tray', id=self.device_name)
@@ -394,16 +394,16 @@ class TestChangeReadOnly(ChangeBaseClass):
     device_name = 'qdev0'
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, old_img, '1440k')
-        qemu_img('create', '-f', iotests.imgfmt, new_img, '1440k')
+        create_test_image(old_img, '1440k')
+        create_test_image(new_img, '1440k')
         self.vm = iotests.VM()
 
     def tearDown(self):
         self.vm.shutdown()
         os.chmod(old_img, 0o666)
         os.chmod(new_img, 0o666)
-        os.remove(old_img)
-        os.remove(new_img)
+        remove_test_image(old_img)
+        remove_test_image(new_img)
 
     def test_ro_ro_retain(self):
         os.chmod(old_img, 0o444)
@@ -640,7 +640,7 @@ class TestBlockJobsAfterCycle(ChangeBaseClass):
     device_name = 'qdev0'
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, old_img, '1440K')
+        create_test_image(old_img, '1440k')
 
         self.vm = iotests.VM()
         self.vm.add_drive_raw("id=drive0,driver=null-co,if=none")
@@ -674,7 +674,7 @@ class TestBlockJobsAfterCycle(ChangeBaseClass):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(old_img)
+        remove_test_image(old_img)
         try:
             os.remove(new_img)
         except OSError:
-- 
2.21.0

[PATCH 26/67] iotests/124: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/124 | 29 +++++++++++++----------------
 1 file changed, 13 insertions(+), 16 deletions(-)

diff --git a/tests/qemu-iotests/124 b/tests/qemu-iotests/124
index d3e851e1ae..145dccb2b0 100755
--- a/tests/qemu-iotests/124
+++ b/tests/qemu-iotests/124
@@ -22,6 +22,7 @@
 
 import os
 import iotests
+from iotests import create_test_image, remove_test_image
 
 
 def io_write_patterns(img, patterns):
@@ -29,13 +30,6 @@ def io_write_patterns(img, patterns):
         iotests.qemu_io('-c', 'write -P%s %s %s' % pattern, img)
 
 
-def try_remove(img):
-    try:
-        os.remove(img)
-    except OSError:
-        pass
-
-
 def transaction_action(action, **kwargs):
     return {
         'type': action,
@@ -82,13 +76,13 @@ class Bitmap:
 
     def del_target(self):
         for image in self.backups.pop():
-            try_remove(image)
+            remove_test_image(image)
         self.num -= 1
 
     def cleanup(self):
         for backup in self.backups:
             for image in backup:
-                try_remove(image)
+                remove_test_image(image)
 
 
 class TestIncrementalBackupBase(iotests.QMPTestCase):
@@ -133,15 +127,14 @@ class TestIncrementalBackupBase(iotests.QMPTestCase):
 
     def img_create(self, img, fmt=iotests.imgfmt, size='64M',
                    parent=None, parentFormat=None, **kwargs):
-        optargs = []
-        for k,v in kwargs.items():
-            optargs = optargs + ['-o', '%s=%s' % (k,v)]
-        args = ['create', '-f', fmt] + optargs + [img, size]
+        opts = []
+        for k, v in kwargs.items():
+            opts.append('%s=%s' % (k, v))
         if parent:
             if parentFormat is None:
                 parentFormat = fmt
-            args = args + ['-b', parent, '-F', parentFormat]
-        iotests.qemu_img(*args)
+        create_test_image(img, size, fmt=fmt, backing_file=parent,
+                          backing_fmt=parentFormat, opts=opts)
         self.files.append(img)
 
 
@@ -287,7 +280,7 @@ class TestIncrementalBackupBase(iotests.QMPTestCase):
         for bitmap in self.bitmaps:
             bitmap.cleanup()
         for filename in self.files:
-            try_remove(filename)
+            remove_test_image(filename)
 
 
 
@@ -414,6 +407,8 @@ class TestIncrementalBackup(TestIncrementalBackupBase):
         self.check_backups()
 
 
+    # blkdebug does not work with an external data file
+    @iotests.skip_for_imgopts(['data_file'])
     def do_transaction_failure_test(self, race=False):
         # Create a second drive, with pattern:
         drive1 = self.add_node('drive1')
@@ -608,6 +603,8 @@ class TestIncrementalBackup(TestIncrementalBackupBase):
 class TestIncrementalBackupBlkdebug(TestIncrementalBackupBase):
     '''Incremental backup tests that utilize a BlkDebug filter on drive0.'''
 
+    # blkdebug does not work with an external data file
+    @iotests.skip_for_imgopts(['data_file'])
     def setUp(self):
         drive0 = self.add_node('drive0')
         self.img_create(drive0['file'], drive0['fmt'])
-- 
2.21.0

[PATCH 27/67] iotests/129: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/129 | 11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/129 b/tests/qemu-iotests/129
index cd6b9e9ce7..379719d85b 100755
--- a/tests/qemu-iotests/129
+++ b/tests/qemu-iotests/129
@@ -20,6 +20,7 @@
 
 import os
 import iotests
+from iotests import create_test_image, remove_test_image
 import time
 
 class TestStopWithBlockJob(iotests.QMPTestCase):
@@ -28,8 +29,8 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
     base_img = os.path.join(iotests.test_dir, 'base.img')
 
     def setUp(self):
-        iotests.qemu_img('create', '-f', iotests.imgfmt, self.base_img, "1G")
-        iotests.qemu_img('create', '-f', iotests.imgfmt, self.test_img, "-b", self.base_img)
+        create_test_image(self.base_img, '1G')
+        create_test_image(self.test_img, backing_file=self.base_img)
         iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 1M 128M', self.test_img)
         self.vm = iotests.VM().add_drive(self.test_img)
         self.vm.launch()
@@ -46,6 +47,12 @@ class TestStopWithBlockJob(iotests.QMPTestCase):
         result = self.vm.qmp("block_set_io_throttle", conv_keys=False,
                              **params)
         self.vm.shutdown()
+        remove_test_image(self.test_img)
+        remove_test_image(self.base_img)
+        try:
+            os.remove(self.target_img)
+        except OSError:
+            pass
 
     def do_test_stop(self, cmd, **args):
         """Test 'stop' while block job is running on a throttled drive.
-- 
2.21.0

[PATCH 28/67] iotests/132: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/132 | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/132 b/tests/qemu-iotests/132
index 0f2a106c81..9c7b773f09 100755
--- a/tests/qemu-iotests/132
+++ b/tests/qemu-iotests/132
@@ -21,7 +21,7 @@
 import time
 import os
 import iotests
-from iotests import qemu_img, qemu_io
+from iotests import create_test_image, remove_test_image, qemu_io
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
 target_img = os.path.join(iotests.test_dir, 'target.img')
@@ -31,7 +31,7 @@ class TestSingleDrive(iotests.QMPTestCase):
 
     def setUp(self):
         # Write data to the image so we can compare later
-        qemu_img('create', '-f', iotests.imgfmt, test_img, str(TestSingleDrive.image_len))
+        create_test_image(test_img, TestSingleDrive.image_len)
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P0x5d 0 2M', test_img)
 
         self.vm = iotests.VM().add_drive(test_img, 'discard=unmap')
@@ -39,7 +39,7 @@ class TestSingleDrive(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
+        remove_test_image(test_img)
         try:
             os.remove(target_img)
         except OSError:
-- 
2.21.0

[PATCH 29/67] iotests/139: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/139 | 15 +++++++--------
 1 file changed, 7 insertions(+), 8 deletions(-)

diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
index cbb5a76530..f4cb553116 100755
--- a/tests/qemu-iotests/139
+++ b/tests/qemu-iotests/139
@@ -21,6 +21,7 @@
 
 import os
 import iotests
+from iotests import create_test_image, remove_test_image
 import time
 
 base_img = os.path.join(iotests.test_dir, 'base.img')
@@ -33,7 +34,7 @@ else:
 class TestBlockdevDel(iotests.QMPTestCase):
 
     def setUp(self):
-        iotests.qemu_img('create', '-f', iotests.imgfmt, base_img, '1M')
+        create_test_image(base_img, '1M')
         self.vm = iotests.VM()
         self.vm.add_device("{},id=virtio-scsi".format(
             iotests.get_virtio_scsi_device()))
@@ -41,9 +42,8 @@ class TestBlockdevDel(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(base_img)
-        if os.path.isfile(new_img):
-            os.remove(new_img)
+        remove_test_image(base_img)
+        remove_test_image(new_img)
 
     # Check whether a BlockDriverState exists
     def checkBlockDriverState(self, node, must_exist = True):
@@ -70,8 +70,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
     # Add a BlockDriverState that will be used as overlay for the base_img BDS
     def addBlockDriverStateOverlay(self, node):
         self.checkBlockDriverState(node, False)
-        iotests.qemu_img('create', '-u', '-f', iotests.imgfmt,
-                         '-b', base_img, new_img, '1M')
+        create_test_image(new_img, '1M', backing_file=base_img, unsafe=True)
         opts = {'driver': iotests.imgfmt,
                 'node-name': node,
                 'backing': None,
@@ -202,7 +201,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
         self.checkBlockDriverState(test, False)
         self.checkBlockDriverState(raw, False)
         self.checkBlockDriverState(blkverify, False)
-        iotests.qemu_img('create', '-f', iotests.imgfmt, new_img, '1M')
+        create_test_image(new_img, '1M')
         node_0 = {'driver': iotests.imgfmt,
                   'node-name': test,
                   'file': {'driver': 'file',
@@ -226,7 +225,7 @@ class TestBlockdevDel(iotests.QMPTestCase):
         self.checkBlockDriverState(child0, False)
         self.checkBlockDriverState(child1, False)
         self.checkBlockDriverState(quorum, False)
-        iotests.qemu_img('create', '-f', iotests.imgfmt, new_img, '1M')
+        create_test_image(new_img, '1M')
         child_0 = {'driver': iotests.imgfmt,
                    'node-name': child0,
                    'file': {'driver': 'file',
-- 
2.21.0

[PATCH 30/67] iotests/147: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/147 | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
index ab8480b9a4..920e44a65c 100755
--- a/tests/qemu-iotests/147
+++ b/tests/qemu-iotests/147
@@ -24,7 +24,8 @@ import socket
 import stat
 import time
 import iotests
-from iotests import cachemode, imgfmt, qemu_img, qemu_nbd, qemu_nbd_early_pipe
+from iotests import cachemode, imgfmt, create_test_image, remove_test_image, \
+        qemu_nbd, qemu_nbd_early_pipe
 
 NBD_PORT_START      = 32768
 NBD_PORT_END        = NBD_PORT_START + 1024
@@ -80,13 +81,13 @@ class NBDBlockdevAddBase(iotests.QMPTestCase):
 
 class QemuNBD(NBDBlockdevAddBase):
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, test_img, '64k')
+        create_test_image(test_img, '64k')
         self.vm = iotests.VM()
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
+        remove_test_image(test_img)
         try:
             os.remove(unix_socket)
         except OSError:
@@ -127,7 +128,7 @@ class QemuNBD(NBDBlockdevAddBase):
 
 class BuiltinNBD(NBDBlockdevAddBase):
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, test_img, '64k')
+        create_test_image(test_img, '64k')
         self.vm = iotests.VM()
         self.vm.launch()
         self.server = iotests.VM('.server')
@@ -140,7 +141,7 @@ class BuiltinNBD(NBDBlockdevAddBase):
     def tearDown(self):
         self.vm.shutdown()
         self.server.shutdown()
-        os.remove(test_img)
+        remove_test_image(test_img)
         try:
             os.remove(unix_socket)
         except OSError:
-- 
2.21.0

[PATCH 31/67] iotests/148: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/148 | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/148 b/tests/qemu-iotests/148
index 8c11c53cba..a6df0ed983 100755
--- a/tests/qemu-iotests/148
+++ b/tests/qemu-iotests/148
@@ -21,6 +21,7 @@
 
 import os
 import iotests
+from iotests import create_test_image, remove_test_image
 
 imgs = (os.path.join(iotests.test_dir, 'quorum0.img'),
         os.path.join(iotests.test_dir, 'quorum1.img'),
@@ -51,7 +52,7 @@ sector = "%d"
         driveopts = ['driver=quorum', 'vote-threshold=2']
         driveopts.append('read-pattern=%s' % self.read_pattern)
         for i in range(len(imgs)):
-            iotests.qemu_img('create', '-f', iotests.imgfmt, imgs[i], '1M')
+            create_test_image(imgs[i], '1M')
             self.create_blkdebug_file(img_conf[i], i + offset)
             driveopts.append('children.%d.driver=%s' % (i, iotests.imgfmt))
             driveopts.append('children.%d.file.driver=blkdebug' % i)
@@ -65,7 +66,7 @@ sector = "%d"
     def tearDown(self):
         self.vm.shutdown()
         for i in range(len(imgs)):
-            os.remove(imgs[i])
+            remove_test_image(imgs[i])
             os.remove(img_conf[i])
 
     def do_check_event(self, node, sector = 0):
-- 
2.21.0

[PATCH 32/67] iotests/151: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/151 | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/151 b/tests/qemu-iotests/151
index 76ae265cc1..bfe2bbb200 100755
--- a/tests/qemu-iotests/151
+++ b/tests/qemu-iotests/151
@@ -20,7 +20,7 @@
 
 import os
 import iotests
-from iotests import qemu_img
+from iotests import create_test_image, remove_test_image
 
 source_img = os.path.join(iotests.test_dir, 'source.' + iotests.imgfmt)
 target_img = os.path.join(iotests.test_dir, 'target.' + iotests.imgfmt)
@@ -30,8 +30,8 @@ class TestActiveMirror(iotests.QMPTestCase):
     potential_writes_in_flight = True
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, source_img, '128M')
-        qemu_img('create', '-f', iotests.imgfmt, target_img, '128M')
+        create_test_image(source_img, '128M')
+        create_test_image(target_img, '128M')
 
         blk_source = {'id': 'source',
                       'if': 'none',
@@ -58,8 +58,8 @@ class TestActiveMirror(iotests.QMPTestCase):
             self.assertTrue(iotests.compare_images(source_img, target_img),
                             'mirror target does not match source')
 
-        os.remove(source_img)
-        os.remove(target_img)
+        remove_test_image(source_img)
+        remove_test_image(target_img)
 
     def doActiveIO(self, sync_source_and_target):
         # Fill the source image
-- 
2.21.0

[PATCH 33/67] iotests/152: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/152 | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/152 b/tests/qemu-iotests/152
index 732bf5f062..123ca0cd5d 100755
--- a/tests/qemu-iotests/152
+++ b/tests/qemu-iotests/152
@@ -20,20 +20,20 @@
 
 import os
 import iotests
-from iotests import qemu_img
+from iotests import create_test_image, remove_test_image
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
 target_img = os.path.join(iotests.test_dir, 'target.img')
 
 class TestUnaligned(iotests.QMPTestCase):
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, test_img, '512')
+        create_test_image(test_img, 512)
         self.vm = iotests.VM().add_drive(test_img)
         self.vm.launch()
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(test_img)
+        remove_test_image(test_img)
         try:
             os.remove(target_img)
         except OSError:
-- 
2.21.0

[PATCH 34/67] iotests/155: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/155 | 29 +++++++++++++----------------
 1 file changed, 13 insertions(+), 16 deletions(-)

diff --git a/tests/qemu-iotests/155 b/tests/qemu-iotests/155
index e19485911c..4efe0e2237 100755
--- a/tests/qemu-iotests/155
+++ b/tests/qemu-iotests/155
@@ -23,7 +23,7 @@
 
 import os
 import iotests
-from iotests import qemu_img
+from iotests import create_test_image, remove_test_image
 
 back0_img = os.path.join(iotests.test_dir, 'back0.' + iotests.imgfmt)
 back1_img = os.path.join(iotests.test_dir, 'back1.' + iotests.imgfmt)
@@ -51,10 +51,10 @@ class BaseClass(iotests.QMPTestCase):
     target_real_backing = None
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, back0_img, '1440K')
-        qemu_img('create', '-f', iotests.imgfmt, '-b', back0_img, back1_img)
-        qemu_img('create', '-f', iotests.imgfmt, '-b', back1_img, back2_img)
-        qemu_img('create', '-f', iotests.imgfmt, '-b', back2_img, source_img)
+        create_test_image(back0_img, '1440K')
+        create_test_image(back1_img, backing_file=back0_img)
+        create_test_image(back2_img, backing_file=back1_img)
+        create_test_image(source_img, backing_file=back2_img)
 
         self.vm = iotests.VM()
         # Add the BDS via blockdev-add so it stays around after the mirror block
@@ -71,10 +71,10 @@ class BaseClass(iotests.QMPTestCase):
 
         if self.existing:
             if self.target_backing:
-                qemu_img('create', '-f', iotests.imgfmt,
-                         '-b', self.target_backing, target_img, '1440K')
+                create_test_image(target_img, '1440K', \
+                                  backing_file=self.target_backing)
             else:
-                qemu_img('create', '-f', iotests.imgfmt, target_img, '1440K')
+                create_test_image(target_img, '1440K')
 
             if self.cmd == 'blockdev-mirror':
                 options = { 'node-name': 'target',
@@ -89,14 +89,11 @@ class BaseClass(iotests.QMPTestCase):
 
     def tearDown(self):
         self.vm.shutdown()
-        os.remove(source_img)
-        os.remove(back2_img)
-        os.remove(back1_img)
-        os.remove(back0_img)
-        try:
-            os.remove(target_img)
-        except OSError:
-            pass
+        remove_test_image(source_img)
+        remove_test_image(back2_img)
+        remove_test_image(back1_img)
+        remove_test_image(back0_img)
+        remove_test_image(target_img)
 
     def findBlockNode(self, node_name, qdev=None):
         if qdev:
-- 
2.21.0

[PATCH 35/67] iotests/163: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/163 | 24 +++++++++++-------------
 1 file changed, 11 insertions(+), 13 deletions(-)

diff --git a/tests/qemu-iotests/163 b/tests/qemu-iotests/163
index 92633f1b21..5c5e23f9fc 100755
--- a/tests/qemu-iotests/163
+++ b/tests/qemu-iotests/163
@@ -19,7 +19,8 @@
 #
 
 import os, random, iotests, struct, qcow2, sys
-from iotests import qemu_img, qemu_io, image_size
+from iotests import create_test_image, remove_test_image, qemu_img, qemu_io, \
+        image_size
 
 test_img = os.path.join(iotests.test_dir, 'test.img')
 check_img = os.path.join(iotests.test_dir, 'check.img')
@@ -82,22 +83,19 @@ class ShrinkBaseClass(iotests.QMPTestCase):
 
     def setUp(self):
         if iotests.imgfmt == 'raw':
-            qemu_img('create', '-f', iotests.imgfmt, test_img, self.image_len)
-            qemu_img('create', '-f', iotests.imgfmt, check_img,
-                     self.shrink_size)
+            create_test_image(test_img, self.image_len)
+            create_test_image(check_img, self.shrink_size)
         else:
-            qemu_img('create', '-f', iotests.imgfmt,
-                     '-o', 'cluster_size=' + self.cluster_size +
-                     ',refcount_bits=' + self.refcount_bits,
-                     test_img, self.image_len)
-            qemu_img('create', '-f', iotests.imgfmt,
-                     '-o', 'cluster_size=%s'% self.cluster_size,
-                     check_img, self.shrink_size)
+            create_test_image(test_img, self.image_len,
+                              opts=['cluster_size=' + self.cluster_size,
+                                    'refcount_bits=' + self.refcount_bits])
+            create_test_image(check_img, self.shrink_size,
+                              opts=['cluster_size=' + self.cluster_size])
         qemu_io('-c', 'write -P 0xff 0 ' + self.shrink_size, check_img)
 
     def tearDown(self):
-        os.remove(test_img)
-        os.remove(check_img)
+        remove_test_image(test_img)
+        remove_test_image(check_img)
 
     def image_verify(self):
         self.assertEqual(image_size(test_img), image_size(check_img),
-- 
2.21.0

[PATCH 36/67] iotests/165: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/165 | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/165 b/tests/qemu-iotests/165
index 97dd7102c3..0c92947f72 100755
--- a/tests/qemu-iotests/165
+++ b/tests/qemu-iotests/165
@@ -22,7 +22,7 @@ from __future__ import print_function
 import os
 import re
 import iotests
-from iotests import qemu_img
+from iotests import create_test_image, remove_test_image
 
 disk = os.path.join(iotests.test_dir, 'disk')
 disk_size = 0x40000000 # 1G
@@ -37,10 +37,10 @@ regions2 = ((0x10000000, 0x20000),
 class TestPersistentDirtyBitmap(iotests.QMPTestCase):
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, disk, str(disk_size))
+        create_test_image(disk, disk_size)
 
     def tearDown(self):
-        os.remove(disk)
+        remove_test_image(disk)
 
     def mkVm(self):
         return iotests.VM().add_drive(disk)
-- 
2.21.0

[PATCH 37/67] iotests/169: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/169 | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
index 5e978b7b7d..8d41463567 100755
--- a/tests/qemu-iotests/169
+++ b/tests/qemu-iotests/169
@@ -24,7 +24,7 @@ import time
 import itertools
 import operator
 import re
-from iotests import qemu_img
+from iotests import create_test_image, remove_test_image
 
 
 disk_a = os.path.join(iotests.test_dir, 'disk_a')
@@ -39,16 +39,16 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
     def tearDown(self):
         self.vm_a.shutdown()
         self.vm_b.shutdown()
-        os.remove(disk_a)
-        os.remove(disk_b)
+        remove_test_image(disk_a)
+        remove_test_image(disk_b)
         try:
             os.remove(mig_file)
         except OSError:
             pass
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, disk_a, size)
-        qemu_img('create', '-f', iotests.imgfmt, disk_b, size)
+        create_test_image(disk_a, size)
+        create_test_image(disk_b, size)
 
         self.vm_a = iotests.VM(path_suffix='a').add_drive(disk_a)
         self.vm_a.launch()
-- 
2.21.0

[PATCH 38/67] iotests/194: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/194 | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/194 b/tests/qemu-iotests/194
index c8aeb6d0e4..70cae2d8ef 100755
--- a/tests/qemu-iotests/194
+++ b/tests/qemu-iotests/194
@@ -20,19 +20,20 @@
 # Non-shared storage migration test using NBD server and drive-mirror
 
 import iotests
+from iotests import create_test_image
 
 iotests.script_initialize(supported_fmts=['qcow2', 'qed', 'raw'])
 
-with iotests.FilePath('source.img') as source_img_path, \
-     iotests.FilePath('dest.img') as dest_img_path, \
+with iotests.ImagePath('source.img') as source_img_path, \
+     iotests.ImagePath('dest.img') as dest_img_path, \
      iotests.FilePath('migration.sock') as migration_sock_path, \
      iotests.FilePath('nbd.sock') as nbd_sock_path, \
      iotests.VM('source') as source_vm, \
      iotests.VM('dest') as dest_vm:
 
     img_size = '1G'
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, source_img_path, img_size)
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, dest_img_path, img_size)
+    create_test_image(source_img_path, img_size)
+    create_test_image(dest_img_path, img_size)
 
     iotests.log('Launching VMs...')
     (source_vm.add_drive(source_img_path)
-- 
2.21.0

[PATCH 39/67] iotests/196: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/196 | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/196 b/tests/qemu-iotests/196
index ec4852a19c..4d6a5fac05 100755
--- a/tests/qemu-iotests/196
+++ b/tests/qemu-iotests/196
@@ -21,7 +21,7 @@
 
 import os
 import iotests
-from iotests import qemu_img
+from iotests import create_test_image, remove_test_image
 
 disk = os.path.join(iotests.test_dir, 'disk')
 migfile = os.path.join(iotests.test_dir, 'migfile')
@@ -31,11 +31,11 @@ class TestInvalidateAutoclear(iotests.QMPTestCase):
     def tearDown(self):
         self.vm_a.shutdown()
         self.vm_b.shutdown()
-        os.remove(disk)
+        remove_test_image(disk)
         os.remove(migfile)
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, disk, '1M')
+        create_test_image(disk, '1M')
 
         self.vm_a = iotests.VM(path_suffix='a').add_drive(disk)
         self.vm_a.launch()
-- 
2.21.0

[PATCH 40/67] iotests/199: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/199 | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/199 b/tests/qemu-iotests/199
index a2c8ecab5a..ec028e1149 100755
--- a/tests/qemu-iotests/199
+++ b/tests/qemu-iotests/199
@@ -21,7 +21,7 @@
 import os
 import iotests
 import time
-from iotests import qemu_img
+from iotests import create_test_image, remove_test_image
 
 disk_a = os.path.join(iotests.test_dir, 'disk_a')
 disk_b = os.path.join(iotests.test_dir, 'disk_b')
@@ -33,14 +33,14 @@ class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
     def tearDown(self):
         self.vm_a.shutdown()
         self.vm_b.shutdown()
-        os.remove(disk_a)
-        os.remove(disk_b)
+        remove_test_image(disk_a)
+        remove_test_image(disk_b)
         os.remove(fifo)
 
     def setUp(self):
         os.mkfifo(fifo)
-        qemu_img('create', '-f', iotests.imgfmt, disk_a, size)
-        qemu_img('create', '-f', iotests.imgfmt, disk_b, size)
+        create_test_image(disk_a, size)
+        create_test_image(disk_b, size)
         self.vm_a = iotests.VM(path_suffix='a').add_drive(disk_a)
         self.vm_b = iotests.VM(path_suffix='b').add_drive(disk_b)
         self.vm_b.add_incoming("exec: cat '" + fifo + "'")
-- 
2.21.0

[PATCH 41/67] iotests/202: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/202 | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/202 b/tests/qemu-iotests/202
index 1271ac9459..f2a77fbe9b 100755
--- a/tests/qemu-iotests/202
+++ b/tests/qemu-iotests/202
@@ -23,18 +23,19 @@
 # against regressions.
 
 import iotests
+from iotests import create_test_image
 
 iotests.script_initialize(supported_fmts=['qcow2'])
 
-with iotests.FilePath('disk0.img') as disk0_img_path, \
-     iotests.FilePath('disk1.img') as disk1_img_path, \
+with iotests.ImagePath('disk0.img') as disk0_img_path, \
+     iotests.ImagePath('disk1.img') as disk1_img_path, \
      iotests.FilePath('disk0-snap.img') as disk0_snap_img_path, \
      iotests.FilePath('disk1-snap.img') as disk1_snap_img_path, \
      iotests.VM() as vm:
 
     img_size = '10M'
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, disk0_img_path, img_size)
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, disk1_img_path, img_size)
+    create_test_image(disk0_img_path, img_size)
+    create_test_image(disk1_img_path, img_size)
 
     iotests.log('Launching VM...')
     vm.launch()
-- 
2.21.0

[PATCH 42/67] iotests/203: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/203 | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/203 b/tests/qemu-iotests/203
index c40fe231ea..ce4bc05e08 100755
--- a/tests/qemu-iotests/203
+++ b/tests/qemu-iotests/203
@@ -23,16 +23,17 @@
 # BDRV_POLL_WHILE().
 
 import iotests
+from iotests import create_test_image
 
 iotests.script_initialize(supported_fmts=['qcow2'])
 
-with iotests.FilePath('disk0.img') as disk0_img_path, \
-     iotests.FilePath('disk1.img') as disk1_img_path, \
+with iotests.ImagePath('disk0.img') as disk0_img_path, \
+     iotests.ImagePath('disk1.img') as disk1_img_path, \
      iotests.VM() as vm:
 
     img_size = '10M'
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, disk0_img_path, img_size)
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, disk1_img_path, img_size)
+    create_test_image(disk0_img_path, img_size)
+    create_test_image(disk1_img_path, img_size)
 
     iotests.log('Launching VM...')
     (vm.add_object('iothread,id=iothread0')
-- 
2.21.0

[PATCH 43/67] iotests/205: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/205 | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/205 b/tests/qemu-iotests/205
index 76f6c5fa2b..343fbc296e 100755
--- a/tests/qemu-iotests/205
+++ b/tests/qemu-iotests/205
@@ -22,7 +22,8 @@ import os
 import sys
 import iotests
 import time
-from iotests import qemu_img_create, qemu_io, filter_qemu_io, QemuIoInteractive
+from iotests import create_test_image, remove_test_image, \
+        qemu_io, filter_qemu_io, QemuIoInteractive
 
 nbd_sock = os.path.join(iotests.test_dir, 'nbd_sock')
 nbd_uri = 'nbd+unix:///exp?socket=' + nbd_sock
@@ -31,7 +32,7 @@ disk = os.path.join(iotests.test_dir, 'disk')
 
 class TestNbdServerRemove(iotests.QMPTestCase):
     def setUp(self):
-        qemu_img_create('-f', iotests.imgfmt, disk, '1M')
+        create_test_image(disk, '1M')
 
         self.vm = iotests.VM().add_drive(disk)
         self.vm.launch()
@@ -51,7 +52,7 @@ class TestNbdServerRemove(iotests.QMPTestCase):
     def tearDown(self):
         self.vm.shutdown()
         os.remove(nbd_sock)
-        os.remove(disk)
+        remove_test_image(disk)
 
     def remove_export(self, name, mode=None):
         if mode is None:
-- 
2.21.0

[PATCH 44/67] iotests/208: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/208 | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/208 b/tests/qemu-iotests/208
index dfce6f9fe4..cd565eea43 100755
--- a/tests/qemu-iotests/208
+++ b/tests/qemu-iotests/208
@@ -21,16 +21,17 @@
 # blockdev-snapshot-sync.
 
 import iotests
+from iotests import create_test_image
 
 iotests.script_initialize(supported_fmts=['generic'])
 
-with iotests.FilePath('disk.img') as disk_img_path, \
+with iotests.ImagePath('disk.img') as disk_img_path, \
      iotests.FilePath('disk-snapshot.img') as disk_snapshot_img_path, \
      iotests.FilePath('nbd.sock') as nbd_sock_path, \
      iotests.VM() as vm:
 
     img_size = '10M'
-    iotests.qemu_img_create('-f', iotests.imgfmt, disk_img_path, img_size)
+    create_test_image(disk_img_path, img_size)
 
     iotests.log('Launching VM...')
     (vm.add_drive(disk_img_path, 'node-name=drive0-node', interface='none')
-- 
2.21.0

[PATCH 45/67] iotests/208: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/209 | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/209 b/tests/qemu-iotests/209
index a77f884166..eb1215afd7 100755
--- a/tests/qemu-iotests/209
+++ b/tests/qemu-iotests/209
@@ -19,15 +19,16 @@
 #
 
 import iotests
-from iotests import qemu_img_create, qemu_io, qemu_img_verbose, qemu_nbd, \
-                    file_path
+from iotests import create_test_image, qemu_io, qemu_img_verbose, qemu_nbd, \
+                    file_path, image_path
 
 iotests.script_initialize(supported_fmts=['qcow2'])
 
-disk, nbd_sock = file_path('disk', 'nbd-sock')
+nbd_sock = file_path('nbd-sock')
+disk = image_path('disk')
 nbd_uri = 'nbd+unix:///exp?socket=' + nbd_sock
 
-qemu_img_create('-f', iotests.imgfmt, disk, '1M')
+create_test_image(disk, '1M')
 qemu_io('-f', iotests.imgfmt, '-c', 'write 0 512K', disk)
 
 qemu_nbd('-k', nbd_sock, '-x', 'exp', '-f', iotests.imgfmt, disk)
-- 
2.21.0

[PATCH 46/67] iotests/216: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/216 | 11 +++++------
 1 file changed, 5 insertions(+), 6 deletions(-)

diff --git a/tests/qemu-iotests/216 b/tests/qemu-iotests/216
index 7574bcc09f..18f2611da6 100755
--- a/tests/qemu-iotests/216
+++ b/tests/qemu-iotests/216
@@ -20,7 +20,7 @@
 # Creator/Owner: Max Reitz <mreitz@redhat.com>
 
 import iotests
-from iotests import log, qemu_img, qemu_io_silent
+from iotests import log, create_test_image, qemu_io_silent
 
 # Need backing file support
 iotests.script_initialize(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk'])
@@ -42,17 +42,16 @@ log('')
 # A COR filter node, however, can request the proper permissions for
 # its child and therefore is not hit by this issue.
 
-with iotests.FilePath('base.img') as base_img_path, \
-     iotests.FilePath('top.img') as top_img_path, \
+with iotests.ImagePath('base.img') as base_img_path, \
+     iotests.ImagePath('top.img') as top_img_path, \
      iotests.VM() as vm:
 
     log('--- Setting up images ---')
     log('')
 
-    assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
+    assert create_test_image(base_img_path, '64M') == 0
     assert qemu_io_silent(base_img_path, '-c', 'write -P 1 0M 1M') == 0
-    assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
-                    top_img_path) == 0
+    assert create_test_image(top_img_path, backing_file=base_img_path) == 0
     assert qemu_io_silent(top_img_path,  '-c', 'write -P 2 1M 1M') == 0
 
     log('Done')
-- 
2.21.0

[PATCH 47/67] iotests/218: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/218 | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/218 b/tests/qemu-iotests/218
index e18e31076b..6f6fe6588e 100755
--- a/tests/qemu-iotests/218
+++ b/tests/qemu-iotests/218
@@ -27,7 +27,7 @@
 # Creator/Owner: Max Reitz <mreitz@redhat.com>
 
 import iotests
-from iotests import log, qemu_img, qemu_io_silent
+from iotests import log, create_test_image, qemu_io_silent
 
 iotests.script_initialize(supported_fmts=['qcow2', 'raw'])
 
@@ -142,9 +142,9 @@ log('=== Cancel mirror job from throttled node by quitting ===')
 log('')
 
 with iotests.VM() as vm, \
-     iotests.FilePath('src.img') as src_img_path:
+     iotests.ImagePath('src.img') as src_img_path:
 
-    assert qemu_img('create', '-f', iotests.imgfmt, src_img_path, '64M') == 0
+    assert create_test_image(src_img_path, '64M') == 0
     assert qemu_io_silent('-f', iotests.imgfmt, src_img_path,
                           '-c', 'write -P 42 0M 64M') == 0
 
-- 
2.21.0

[PATCH 48/67] iotests/219: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/219 | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/219 b/tests/qemu-iotests/219
index 9ae27cb04e..37e3c70aef 100755
--- a/tests/qemu-iotests/219
+++ b/tests/qemu-iotests/219
@@ -20,6 +20,7 @@
 # Check using the job-* QMP commands with block jobs
 
 import iotests
+from iotests import create_test_image
 
 iotests.script_initialize(supported_fmts=['qcow2'])
 
@@ -177,11 +178,11 @@ def test_job_lifecycle(vm, job, job_args, has_ready=False):
     iotests.log(vm.qmp('query-jobs'))
 
 
-with iotests.FilePath('disk.img') as disk_path, \
+with iotests.ImagePath('disk.img') as disk_path, \
      iotests.FilePath('copy.img') as copy_path, \
      iotests.VM() as vm:
 
-    iotests.qemu_img_create('-f', iotests.imgfmt, disk_path, str(img_size))
+    create_test_image(disk_path, img_size)
     iotests.qemu_io('-c', 'write 0 %i' % (img_size),
                     '-f', iotests.imgfmt, disk_path)
 
-- 
2.21.0

[PATCH 49/67] iotests/222: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/222 | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/222 b/tests/qemu-iotests/222
index ffa88d4bb9..7ae76c1be1 100644
--- a/tests/qemu-iotests/222
+++ b/tests/qemu-iotests/222
@@ -22,7 +22,7 @@
 # Creator/Owner: John Snow <jsnow@redhat.com>
 
 import iotests
-from iotests import log, qemu_img, qemu_io, qemu_io_silent
+from iotests import log, create_test_image, qemu_io, qemu_io_silent
 
 iotests.script_initialize(supported_fmts=['qcow2', 'qcow', 'qed', 'vmdk',
                                           'vhdx', 'raw'],
@@ -46,16 +46,16 @@ remainder = [("0xd5", "0x108000",  "32k"), # Right-end of partial-left [1]
              ("0xdc", "32M",       "32k"), # Left-end of partial-right [2]
              ("0xcd", "0x3ff0000", "64k")] # patterns[3]
 
-with iotests.FilePath('base.img') as base_img_path, \
-     iotests.FilePath('fleece.img') as fleece_img_path, \
+with iotests.ImagePath('base.img') as base_img_path, \
+     iotests.ImagePath('fleece.img') as fleece_img_path, \
      iotests.FilePath('nbd.sock') as nbd_sock_path, \
      iotests.VM() as vm:
 
     log('--- Setting up images ---')
     log('')
 
-    assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
-    assert qemu_img('create', '-f', "qcow2", fleece_img_path, '64M') == 0
+    assert create_test_image(base_img_path, '64M') == 0
+    assert create_test_image(fleece_img_path, '64M', fmt='qcow2') == 0
 
     for p in patterns:
         qemu_io('-f', iotests.imgfmt,
-- 
2.21.0

[PATCH 50/67] iotests/224: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/224 | 20 +++++++++-----------
 1 file changed, 9 insertions(+), 11 deletions(-)

diff --git a/tests/qemu-iotests/224 b/tests/qemu-iotests/224
index e227d3ab3c..e9f5729c33 100755
--- a/tests/qemu-iotests/224
+++ b/tests/qemu-iotests/224
@@ -21,8 +21,9 @@
 # Creator/Owner: Max Reitz <mreitz@redhat.com>
 
 import iotests
-from iotests import log, qemu_img, qemu_io_silent, filter_qmp_testfiles, \
-                    filter_qmp_imgfmt, filter_json_filename
+from iotests import log, create_test_image, qemu_io_silent, \
+                    filter_qmp_testfiles, filter_qmp_imgfmt, \
+                    filter_json_filename
 import json
 import re
 
@@ -47,17 +48,14 @@ for filter_node_name in False, True:
     log('--- filter_node_name: %s ---' % filter_node_name)
     log('')
 
-    with iotests.FilePath('base.img') as base_img_path, \
-         iotests.FilePath('mid.img') as mid_img_path, \
-         iotests.FilePath('top.img') as top_img_path, \
+    with iotests.ImagePath('base.img') as base_img_path, \
+         iotests.ImagePath('mid.img') as mid_img_path, \
+         iotests.ImagePath('top.img') as top_img_path, \
          iotests.VM() as vm:
 
-        assert qemu_img('create', '-f', iotests.imgfmt,
-                        base_img_path, '64M') == 0
-        assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
-                        mid_img_path) == 0
-        assert qemu_img('create', '-f', iotests.imgfmt, '-b', mid_img_path,
-                        top_img_path) == 0
+        assert create_test_image(base_img_path, '64M') == 0
+        assert create_test_image(mid_img_path, backing_file=base_img_path) == 0
+        assert create_test_image(top_img_path, backing_file=mid_img_path) == 0
 
         # Something to commit
         assert qemu_io_silent(mid_img_path, '-c', 'write -P 1 0 1M') == 0
-- 
2.21.0

[PATCH 51/67] iotests/228: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/228 | 19 ++++++++++---------
 1 file changed, 10 insertions(+), 9 deletions(-)

diff --git a/tests/qemu-iotests/228 b/tests/qemu-iotests/228
index e507db4f27..ad7966fc8f 100755
--- a/tests/qemu-iotests/228
+++ b/tests/qemu-iotests/228
@@ -21,8 +21,9 @@
 # Creator/Owner: Max Reitz <mreitz@redhat.com>
 
 import iotests
-from iotests import log, qemu_img, filter_testfiles, filter_imgfmt, \
-        filter_qmp_testfiles, filter_qmp_imgfmt, filter_json_filename
+from iotests import log, create_test_image, remove_test_image, \
+        filter_testfiles, filter_imgfmt, filter_qmp_testfiles, \
+        filter_qmp_imgfmt, filter_json_filename
 
 # Need backing file and change-backing-file support
 iotests.script_initialize(supported_fmts=['qcow2', 'qed'])
@@ -46,14 +47,14 @@ def log_node_info(node):
     log('')
 
 
-with iotests.FilePath('base.img') as base_img_path, \
-     iotests.FilePath('top.img') as top_img_path, \
+with iotests.ImagePath('base.img') as base_img_path, \
+     iotests.ImagePath('top.img') as top_img_path, \
      iotests.VM() as vm:
 
-    assert qemu_img('create', '-f', iotests.imgfmt, base_img_path, '64M') == 0
+    assert create_test_image(base_img_path, '64M') == 0
     # Choose a funny way to describe the backing filename
-    assert qemu_img('create', '-f', iotests.imgfmt, '-b',
-                    'file:' + base_img_path, top_img_path) == 0
+    assert create_test_image(top_img_path,
+                             backing_file=('file:' + base_img_path)) == 0
 
     vm.launch()
 
@@ -167,8 +168,8 @@ with iotests.FilePath('base.img') as base_img_path, \
     # (because qemu cannot "canonicalize"/"resolve" the backing
     # filename unless the backing file is opened implicitly with the
     # overlay)
-    assert qemu_img('create', '-f', iotests.imgfmt, '-b', base_img_path,
-                    top_img_path) == 0
+    remove_test_image(top_img_path)
+    assert create_test_image(top_img_path, backing_file=base_img_path) == 0
 
     # You can only reliably override backing options by using a node
     # reference (or by specifying file.filename, but, well...)
-- 
2.21.0

[PATCH 52/67] iotests/234: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/234 | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/234 b/tests/qemu-iotests/234
index 3de6ab2341..f200af3f27 100755
--- a/tests/qemu-iotests/234
+++ b/tests/qemu-iotests/234
@@ -21,19 +21,20 @@
 # that is built with individually created nodes
 
 import iotests
+from iotests import create_test_image
 import os
 
 iotests.script_initialize(supported_fmts=['qcow2'])
 
-with iotests.FilePath('img') as img_path, \
-     iotests.FilePath('backing') as backing_path, \
+with iotests.ImagePath('img') as img_path, \
+     iotests.ImagePath('backing') as backing_path, \
      iotests.FilePath('mig_fifo_a') as fifo_a, \
      iotests.FilePath('mig_fifo_b') as fifo_b, \
      iotests.VM(path_suffix='a') as vm_a, \
      iotests.VM(path_suffix='b') as vm_b:
 
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, backing_path, '64M')
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, img_path, '64M')
+    create_test_image(backing_path, '64M')
+    create_test_image(img_path, '64M')
 
     os.mkfifo(fifo_a)
     os.mkfifo(fifo_b)
-- 
2.21.0

[PATCH 53/67] iotests/235: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/235 | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/235 b/tests/qemu-iotests/235
index 9e88c65b93..3326d205b7 100755
--- a/tests/qemu-iotests/235
+++ b/tests/qemu-iotests/235
@@ -21,7 +21,7 @@
 import sys
 import os
 import iotests
-from iotests import qemu_img_create, qemu_io, file_path, log
+from iotests import create_test_image, qemu_io, image_path, log
 
 sys.path.append(os.path.join(os.path.dirname(__file__), '..', '..', 'python'))
 
@@ -42,11 +42,10 @@ iotests.script_initialize(supported_fmts=['qcow2'])
 
 size = 1 * 1024 * 1024 * 1024
 
-disk = file_path('disk')
+disk = image_path('disk')
 
 # prepare source image
-qemu_img_create('-f', iotests.imgfmt, '-o', 'preallocation=metadata', disk,
-                str(size))
+create_test_image(disk, size, opts=['preallocation=metadata'])
 
 vm = QEMUMachine(iotests.qemu_prog)
 vm.add_args('-machine', 'accel=kvm:tcg')
-- 
2.21.0

[PATCH 54/67] iotests/236: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/236 | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/236 b/tests/qemu-iotests/236
index b88779eb0b..ef1d732510 100755
--- a/tests/qemu-iotests/236
+++ b/tests/qemu-iotests/236
@@ -20,7 +20,7 @@
 # owner=jsnow@redhat.com
 
 import iotests
-from iotests import log
+from iotests import log, create_test_image
 
 iotests.script_initialize(supported_fmts=['generic'])
 size = 64 * 1024 * 1024
@@ -41,11 +41,11 @@ def query_bitmaps(vm):
     return { "bitmaps": { device['device']: device.get('dirty-bitmaps', []) for
                           device in res['return'] } }
 
-with iotests.FilePath('img') as img_path, \
+with iotests.ImagePath('img') as img_path, \
      iotests.VM() as vm:
 
     log('--- Preparing image & VM ---\n')
-    iotests.qemu_img_create('-f', iotests.imgfmt, img_path, str(size))
+    create_test_image(img_path, size)
     vm.add_drive(img_path)
     vm.launch()
 
-- 
2.21.0

[PATCH 55/67] iotests/237: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/237     | 11 +++++------
 tests/qemu-iotests/237.out |  6 ------
 2 files changed, 5 insertions(+), 12 deletions(-)

diff --git a/tests/qemu-iotests/237 b/tests/qemu-iotests/237
index 34ff4b55db..179eae87dc 100755
--- a/tests/qemu-iotests/237
+++ b/tests/qemu-iotests/237
@@ -22,7 +22,7 @@
 
 import math
 import iotests
-from iotests import imgfmt
+from iotests import imgfmt, create_test_image
 
 iotests.script_initialize(supported_fmts=['vmdk'],
                           # All options are ignored
@@ -38,9 +38,9 @@ def blockdev_create(vm, options):
     iotests.log("")
 
 with iotests.FilePath('t.vmdk') as disk_path, \
-     iotests.FilePath('t.vmdk.1') as extent1_path, \
-     iotests.FilePath('t.vmdk.2') as extent2_path, \
-     iotests.FilePath('t.vmdk.3') as extent3_path, \
+     iotests.ImagePath('t.vmdk.1') as extent1_path, \
+     iotests.ImagePath('t.vmdk.2') as extent2_path, \
+     iotests.ImagePath('t.vmdk.3') as extent3_path, \
      iotests.VM() as vm:
 
     #
@@ -175,8 +175,7 @@ with iotests.FilePath('t.vmdk') as disk_path, \
     iotests.log("")
 
     for path in [ extent1_path, extent2_path, extent3_path ]:
-        msg = iotests.qemu_img_pipe('create', '-f', imgfmt, path, '0')
-        iotests.log(msg, [iotests.filter_testfiles])
+        assert create_test_image(path, 0) == 0
 
     vm.add_blockdev('driver=file,filename=%s,node-name=ext1' % (extent1_path))
     vm.add_blockdev('driver=file,filename=%s,node-name=ext2' % (extent2_path))
diff --git a/tests/qemu-iotests/237.out b/tests/qemu-iotests/237.out
index a8c800bfad..c32b970363 100644
--- a/tests/qemu-iotests/237.out
+++ b/tests/qemu-iotests/237.out
@@ -129,12 +129,6 @@ Job failed: Cannot find device=this doesn't exist nor node_name=this doesn't exi
 
 === Other subformats ===
 
-Formatting 'TEST_DIR/PID-t.vmdk.1', fmt=vmdk size=0 compat6=off hwversion=undefined
-
-Formatting 'TEST_DIR/PID-t.vmdk.2', fmt=vmdk size=0 compat6=off hwversion=undefined
-
-Formatting 'TEST_DIR/PID-t.vmdk.3', fmt=vmdk size=0 compat6=off hwversion=undefined
-
 == Missing extent ==
 
 {"execute": "blockdev-create", "arguments": {"job-id": "job0", "options": {"driver": "vmdk", "file": "node0", "size": 33554432, "subformat": "monolithicFlat"}}}
-- 
2.21.0

[PATCH 56/67] iotests/242: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/242 | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/242 b/tests/qemu-iotests/242
index 333fb8b56c..bccd8a9fdf 100755
--- a/tests/qemu-iotests/242
+++ b/tests/qemu-iotests/242
@@ -21,14 +21,14 @@
 import iotests
 import json
 import struct
-from iotests import qemu_img_create, qemu_io, qemu_img_pipe, \
-    file_path, img_info_log, log, filter_qemu_io
+from iotests import create_test_image, remove_test_image, qemu_io, \
+    qemu_img_pipe, image_path, img_info_log, log, filter_qemu_io
 
 iotests.script_initialize(supported_fmts=['qcow2'],
                           # compat=1.1 is needed for dirty bitmaps
                           unsupported_imgopts=['compat=0.10'])
 
-disk = file_path('disk')
+disk = image_path('disk')
 chunk = 256 * 1024
 bitmap_flag_unknown = 1 << 2
 # flag_offset = 5*cluster_size + flag_offset_in_bitmap_directory_entry
@@ -74,7 +74,7 @@ def toggle_flag(offset):
         f.write(struct.pack("B", toggled))
 
 
-qemu_img_create('-f', iotests.imgfmt, disk, '1M')
+create_test_image(disk, '1M')
 
 for num in range(1, 4):
     disabled = False
@@ -95,7 +95,8 @@ vm.shutdown()
 
 num += 1
 log('\nTest {}'.format(num))
-qemu_img_create('-f', iotests.imgfmt, disk, '1M')
+remove_test_image(disk)
+create_test_image(disk, '1M')
 add_bitmap(1, True, False)
 log('Write an unknown bitmap flag \'{}\' into a new QCOW2 image at offset {}'
     .format(hex(bitmap_flag_unknown), flag_offset))
-- 
2.21.0

[PATCH 57/67] iotests/245: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/245 | 17 +++++++++--------
 1 file changed, 9 insertions(+), 8 deletions(-)

diff --git a/tests/qemu-iotests/245 b/tests/qemu-iotests/245
index 72ab2a248d..af6bc40575 100644
--- a/tests/qemu-iotests/245
+++ b/tests/qemu-iotests/245
@@ -24,7 +24,7 @@ import re
 import iotests
 import copy
 import json
-from iotests import qemu_img, qemu_io
+from iotests import create_test_image, remove_test_image, qemu_io
 
 hd_path = [
     os.path.join(iotests.test_dir, 'hd0.img'),
@@ -43,9 +43,9 @@ class TestBlockdevReopen(iotests.QMPTestCase):
     total_io_cmds = 0
 
     def setUp(self):
-        qemu_img('create', '-f', iotests.imgfmt, hd_path[0], '3M')
-        qemu_img('create', '-f', iotests.imgfmt, '-b', hd_path[0], hd_path[1])
-        qemu_img('create', '-f', iotests.imgfmt, hd_path[2], '3M')
+        create_test_image(hd_path[0], '3M')
+        create_test_image(hd_path[1], backing_file=hd_path[0])
+        create_test_image(hd_path[2], '3M')
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xa0  0 1M', hd_path[0])
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xa1 1M 1M', hd_path[1])
         qemu_io('-f', iotests.imgfmt, '-c', 'write -P 0xa2 2M 1M', hd_path[2])
@@ -55,9 +55,9 @@ class TestBlockdevReopen(iotests.QMPTestCase):
     def tearDown(self):
         self.vm.shutdown()
         self.check_qemu_io_errors()
-        os.remove(hd_path[0])
-        os.remove(hd_path[1])
-        os.remove(hd_path[2])
+        remove_test_image(hd_path[0])
+        remove_test_image(hd_path[1])
+        remove_test_image(hd_path[2])
 
     # The output of qemu-io is not returned by vm.hmp_qemu_io() but
     # it's stored in the log and can only be read when the VM has been
@@ -328,7 +328,8 @@ class TestBlockdevReopen(iotests.QMPTestCase):
 
         # First we create a 2MB raw file, and fill each half with a
         # different value
-        qemu_img('create', '-f', 'raw', hd_path[0], '2M')
+        remove_test_image(hd_path[0])
+        create_test_image(hd_path[0], '2M', fmt='raw')
         qemu_io('-f', 'raw', '-c', 'write -P 0xa0  0 1M', hd_path[0])
         qemu_io('-f', 'raw', '-c', 'write -P 0xa1 1M 1M', hd_path[0])
 
-- 
2.21.0

[PATCH 58/67] iotests/246: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/246 | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/246 b/tests/qemu-iotests/246
index d1364d5901..2224c61545 100755
--- a/tests/qemu-iotests/246
+++ b/tests/qemu-iotests/246
@@ -20,7 +20,7 @@
 # owner=jsnow@redhat.com
 
 import iotests
-from iotests import log
+from iotests import log, create_test_image
 
 iotests.script_initialize(supported_fmts=['qcow2'],
                           # compat=1.1 is needed for dirty bitmaps
@@ -35,11 +35,11 @@ def query_bitmaps(vm):
     return { "bitmaps": { device['device']: device.get('dirty-bitmaps', []) for
                           device in res['return'] } }
 
-with iotests.FilePath('img') as img_path, \
+with iotests.ImagePath('img') as img_path, \
      iotests.VM() as vm:
 
     log('--- Preparing image & VM ---\n')
-    iotests.qemu_img_create('-f', iotests.imgfmt, img_path, str(size))
+    create_test_image(img_path, size)
     vm.add_drive(img_path)
 
 
-- 
2.21.0

[PATCH 59/67] iotests/248: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/248 | 8 ++++----
 1 file changed, 4 insertions(+), 4 deletions(-)

diff --git a/tests/qemu-iotests/248 b/tests/qemu-iotests/248
index 0adc4da802..90821eaeda 100755
--- a/tests/qemu-iotests/248
+++ b/tests/qemu-iotests/248
@@ -19,7 +19,7 @@
 #
 
 import iotests
-from iotests import qemu_img_create, qemu_io, file_path, filter_qmp_testfiles
+from iotests import create_test_image, qemu_io, image_path, filter_qmp_testfiles
 
 iotests.script_initialize(supported_fmts=['qcow2'],
                           # With an external data file, we would need
@@ -27,12 +27,12 @@ iotests.script_initialize(supported_fmts=['qcow2'],
                           # of @file
                           unsupported_imgopts=['data_file'])
 
-source, target = file_path('source', 'target')
+source, target = image_path('source', 'target')
 size = 5 * 1024 * 1024
 limit = 2 * 1024 * 1024
 
-qemu_img_create('-f', iotests.imgfmt, source, str(size))
-qemu_img_create('-f', iotests.imgfmt, target, str(size))
+create_test_image(source, size)
+create_test_image(target, size)
 qemu_io('-c', 'write 0 {}'.format(size), source)
 
 # raw format don't like empty files
-- 
2.21.0

[PATCH 60/67] iotests/254: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/254 | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/254 b/tests/qemu-iotests/254
index 0404faf853..b44771fbc6 100755
--- a/tests/qemu-iotests/254
+++ b/tests/qemu-iotests/254
@@ -19,16 +19,16 @@
 #
 
 import iotests
-from iotests import qemu_img_create, file_path, log
+from iotests import create_test_image, image_path, log
 
 iotests.script_initialize(supported_fmts=['qcow2'],
                           # compat=1.1 is needed for dirty bitmaps
                           unsupported_imgopts=['compat=0.10'])
 
-disk, top = file_path('disk', 'top')
+disk, top = image_path('disk', 'top')
 size = 1024 * 1024
 
-qemu_img_create('-f', iotests.imgfmt, disk, str(size))
+create_test_image(disk, size)
 
 vm = iotests.VM().add_drive(disk, opts='node-name=base')
 vm.launch()
-- 
2.21.0

[PATCH 61/67] iotests/255: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/255     | 20 +++++++++-----------
 tests/qemu-iotests/255.out |  8 --------
 2 files changed, 9 insertions(+), 19 deletions(-)

diff --git a/tests/qemu-iotests/255 b/tests/qemu-iotests/255
index ff16402268..4e745b68fa 100755
--- a/tests/qemu-iotests/255
+++ b/tests/qemu-iotests/255
@@ -21,7 +21,7 @@
 #
 
 import iotests
-from iotests import imgfmt
+from iotests import imgfmt, create_test_image
 
 iotests.script_initialize(supported_fmts=['qcow2'])
 
@@ -39,8 +39,8 @@ iotests.log('Finishing a commit job with background reads')
 iotests.log('============================================')
 iotests.log('')
 
-with iotests.FilePath('t.qcow2') as disk_path, \
-     iotests.FilePath('t.qcow2.mid') as mid_path, \
+with iotests.ImagePath('t.qcow2') as disk_path, \
+     iotests.ImagePath('t.qcow2.mid') as mid_path, \
      iotests.FilePath('t.qcow2.base') as base_path, \
      iotests.VM() as vm:
 
@@ -48,11 +48,10 @@ with iotests.FilePath('t.qcow2') as disk_path, \
     iotests.log("")
 
     size = 128 * 1024 * 1024
-    size_str = str(size)
 
     iotests.create_image(base_path, size)
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, mid_path, size_str)
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, disk_path, size_str)
+    assert create_test_image(mid_path, size) == 0
+    assert create_test_image(disk_path, size) == 0
 
     # Create a backing chain like this:
     # base <- [throttled: bps-read=4096] <- mid <- overlay
@@ -91,18 +90,17 @@ iotests.log('Closing the VM while a job is being cancelled')
 iotests.log('=============================================')
 iotests.log('')
 
-with iotests.FilePath('src.qcow2') as src_path, \
-     iotests.FilePath('dst.qcow2') as dst_path, \
+with iotests.ImagePath('src.qcow2') as src_path, \
+     iotests.ImagePath('dst.qcow2') as dst_path, \
      iotests.VM() as vm:
 
     iotests.log('=== Create images and start VM ===')
     iotests.log('')
 
     size = 128 * 1024 * 1024
-    size_str = str(size)
 
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, src_path, size_str)
-    iotests.qemu_img_log('create', '-f', iotests.imgfmt, dst_path, size_str)
+    assert create_test_image(src_path, size) == 0
+    assert create_test_image(dst_path, size) == 0
 
     iotests.log(iotests.qemu_io('-f', iotests.imgfmt, '-c', 'write 0 1M',
                                 src_path),
diff --git a/tests/qemu-iotests/255.out b/tests/qemu-iotests/255.out
index 348909fdef..2e837cbb5f 100644
--- a/tests/qemu-iotests/255.out
+++ b/tests/qemu-iotests/255.out
@@ -3,10 +3,6 @@ Finishing a commit job with background reads
 
 === Create backing chain and start VM ===
 
-Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
-
-Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
-
 === Start background read requests ===
 
 === Run a commit job ===
@@ -23,10 +19,6 @@ Closing the VM while a job is being cancelled
 
 === Create images and start VM ===
 
-Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
-
-Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
-
 wrote 1048576/1048576 bytes at offset 0
 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 
-- 
2.21.0

[PATCH 62/67] iotests/256: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/256 | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/tests/qemu-iotests/256 b/tests/qemu-iotests/256
index d2f9212e5a..7c78a123ad 100755
--- a/tests/qemu-iotests/256
+++ b/tests/qemu-iotests/256
@@ -21,13 +21,13 @@
 
 import os
 import iotests
-from iotests import log
+from iotests import log, create_test_image
 
 iotests.script_initialize(supported_fmts=['qcow2'])
 size = 64 * 1024 * 1024
 
-with iotests.FilePath('img0') as img0_path, \
-     iotests.FilePath('img1') as img1_path, \
+with iotests.ImagePath('img0') as img0_path, \
+     iotests.ImagePath('img1') as img1_path, \
      iotests.FilePath('img0-full') as img0_full_path, \
      iotests.FilePath('img1-full') as img1_full_path, \
      iotests.FilePath('img0-incr') as img0_incr_path, \
@@ -62,8 +62,8 @@ with iotests.FilePath('img0') as img0_path, \
     vm.add_object('iothread,id=iothread1')
     vm.add_device('virtio-scsi-pci,id=scsi0,iothread=iothread0')
     vm.add_device('virtio-scsi-pci,id=scsi1,iothread=iothread1')
-    iotests.qemu_img_create('-f', iotests.imgfmt, img0_path, str(size))
-    iotests.qemu_img_create('-f', iotests.imgfmt, img1_path, str(size))
+    create_test_image(img0_path, size)
+    create_test_image(img1_path, size)
     vm.add_drive(img0_path, interface='none')
     vm.add_device('scsi-hd,id=device0,drive=drive0,bus=scsi0.0')
     vm.add_drive(img1_path, interface='none')
-- 
2.21.0

[PATCH 63/67] iotests/257: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/257 | 14 +++++++-------
 1 file changed, 7 insertions(+), 7 deletions(-)

diff --git a/tests/qemu-iotests/257 b/tests/qemu-iotests/257
index 82c10e9b52..ea38945a71 100755
--- a/tests/qemu-iotests/257
+++ b/tests/qemu-iotests/257
@@ -23,7 +23,7 @@ import math
 import os
 
 import iotests
-from iotests import log, qemu_img
+from iotests import log, create_test_image, qemu_img
 
 SIZE = 64 * 1024 * 1024
 GRANULARITY = 64 * 1024
@@ -152,7 +152,7 @@ class Drive:
     def img_create(self, fmt, size):
         self.fmt = fmt
         self.size = size
-        iotests.qemu_img_create('-f', self.fmt, self.path, str(self.size))
+        create_test_image(self.path, self.size)
 
     def create_target(self, name, fmt, size):
         basename = os.path.basename(self.path)
@@ -273,10 +273,10 @@ def test_bitmap_sync(bsync_mode, msync_mode='bitmap', failure=None):
                         an incomplete backup. Testing limitations prevent
                         testing competing writes.
     """
-    with iotests.FilePaths(['img', 'bsync1', 'bsync2',
+    with iotests.ImagePath('img') as img_path, \
+        iotests.FilePaths(['bsync1', 'bsync2',
                             'fbackup0', 'fbackup1', 'fbackup2']) as \
-                            (img_path, bsync1, bsync2,
-                             fbackup0, fbackup1, fbackup2), \
+                            (bsync1, bsync2, fbackup0, fbackup1, fbackup2), \
          iotests.VM() as vm:
 
         mode = "Mode {:s}; Bitmap Sync {:s}".format(msync_mode, bsync_mode)
@@ -439,8 +439,8 @@ def test_backup_api():
     """
     Test malformed and prohibited invocations of the backup API.
     """
-    with iotests.FilePaths(['img', 'bsync1']) as \
-         (img_path, backup_path), \
+    with iotests.ImagePath('img') as img_path, \
+         iotests.FilePath('bsync1') as backup_path, \
          iotests.VM() as vm:
 
         log("\n=== API failure tests ===\n")
-- 
2.21.0

[PATCH 64/67] iotests/258: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/258 | 16 ++++++++--------
 1 file changed, 8 insertions(+), 8 deletions(-)

diff --git a/tests/qemu-iotests/258 b/tests/qemu-iotests/258
index 1372522c7a..3e9505d8d4 100755
--- a/tests/qemu-iotests/258
+++ b/tests/qemu-iotests/258
@@ -20,7 +20,7 @@
 # Creator/Owner: Max Reitz <mreitz@redhat.com>
 
 import iotests
-from iotests import log, qemu_img, qemu_io_silent, \
+from iotests import log, create_test_image, qemu_io_silent, \
         filter_qmp_testfiles, filter_qmp_imgfmt
 
 # Returns a node for blockdev-add
@@ -65,22 +65,22 @@ def test_concurrent_finish(write_to_stream_node):
     # and the other way around, depending on whether the commit job
     # is finalized before stream completes or not.
 
-    with iotests.FilePath('node4.img') as node4_path, \
-         iotests.FilePath('node3.img') as node3_path, \
-         iotests.FilePath('node2.img') as node2_path, \
-         iotests.FilePath('node1.img') as node1_path, \
-         iotests.FilePath('node0.img') as node0_path, \
+    with iotests.ImagePath('node4.img') as node4_path, \
+         iotests.ImagePath('node3.img') as node3_path, \
+         iotests.ImagePath('node2.img') as node2_path, \
+         iotests.ImagePath('node1.img') as node1_path, \
+         iotests.ImagePath('node0.img') as node0_path, \
          iotests.VM() as vm:
 
         # It is important to use raw for the base layer (so that
         # permissions are just handed through to the protocol layer)
-        assert qemu_img('create', '-f', 'raw', node0_path, '64M') == 0
+        assert create_test_image(node0_path, '64M', fmt='raw') == 0
 
         stream_throttle=None
         commit_throttle=None
 
         for path in [node1_path, node2_path, node3_path, node4_path]:
-            assert qemu_img('create', '-f', iotests.imgfmt, path, '64M') == 0
+            assert create_test_image(path, '64M') == 0
 
         if write_to_stream_node:
             # This is what (most of the time) makes commit finish
-- 
2.21.0

[PATCH 65/67] iotests/262: Honor $IMGOPTS

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/262 | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/262 b/tests/qemu-iotests/262
index 6879665f99..b436fb3e14 100755
--- a/tests/qemu-iotests/262
+++ b/tests/qemu-iotests/262
@@ -21,11 +21,12 @@
 # iothread just for fun.
 
 import iotests
+from iotests import create_test_image
 import os
 
 iotests.script_initialize(supported_fmts=['qcow2'])
 
-with iotests.FilePath('img') as img_path, \
+with iotests.ImagePath('img') as img_path, \
      iotests.FilePath('mig_fifo') as fifo, \
      iotests.VM(path_suffix='a') as vm_a, \
      iotests.VM(path_suffix='b') as vm_b:
@@ -49,7 +50,7 @@ with iotests.FilePath('img') as img_path, \
 
         vm.add_device('virtio-blk,drive=%s,iothread=iothread0' % root)
 
-    iotests.qemu_img_pipe('create', '-f', iotests.imgfmt, img_path, '64M')
+    create_test_image(img_path, '64M')
 
     os.mkfifo(fifo)
 
-- 
2.21.0

[PATCH 66/67] iotests.py: Forbid qemu_img*('create', ...)

[Max Reitz]

All tests should use create_test_image().

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 36 +++++++++++++++++++++++++++--------
 1 file changed, 28 insertions(+), 8 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index f3c80ba4e9..2e377b8379 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -82,12 +82,20 @@ luks_default_secret_object = 'secret,id=keysec0,data=' + \
 luks_default_key_secret_opt = 'key-secret=keysec0'
 
 
-def qemu_img(*args):
+# @from_create_test_image is purely for use in create_test_image
+def qemu_img(*args, from_create_test_image=False):
     '''Run qemu-img and return the exit code'''
+    args = list(args)
+
+    assert args[0] != 'create' or from_create_test_image, \
+        'Use iotests.create_test_image for image creation'
+
     devnull = open('/dev/null', 'r+')
-    exitcode = subprocess.call(qemu_img_args + list(args), stdin=devnull, stdout=devnull)
+    exitcode = subprocess.call(qemu_img_args + args,
+                               stdin=devnull, stdout=devnull)
     if exitcode < 0:
-        sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
+        sys.stderr.write('qemu-img received signal %i: %s\n' %
+                         (-exitcode, ' '.join(qemu_img_args + args)))
     return exitcode
 
 def ordered_qmp(qmsg, conv_keys=True):
@@ -162,7 +170,7 @@ def create_test_image(filename, size=None, fmt=imgfmt, opts=[],
     if size is not None:
         args.append(str(size))
 
-    return qemu_img(*args)
+    return qemu_img(*args, from_create_test_image=True)
 
 # Use this to remove images create with create_test_image in the
 # default image format (iotests.imgfmt)
@@ -180,20 +188,32 @@ def remove_test_image(filename):
 
 def qemu_img_verbose(*args):
     '''Run qemu-img without suppressing its output and return the exit code'''
-    exitcode = subprocess.call(qemu_img_args + list(args))
+    args = list(args)
+
+    assert args[0] != 'create', \
+            'Use iotests.create_test_image for image creation'
+
+    exitcode = subprocess.call(qemu_img_args + args)
     if exitcode < 0:
-        sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
+        sys.stderr.write('qemu-img received signal %i: %s\n' %
+                         (-exitcode, ' '.join(qemu_img_args + args)))
     return exitcode
 
 def qemu_img_pipe(*args):
     '''Run qemu-img and return its output'''
-    subp = subprocess.Popen(qemu_img_args + list(args),
+    args = list(args)
+
+    assert args[0] != 'create', \
+            'Use iotests.create_test_image for image creation'
+
+    subp = subprocess.Popen(qemu_img_args + args,
                             stdout=subprocess.PIPE,
                             stderr=subprocess.STDOUT,
                             universal_newlines=True)
     exitcode = subp.wait()
     if exitcode < 0:
-        sys.stderr.write('qemu-img received signal %i: %s\n' % (-exitcode, ' '.join(qemu_img_args + list(args))))
+        sys.stderr.write('qemu-img received signal %i: %s\n' %
+                         (-exitcode, ' '.join(qemu_img_args + args)))
     return subp.communicate()[0]
 
 def qemu_img_log(*args):
-- 
2.21.0

[PATCH 67/67] iotests.py: Drop qemu_img_log(), qemu_img_create()

[Max Reitz]

They are both unused now.

Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/iotests.py | 24 ------------------------
 1 file changed, 24 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 2e377b8379..2476f3744a 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -111,25 +111,6 @@ def ordered_qmp(qmsg, conv_keys=True):
         return od
     return qmsg
 
-def qemu_img_create(*args):
-    args = list(args)
-
-    # default luks support
-    if '-f' in args and args[args.index('-f') + 1] == 'luks':
-        if '-o' in args:
-            i = args.index('-o')
-            if 'key-secret' not in args[i + 1]:
-                args[i + 1].append(luks_default_key_secret_opt)
-                args.insert(i + 2, '--object')
-                args.insert(i + 3, luks_default_secret_object)
-        else:
-            args = ['-o', luks_default_key_secret_opt,
-                    '--object', luks_default_secret_object] + args
-
-    args.insert(0, 'create')
-
-    return qemu_img(*args)
-
 def create_test_image(filename, size=None, fmt=imgfmt, opts=[],
                       backing_file=None, backing_fmt=None,
                       objects=[], unsafe=False):
@@ -216,11 +197,6 @@ def qemu_img_pipe(*args):
                          (-exitcode, ' '.join(qemu_img_args + args)))
     return subp.communicate()[0]
 
-def qemu_img_log(*args):
-    result = qemu_img_pipe(*args)
-    log(result, filters=[filter_testfiles])
-    return result
-
 def img_info_log(filename, filter_path=None, imgopts=False, extra_args=[],
                  hide_fields=[]):
     args = [ 'info' ]
-- 
2.21.0

Re: [PATCH 00/67] iotests: Honor $IMGOPTS in Python tests

[John Snow]

On 10/1/19 3:46 PM, Max Reitz wrote:
> First of all: Sorry.
> 

Thank you for finding the time to do it.

> 
> Second:
> 
> Based-on: My block branch
>           (https://github.com/XanClic/qemu.git block)
> 
> Based-on: 20190917234549.22910-1-jsnow@redhat.com
>           (“iotests: use python logging”)
> 
> Based-on: 20190927094242.11152-1-mreitz@redhat.com
>           (“iotests: Allow ./check -o data_file”)
> 
> Based-on: 20190917092004.999-1-mreitz@redhat.com
>           (“iotests: Selfish patches”)
> 
> Based-on: 20191001174827.11081-1-mreitz@redhat.com
>           (“block: Skip COR for inactive nodes”)
> 
> 
> OK, now:
> 
> Hi,
> 
> My recent series “iotests: Allow ./check -o data_file” enabled our bash
> tests to interpret the data_file qcow2 option.  It didn’t do anything
> for Python tests because those currently completely ignore all image
> format options.
> 
> This is where it gets hairy.  To do so, we need two things: First of
> all, whatever way Python tests use to create images needs to interpret
> $IMGOPTS.  Second, when deleting image files, they must not use a plain
> os.remove(), but a special function that will clean up data files, too.
> 
> The heap of patches in this series comes from making the Python tests
> use these new functions.
> 
> Most Python tests just run qemu-img through a helper function that does
> not care about the exact subcommand to create images.  I could add
> $IMGOPTS support to it, but that doesn’t feel quite right to me, and it
> wouldn’t reduce the patch count because we still need a special removal
> function.
> 
> 
> This series is structured as follows:
> - Patches 1 through 7 add support to handle image files differently from
>   other files (consider $IMGOPTS when creating them, consider data files
>   when deleting them, separate ImagePaths from FilePaths, and so on)

OK, that makes sense. I suppose we've been playing a bit fast and loose
with these such things.

> 
> - Patches 8 and 9 add two filters we’ll need in the next range:
> 
> - Patches 10 through 13 address some issues in a handful of tests that
>   just need to be changed a little so they can overall work with some
>   format options
> 
> - Patch 14 makes all tests pass unsupported_imgopts where there are
>   options that they cannot support.
> 
> - Patches 15 through 65 make all Python tests use the new functions
>   introduced in the first 7 patches so they no longer ignore $IMGOPTS.
> 
>   I felt like this is much better than munching everything together into
>   a single big commit (better to rebase, better to review), and I don’t
>   really like ideas like “Just do five patches that each address ten
>   iotests”.
> 

This is the right approach, for the exact reasons you specify.

>   But I’m still very much open to suggestions on how to combine these
>   many small patches to reduce the overall patch count.
> 

You could group them by release windows, if you really wanted to;

- [...etc...]
- Update iotests added for 3.2
- Update iotests added for 4.0
- Update iotests added for 4.1
- Individual patches thereafter

But maybe that doesn't really solve anything for anyone. If you didn't
find a more obvious grouping for these, I'd just leave it alone. I'll
get to reviewing them.

> - Patch 66 ensures that Python tests always use the new function to
>   create test images so they won’t bypass $IMGOPTS.
> 
> - Patch 67 cleans up.  qemu_img_log() is only used for image creation,
>   and I don’t see the point in that.  The output is predictable and it
>   is very unlikely to fail.  We can see in the bash tests that regularly
>   we basically just filter everything from it anyway.
>   (So this series replaces log(qemu_img_pipe()) instances by asserting
>   that image creation did not fail.)
>   ((qemu_img_create() obviously no longer has any use after this
>   series.))
> 
> 
> After this series, running the iotests with -o compat=0.10,
> -o refcount_bits=1, and -o 'data_file=$TEST_IMG.data_file' does
> something sensible even for the Python tests, and it passes.
> 

No minor accomplishment.

I'll make sure to review at least 1-14, but not before Friday.

> 
> Max Reitz (67):
>   iotests.py: Read $IMGOPTS
>   iotests.py: Add @skip_for_imgopts()
>   iotests.py: Add unsupported_imgopts
>   iotests.py: create_test_image, remove_test_image
>   iotests.py: Add ImagePaths
>   iotests.py: Add image_path()
>   iotests.py: Filter data_file in filter_img_info
>   iotests.py: Add filter_json_filename()
>   iotests.py: Add @hide_fields to img_info_log
>   iotests/169: Skip persistent cases for compat=0.10
>   iotests/224: Filter json:{} from commit command
>   iotests/228: Filter json:{} filenames
>   iotests/242: Hide refcount bit information
>   iotests: Use unsupported_imgopts in Python tests
>   iotests/030: Honor $IMGOPTS
>   iotests/040: Honor $IMGOPTS
>   iotests/041: Honor $IMGOPTS
>   iotests/044: Honor $IMGOPTS
>   iotests/045: Honor $IMGOPTS
>   iotests/055: Honor $IMGOPTS
>   iotests/056: Honor $IMGOPTS
>   iotests/057: Honor $IMGOPTS
>   iotests/065: Honor $IMGOPTS
>   iotests/096: Honor $IMGOPTS
>   iotests/118: Honor $IMGOPTS
>   iotests/124: Honor $IMGOPTS
>   iotests/129: Honor $IMGOPTS
>   iotests/132: Honor $IMGOPTS
>   iotests/139: Honor $IMGOPTS
>   iotests/147: Honor $IMGOPTS
>   iotests/148: Honor $IMGOPTS
>   iotests/151: Honor $IMGOPTS
>   iotests/152: Honor $IMGOPTS
>   iotests/155: Honor $IMGOPTS
>   iotests/163: Honor $IMGOPTS
>   iotests/165: Honor $IMGOPTS
>   iotests/169: Honor $IMGOPTS
>   iotests/194: Honor $IMGOPTS
>   iotests/196: Honor $IMGOPTS
>   iotests/199: Honor $IMGOPTS
>   iotests/202: Honor $IMGOPTS
>   iotests/203: Honor $IMGOPTS
>   iotests/205: Honor $IMGOPTS
>   iotests/208: Honor $IMGOPTS
>   iotests/208: Honor $IMGOPTS
>   iotests/216: Honor $IMGOPTS
>   iotests/218: Honor $IMGOPTS
>   iotests/219: Honor $IMGOPTS
>   iotests/222: Honor $IMGOPTS
>   iotests/224: Honor $IMGOPTS
>   iotests/228: Honor $IMGOPTS
>   iotests/234: Honor $IMGOPTS
>   iotests/235: Honor $IMGOPTS
>   iotests/236: Honor $IMGOPTS
>   iotests/237: Honor $IMGOPTS
>   iotests/242: Honor $IMGOPTS
>   iotests/245: Honor $IMGOPTS
>   iotests/246: Honor $IMGOPTS
>   iotests/248: Honor $IMGOPTS
>   iotests/254: Honor $IMGOPTS
>   iotests/255: Honor $IMGOPTS
>   iotests/256: Honor $IMGOPTS
>   iotests/257: Honor $IMGOPTS
>   iotests/258: Honor $IMGOPTS
>   iotests/262: Honor $IMGOPTS
>   iotests.py: Forbid qemu_img*('create', ...)
>   iotests.py: Drop qemu_img_log(), qemu_img_create()
> 
>  tests/qemu-iotests/030        |  69 ++++++------
>  tests/qemu-iotests/040        |  42 ++++----
>  tests/qemu-iotests/041        | 108 +++++++++----------
>  tests/qemu-iotests/044        |  11 +-
>  tests/qemu-iotests/045        |  26 ++---
>  tests/qemu-iotests/055        |  41 +++----
>  tests/qemu-iotests/056        |  30 +++---
>  tests/qemu-iotests/057        |  10 +-
>  tests/qemu-iotests/065        |  21 ++--
>  tests/qemu-iotests/096        |   5 +-
>  tests/qemu-iotests/118        |  26 ++---
>  tests/qemu-iotests/124        |  29 +++--
>  tests/qemu-iotests/129        |  11 +-
>  tests/qemu-iotests/132        |   6 +-
>  tests/qemu-iotests/139        |  15 ++-
>  tests/qemu-iotests/147        |  11 +-
>  tests/qemu-iotests/148        |   5 +-
>  tests/qemu-iotests/151        |  10 +-
>  tests/qemu-iotests/152        |   6 +-
>  tests/qemu-iotests/155        |  29 +++--
>  tests/qemu-iotests/163        |  29 ++---
>  tests/qemu-iotests/165        |  10 +-
>  tests/qemu-iotests/169        |  23 ++--
>  tests/qemu-iotests/194        |   9 +-
>  tests/qemu-iotests/196        |  10 +-
>  tests/qemu-iotests/199        |  10 +-
>  tests/qemu-iotests/202        |   9 +-
>  tests/qemu-iotests/203        |   9 +-
>  tests/qemu-iotests/205        |   7 +-
>  tests/qemu-iotests/206        |   5 +-
>  tests/qemu-iotests/208        |   5 +-
>  tests/qemu-iotests/209        |   9 +-
>  tests/qemu-iotests/216        |  11 +-
>  tests/qemu-iotests/218        |   6 +-
>  tests/qemu-iotests/219        |   5 +-
>  tests/qemu-iotests/222        |  13 +--
>  tests/qemu-iotests/224        |  33 +++---
>  tests/qemu-iotests/224.out    |   4 +-
>  tests/qemu-iotests/228        |  25 ++---
>  tests/qemu-iotests/228.out    |   8 +-
>  tests/qemu-iotests/234        |   9 +-
>  tests/qemu-iotests/235        |   7 +-
>  tests/qemu-iotests/236        |   6 +-
>  tests/qemu-iotests/237        |  15 +--
>  tests/qemu-iotests/237.out    |   6 --
>  tests/qemu-iotests/242        |  21 ++--
>  tests/qemu-iotests/242.out    |   5 -
>  tests/qemu-iotests/245        |  21 ++--
>  tests/qemu-iotests/246        |  11 +-
>  tests/qemu-iotests/248        |  14 ++-
>  tests/qemu-iotests/254        |  10 +-
>  tests/qemu-iotests/255        |  20 ++--
>  tests/qemu-iotests/255.out    |   8 --
>  tests/qemu-iotests/256        |  10 +-
>  tests/qemu-iotests/257        |  18 ++--
>  tests/qemu-iotests/258        |  16 +--
>  tests/qemu-iotests/262        |   5 +-
>  tests/qemu-iotests/iotests.py | 197 +++++++++++++++++++++++++++-------
>  58 files changed, 654 insertions(+), 496 deletions(-)
> 

Re: [PATCH 01/67] iotests.py: Read $IMGOPTS

[John Snow]

On 10/1/19 3:46 PM, Max Reitz wrote:
> We do not do anything with yet, but this is the first step.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/iotests.py | 6 ++++++
>  1 file changed, 6 insertions(+)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 1c5fce3e9e..7030900807 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -69,6 +69,12 @@ output_dir = os.environ.get('OUTPUT_DIR', '.')
>  cachemode = os.environ.get('CACHEMODE')
>  qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
>  
> +imgopts = os.environ.get('IMGOPTS', '')
> +if len(imgopts) == 0:
> +    imgopts = []
> +else:
> +    imgopts = imgopts.split(',')
> +

Sometimes I think about the type of person I'd like to meet, and I ask
myself if it's the type of person who would quote pep8, or say things
like "idiomatic python" in a code review when the existing form has no
technical problem whatsoever.

I wonder what type of person I am.

Well, questions for another day.

>  socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
>  
>  luks_default_secret_object = 'secret,id=keysec0,data=' + \
> 

Reviewed-by: John Snow <jsnow@redhat.com>

Re: [PATCH 02/67] iotests.py: Add @skip_for_imgopts()

[John Snow]

On 10/1/19 3:46 PM, Max Reitz wrote:
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/iotests.py | 13 +++++++++++++
>  1 file changed, 13 insertions(+)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 7030900807..cdcb62c4ac 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -950,6 +950,19 @@ def skip_if_unsupported(required_formats=[], read_only=False):
>          return func_wrapper
>      return skip_test_decorator
>  
> +def skip_for_imgopts(unsupported_opts=[]):
> +    '''Skip Test Decorator
> +       Skips the test if imgopts contains any of the given options'''
> +    def skip_test_decorator(func):
> +        def func_wrapper(test_case: QMPTestCase, *args, **kwargs):

:D

> +            for opt in imgopts:
> +                if any(unsupported in opt for unsupported in unsupported_opts):
> +                    test_case.case_skip('{}: Option {} is unsupported'.format(
> +                                        test_case, opt))
> +            return func(test_case, *args, **kwargs)
> +        return func_wrapper
> +    return skip_test_decorator
> +
>  def execute_unittest(debug=False):
>      """Executes unittests within the calling module."""
>  
> 

Reviewed-by: John Snow <jsnow@redhat.com>

Re: [PATCH 03/67] iotests.py: Add unsupported_imgopts

[John Snow]

On 10/1/19 3:46 PM, Max Reitz wrote:
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/iotests.py | 7 ++++++-
>  1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index cdcb62c4ac..b5ea424de4 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -998,7 +998,8 @@ def execute_setup_common(supported_fmts=[],
>                           supported_cache_modes=[],
>                           unsupported_fmts=[],
>                           supported_protocols=[],
> -                         unsupported_protocols=[]):
> +                         unsupported_protocols=[],
> +                         unsupported_imgopts=[]):
>      """
>      Perform necessary setup for either script-style or unittest-style tests.
>      """
> @@ -1016,6 +1017,10 @@ def execute_setup_common(supported_fmts=[],
>      verify_platform(supported=supported_platforms)
>      verify_cache_mode(supported_cache_modes)
>  
> +    for opt in imgopts:
> +        if any(unsupported in opt for unsupported in unsupported_imgopts):

any() is cool and I don't use it as often as I should.

> +            notrun('not suitable for this option: %s' % opt)
> +
>      debug = '-d' in sys.argv
>      if debug:
>          sys.argv.remove('-d')
> 

Reviewed-by: John Snow <jsnow@redhat.com>

Re: [PATCH 04/67] iotests.py: create_test_image, remove_test_image

[John Snow]

On 10/1/19 3:46 PM, Max Reitz wrote:
> Python tests should use these two new functions instead of
> qemu_img('create', ...) + os.remove(), so that user-supplied image
> options are interpreted and handled correctly.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>  tests/qemu-iotests/iotests.py | 56 +++++++++++++++++++++++++++++++++++
>  1 file changed, 56 insertions(+)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index b5ea424de4..fce1ab04c9 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -122,6 +122,62 @@ def qemu_img_create(*args):
>  
>      return qemu_img(*args)
>  
> +def create_test_image(filename, size=None, fmt=imgfmt, opts=[],
> +                      backing_file=None, backing_fmt=None,
> +                      objects=[], unsafe=False):

Python! It's the language that everybody loves and can do no wrong!

Ah, wait, no, maybe the opposite.

You want this:

(..., opts=None, ...):
    opts = opts or []

because, unfortunately, default parameters are bound at definition time
and not at call time, so the default list here is like a static local.

> +    if fmt == imgfmt:
> +        # Only use imgopts for the default format
> +        opts = imgopts + opts
> +
> +    for i, opt in enumerate(opts):
> +        if '$TEST_IMG' in opt:
> +            opts[i] = opt.replace('$TEST_IMG', filename)
> +
> +    # default luks support
> +    if fmt == 'luks':
> +        if not any('key-secret' in opt for opt in opts):

You can write "if not 'key-secret' in opts"

> +            opts.append(luks_default_key_secret_opt)

And here we might modify that default list.

> +        objects.append(luks_default_secret_object)
> +
> +    args = ['create', '-f', fmt]
> +
> +    if len(opts) > 0:
> +        args += ['-o', ','.join(opts)]
> +
> +    if backing_file is not None:
> +        args += ['-b', backing_file]
> +
> +    if backing_fmt is not None:
> +        args += ['-F', backing_fmt]
> +
> +    if len(objects) > 0:
> +        # Generate a [['--object', $obj], [...], ...] list and flatten it
> +        args += [arg for objarg in (['--object', obj] for obj in objects) \
> +                     for arg in objarg]

I may have mentioned at one point that I love comprehensions, but
dislike nested comprehensions. At this point, I think it's far simpler
to say:

for obj in objects:
    args.extend(['--object', obj])

or, even shorter:
    args += ['--object', obj]

> +
> +    if unsafe:
> +        args.append('-u')
> +
> +    args.append(filename)
> +    if size is not None:
> +        args.append(str(size))
> +
> +    return qemu_img(*args)
> +
> +# Use this to remove images create with create_test_image in the

created

and you might as well move the # comment to a """docstring""" while
you're here.

> +# default image format (iotests.imgfmt)
> +def remove_test_image(filename):
> +    try:
> +        os.remove(filename)
> +
> +        data_file = next(opt.replace('data_file=', '') \
> +                            .replace('$TEST_IMG', filename) \
> +                         for opt in imgopts if opt.startswith('data_file='))
> +

Learned something today: you can use next() to get the first value from
a generator expression.

> +        os.remove(data_file)

Keep in mind that if the generator expression returns no results, that
next() will throw an exception and we won't make it here. That's ok, but,

> +    except:
> +        pass
> +

The unqualified except doesn't help me know which errors you expected
and which you didn't.

We have a function like this elsewhere in the python directory:

def remove_if_exists(filename):
    try:
        os.remove(filename)
    except FileNotFoundError:
        pass

Can we use that here and remove the try:/except: from this function? It
will require you to change the list search to something like this instead:

remove_if_exists(filename)
for opt in (x for x in imgopts if etc):
    data_file = opt.replace('etc', 'etc')
    remove_if_exists(data_file)

to avoid the exception when you call next().

>  def qemu_img_verbose(*args):
>      '''Run qemu-img without suppressing its output and return the exit code'''
>      exitcode = subprocess.call(qemu_img_args + list(args))
> 

My fussiness with the remove() function is just optional picky stuff,
but the rest matters, I think.

--js

Re: [PATCH 04/67] iotests.py: create_test_image, remove_test_image

[John Snow]

On 10/1/19 7:20 PM, John Snow wrote:
> You can write "if not 'key-secret' in opts"

Ah nah, because it's key-secret= whatever and I'm just being too hasty
with what I thought is actually in the opts list.

Carry on ...

Re: [PATCH 04/67] iotests.py: create_test_image, remove_test_image

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 6123 bytes --]

On 02.10.19 01:20, John Snow wrote:
> 
> 
> On 10/1/19 3:46 PM, Max Reitz wrote:
>> Python tests should use these two new functions instead of
>> qemu_img('create', ...) + os.remove(), so that user-supplied image
>> options are interpreted and handled correctly.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>  tests/qemu-iotests/iotests.py | 56 +++++++++++++++++++++++++++++++++++
>>  1 file changed, 56 insertions(+)
>>
>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>> index b5ea424de4..fce1ab04c9 100644
>> --- a/tests/qemu-iotests/iotests.py
>> +++ b/tests/qemu-iotests/iotests.py
>> @@ -122,6 +122,62 @@ def qemu_img_create(*args):
>>  
>>      return qemu_img(*args)
>>  
>> +def create_test_image(filename, size=None, fmt=imgfmt, opts=[],
>> +                      backing_file=None, backing_fmt=None,
>> +                      objects=[], unsafe=False):
> 
> Python! It's the language that everybody loves and can do no wrong!
> 
> Ah, wait, no, maybe the opposite.
> 
> You want this:
> 
> (..., opts=None, ...):
>     opts = opts or []
> 
> because, unfortunately, default parameters are bound at definition time
> and not at call time, so the default list here is like a static local.

OK.  Interesting.

I suppose the same goes for @objects, then.

>> +    if fmt == imgfmt:
>> +        # Only use imgopts for the default format
>> +        opts = imgopts + opts
>> +
>> +    for i, opt in enumerate(opts):
>> +        if '$TEST_IMG' in opt:
>> +            opts[i] = opt.replace('$TEST_IMG', filename)
>> +
>> +    # default luks support
>> +    if fmt == 'luks':
>> +        if not any('key-secret' in opt for opt in opts):
> 
> You can write "if not 'key-secret' in opts"

Oh, that’s recursive?

>> +            opts.append(luks_default_key_secret_opt)
> 
> And here we might modify that default list.
> 
>> +        objects.append(luks_default_secret_object)
>> +
>> +    args = ['create', '-f', fmt]
>> +
>> +    if len(opts) > 0:
>> +        args += ['-o', ','.join(opts)]
>> +
>> +    if backing_file is not None:
>> +        args += ['-b', backing_file]
>> +
>> +    if backing_fmt is not None:
>> +        args += ['-F', backing_fmt]
>> +
>> +    if len(objects) > 0:
>> +        # Generate a [['--object', $obj], [...], ...] list and flatten it
>> +        args += [arg for objarg in (['--object', obj] for obj in objects) \
>> +                     for arg in objarg]
> 
> I may have mentioned at one point that I love comprehensions, but
> dislike nested comprehensions.

I can’t remember but I do remember writing this piece of code, being sad
that there is no .flatten, and wanting everyone to see the monster that
arises.

> At this point, I think it's far simpler
> to say:
> 
> for obj in objects:
>     args.extend(['--object', obj])
> 
> or, even shorter:
>     args += ['--object', obj]

OK, so now you saw it, I’m glad to make the flattening more flattering
to read.

>> +
>> +    if unsafe:
>> +        args.append('-u')
>> +
>> +    args.append(filename)
>> +    if size is not None:
>> +        args.append(str(size))
>> +
>> +    return qemu_img(*args)
>> +
>> +# Use this to remove images create with create_test_image in the
> 
> created
> 
> and you might as well move the # comment to a """docstring""" while
> you're here.
> 
>> +# default image format (iotests.imgfmt)
>> +def remove_test_image(filename):
>> +    try:
>> +        os.remove(filename)
>> +
>> +        data_file = next(opt.replace('data_file=', '') \
>> +                            .replace('$TEST_IMG', filename) \
>> +                         for opt in imgopts if opt.startswith('data_file='))
>> +
> 
> Learned something today: you can use next() to get the first value from
> a generator expression.

I was sad for a bit that Python doesn’t have a find(), but then I
noticed this works as well.  (Already used extensively in “iotests: Add
VM.assert_block_path()” from my “block: Fix check_to_replace_node()”
series.)

>> +        os.remove(data_file)
> 
> Keep in mind that if the generator expression returns no results, that
> next() will throw an exception and we won't make it here. That's ok, but,

I did.  If there are no results, it’s good we won’t get here.

This code would be wrong if the next() didn’t throw an exception.

>> +    except:
>> +        pass
>> +
> 
> The unqualified except doesn't help me know which errors you expected
> and which you didn't.

What I’m expecting: FileNotFound, StopIteration.

But the thing is that I feel like maybe removing a file should always
pass, regardless of the exact exception.  (I can imagine to be wrong.)

> We have a function like this elsewhere in the python directory:
> 
> def remove_if_exists(filename):
>     try:
>         os.remove(filename)
>     except FileNotFoundError:
>         pass

We do?  I can’t find it.  I find a _remove_if_exists in machine.py,
which I’m not sure whether it’s supposed to be used outside, and it
works a bit different, actually (but probably to the same effect).

> Can we use that here and remove the try:/except: from this function? It
> will require you to change the list search to something like this instead:
> 
> remove_if_exists(filename)
> for opt in (x for x in imgopts if etc):
>     data_file = opt.replace('etc', 'etc')
>     remove_if_exists(data_file)
> 
> to avoid the exception when you call next().

I don’t know why I’d avoid the exception, though.

This is probably because I don’t like pythonic code, again, but I prefer
a next() + exception over a for loop that just iterates once or not at all.

>>  def qemu_img_verbose(*args):
>>      '''Run qemu-img without suppressing its output and return the exit code'''
>>      exitcode = subprocess.call(qemu_img_args + list(args))
>>
> 
> My fussiness with the remove() function is just optional picky stuff,
> but the rest matters, I think.

OK.  Indeed it does!

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PATCH 04/67] iotests.py: create_test_image, remove_test_image

[John Snow]

On 10/2/19 7:00 AM, Max Reitz wrote:
> On 02.10.19 01:20, John Snow wrote:
>>
>>
>> On 10/1/19 3:46 PM, Max Reitz wrote:
>>> Python tests should use these two new functions instead of
>>> qemu_img('create', ...) + os.remove(), so that user-supplied image
>>> options are interpreted and handled correctly.
>>>
>>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>>> ---
>>>  tests/qemu-iotests/iotests.py | 56 +++++++++++++++++++++++++++++++++++
>>>  1 file changed, 56 insertions(+)
>>>
>>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>>> index b5ea424de4..fce1ab04c9 100644
>>> --- a/tests/qemu-iotests/iotests.py
>>> +++ b/tests/qemu-iotests/iotests.py
>>> @@ -122,6 +122,62 @@ def qemu_img_create(*args):
>>>  
>>>      return qemu_img(*args)
>>>  
>>> +def create_test_image(filename, size=None, fmt=imgfmt, opts=[],
>>> +                      backing_file=None, backing_fmt=None,
>>> +                      objects=[], unsafe=False):
>>
>> Python! It's the language that everybody loves and can do no wrong!
>>
>> Ah, wait, no, maybe the opposite.
>>
>> You want this:
>>
>> (..., opts=None, ...):
>>     opts = opts or []
>>
>> because, unfortunately, default parameters are bound at definition time
>> and not at call time, so the default list here is like a static local.
> 
> OK.  Interesting.
> 
> I suppose the same goes for @objects, then.
> 

It is by far the WORST thing about Python.

I realize we use this pattern a few places in iotests, but I think it's
also usually where we don't modify the list, so it's actually OK, but
serves as an example of a bad habit.

>>> +    if fmt == imgfmt:
>>> +        # Only use imgopts for the default format
>>> +        opts = imgopts + opts
>>> +
>>> +    for i, opt in enumerate(opts):
>>> +        if '$TEST_IMG' in opt:
>>> +            opts[i] = opt.replace('$TEST_IMG', filename)
>>> +
>>> +    # default luks support
>>> +    if fmt == 'luks':
>>> +        if not any('key-secret' in opt for opt in opts):
>>
>> You can write "if not 'key-secret' in opts"
> 
> Oh, that’s recursive?
> 

No, I was just mistaken about the shape of the data.
You are looking for 'key-secret=XXX', I was thinking that there was a
token that was really just 'key-secret'.

What you wrote is correct and good and I am wrong and bad.

>>> +            opts.append(luks_default_key_secret_opt)
>>
>> And here we might modify that default list.
>>
>>> +        objects.append(luks_default_secret_object)
>>> +
>>> +    args = ['create', '-f', fmt]
>>> +
>>> +    if len(opts) > 0:
>>> +        args += ['-o', ','.join(opts)]
>>> +
>>> +    if backing_file is not None:
>>> +        args += ['-b', backing_file]
>>> +
>>> +    if backing_fmt is not None:
>>> +        args += ['-F', backing_fmt]
>>> +
>>> +    if len(objects) > 0:
>>> +        # Generate a [['--object', $obj], [...], ...] list and flatten it
>>> +        args += [arg for objarg in (['--object', obj] for obj in objects) \
>>> +                     for arg in objarg]
>>
>> I may have mentioned at one point that I love comprehensions, but
>> dislike nested comprehensions.
> 
> I can’t remember but I do remember writing this piece of code, being sad
> that there is no .flatten, and wanting everyone to see the monster that
> arises.
> 
>> At this point, I think it's far simpler
>> to say:
>>
>> for obj in objects:
>>     args.extend(['--object', obj])
>>
>> or, even shorter:
>>     args += ['--object', obj]
> 
> OK, so now you saw it, I’m glad to make the flattening more flattering
> to read.
> 

I am sorry I ever mentioned liking Python. I will accept your punishments.

>>> +
>>> +    if unsafe:
>>> +        args.append('-u')
>>> +
>>> +    args.append(filename)
>>> +    if size is not None:
>>> +        args.append(str(size))
>>> +
>>> +    return qemu_img(*args)
>>> +
>>> +# Use this to remove images create with create_test_image in the
>>
>> created
>>
>> and you might as well move the # comment to a """docstring""" while
>> you're here.
>>
>>> +# default image format (iotests.imgfmt)
>>> +def remove_test_image(filename):
>>> +    try:
>>> +        os.remove(filename)
>>> +
>>> +        data_file = next(opt.replace('data_file=', '') \
>>> +                            .replace('$TEST_IMG', filename) \
>>> +                         for opt in imgopts if opt.startswith('data_file='))
>>> +
>>
>> Learned something today: you can use next() to get the first value from
>> a generator expression.
> 
> I was sad for a bit that Python doesn’t have a find(), but then I
> noticed this works as well.  (Already used extensively in “iotests: Add
> VM.assert_block_path()” from my “block: Fix check_to_replace_node()”
> series.)
> 

I honestly tried to rewrite this a few times because it looks so chunky,
but realized there isn't ... a great way to do this without implying
that you might find more than one result.

You can filter to a new list and assert that the length is one, but
that's not less chunky.

>>> +        os.remove(data_file)
>>
>> Keep in mind that if the generator expression returns no results, that
>> next() will throw an exception and we won't make it here. That's ok, but,
> 
> I did.  If there are no results, it’s good we won’t get here.
> 
> This code would be wrong if the next() didn’t throw an exception.
> 

It just wasn't clear, because the except is doing the lifting for both
the remove and the finding.

Oh well, it's not really important.

>>> +    except:
>>> +        pass
>>> +
>>
>> The unqualified except doesn't help me know which errors you expected
>> and which you didn't.
> 
> What I’m expecting: FileNotFound, StopIteration.
> 
> But the thing is that I feel like maybe removing a file should always
> pass, regardless of the exact exception.  (I can imagine to be wrong.)
> 

I wonder if that's true ... I just don't know what the full set of
errors we might get are. I don't really like exception driven code,
honestly.

"It's wrong to catch ANY exception because you might suppress errors too
broadly."

"It's wrong to be too specific, because you'll miss cases you meant to
catch."

Awful.

Anyway, like I said I was just being fiddly because I found this odd to
read, but really don't have suggestions that are clearly nicer, so ...
carry on.

>> We have a function like this elsewhere in the python directory:
>>
>> def remove_if_exists(filename):
>>     try:
>>         os.remove(filename)
>>     except FileNotFoundError:
>>         pass
> 
> We do?  I can’t find it.  I find a _remove_if_exists in machine.py,
> which I’m not sure whether it’s supposed to be used outside, and it
> works a bit different, actually (but probably to the same effect).
> 

Yeah, that's the one. Don't worry about plucking it out here for this,
just nothing that we do this in a few places. We might want a util
eventually that gets it exactly right.

Or not, because what's "exactly right" anyway. Ah, ah, ah.

>> Can we use that here and remove the try:/except: from this function? It
>> will require you to change the list search to something like this instead:
>>
>> remove_if_exists(filename)
>> for opt in (x for x in imgopts if etc):
>>     data_file = opt.replace('etc', 'etc')
>>     remove_if_exists(data_file)
>>
>> to avoid the exception when you call next().
> 
> I don’t know why I’d avoid the exception, though.
> 
> This is probably because I don’t like pythonic code, again, but I prefer
> a next() + exception over a for loop that just iterates once or not at all.
> 

Nah, Python people LOVE exceptions. They don't like "bare except"
statements, though. I am the weird person in that I like to avoid
exceptions whenever it's elegant and pretty to do so.

I find exceptions as normal control flow to be quite hard to deal with;
but Pythonistas seem to love it.

>>>  def qemu_img_verbose(*args):
>>>      '''Run qemu-img without suppressing its output and return the exit code'''
>>>      exitcode = subprocess.call(qemu_img_args + list(args))
>>>
>>
>> My fussiness with the remove() function is just optional picky stuff,
>> but the rest matters, I think.
> 
> OK.  Indeed it does!
> 
> Max
> 

Re: [PATCH 01/67] iotests.py: Read $IMGOPTS

[Vladimir Sementsov-Ogievskiy]

02.10.2019 1:16, John Snow wrote:
> 
> 
> On 10/1/19 3:46 PM, Max Reitz wrote:
>> We do not do anything with yet, but this is the first step.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>   tests/qemu-iotests/iotests.py | 6 ++++++
>>   1 file changed, 6 insertions(+)
>>
>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>> index 1c5fce3e9e..7030900807 100644
>> --- a/tests/qemu-iotests/iotests.py
>> +++ b/tests/qemu-iotests/iotests.py
>> @@ -69,6 +69,12 @@ output_dir = os.environ.get('OUTPUT_DIR', '.')
>>   cachemode = os.environ.get('CACHEMODE')
>>   qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
>>   
>> +imgopts = os.environ.get('IMGOPTS', '')
>> +if len(imgopts) == 0:
>> +    imgopts = []
>> +else:
>> +    imgopts = imgopts.split(',')
>> +
> 
> Sometimes I think about the type of person I'd like to meet, and I ask
> myself if it's the type of person who would quote pep8, or say things
> like "idiomatic python" in a code review when the existing form has no
> technical problem whatsoever.
> 
> I wonder what type of person I am.
> 
> Well, questions for another day.

Hope you don't say about quoting, as pep8 "does not make a recommendation for this".

> 
>>   socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
>>   
>>   luks_default_secret_object = 'secret,id=keysec0,data=' + \
>>
> 
> Reviewed-by: John Snow <jsnow@redhat.com>
> 


-- 
Best regards,
Vladimir

Re: [PATCH 01/67] iotests.py: Read $IMGOPTS

[Vladimir Sementsov-Ogievskiy]

01.10.2019 22:46, Max Reitz wrote:
> We do not do anything with yet, but this is the first step.
> 
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>   tests/qemu-iotests/iotests.py | 6 ++++++
>   1 file changed, 6 insertions(+)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 1c5fce3e9e..7030900807 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -69,6 +69,12 @@ output_dir = os.environ.get('OUTPUT_DIR', '.')
>   cachemode = os.environ.get('CACHEMODE')
>   qemu_default_machine = os.environ.get('QEMU_DEFAULT_MACHINE')
>   
> +imgopts = os.environ.get('IMGOPTS', '')
> +if len(imgopts) == 0:
> +    imgopts = []
> +else:
> +    imgopts = imgopts.split(',')
> +
>   socket_scm_helper = os.environ.get('SOCKET_SCM_HELPER', 'socket_scm_helper')
>   
>   luks_default_secret_object = 'secret,id=keysec0,data=' + \
> 

Note, that empty sequences are always false, so you may use just "if not imgopts"

Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

-- 
Best regards,
Vladimir

Re: [PATCH 02/67] iotests.py: Add @skip_for_imgopts()

[Vladimir Sementsov-Ogievskiy]

01.10.2019 22:46, Max Reitz wrote:
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>   tests/qemu-iotests/iotests.py | 13 +++++++++++++
>   1 file changed, 13 insertions(+)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 7030900807..cdcb62c4ac 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -950,6 +950,19 @@ def skip_if_unsupported(required_formats=[], read_only=False):
>           return func_wrapper
>       return skip_test_decorator
>   
> +def skip_for_imgopts(unsupported_opts=[]):
> +    '''Skip Test Decorator
> +       Skips the test if imgopts contains any of the given options'''
> +    def skip_test_decorator(func):
> +        def func_wrapper(test_case: QMPTestCase, *args, **kwargs):

how about

unsup = set(imgopts) & set(unsupported_opts)
if unsup:
    test_case.case_skip('... Options {} are ...', format(..., ', '.join(map(str, unsup)))

> +            for opt in imgopts:
> +                if any(unsupported in opt for unsupported in unsupported_opts):
> +                    test_case.case_skip('{}: Option {} is unsupported'.format(
> +                                        test_case, opt))
> +            return func(test_case, *args, **kwargs)
> +        return func_wrapper
> +    return skip_test_decorator
> +
>   def execute_unittest(debug=False):
>       """Executes unittests within the calling module."""
>   
> 


-- 
Best regards,
Vladimir

Re: [PATCH 03/67] iotests.py: Add unsupported_imgopts

[Vladimir Sementsov-Ogievskiy]

01.10.2019 22:46, Max Reitz wrote:
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>   tests/qemu-iotests/iotests.py | 7 ++++++-
>   1 file changed, 6 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index cdcb62c4ac..b5ea424de4 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -998,7 +998,8 @@ def execute_setup_common(supported_fmts=[],
>                            supported_cache_modes=[],
>                            unsupported_fmts=[],
>                            supported_protocols=[],
> -                         unsupported_protocols=[]):
> +                         unsupported_protocols=[],
> +                         unsupported_imgopts=[]):
>       """
>       Perform necessary setup for either script-style or unittest-style tests.
>       """
> @@ -1016,6 +1017,10 @@ def execute_setup_common(supported_fmts=[],
>       verify_platform(supported=supported_platforms)
>       verify_cache_mode(supported_cache_modes)
>   
> +    for opt in imgopts:
> +        if any(unsupported in opt for unsupported in unsupported_imgopts):
> +            notrun('not suitable for this option: %s' % opt)
> +

same suggestion

>       debug = '-d' in sys.argv
>       if debug:
>           sys.argv.remove('-d')
> 


-- 
Best regards,
Vladimir

Re: [PATCH 02/67] iotests.py: Add @skip_for_imgopts()

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 1074 bytes --]

On 03.10.19 17:19, Vladimir Sementsov-Ogievskiy wrote:
> 01.10.2019 22:46, Max Reitz wrote:
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>   tests/qemu-iotests/iotests.py | 13 +++++++++++++
>>   1 file changed, 13 insertions(+)
>>
>> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
>> index 7030900807..cdcb62c4ac 100644
>> --- a/tests/qemu-iotests/iotests.py
>> +++ b/tests/qemu-iotests/iotests.py
>> @@ -950,6 +950,19 @@ def skip_if_unsupported(required_formats=[], read_only=False):
>>           return func_wrapper
>>       return skip_test_decorator
>>   
>> +def skip_for_imgopts(unsupported_opts=[]):
>> +    '''Skip Test Decorator
>> +       Skips the test if imgopts contains any of the given options'''
>> +    def skip_test_decorator(func):
>> +        def func_wrapper(test_case: QMPTestCase, *args, **kwargs):
> 
> how about
> 
> unsup = set(imgopts) & set(unsupported_opts)
> if unsup:
>     test_case.case_skip('... Options {} are ...', format(..., ', '.join(map(str, unsup)))

Sure.

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]