[PULL 00/18] Block patches

[PULL 00/18] Block patches

[Max Reitz]

The following changes since commit 672f9d0df10a68a5c5f2b32cbc8284abf9f5ee18:

  Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2020-02-18 14:23:43 +0000)

are available in the Git repository at:

  https://github.com/XanClic/qemu.git tags/pull-block-2020-02-20

for you to fetch changes up to dff8d44c96f128480430b0c59ed8760917dbd427:

  iotests: Test snapshot -l field separation (2020-02-20 16:43:42 +0100)

----------------------------------------------------------------
Block patches:
- qemu-img convert: New --target-is-zero parameter
- qcow2: Specify non-default compression type flag
- optionally flat output for query-named-block-nodes
- some fixes
- pseudo-creation of images on block devices is now done by a generic
  block layer function

----------------------------------------------------------------
Daniel P. Berrangé (1):
  block: always fill entire LUKS header space with zeros

David Edmondson (1):
  qemu-img: Add --target-is-zero to convert

Max Reitz (11):
  iotests/147: Fix drive parameters
  iotests/279: Fix for non-qcow2 formats
  block/nbd: Fix hang in .bdrv_close()
  block: Generic file creation fallback
  file-posix: Drop hdev_co_create_opts()
  iscsi: Drop iscsi_co_create_opts()
  iotests: Add test for image creation fallback
  qemu-img: Fix convert -n -B for backing-less targets
  iotests: Test convert -n -B to backing-less target
  block: Fix VM size field width in snapshot dump
  iotests: Test snapshot -l field separation

Peter Krempa (1):
  qapi: Allow getting flat output from 'query-named-block-nodes'

Thomas Huth (1):
  iotests: Remove the superfluous 2nd check for the availability of
    quorum

Vladimir Sementsov-Ogievskiy (3):
  docs: improve qcow2 spec about extending image header
  docs: qcow2: introduce compression type feature
  block/backup-top: fix flags handling

 block.c                    | 164 +++++++++++++++++++++++++++++++++----
 block/backup-top.c         |  31 ++++---
 block/file-posix.c         |  67 ---------------
 block/iscsi.c              |  56 -------------
 block/nbd.c                |  14 +++-
 block/qapi.c               |  15 +++-
 block/qcow2.c              |  11 ++-
 blockdev.c                 |   8 +-
 docs/interop/qcow2.txt     |  64 ++++++++++++++-
 docs/interop/qemu-img.rst  |   9 +-
 include/block/block.h      |   2 +-
 include/block/qapi.h       |   4 +-
 monitor/hmp-cmds.c         |   2 +-
 qapi/block-core.json       |   7 +-
 qemu-img-cmds.hx           |   4 +-
 qemu-img.c                 |  28 ++++++-
 tests/qemu-iotests/122     |  14 ++++
 tests/qemu-iotests/122.out |   5 ++
 tests/qemu-iotests/139     |   3 -
 tests/qemu-iotests/147     |   2 +-
 tests/qemu-iotests/259     |  62 ++++++++++++++
 tests/qemu-iotests/259.out |  14 ++++
 tests/qemu-iotests/279     |   7 +-
 tests/qemu-iotests/284     |  97 ++++++++++++++++++++++
 tests/qemu-iotests/284.out |  62 ++++++++++++++
 tests/qemu-iotests/286     |  76 +++++++++++++++++
 tests/qemu-iotests/286.out |   8 ++
 tests/qemu-iotests/group   |   3 +
 28 files changed, 659 insertions(+), 180 deletions(-)
 create mode 100755 tests/qemu-iotests/259
 create mode 100644 tests/qemu-iotests/259.out
 create mode 100755 tests/qemu-iotests/284
 create mode 100644 tests/qemu-iotests/284.out
 create mode 100755 tests/qemu-iotests/286
 create mode 100644 tests/qemu-iotests/286.out

-- 
2.24.1

[PULL 01/18] docs: improve qcow2 spec about extending image header

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Make it more obvious how to add new fields to the version 3 header and
how to interpret them.

The specification is adjusted so that for new defined optional fields:

1. Software may support some of these optional fields and ignore the
   others, which means that features may be backported to downstream
   Qemu independently.
2. If we want to add incompatible field (or a field, for which some of
   its values would be incompatible), it must be accompanied by
   incompatible feature bit.

Also the concept of "default is zero" is clarified, as it's strange to
say that the value of the field is assumed to be zero for the software
version which don't know about the field at all and don't know how to
treat it be it zero or not.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Message-Id: <20200131142219.3264-2-vsementsov@virtuozzo.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
[mreitz: s/some its/some of its/]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 docs/interop/qcow2.txt | 45 +++++++++++++++++++++++++++++++++++++++---
 1 file changed, 42 insertions(+), 3 deletions(-)

diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index af5711e533..823cc266e0 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -79,9 +79,9 @@ The first cluster of a qcow2 image contains the file header:
                     Offset into the image file at which the snapshot table
                     starts. Must be aligned to a cluster boundary.
 
-If the version is 3 or higher, the header has the following additional fields.
-For version 2, the values are assumed to be zero, unless specified otherwise
-in the description of a field.
+For version 2, the header is exactly 72 bytes in length, and finishes here.
+For version 3 or higher, the header length is at least 104 bytes, including
+the next fields through header_length.
 
          72 -  79:  incompatible_features
                     Bitmask of incompatible features. An implementation must
@@ -164,6 +164,45 @@ in the description of a field.
         100 - 103:  header_length
                     Length of the header structure in bytes. For version 2
                     images, the length is always assumed to be 72 bytes.
+                    For version 3 it's at least 104 bytes and must be a multiple
+                    of 8.
+
+
+=== Additional fields (version 3 and higher) ===
+
+In general, these fields are optional and may be safely ignored by the software,
+as well as filled by zeros (which is equal to field absence), if software needs
+to set field B, but does not care about field A which precedes B. More
+formally, additional fields have the following compatibility rules:
+
+1. If the value of the additional field must not be ignored for correct
+handling of the file, it will be accompanied by a corresponding incompatible
+feature bit.
+
+2. If there are no unrecognized incompatible feature bits set, an unknown
+additional field may be safely ignored other than preserving its value when
+rewriting the image header.
+
+3. An explicit value of 0 will have the same behavior as when the field is not
+present*, if not altered by a specific incompatible bit.
+
+*. A field is considered not present when header_length is less than or equal
+to the field's offset. Also, all additional fields are not present for
+version 2.
+
+        < ... No additional fields in the header currently ... >
+
+
+=== Header padding ===
+
+@header_length must be a multiple of 8, which means that if the end of the last
+additional field is not aligned, some padding is needed. This padding must be
+zeroed, so that if some existing (or future) additional field will fall into
+the padding, it will be interpreted accordingly to point [3.] of the previous
+paragraph, i.e.  in the same manner as when this field is not present.
+
+
+=== Header extensions ===
 
 Directly after the image header, optional sections called header extensions can
 be stored. Each extension has a structure like the following:
-- 
2.24.1

[PULL 02/18] docs: qcow2: introduce compression type feature

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

The patch adds a new additional field to the qcow2 header: compression_type,
which specifies compression type. If field is absent or zero, default
compression type is set: ZLIB, which corresponds to current behavior.

New compression type (ZSTD) is to be added in further commit.

Suggested-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200131142219.3264-3-vsementsov@virtuozzo.com>
[mreitz: s/Bits 3-63:  Reserved/Bits 4-63:  Reserved/]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 docs/interop/qcow2.txt | 21 +++++++++++++++++++--
 1 file changed, 19 insertions(+), 2 deletions(-)

diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 823cc266e0..5597e24474 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -109,7 +109,12 @@ the next fields through header_length.
                                 An External Data File Name header extension may
                                 be present if this bit is set.
 
-                    Bits 3-63:  Reserved (set to 0)
+                    Bit 3:      Compression type bit.  If this bit is set,
+                                a non-default compression is used for compressed
+                                clusters. The compression_type field must be
+                                present and not zero.
+
+                    Bits 4-63:  Reserved (set to 0)
 
          80 -  87:  compatible_features
                     Bitmask of compatible features. An implementation can
@@ -190,7 +195,19 @@ present*, if not altered by a specific incompatible bit.
 to the field's offset. Also, all additional fields are not present for
 version 2.
 
-        < ... No additional fields in the header currently ... >
+              104:  compression_type
+
+                    Defines the compression method used for compressed clusters.
+                    All compressed clusters in an image use the same compression
+                    type.
+
+                    If the incompatible bit "Compression type" is set: the field
+                    must be present and non-zero (which means non-zlib
+                    compression type). Otherwise, this field must not be present
+                    or must be zero (which means zlib).
+
+                    Available compression type values:
+                        0: zlib <https://www.zlib.net/>
 
 
 === Header padding ===
-- 
2.24.1

[PULL 03/18] iotests: Remove the superfluous 2nd check for the availability of quorum

[Max Reitz]

From: Thomas Huth <thuth@redhat.com>

Commit d9df28e7b07 ("iotests: check whitelisted formats") added the
modern @iotests.skip_if_unsupported() to the functions in this test,
so we don't need the old explicit test here anymore.

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20200129141751.32652-1-thuth@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Reviewed-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Tested-by: Andrey Shinkevich <andrey.shinkevich@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/139 | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/tests/qemu-iotests/139 b/tests/qemu-iotests/139
index 6b1a444364..7120d3142b 100755
--- a/tests/qemu-iotests/139
+++ b/tests/qemu-iotests/139
@@ -344,9 +344,6 @@ class TestBlockdevDel(iotests.QMPTestCase):
 
     @iotests.skip_if_unsupported(['quorum'])
     def testQuorum(self):
-        if not iotests.supports_quorum():
-            return
-
         self.addQuorum('quorum0', 'node0', 'node1')
         # We cannot remove the children of a Quorum device
         self.delBlockDriverState('node0', expect_error = True)
-- 
2.24.1

[PULL 04/18] iotests/147: Fix drive parameters

[Max Reitz]

8dff69b94 added an aio parameter to the drive parameter but forgot to
add a comma before, thus breaking the test.  Fix it again.

Fixes: 8dff69b9415b4287e900358744b732195e1ab2e2
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200206130812.612960-1-mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Tested-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/147 | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/147 b/tests/qemu-iotests/147
index f4b0a11dba..d7a9f31089 100755
--- a/tests/qemu-iotests/147
+++ b/tests/qemu-iotests/147
@@ -134,7 +134,7 @@ class BuiltinNBD(NBDBlockdevAddBase):
         self.server.add_drive_raw('if=none,id=nbd-export,' +
                                   'file=%s,' % test_img +
                                   'format=%s,' % imgfmt +
-                                  'cache=%s' % cachemode +
+                                  'cache=%s,' % cachemode +
                                   'aio=%s' % aiomode)
         self.server.launch()
 
-- 
2.24.1

[PULL 05/18] qapi: Allow getting flat output from 'query-named-block-nodes'

[Max Reitz]

From: Peter Krempa <pkrempa@redhat.com>

When a management application manages node names there's no reason to
recurse into backing images in the output of query-named-block-nodes.

Add a parameter to the command which will return just the top level
structs.

Signed-off-by: Peter Krempa <pkrempa@redhat.com>
Message-Id: <4470f8c779abc404dcf65e375db195cd91a80651.1579509782.git.pkrempa@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[mreitz: Fixed coding style]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c               |  5 +++--
 block/qapi.c          | 11 +++++++++--
 blockdev.c            |  8 ++++++--
 include/block/block.h |  2 +-
 include/block/qapi.h  |  4 +++-
 monitor/hmp-cmds.c    |  2 +-
 qapi/block-core.json  |  7 ++++++-
 7 files changed, 29 insertions(+), 10 deletions(-)

diff --git a/block.c b/block.c
index 946e3c896e..08372ced26 100644
--- a/block.c
+++ b/block.c
@@ -4807,14 +4807,15 @@ BlockDriverState *bdrv_find_node(const char *node_name)
 }
 
 /* Put this QMP function here so it can access the static graph_bdrv_states. */
-BlockDeviceInfoList *bdrv_named_nodes_list(Error **errp)
+BlockDeviceInfoList *bdrv_named_nodes_list(bool flat,
+                                           Error **errp)
 {
     BlockDeviceInfoList *list, *entry;
     BlockDriverState *bs;
 
     list = NULL;
     QTAILQ_FOREACH(bs, &graph_bdrv_states, node_list) {
-        BlockDeviceInfo *info = bdrv_block_device_info(NULL, bs, errp);
+        BlockDeviceInfo *info = bdrv_block_device_info(NULL, bs, flat, errp);
         if (!info) {
             qapi_free_BlockDeviceInfoList(list);
             return NULL;
diff --git a/block/qapi.c b/block/qapi.c
index 9a5d0c9b27..3f09477cc5 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -42,7 +42,9 @@
 #include "qemu/cutils.h"
 
 BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
-                                        BlockDriverState *bs, Error **errp)
+                                        BlockDriverState *bs,
+                                        bool flat,
+                                        Error **errp)
 {
     ImageInfo **p_image_info;
     BlockDriverState *bs0;
@@ -156,6 +158,11 @@ BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
             return NULL;
         }
 
+        /* stop gathering data for flat output */
+        if (flat) {
+            break;
+        }
+
         if (bs0->drv && bs0->backing) {
             info->backing_file_depth++;
             bs0 = bs0->backing->bs;
@@ -389,7 +396,7 @@ static void bdrv_query_info(BlockBackend *blk, BlockInfo **p_info,
 
     if (bs && bs->drv) {
         info->has_inserted = true;
-        info->inserted = bdrv_block_device_info(blk, bs, errp);
+        info->inserted = bdrv_block_device_info(blk, bs, false, errp);
         if (info->inserted == NULL) {
             goto err;
         }
diff --git a/blockdev.c b/blockdev.c
index 45de0ba37e..011dcfec27 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -3734,9 +3734,13 @@ void qmp_drive_backup(DriveBackup *backup, Error **errp)
     blockdev_do_action(&action, errp);
 }
 
-BlockDeviceInfoList *qmp_query_named_block_nodes(Error **errp)
+BlockDeviceInfoList *qmp_query_named_block_nodes(bool has_flat,
+                                                 bool flat,
+                                                 Error **errp)
 {
-    return bdrv_named_nodes_list(errp);
+    bool return_flat = has_flat && flat;
+
+    return bdrv_named_nodes_list(return_flat, errp);
 }
 
 XDbgBlockGraph *qmp_x_debug_query_block_graph(Error **errp)
diff --git a/include/block/block.h b/include/block/block.h
index 314ce63ed9..cd6b5b95aa 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -454,7 +454,7 @@ void bdrv_lock_medium(BlockDriverState *bs, bool locked);
 void bdrv_eject(BlockDriverState *bs, bool eject_flag);
 const char *bdrv_get_format_name(BlockDriverState *bs);
 BlockDriverState *bdrv_find_node(const char *node_name);
-BlockDeviceInfoList *bdrv_named_nodes_list(Error **errp);
+BlockDeviceInfoList *bdrv_named_nodes_list(bool flat, Error **errp);
 XDbgBlockGraph *bdrv_get_xdbg_block_graph(Error **errp);
 BlockDriverState *bdrv_lookup_bs(const char *device,
                                  const char *node_name,
diff --git a/include/block/qapi.h b/include/block/qapi.h
index cd9410dee3..22c7807c89 100644
--- a/include/block/qapi.h
+++ b/include/block/qapi.h
@@ -29,7 +29,9 @@
 #include "block/snapshot.h"
 
 BlockDeviceInfo *bdrv_block_device_info(BlockBackend *blk,
-                                        BlockDriverState *bs, Error **errp);
+                                        BlockDriverState *bs,
+                                        bool flat,
+                                        Error **errp);
 int bdrv_query_snapshot_info_list(BlockDriverState *bs,
                                   SnapshotInfoList **p_list,
                                   Error **errp);
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index b237613e0d..53bc3f76c4 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -620,7 +620,7 @@ void hmp_info_block(Monitor *mon, const QDict *qdict)
     }
 
     /* Print node information */
-    blockdev_list = qmp_query_named_block_nodes(NULL);
+    blockdev_list = qmp_query_named_block_nodes(false, false, NULL);
     for (blockdev = blockdev_list; blockdev; blockdev = blockdev->next) {
         assert(blockdev->value->has_node_name);
         if (device && strcmp(device, blockdev->value->node_name)) {
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 37d7ea7295..85e27bb61f 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -1758,6 +1758,9 @@
 #
 # Get the named block driver list
 #
+# @flat: Omit the nested data about backing image ("backing-image" key) if true.
+#        Default is false (Since 5.0)
+#
 # Returns: the list of BlockDeviceInfo
 #
 # Since: 2.0
@@ -1811,7 +1814,9 @@
 #                    } } ] }
 #
 ##
-{ 'command': 'query-named-block-nodes', 'returns': [ 'BlockDeviceInfo' ] }
+{ 'command': 'query-named-block-nodes',
+  'returns': [ 'BlockDeviceInfo' ],
+  'data': { '*flat': 'bool' } }
 
 ##
 # @XDbgBlockGraphNodeType:
-- 
2.24.1

[PULL 06/18] qemu-img: Add --target-is-zero to convert

[Max Reitz]

From: David Edmondson <david.edmondson@oracle.com>

In many cases the target of a convert operation is a newly provisioned
target that the user knows is blank (reads as zero). In this situation
there is no requirement for qemu-img to wastefully zero out the entire
device.

Add a new option, --target-is-zero, allowing the user to indicate that
an existing target device will return zeros for all reads.

Signed-off-by: David Edmondson <david.edmondson@oracle.com>
Message-Id: <20200205110248.2009589-2-david.edmondson@oracle.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 docs/interop/qemu-img.rst |  9 ++++++++-
 qemu-img-cmds.hx          |  4 ++--
 qemu-img.c                | 26 +++++++++++++++++++++++---
 3 files changed, 33 insertions(+), 6 deletions(-)

diff --git a/docs/interop/qemu-img.rst b/docs/interop/qemu-img.rst
index 42e4451db4..5f40137c10 100644
--- a/docs/interop/qemu-img.rst
+++ b/docs/interop/qemu-img.rst
@@ -214,6 +214,13 @@ Parameters to convert subcommand:
   will still be printed.  Areas that cannot be read from the source will be
   treated as containing only zeroes.
 
+.. option:: --target-is-zero
+
+  Assume that reading the destination image will always return
+  zeros. This parameter is mutually exclusive with a destination image
+  that has a backing file. It is required to also use the ``-n``
+  parameter to skip image creation.
+
 Parameters to dd subcommand:
 
 .. program:: qemu-img-dd
@@ -366,7 +373,7 @@ Command description:
   4
     Error on reading data
 
-.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
+.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
 
   Convert the disk image *FILENAME* or a snapshot *SNAPSHOT_PARAM*
   to disk image *OUTPUT_FILENAME* using format *OUTPUT_FMT*. It can
diff --git a/qemu-img-cmds.hx b/qemu-img-cmds.hx
index d7fbc6b1f4..c9c54de1df 100644
--- a/qemu-img-cmds.hx
+++ b/qemu-img-cmds.hx
@@ -39,9 +39,9 @@ SRST
 ERST
 
 DEF("convert", img_convert,
-    "convert [--object objectdef] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
+    "convert [--object objectdef] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f fmt] [-t cache] [-T src_cache] [-O output_fmt] [-B backing_file] [-o options] [-l snapshot_param] [-S sparse_size] [-m num_coroutines] [-W] [--salvage] filename [filename2 [...]] output_filename")
 SRST
-.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
+.. option:: convert [--object OBJECTDEF] [--image-opts] [--target-image-opts] [--target-is-zero] [-U] [-C] [-c] [-p] [-q] [-n] [-f FMT] [-t CACHE] [-T SRC_CACHE] [-O OUTPUT_FMT] [-B BACKING_FILE] [-o OPTIONS] [-l SNAPSHOT_PARAM] [-S SPARSE_SIZE] [-m NUM_COROUTINES] [-W] [--salvage] FILENAME [FILENAME2 [...]] OUTPUT_FILENAME
 ERST
 
 DEF("create", img_create,
diff --git a/qemu-img.c b/qemu-img.c
index 2b4562b9d9..0faf2cd2f5 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -70,6 +70,7 @@ enum {
     OPTION_PREALLOCATION = 265,
     OPTION_SHRINK = 266,
     OPTION_SALVAGE = 267,
+    OPTION_TARGET_IS_ZERO = 268,
 };
 
 typedef enum OutputFormat {
@@ -1984,10 +1985,9 @@ static int convert_do_copy(ImgConvertState *s)
     int64_t sector_num = 0;
 
     /* Check whether we have zero initialisation or can get it efficiently */
-    if (s->target_is_new && s->min_sparse && !s->target_has_backing) {
+    if (!s->has_zero_init && s->target_is_new && s->min_sparse &&
+        !s->target_has_backing) {
         s->has_zero_init = bdrv_has_zero_init(blk_bs(s->target));
-    } else {
-        s->has_zero_init = false;
     }
 
     if (!s->has_zero_init && !s->target_has_backing &&
@@ -2086,6 +2086,7 @@ static int img_convert(int argc, char **argv)
             {"force-share", no_argument, 0, 'U'},
             {"target-image-opts", no_argument, 0, OPTION_TARGET_IMAGE_OPTS},
             {"salvage", no_argument, 0, OPTION_SALVAGE},
+            {"target-is-zero", no_argument, 0, OPTION_TARGET_IS_ZERO},
             {0, 0, 0, 0}
         };
         c = getopt_long(argc, argv, ":hf:O:B:Cco:l:S:pt:T:qnm:WU",
@@ -2209,6 +2210,14 @@ static int img_convert(int argc, char **argv)
         case OPTION_TARGET_IMAGE_OPTS:
             tgt_image_opts = true;
             break;
+        case OPTION_TARGET_IS_ZERO:
+            /*
+             * The user asserting that the target is blank has the
+             * same effect as the target driver supporting zero
+             * initialisation.
+             */
+            s.has_zero_init = true;
+            break;
         }
     }
 
@@ -2247,6 +2256,11 @@ static int img_convert(int argc, char **argv)
         warn_report("This will become an error in future QEMU versions.");
     }
 
+    if (s.has_zero_init && !skip_create) {
+        error_report("--target-is-zero requires use of -n flag");
+        goto fail_getopt;
+    }
+
     s.src_num = argc - optind - 1;
     out_filename = s.src_num >= 1 ? argv[argc - 1] : NULL;
 
@@ -2380,6 +2394,12 @@ static int img_convert(int argc, char **argv)
     }
     s.target_has_backing = (bool) out_baseimg;
 
+    if (s.has_zero_init && s.target_has_backing) {
+        error_report("Cannot use --target-is-zero when the destination "
+                     "image has a backing file");
+        goto out;
+    }
+
     if (s.src_num > 1 && out_baseimg) {
         error_report("Having a backing file for the target makes no sense when "
                      "concatenating multiple input images");
-- 
2.24.1

[PULL 07/18] block: always fill entire LUKS header space with zeros

[Max Reitz]

From: Daniel P. Berrangé <berrange@redhat.com>

When initializing the LUKS header the size with default encryption
parameters will currently be 2068480 bytes. This is rounded up to
a multiple of the cluster size, 2081792, with 64k sectors. If the
end of the header is not the same as the end of the cluster we fill
the extra space with zeros. This was forgetting that not even the
space allocated for the header will be fully initialized, as we
only write key material for the first key slot. The space left
for the other 7 slots is never written to.

An optimization to the ref count checking code:

  commit a5fff8d4b4d928311a5005efa12d0991fe3b66f9 (refs/bisect/bad)
  Author: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
  Date:   Wed Feb 27 16:14:30 2019 +0300

    qcow2-refcount: avoid eating RAM

made the assumption that every cluster which was allocated would
have at least some data written to it. This was violated by way
the LUKS header is only partially written, with much space simply
reserved for future use.

Depending on the cluster size this problem was masked by the
logic which wrote zeros between the end of the LUKS header and
the end of the cluster.

$ qemu-img create --object secret,id=cluster_encrypt0,data=123456 \
   -f qcow2 -o cluster_size=2k,encrypt.iter-time=1,\
               encrypt.format=luks,encrypt.key-secret=cluster_encrypt0 \
               cluster_size_check.qcow2 100M
  Formatting 'cluster_size_check.qcow2', fmt=qcow2 size=104857600
    encrypt.format=luks encrypt.key-secret=cluster_encrypt0
    encrypt.iter-time=1 cluster_size=2048 lazy_refcounts=off refcount_bits=16

$ qemu-img check --object secret,id=cluster_encrypt0,data=redhat \
    'json:{"driver": "qcow2", "encrypt.format": "luks", \
           "encrypt.key-secret": "cluster_encrypt0", \
           "file.driver": "file", "file.filename": "cluster_size_check.qcow2"}'
ERROR: counting reference for region exceeding the end of the file by one cluster or more: offset 0x2000 size 0x1f9000
Leaked cluster 4 refcount=1 reference=0
...snip...
Leaked cluster 130 refcount=1 reference=0

1 errors were found on the image.
Data may be corrupted, or further writes to the image may corrupt it.

127 leaked clusters were found on the image.
This means waste of disk space, but no harm to data.
Image end offset: 268288

The problem only exists when the disk image is entirely empty. Writing
data to the disk image payload will solve the problem by causing the
end of the file to be extended further.

The change fixes it by ensuring that the entire allocated LUKS header
region is fully initialized with zeros. The qemu-img check will still
fail for any pre-existing disk images created prior to this change,
unless at least 1 byte of the payload is written to.

Fully writing zeros to the entire LUKS header is a good idea regardless
as it ensures that space has been allocated on the host filesystem (or
whatever block storage backend is used).

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20200207135520.2669430-1-berrange@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/qcow2.c              | 11 +++--
 tests/qemu-iotests/284     | 97 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/284.out | 62 ++++++++++++++++++++++++
 tests/qemu-iotests/group   |  1 +
 4 files changed, 167 insertions(+), 4 deletions(-)
 create mode 100755 tests/qemu-iotests/284
 create mode 100644 tests/qemu-iotests/284.out

diff --git a/block/qcow2.c b/block/qcow2.c
index 8dcee5efec..3c754f616b 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -135,13 +135,16 @@ static ssize_t qcow2_crypto_hdr_init_func(QCryptoBlock *block, size_t headerlen,
     s->crypto_header.length = headerlen;
     s->crypto_header.offset = ret;
 
-    /* Zero fill remaining space in cluster so it has predictable
-     * content in case of future spec changes */
+    /*
+     * Zero fill all space in cluster so it has predictable
+     * content, as we may not initialize some regions of the
+     * header (eg only 1 out of 8 key slots will be initialized)
+     */
     clusterlen = size_to_clusters(s, headerlen) * s->cluster_size;
     assert(qcow2_pre_write_overlap_check(bs, 0, ret, clusterlen, false) == 0);
     ret = bdrv_pwrite_zeroes(bs->file,
-                             ret + headerlen,
-                             clusterlen - headerlen, 0);
+                             ret,
+                             clusterlen, 0);
     if (ret < 0) {
         error_setg_errno(errp, -ret, "Could not zero fill encryption header");
         return -1;
diff --git a/tests/qemu-iotests/284 b/tests/qemu-iotests/284
new file mode 100755
index 0000000000..071e89b33e
--- /dev/null
+++ b/tests/qemu-iotests/284
@@ -0,0 +1,97 @@
+#!/usr/bin/env bash
+#
+# Test ref count checks on encrypted images
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=berrange@redhat.com
+
+seq=`basename $0`
+echo "QA output created by $seq"
+
+status=1        # failure is the default!
+
+_cleanup()
+{
+        _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt qcow2
+_supported_proto generic
+_supported_os Linux
+
+
+size=1M
+
+SECRET="secret,id=sec0,data=astrochicken"
+
+IMGSPEC="driver=$IMGFMT,file.filename=$TEST_IMG,encrypt.key-secret=sec0"
+QEMU_IO_OPTIONS=$QEMU_IO_OPTIONS_NO_FMT
+
+_run_test()
+{
+        IMGOPTSSYNTAX=true
+        OLD_TEST_IMG="$TEST_IMG"
+        TEST_IMG="driver=$IMGFMT,file.filename=$TEST_IMG,encrypt.key-secret=sec0"
+        QEMU_IMG_EXTRA_ARGS="--image-opts --object $SECRET"
+
+        echo
+        echo "== cluster size $csize"
+        echo "== checking image refcounts =="
+        _check_test_img
+
+        echo
+        echo "== writing some data =="
+        $QEMU_IO -c "write -P 0x9 0 1"  $QEMU_IMG_EXTRA_ARGS $TEST_IMG | _filter_qemu_io | _filter_testdir
+        echo
+        echo "== rechecking image refcounts =="
+        _check_test_img
+
+        echo
+        echo "== writing some more data =="
+        $QEMU_IO -c "write -P 0x9 $csize 1" $QEMU_IMG_EXTRA_ARGS $TEST_IMG | _filter_qemu_io | _filter_testdir
+        echo
+        echo "== rechecking image refcounts =="
+        _check_test_img
+
+        TEST_IMG="$OLD_TEST_IMG"
+        QEMU_IMG_EXTRA_ARGS=
+        IMGOPTSSYNTAX=
+}
+
+
+echo
+echo "testing LUKS qcow2 encryption"
+echo
+
+for csize in 512 2048 32768
+do
+  _make_test_img --object $SECRET -o "encrypt.format=luks,encrypt.key-secret=sec0,encrypt.iter-time=10,cluster_size=$csize" $size
+  _run_test
+  _cleanup_test_img
+done
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/284.out b/tests/qemu-iotests/284.out
new file mode 100644
index 0000000000..48216f5742
--- /dev/null
+++ b/tests/qemu-iotests/284.out
@@ -0,0 +1,62 @@
+QA output created by 284
+
+testing LUKS qcow2 encryption
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10
+
+== cluster size 512
+== checking image refcounts ==
+No errors were found on the image.
+
+== writing some data ==
+wrote 1/1 bytes at offset 0
+1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== rechecking image refcounts ==
+No errors were found on the image.
+
+== writing some more data ==
+wrote 1/1 bytes at offset 512
+1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== rechecking image refcounts ==
+No errors were found on the image.
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10
+
+== cluster size 2048
+== checking image refcounts ==
+No errors were found on the image.
+
+== writing some data ==
+wrote 1/1 bytes at offset 0
+1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== rechecking image refcounts ==
+No errors were found on the image.
+
+== writing some more data ==
+wrote 1/1 bytes at offset 2048
+1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== rechecking image refcounts ==
+No errors were found on the image.
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=1048576 encrypt.format=luks encrypt.key-secret=sec0 encrypt.iter-time=10
+
+== cluster size 32768
+== checking image refcounts ==
+No errors were found on the image.
+
+== writing some data ==
+wrote 1/1 bytes at offset 0
+1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== rechecking image refcounts ==
+No errors were found on the image.
+
+== writing some more data ==
+wrote 1/1 bytes at offset 32768
+1 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+== rechecking image refcounts ==
+No errors were found on the image.
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 1904223020..818380a8f0 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -290,3 +290,4 @@
 280 rw migration quick
 281 rw quick
 283 auto quick
+284 rw
-- 
2.24.1

[PULL 08/18] block/backup-top: fix flags handling

[Max Reitz]

From: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

backup-top "supports" write-unchanged, by skipping CBW operation in
backup_top_co_pwritev. But it forgets to do the same in
backup_top_co_pwrite_zeroes, as well as declare support for
BDRV_REQ_WRITE_UNCHANGED.

Fix this, and, while being here, declare also support for flags
supported by source child.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20200207161231.32707-1-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/backup-top.c | 31 ++++++++++++++++++++-----------
 1 file changed, 20 insertions(+), 11 deletions(-)

diff --git a/block/backup-top.c b/block/backup-top.c
index fa78f3256d..1bfb360bd3 100644
--- a/block/backup-top.c
+++ b/block/backup-top.c
@@ -48,11 +48,17 @@ static coroutine_fn int backup_top_co_preadv(
 }
 
 static coroutine_fn int backup_top_cbw(BlockDriverState *bs, uint64_t offset,
-                                       uint64_t bytes)
+                                       uint64_t bytes, BdrvRequestFlags flags)
 {
     BDRVBackupTopState *s = bs->opaque;
-    uint64_t end = QEMU_ALIGN_UP(offset + bytes, s->bcs->cluster_size);
-    uint64_t off = QEMU_ALIGN_DOWN(offset, s->bcs->cluster_size);
+    uint64_t off, end;
+
+    if (flags & BDRV_REQ_WRITE_UNCHANGED) {
+        return 0;
+    }
+
+    off = QEMU_ALIGN_DOWN(offset, s->bcs->cluster_size);
+    end = QEMU_ALIGN_UP(offset + bytes, s->bcs->cluster_size);
 
     return block_copy(s->bcs, off, end - off, NULL);
 }
@@ -60,7 +66,7 @@ static coroutine_fn int backup_top_cbw(BlockDriverState *bs, uint64_t offset,
 static int coroutine_fn backup_top_co_pdiscard(BlockDriverState *bs,
                                                int64_t offset, int bytes)
 {
-    int ret = backup_top_cbw(bs, offset, bytes);
+    int ret = backup_top_cbw(bs, offset, bytes, 0);
     if (ret < 0) {
         return ret;
     }
@@ -71,7 +77,7 @@ static int coroutine_fn backup_top_co_pdiscard(BlockDriverState *bs,
 static int coroutine_fn backup_top_co_pwrite_zeroes(BlockDriverState *bs,
         int64_t offset, int bytes, BdrvRequestFlags flags)
 {
-    int ret = backup_top_cbw(bs, offset, bytes);
+    int ret = backup_top_cbw(bs, offset, bytes, flags);
     if (ret < 0) {
         return ret;
     }
@@ -84,11 +90,9 @@ static coroutine_fn int backup_top_co_pwritev(BlockDriverState *bs,
                                               uint64_t bytes,
                                               QEMUIOVector *qiov, int flags)
 {
-    if (!(flags & BDRV_REQ_WRITE_UNCHANGED)) {
-        int ret = backup_top_cbw(bs, offset, bytes);
-        if (ret < 0) {
-            return ret;
-        }
+    int ret = backup_top_cbw(bs, offset, bytes, flags);
+    if (ret < 0) {
+        return ret;
     }
 
     return bdrv_co_pwritev(bs->backing, offset, bytes, qiov, flags);
@@ -196,8 +200,13 @@ BlockDriverState *bdrv_backup_top_append(BlockDriverState *source,
         return NULL;
     }
 
-    top->total_sectors = source->total_sectors;
     state = top->opaque;
+    top->total_sectors = source->total_sectors;
+    top->supported_write_flags = BDRV_REQ_WRITE_UNCHANGED |
+            (BDRV_REQ_FUA & source->supported_write_flags);
+    top->supported_zero_flags = BDRV_REQ_WRITE_UNCHANGED |
+            ((BDRV_REQ_FUA | BDRV_REQ_MAY_UNMAP | BDRV_REQ_NO_FALLBACK) &
+             source->supported_zero_flags);
 
     bdrv_ref(target);
     state->target = bdrv_attach_child(top, target, "target", &child_file, errp);
-- 
2.24.1

[PULL 09/18] iotests/279: Fix for non-qcow2 formats

[Max Reitz]

First, driver=qcow2 will not work so well with non-qcow2 formats (and
this test claims to support qcow, qed, and vmdk).

Second, vmdk will always report the backing file format to be vmdk.
Filter that out so the output looks like for all other formats.

Third, the flat vmdk subformats do not support backing files, so they
will not work with this test.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20191219144243.1763246-1-mreitz@redhat.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/279 | 7 +++++--
 1 file changed, 5 insertions(+), 2 deletions(-)

diff --git a/tests/qemu-iotests/279 b/tests/qemu-iotests/279
index 6682376808..30d29b1cb2 100755
--- a/tests/qemu-iotests/279
+++ b/tests/qemu-iotests/279
@@ -38,6 +38,8 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
 _supported_fmt qcow qcow2 vmdk qed
 _supported_proto file
 _supported_os Linux
+_unsupported_imgopts "subformat=monolithicFlat" \
+                     "subformat=twoGbMaxExtentFlat" \
 
 TEST_IMG="$TEST_IMG.base" _make_test_img 64M
 TEST_IMG="$TEST_IMG.mid" _make_test_img -b "$TEST_IMG.base"
@@ -45,11 +47,12 @@ _make_test_img -b "$TEST_IMG.mid"
 
 echo
 echo '== qemu-img info --backing-chain =='
-_img_info --backing-chain | _filter_img_info
+_img_info --backing-chain | _filter_img_info | grep -v 'backing file format'
 
 echo
 echo '== qemu-img info --backing-chain --image-opts =='
-TEST_IMG="driver=qcow2,file.driver=file,file.filename=$TEST_IMG" _img_info --backing-chain --image-opts | _filter_img_info
+TEST_IMG="driver=$IMGFMT,file.driver=file,file.filename=$TEST_IMG" _img_info --backing-chain --image-opts \
+    | _filter_img_info | grep -v 'backing file format'
 
 # success, all done
 echo "*** done"
-- 
2.24.1

[PULL 10/18] block/nbd: Fix hang in .bdrv_close()

[Max Reitz]

When nbd_close() is called from a coroutine, the connection_co never
gets to run, and thus nbd_teardown_connection() hangs.

This is because aio_co_enter() only puts the connection_co into the main
coroutine's wake-up queue, so this main coroutine needs to yield and
wait for connection_co to terminate.

Suggested-by: Kevin Wolf <kwolf@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200122164532.178040-2-mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/nbd.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/block/nbd.c b/block/nbd.c
index d085554f21..6d3b22f844 100644
--- a/block/nbd.c
+++ b/block/nbd.c
@@ -70,6 +70,7 @@ typedef struct BDRVNBDState {
     CoMutex send_mutex;
     CoQueue free_sema;
     Coroutine *connection_co;
+    Coroutine *teardown_co;
     QemuCoSleepState *connection_co_sleep_ns_state;
     bool drained;
     bool wait_drained_end;
@@ -203,7 +204,15 @@ static void nbd_teardown_connection(BlockDriverState *bs)
             qemu_co_sleep_wake(s->connection_co_sleep_ns_state);
         }
     }
-    BDRV_POLL_WHILE(bs, s->connection_co);
+    if (qemu_in_coroutine()) {
+        s->teardown_co = qemu_coroutine_self();
+        /* connection_co resumes us when it terminates */
+        qemu_coroutine_yield();
+        s->teardown_co = NULL;
+    } else {
+        BDRV_POLL_WHILE(bs, s->connection_co);
+    }
+    assert(!s->connection_co);
 }
 
 static bool nbd_client_connecting(BDRVNBDState *s)
@@ -395,6 +404,9 @@ static coroutine_fn void nbd_connection_entry(void *opaque)
         s->ioc = NULL;
     }
 
+    if (s->teardown_co) {
+        aio_co_wake(s->teardown_co);
+    }
     aio_wait_kick();
 }
 
-- 
2.24.1

[PULL 11/18] block: Generic file creation fallback

[Max Reitz]

If a protocol driver does not support image creation, we can see whether
maybe the file exists already.  If so, just truncating it will be
sufficient.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200122164532.178040-3-mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block.c | 159 +++++++++++++++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 147 insertions(+), 12 deletions(-)

diff --git a/block.c b/block.c
index 08372ced26..308a91c96b 100644
--- a/block.c
+++ b/block.c
@@ -532,20 +532,139 @@ out:
     return ret;
 }
 
-int bdrv_create_file(const char *filename, QemuOpts *opts, Error **errp)
+/**
+ * Helper function for bdrv_create_file_fallback(): Resize @blk to at
+ * least the given @minimum_size.
+ *
+ * On success, return @blk's actual length.
+ * Otherwise, return -errno.
+ */
+static int64_t create_file_fallback_truncate(BlockBackend *blk,
+                                             int64_t minimum_size, Error **errp)
 {
-    BlockDriver *drv;
+    Error *local_err = NULL;
+    int64_t size;
+    int ret;
+
+    ret = blk_truncate(blk, minimum_size, false, PREALLOC_MODE_OFF, &local_err);
+    if (ret < 0 && ret != -ENOTSUP) {
+        error_propagate(errp, local_err);
+        return ret;
+    }
+
+    size = blk_getlength(blk);
+    if (size < 0) {
+        error_free(local_err);
+        error_setg_errno(errp, -size,
+                         "Failed to inquire the new image file's length");
+        return size;
+    }
+
+    if (size < minimum_size) {
+        /* Need to grow the image, but we failed to do that */
+        error_propagate(errp, local_err);
+        return -ENOTSUP;
+    }
+
+    error_free(local_err);
+    local_err = NULL;
+
+    return size;
+}
+
+/**
+ * Helper function for bdrv_create_file_fallback(): Zero the first
+ * sector to remove any potentially pre-existing image header.
+ */
+static int create_file_fallback_zero_first_sector(BlockBackend *blk,
+                                                  int64_t current_size,
+                                                  Error **errp)
+{
+    int64_t bytes_to_clear;
+    int ret;
+
+    bytes_to_clear = MIN(current_size, BDRV_SECTOR_SIZE);
+    if (bytes_to_clear) {
+        ret = blk_pwrite_zeroes(blk, 0, bytes_to_clear, BDRV_REQ_MAY_UNMAP);
+        if (ret < 0) {
+            error_setg_errno(errp, -ret,
+                             "Failed to clear the new image's first sector");
+            return ret;
+        }
+    }
+
+    return 0;
+}
+
+static int bdrv_create_file_fallback(const char *filename, BlockDriver *drv,
+                                     QemuOpts *opts, Error **errp)
+{
+    BlockBackend *blk;
+    QDict *options = qdict_new();
+    int64_t size = 0;
+    char *buf = NULL;
+    PreallocMode prealloc;
     Error *local_err = NULL;
     int ret;
 
+    size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
+    buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
+    prealloc = qapi_enum_parse(&PreallocMode_lookup, buf,
+                               PREALLOC_MODE_OFF, &local_err);
+    g_free(buf);
+    if (local_err) {
+        error_propagate(errp, local_err);
+        return -EINVAL;
+    }
+
+    if (prealloc != PREALLOC_MODE_OFF) {
+        error_setg(errp, "Unsupported preallocation mode '%s'",
+                   PreallocMode_str(prealloc));
+        return -ENOTSUP;
+    }
+
+    qdict_put_str(options, "driver", drv->format_name);
+
+    blk = blk_new_open(filename, NULL, options,
+                       BDRV_O_RDWR | BDRV_O_RESIZE, errp);
+    if (!blk) {
+        error_prepend(errp, "Protocol driver '%s' does not support image "
+                      "creation, and opening the image failed: ",
+                      drv->format_name);
+        return -EINVAL;
+    }
+
+    size = create_file_fallback_truncate(blk, size, errp);
+    if (size < 0) {
+        ret = size;
+        goto out;
+    }
+
+    ret = create_file_fallback_zero_first_sector(blk, size, errp);
+    if (ret < 0) {
+        goto out;
+    }
+
+    ret = 0;
+out:
+    blk_unref(blk);
+    return ret;
+}
+
+int bdrv_create_file(const char *filename, QemuOpts *opts, Error **errp)
+{
+    BlockDriver *drv;
+
     drv = bdrv_find_protocol(filename, true, errp);
     if (drv == NULL) {
         return -ENOENT;
     }
 
-    ret = bdrv_create(drv, filename, opts, &local_err);
-    error_propagate(errp, local_err);
-    return ret;
+    if (drv->bdrv_co_create_opts) {
+        return bdrv_create(drv, filename, opts, errp);
+    } else {
+        return bdrv_create_file_fallback(filename, drv, opts, errp);
+    }
 }
 
 /**
@@ -1444,6 +1563,24 @@ QemuOptsList bdrv_runtime_opts = {
     },
 };
 
+static QemuOptsList fallback_create_opts = {
+    .name = "fallback-create-opts",
+    .head = QTAILQ_HEAD_INITIALIZER(fallback_create_opts.head),
+    .desc = {
+        {
+            .name = BLOCK_OPT_SIZE,
+            .type = QEMU_OPT_SIZE,
+            .help = "Virtual disk size"
+        },
+        {
+            .name = BLOCK_OPT_PREALLOC,
+            .type = QEMU_OPT_STRING,
+            .help = "Preallocation mode (allowed values: off)"
+        },
+        { /* end of list */ }
+    }
+};
+
 /*
  * Common part for opening disk images and files
  *
@@ -5772,15 +5909,13 @@ void bdrv_img_create(const char *filename, const char *fmt,
         return;
     }
 
-    if (!proto_drv->create_opts) {
-        error_setg(errp, "Protocol driver '%s' does not support image creation",
-                   proto_drv->format_name);
-        return;
-    }
-
     /* Create parameter list */
     create_opts = qemu_opts_append(create_opts, drv->create_opts);
-    create_opts = qemu_opts_append(create_opts, proto_drv->create_opts);
+    if (proto_drv->create_opts) {
+        create_opts = qemu_opts_append(create_opts, proto_drv->create_opts);
+    } else {
+        create_opts = qemu_opts_append(create_opts, &fallback_create_opts);
+    }
 
     opts = qemu_opts_create(create_opts, NULL, 0, &error_abort);
 
-- 
2.24.1

[PULL 12/18] file-posix: Drop hdev_co_create_opts()

[Max Reitz]

The generic fallback implementation effectively does the same.

Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200122164532.178040-4-mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/file-posix.c | 67 ----------------------------------------------
 1 file changed, 67 deletions(-)

diff --git a/block/file-posix.c b/block/file-posix.c
index ab82ee1a67..6345477112 100644
--- a/block/file-posix.c
+++ b/block/file-posix.c
@@ -3477,67 +3477,6 @@ static coroutine_fn int hdev_co_pwrite_zeroes(BlockDriverState *bs,
     return raw_do_pwrite_zeroes(bs, offset, bytes, flags, true);
 }
 
-static int coroutine_fn hdev_co_create_opts(const char *filename, QemuOpts *opts,
-                                            Error **errp)
-{
-    int fd;
-    int ret = 0;
-    struct stat stat_buf;
-    int64_t total_size = 0;
-    bool has_prefix;
-
-    /* This function is used by both protocol block drivers and therefore either
-     * of these prefixes may be given.
-     * The return value has to be stored somewhere, otherwise this is an error
-     * due to -Werror=unused-value. */
-    has_prefix =
-        strstart(filename, "host_device:", &filename) ||
-        strstart(filename, "host_cdrom:" , &filename);
-
-    (void)has_prefix;
-
-    ret = raw_normalize_devicepath(&filename, errp);
-    if (ret < 0) {
-        return ret;
-    }
-
-    /* Read out options */
-    total_size = ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
-                          BDRV_SECTOR_SIZE);
-
-    fd = qemu_open(filename, O_WRONLY | O_BINARY);
-    if (fd < 0) {
-        ret = -errno;
-        error_setg_errno(errp, -ret, "Could not open device");
-        return ret;
-    }
-
-    if (fstat(fd, &stat_buf) < 0) {
-        ret = -errno;
-        error_setg_errno(errp, -ret, "Could not stat device");
-    } else if (!S_ISBLK(stat_buf.st_mode) && !S_ISCHR(stat_buf.st_mode)) {
-        error_setg(errp,
-                   "The given file is neither a block nor a character device");
-        ret = -ENODEV;
-    } else if (lseek(fd, 0, SEEK_END) < total_size) {
-        error_setg(errp, "Device is too small");
-        ret = -ENOSPC;
-    }
-
-    if (!ret && total_size) {
-        uint8_t buf[BDRV_SECTOR_SIZE] = { 0 };
-        int64_t zero_size = MIN(BDRV_SECTOR_SIZE, total_size);
-        if (lseek(fd, 0, SEEK_SET) == -1) {
-            ret = -errno;
-        } else {
-            ret = qemu_write_full(fd, buf, zero_size);
-            ret = ret == zero_size ? 0 : -errno;
-        }
-    }
-    qemu_close(fd);
-    return ret;
-}
-
 static BlockDriver bdrv_host_device = {
     .format_name        = "host_device",
     .protocol_name        = "host_device",
@@ -3550,8 +3489,6 @@ static BlockDriver bdrv_host_device = {
     .bdrv_reopen_prepare = raw_reopen_prepare,
     .bdrv_reopen_commit  = raw_reopen_commit,
     .bdrv_reopen_abort   = raw_reopen_abort,
-    .bdrv_co_create_opts = hdev_co_create_opts,
-    .create_opts         = &raw_create_opts,
     .mutable_opts        = mutable_opts,
     .bdrv_co_invalidate_cache = raw_co_invalidate_cache,
     .bdrv_co_pwrite_zeroes = hdev_co_pwrite_zeroes,
@@ -3678,8 +3615,6 @@ static BlockDriver bdrv_host_cdrom = {
     .bdrv_reopen_prepare = raw_reopen_prepare,
     .bdrv_reopen_commit  = raw_reopen_commit,
     .bdrv_reopen_abort   = raw_reopen_abort,
-    .bdrv_co_create_opts = hdev_co_create_opts,
-    .create_opts         = &raw_create_opts,
     .mutable_opts        = mutable_opts,
     .bdrv_co_invalidate_cache = raw_co_invalidate_cache,
 
@@ -3812,8 +3747,6 @@ static BlockDriver bdrv_host_cdrom = {
     .bdrv_reopen_prepare = raw_reopen_prepare,
     .bdrv_reopen_commit  = raw_reopen_commit,
     .bdrv_reopen_abort   = raw_reopen_abort,
-    .bdrv_co_create_opts = hdev_co_create_opts,
-    .create_opts        = &raw_create_opts,
     .mutable_opts       = mutable_opts,
 
     .bdrv_co_preadv         = raw_co_preadv,
-- 
2.24.1

[PULL 13/18] iscsi: Drop iscsi_co_create_opts()

[Max Reitz]

The generic fallback implementation effectively does the same.

Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200122164532.178040-5-mreitz@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/iscsi.c | 56 ---------------------------------------------------
 1 file changed, 56 deletions(-)

diff --git a/block/iscsi.c b/block/iscsi.c
index c8feaa2f0e..682abd8e09 100644
--- a/block/iscsi.c
+++ b/block/iscsi.c
@@ -2164,58 +2164,6 @@ static int coroutine_fn iscsi_co_truncate(BlockDriverState *bs, int64_t offset,
     return 0;
 }
 
-static int coroutine_fn iscsi_co_create_opts(const char *filename, QemuOpts *opts,
-                                             Error **errp)
-{
-    int ret = 0;
-    int64_t total_size = 0;
-    BlockDriverState *bs;
-    IscsiLun *iscsilun = NULL;
-    QDict *bs_options;
-    Error *local_err = NULL;
-
-    bs = bdrv_new();
-
-    /* Read out options */
-    total_size = DIV_ROUND_UP(qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0),
-                              BDRV_SECTOR_SIZE);
-    bs->opaque = g_new0(struct IscsiLun, 1);
-    iscsilun = bs->opaque;
-
-    bs_options = qdict_new();
-    iscsi_parse_filename(filename, bs_options, &local_err);
-    if (local_err) {
-        error_propagate(errp, local_err);
-        ret = -EINVAL;
-    } else {
-        ret = iscsi_open(bs, bs_options, 0, NULL);
-    }
-    qobject_unref(bs_options);
-
-    if (ret != 0) {
-        goto out;
-    }
-    iscsi_detach_aio_context(bs);
-    if (iscsilun->type != TYPE_DISK) {
-        ret = -ENODEV;
-        goto out;
-    }
-    if (bs->total_sectors < total_size) {
-        ret = -ENOSPC;
-        goto out;
-    }
-
-    ret = 0;
-out:
-    if (iscsilun->iscsi != NULL) {
-        iscsi_destroy_context(iscsilun->iscsi);
-    }
-    g_free(bs->opaque);
-    bs->opaque = NULL;
-    bdrv_unref(bs);
-    return ret;
-}
-
 static int iscsi_get_info(BlockDriverState *bs, BlockDriverInfo *bdi)
 {
     IscsiLun *iscsilun = bs->opaque;
@@ -2486,8 +2434,6 @@ static BlockDriver bdrv_iscsi = {
     .bdrv_parse_filename    = iscsi_parse_filename,
     .bdrv_file_open         = iscsi_open,
     .bdrv_close             = iscsi_close,
-    .bdrv_co_create_opts    = iscsi_co_create_opts,
-    .create_opts            = &iscsi_create_opts,
     .bdrv_reopen_prepare    = iscsi_reopen_prepare,
     .bdrv_reopen_commit     = iscsi_reopen_commit,
     .bdrv_co_invalidate_cache = iscsi_co_invalidate_cache,
@@ -2525,8 +2471,6 @@ static BlockDriver bdrv_iser = {
     .bdrv_parse_filename    = iscsi_parse_filename,
     .bdrv_file_open         = iscsi_open,
     .bdrv_close             = iscsi_close,
-    .bdrv_co_create_opts    = iscsi_co_create_opts,
-    .create_opts            = &iscsi_create_opts,
     .bdrv_reopen_prepare    = iscsi_reopen_prepare,
     .bdrv_reopen_commit     = iscsi_reopen_commit,
     .bdrv_co_invalidate_cache  = iscsi_co_invalidate_cache,
-- 
2.24.1

[PULL 14/18] iotests: Add test for image creation fallback

[Max Reitz]

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200122164532.178040-6-mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Maxim Levitsky <mlevitsk@redhat.com>
[mreitz: Added a note that NBD does not support resizing, which is why
         the second case is expected to fail]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/259     | 62 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/259.out | 14 +++++++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 77 insertions(+)
 create mode 100755 tests/qemu-iotests/259
 create mode 100644 tests/qemu-iotests/259.out

diff --git a/tests/qemu-iotests/259 b/tests/qemu-iotests/259
new file mode 100755
index 0000000000..62e29af05f
--- /dev/null
+++ b/tests/qemu-iotests/259
@@ -0,0 +1,62 @@
+#!/usr/bin/env bash
+#
+# Test generic image creation fallback (by using NBD)
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=mreitz@redhat.com
+
+seq=$(basename $0)
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+
+_supported_fmt raw
+_supported_proto nbd
+_supported_os Linux
+
+
+_make_test_img 64M
+
+echo
+echo '--- Testing creation ---'
+
+$QEMU_IMG create -f qcow2 "$TEST_IMG" 64M | _filter_img_create
+$QEMU_IMG info "$TEST_IMG" | _filter_img_info
+
+echo
+echo '--- Testing creation for which the node would need to grow ---'
+
+# NBD does not support resizing, so this will fail
+$QEMU_IMG create -f qcow2 -o preallocation=metadata "$TEST_IMG" 64M 2>&1 \
+    | _filter_img_create
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/259.out b/tests/qemu-iotests/259.out
new file mode 100644
index 0000000000..ffed19c2a0
--- /dev/null
+++ b/tests/qemu-iotests/259.out
@@ -0,0 +1,14 @@
+QA output created by 259
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+
+--- Testing creation ---
+Formatting 'TEST_DIR/t.IMGFMT', fmt=qcow2 size=67108864
+image: TEST_DIR/t.IMGFMT
+file format: qcow2
+virtual size: 64 MiB (67108864 bytes)
+disk size: unavailable
+
+--- Testing creation for which the node would need to grow ---
+qemu-img: TEST_DIR/t.IMGFMT: Could not resize image: Image format driver does not support resize
+Formatting 'TEST_DIR/t.IMGFMT', fmt=qcow2 size=67108864 preallocation=metadata
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 818380a8f0..b17711d17d 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -273,6 +273,7 @@
 256 rw auto quick
 257 rw
 258 rw quick
+259 rw auto quick
 260 rw quick
 261 rw
 262 rw quick migration
-- 
2.24.1

[PULL 15/18] qemu-img: Fix convert -n -B for backing-less targets

[Max Reitz]

s.target_has_backing does not reflect whether the target BDS has a
backing file; it only tells whether we should use a backing file during
conversion (specified by -B).

As such, if you use convert -n, the target does not necessarily actually
have a backing file, and then dereferencing out_bs->backing fails here.

When converting to an existing file, we should set
target_backing_sectors to a negative value, because first, as the
comment explains, this value is only used for optimization, so it is
always fine to do that.

Second, we use this value to determine where the target must be
initialized to zeroes (overlays are initialized to zero after the end of
their backing file).  When converting to an existing file, we cannot
assume that to be true.

Cc: qemu-stable@nongnu.org
Fixes: 351c8efff9ad809c822d55620df54d575d536f68
       ("qemu-img: Special post-backing convert handling")
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200121155915.98232-2-mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 qemu-img.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/qemu-img.c b/qemu-img.c
index 0faf2cd2f5..804630a368 100644
--- a/qemu-img.c
+++ b/qemu-img.c
@@ -2523,7 +2523,7 @@ static int img_convert(int argc, char **argv)
         }
     }
 
-    if (s.target_has_backing) {
+    if (s.target_has_backing && s.target_is_new) {
         /* Errors are treated as "backing length unknown" (which means
          * s.target_backing_sectors has to be negative, which it will
          * be automatically).  The backing file length is used only
-- 
2.24.1

[PULL 16/18] iotests: Test convert -n -B to backing-less target

[Max Reitz]

This must not crash.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200121155915.98232-3-mreitz@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/122     | 14 ++++++++++++++
 tests/qemu-iotests/122.out |  5 +++++
 2 files changed, 19 insertions(+)

diff --git a/tests/qemu-iotests/122 b/tests/qemu-iotests/122
index dfa350936f..f7a3ae684a 100755
--- a/tests/qemu-iotests/122
+++ b/tests/qemu-iotests/122
@@ -276,6 +276,20 @@ $QEMU_IMG convert -O $IMGFMT -n "$TEST_IMG" "$TEST_IMG".orig
 
 $QEMU_IMG compare "$TEST_IMG" "$TEST_IMG".orig
 
+echo
+echo '=== -n -B to an image without a backing file ==='
+echo
+
+# Base for the output
+TEST_IMG="$TEST_IMG".base _make_test_img 64M
+
+# Output that does have $TEST_IMG.base set as its (implicit) backing file
+TEST_IMG="$TEST_IMG".orig _make_test_img 64M
+
+# Convert with -n, which should not confuse -B with "target BDS has a
+# backing file"
+$QEMU_IMG convert -O $IMGFMT -B "$TEST_IMG".base -n "$TEST_IMG" "$TEST_IMG".orig
+
 # success, all done
 echo '*** done'
 rm -f $seq.full
diff --git a/tests/qemu-iotests/122.out b/tests/qemu-iotests/122.out
index 849b6cc2ef..1a35951a80 100644
--- a/tests/qemu-iotests/122.out
+++ b/tests/qemu-iotests/122.out
@@ -228,4 +228,9 @@ Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864
 wrote 65536/65536 bytes at offset 0
 64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 Images are identical.
+
+=== -n -B to an image without a backing file ===
+
+Formatting 'TEST_DIR/t.IMGFMT.base', fmt=IMGFMT size=67108864
+Formatting 'TEST_DIR/t.IMGFMT.orig', fmt=IMGFMT size=67108864
 *** done
-- 
2.24.1

[PULL 17/18] block: Fix VM size field width in snapshot dump

[Max Reitz]

When printing the snapshot list (e.g. with qemu-img snapshot -l), the VM
size field is only seven characters wide.  As of de38b5005e9, this is
not necessarily sufficient: We generally print three digits, and this
may require a decimal point.  Also, the unit field grew from something
as plain as "M" to " MiB".  This means that number and unit may take up
eight characters in total; but we also want spaces in front.

Considering previously the maximum width was four characters and the
field width was chosen to be three characters wider, let us adjust the
field width to be eleven now.

Fixes: de38b5005e946aa3714963ea4c501e279e7d3666
Buglink: https://bugs.launchpad.net/qemu/+bug/1859989
Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200117105859.241818-2-mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 block/qapi.c | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/block/qapi.c b/block/qapi.c
index 3f09477cc5..afd9f3b4a7 100644
--- a/block/qapi.c
+++ b/block/qapi.c
@@ -664,7 +664,7 @@ void bdrv_snapshot_dump(QEMUSnapshotInfo *sn)
     char *sizing = NULL;
 
     if (!sn) {
-        qemu_printf("%-10s%-20s%7s%20s%15s",
+        qemu_printf("%-10s%-20s%11s%20s%15s",
                     "ID", "TAG", "VM SIZE", "DATE", "VM CLOCK");
     } else {
         ti = sn->date_sec;
@@ -679,7 +679,7 @@ void bdrv_snapshot_dump(QEMUSnapshotInfo *sn)
                  (int)(secs % 60),
                  (int)((sn->vm_clock_nsec / 1000000) % 1000));
         sizing = size_to_str(sn->vm_state_size);
-        qemu_printf("%-10s%-20s%7s%20s%15s",
+        qemu_printf("%-10s%-20s%11s%20s%15s",
                     sn->id_str, sn->name,
                     sizing,
                     date_buf,
-- 
2.24.1

[PULL 18/18] iotests: Test snapshot -l field separation

[Max Reitz]

Add a test that all fields in "qemu-img snapshot -l"s output are
separated by spaces.

Signed-off-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20200117105859.241818-3-mreitz@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
[mreitz: Renamed test from 284 to 286]
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
 tests/qemu-iotests/286     | 76 ++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/286.out |  8 ++++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 85 insertions(+)
 create mode 100755 tests/qemu-iotests/286
 create mode 100644 tests/qemu-iotests/286.out

diff --git a/tests/qemu-iotests/286 b/tests/qemu-iotests/286
new file mode 100755
index 0000000000..f14445ba4a
--- /dev/null
+++ b/tests/qemu-iotests/286
@@ -0,0 +1,76 @@
+#!/usr/bin/env bash
+#
+# Test qemu-img snapshot -l
+#
+# Copyright (C) 2019 Red Hat, Inc.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+seq=$(basename "$0")
+echo "QA output created by $seq"
+
+status=1	# failure is the default!
+
+_cleanup()
+{
+    _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common.rc
+. ./common.filter
+. ./common.qemu
+
+_supported_fmt qcow2
+_supported_proto file
+# Internal snapshots are (currently) impossible with refcount_bits=1,
+# and generally impossible with external data files
+_unsupported_imgopts 'refcount_bits=1[^0-9]' data_file
+
+_make_test_img 64M
+
+# Should be so long as to take up the whole field width
+sn_name=abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
+
+# More memory will give us a larger VM state, i.e. one above 1 MB.
+# This way, we get a number with a decimal point.
+qemu_comm_method=monitor _launch_qemu -m 512 "$TEST_IMG"
+
+_send_qemu_cmd $QEMU_HANDLE "savevm $sn_name" '(qemu)'
+_send_qemu_cmd $QEMU_HANDLE 'quit' '(qemu)'
+wait=yes _cleanup_qemu
+
+# Check that all fields are separated by spaces.
+# We first collapse all space sequences into one space each;
+# then we turn every space-separated field into a '.';
+# and finally, we name the '.'s so the output is not just a confusing
+# sequence of dots.
+
+echo 'Output structure:'
+$QEMU_IMG snapshot -l "$TEST_IMG" | tail -n 1 | tr -s ' ' \
+    | sed -e 's/\S\+/./g' \
+    | sed -e 's/\./(snapshot ID)/' \
+          -e 's/\./(snapshot name)/' \
+          -e 's/\./(VM state size value)/' \
+          -e 's/\./(VM state size unit)/' \
+          -e 's/\./(snapshot date)/' \
+          -e 's/\./(snapshot time)/' \
+          -e 's/\./(VM clock)/'
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/286.out b/tests/qemu-iotests/286.out
new file mode 100644
index 0000000000..39ff07e12c
--- /dev/null
+++ b/tests/qemu-iotests/286.out
@@ -0,0 +1,8 @@
+QA output created by 286
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+QEMU X.Y.Z monitor - type 'help' for more information
+(qemu) savevm abcdefghijklmnopqrstuvwxyzabcdefghijklmnopqrstuvwxyz
+(qemu) quit
+Output structure:
+(snapshot ID) (snapshot name) (VM state size value) (VM state size unit) (snapshot date) (snapshot time) (VM clock)
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index b17711d17d..0317667695 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -292,3 +292,4 @@
 281 rw quick
 283 auto quick
 284 rw
+286 rw quick
-- 
2.24.1

Re: [PULL 06/18] qemu-img: Add --target-is-zero to convert

[Eric Blake]

On 2/20/20 10:06 AM, Max Reitz wrote:
> From: David Edmondson <david.edmondson@oracle.com>
> 
> In many cases the target of a convert operation is a newly provisioned
> target that the user knows is blank (reads as zero). In this situation
> there is no requirement for qemu-img to wastefully zero out the entire
> device.
> 
> Add a new option, --target-is-zero, allowing the user to indicate that
> an existing target device will return zeros for all reads.
> 
> Signed-off-by: David Edmondson <david.edmondson@oracle.com>
> Message-Id: <20200205110248.2009589-2-david.edmondson@oracle.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Reviewed-by: Eric Blake <eblake@redhat.com>
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
>   docs/interop/qemu-img.rst |  9 ++++++++-
>   qemu-img-cmds.hx          |  4 ++--
>   qemu-img.c                | 26 +++++++++++++++++++++++---
>   3 files changed, 33 insertions(+), 6 deletions(-)
> 
> diff --git a/docs/interop/qemu-img.rst b/docs/interop/qemu-img.rst
> index 42e4451db4..5f40137c10 100644
> --- a/docs/interop/qemu-img.rst
> +++ b/docs/interop/qemu-img.rst
> @@ -214,6 +214,13 @@ Parameters to convert subcommand:
>     will still be printed.  Areas that cannot be read from the source will be
>     treated as containing only zeroes.
>   
> +.. option:: --target-is-zero
> +
> +  Assume that reading the destination image will always return
> +  zeros. This parameter is mutually exclusive with a destination image

Late tweak now that this is in a pull request, so we may want a followup 
patch, but:

The image doesn't always return zeros after we write to it, maybe we 
should tweak this sentence:

Assume that reading the destination image will initially return all zeros.

Also, my earlier comment about 'zeroes' one line before 'zeros' still 
applies - although both spellings are valid, we look inconsistent when 
we can't make up our mind within two adjacent paragraphs.

-- 
Eric Blake, Principal Software Engineer
Red Hat, Inc.           +1-919-301-3226
Virtualization:  qemu.org | libvirt.org

Re: [PULL 06/18] qemu-img: Add --target-is-zero to convert

[David Edmondson]

On Thursday, 2020-02-20 at 10:36:04 -06, Eric Blake wrote:

> On 2/20/20 10:06 AM, Max Reitz wrote:
>> From: David Edmondson <david.edmondson@oracle.com>
>> 
>> In many cases the target of a convert operation is a newly provisioned
>> target that the user knows is blank (reads as zero). In this situation
>> there is no requirement for qemu-img to wastefully zero out the entire
>> device.
>> 
>> Add a new option, --target-is-zero, allowing the user to indicate that
>> an existing target device will return zeros for all reads.
>> 
>> Signed-off-by: David Edmondson <david.edmondson@oracle.com>
>> Message-Id: <20200205110248.2009589-2-david.edmondson@oracle.com>
>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> Reviewed-by: Eric Blake <eblake@redhat.com>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>>   docs/interop/qemu-img.rst |  9 ++++++++-
>>   qemu-img-cmds.hx          |  4 ++--
>>   qemu-img.c                | 26 +++++++++++++++++++++++---
>>   3 files changed, 33 insertions(+), 6 deletions(-)
>> 
>> diff --git a/docs/interop/qemu-img.rst b/docs/interop/qemu-img.rst
>> index 42e4451db4..5f40137c10 100644
>> --- a/docs/interop/qemu-img.rst
>> +++ b/docs/interop/qemu-img.rst
>> @@ -214,6 +214,13 @@ Parameters to convert subcommand:
>>     will still be printed.  Areas that cannot be read from the source will be
>>     treated as containing only zeroes.
>>   
>> +.. option:: --target-is-zero
>> +
>> +  Assume that reading the destination image will always return
>> +  zeros. This parameter is mutually exclusive with a destination image
>
> Late tweak now that this is in a pull request, so we may want a followup 
> patch, but:
>
> The image doesn't always return zeros after we write to it, maybe we 
> should tweak this sentence:
>
> Assume that reading the destination image will initially return all zeros.

I will send a patch for this.

> Also, my earlier comment about 'zeroes' one line before 'zeros' still 
> applies - although both spellings are valid, we look inconsistent when 
> we can't make up our mind within two adjacent paragraphs.

If we can agree on one of "zeros" or "zeroes" then I'm happy to send a
patch making it consistent everywhere.

I think that given there are existing functions with "zeroes" in the
name, I'd be inclined to go that way.

dme.
-- 
Why stay in college? Why go to night school?

Re: [PULL 00/18] Block patches

[Peter Maydell]

On Thu, 20 Feb 2020 at 16:07, Max Reitz <mreitz@redhat.com> wrote:
>
> The following changes since commit 672f9d0df10a68a5c5f2b32cbc8284abf9f5ee18:
>
>   Merge remote-tracking branch 'remotes/kevin/tags/for-upstream' into staging (2020-02-18 14:23:43 +0000)
>
> are available in the Git repository at:
>
>   https://github.com/XanClic/qemu.git tags/pull-block-2020-02-20
>
> for you to fetch changes up to dff8d44c96f128480430b0c59ed8760917dbd427:
>
>   iotests: Test snapshot -l field separation (2020-02-20 16:43:42 +0100)
>
> ----------------------------------------------------------------
> Block patches:
> - qemu-img convert: New --target-is-zero parameter
> - qcow2: Specify non-default compression type flag
> - optionally flat output for query-named-block-nodes
> - some fixes
> - pseudo-creation of images on block devices is now done by a generic
>   block layer function
>
> ----------------------------------------------------------------



Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.0
for any user-visible changes.

-- PMM

Re: [PULL 11/18] block: Generic file creation fallback

[Peter Maydell]

On Thu, 20 Feb 2020 at 16:11, Max Reitz <mreitz@redhat.com> wrote:
>
> If a protocol driver does not support image creation, we can see whether
> maybe the file exists already.  If so, just truncating it will be
> sufficient.
>
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> Message-Id: <20200122164532.178040-3-mreitz@redhat.com>
> Signed-off-by: Max Reitz <mreitz@redhat.com>

Hi; Coverity thinks there's a memory leak in the error
codepaths in this function (CID 1419884): is it right?

> +static int bdrv_create_file_fallback(const char *filename, BlockDriver *drv,
> +                                     QemuOpts *opts, Error **errp)
> +{
> +    BlockBackend *blk;
> +    QDict *options = qdict_new();

We create the QDict here...

> +    int64_t size = 0;
> +    char *buf = NULL;
> +    PreallocMode prealloc;
>      Error *local_err = NULL;
>      int ret;
>
> +    size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
> +    buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
> +    prealloc = qapi_enum_parse(&PreallocMode_lookup, buf,
> +                               PREALLOC_MODE_OFF, &local_err);
> +    g_free(buf);
> +    if (local_err) {
> +        error_propagate(errp, local_err);
> +        return -EINVAL;

...but here and in other error return paths we don't
free it (I think that might need a qobject_unref() but
am not sure).

> +    }
> +
> +    if (prealloc != PREALLOC_MODE_OFF) {
> +        error_setg(errp, "Unsupported preallocation mode '%s'",
> +                   PreallocMode_str(prealloc));
> +        return -ENOTSUP;
> +    }

(You could probably postpone qdict_new() to here to avoid
having to change the error handling paths above this point, but
you still need to deal with the error path for blk_new_open failing.)

> +
> +    qdict_put_str(options, "driver", drv->format_name);
> +
> +    blk = blk_new_open(filename, NULL, options,
> +                       BDRV_O_RDWR | BDRV_O_RESIZE, errp);
> +    if (!blk) {
> +        error_prepend(errp, "Protocol driver '%s' does not support image "
> +                      "creation, and opening the image failed: ",
> +                      drv->format_name);
> +        return -EINVAL;
> +    }

I guess for error-paths after blk_new_open() succeeds
that the blk object owns the options dictionary and
will deal with unreffing it for us?

> +
> +    size = create_file_fallback_truncate(blk, size, errp);
> +    if (size < 0) {
> +        ret = size;
> +        goto out;
> +    }
> +
> +    ret = create_file_fallback_zero_first_sector(blk, size, errp);
> +    if (ret < 0) {
> +        goto out;
> +    }
> +
> +    ret = 0;
> +out:
> +    blk_unref(blk);
> +    return ret;
> +}

thanks
-- PMM

Re: [PULL 11/18] block: Generic file creation fallback

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 3114 bytes --]

On 25.02.20 11:05, Peter Maydell wrote:
> On Thu, 20 Feb 2020 at 16:11, Max Reitz <mreitz@redhat.com> wrote:
>>
>> If a protocol driver does not support image creation, we can see whether
>> maybe the file exists already.  If so, just truncating it will be
>> sufficient.
>>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> Message-Id: <20200122164532.178040-3-mreitz@redhat.com>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
> 
> Hi; Coverity thinks there's a memory leak in the error
> codepaths in this function (CID 1419884): is it right?

Yes, it is, I’ll write a patch.

>> +static int bdrv_create_file_fallback(const char *filename, BlockDriver *drv,
>> +                                     QemuOpts *opts, Error **errp)
>> +{
>> +    BlockBackend *blk;
>> +    QDict *options = qdict_new();
> 
> We create the QDict here...
> 
>> +    int64_t size = 0;
>> +    char *buf = NULL;
>> +    PreallocMode prealloc;
>>      Error *local_err = NULL;
>>      int ret;
>>
>> +    size = qemu_opt_get_size_del(opts, BLOCK_OPT_SIZE, 0);
>> +    buf = qemu_opt_get_del(opts, BLOCK_OPT_PREALLOC);
>> +    prealloc = qapi_enum_parse(&PreallocMode_lookup, buf,
>> +                               PREALLOC_MODE_OFF, &local_err);
>> +    g_free(buf);
>> +    if (local_err) {
>> +        error_propagate(errp, local_err);
>> +        return -EINVAL;
> 
> ...but here and in other error return paths we don't
> free it (I think that might need a qobject_unref() but
> am not sure).
> 
>> +    }
>> +
>> +    if (prealloc != PREALLOC_MODE_OFF) {
>> +        error_setg(errp, "Unsupported preallocation mode '%s'",
>> +                   PreallocMode_str(prealloc));
>> +        return -ENOTSUP;
>> +    }
> 
> (You could probably postpone qdict_new() to here to avoid
> having to change the error handling paths above this point, but
> you still need to deal with the error path for blk_new_open failing.)

Indeed.  Or maybe put the unref() under the out label and set @options
to NULL after it’s captured by @blk.  I’ll see.

>> +
>> +    qdict_put_str(options, "driver", drv->format_name);
>> +
>> +    blk = blk_new_open(filename, NULL, options,
>> +                       BDRV_O_RDWR | BDRV_O_RESIZE, errp);
>> +    if (!blk) {
>> +        error_prepend(errp, "Protocol driver '%s' does not support image "
>> +                      "creation, and opening the image failed: ",
>> +                      drv->format_name);
>> +        return -EINVAL;
>> +    }
> 
> I guess for error-paths after blk_new_open() succeeds
> that the blk object owns the options dictionary and
> will deal with unreffing it for us?

Yes.

Max

>> +
>> +    size = create_file_fallback_truncate(blk, size, errp);
>> +    if (size < 0) {
>> +        ret = size;
>> +        goto out;
>> +    }
>> +
>> +    ret = create_file_fallback_zero_first_sector(blk, size, errp);
>> +    if (ret < 0) {
>> +        goto out;
>> +    }
>> +
>> +    ret = 0;
>> +out:
>> +    blk_unref(blk);
>> +    return ret;
>> +}
> 
> thanks
> -- PMM
> 



[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PULL 00/18] Block patches

[Richard Henderson]

On 6/15/22 08:51, Stefan Hajnoczi wrote:
> The following changes since commit 8e6c70b9d4a1b1f3011805947925cfdb31642f7f:
> 
>    Merge tag 'kraxel-20220614-pull-request' of git://git.kraxel.org/qemu into staging (2022-06-14 06:21:46 -0700)
> 
> are available in the Git repository at:
> 
>    https://gitlab.com/stefanha/qemu.git tags/block-pull-request
> 
> for you to fetch changes up to 99b969fbe105117f5af6060d3afef40ca39cc9c1:
> 
>    linux-aio: explain why max batch is checked in laio_io_unplug() (2022-06-15 16:43:42 +0100)
> 
> ----------------------------------------------------------------
> Pull request
> 
> This pull request includes an important aio=native I/O stall fix, the
> experimental vifo-user server, the io_uring_register_ring_fd() optimization for
> aio=io_uring, and an update to Vladimir Sementsov-Ogievskiy's maintainership
> details.

Applied, thanks.  Please update https://wiki.qemu.org/ChangeLog/7.1 as appropriate.


r~


> 
> ----------------------------------------------------------------
> 
> Jagannathan Raman (14):
>    qdev: unplug blocker for devices
>    remote/machine: add HotplugHandler for remote machine
>    remote/machine: add vfio-user property
>    vfio-user: build library
>    vfio-user: define vfio-user-server object
>    vfio-user: instantiate vfio-user context
>    vfio-user: find and init PCI device
>    vfio-user: run vfio-user context
>    vfio-user: handle PCI config space accesses
>    vfio-user: IOMMU support for remote device
>    vfio-user: handle DMA mappings
>    vfio-user: handle PCI BAR accesses
>    vfio-user: handle device interrupts
>    vfio-user: handle reset of remote device
> 
> Sam Li (1):
>    Use io_uring_register_ring_fd() to skip fd operations
> 
> Stefan Hajnoczi (2):
>    linux-aio: fix unbalanced plugged counter in laio_io_unplug()
>    linux-aio: explain why max batch is checked in laio_io_unplug()
> 
> Vladimir Sementsov-Ogievskiy (1):
>    MAINTAINERS: update Vladimir's address and repositories
> 
>   MAINTAINERS                             |  27 +-
>   meson_options.txt                       |   2 +
>   qapi/misc.json                          |  31 +
>   qapi/qom.json                           |  20 +-
>   configure                               |  17 +
>   meson.build                             |  24 +-
>   include/exec/memory.h                   |   3 +
>   include/hw/pci/msi.h                    |   1 +
>   include/hw/pci/msix.h                   |   1 +
>   include/hw/pci/pci.h                    |  13 +
>   include/hw/qdev-core.h                  |  29 +
>   include/hw/remote/iommu.h               |  40 +
>   include/hw/remote/machine.h             |   4 +
>   include/hw/remote/vfio-user-obj.h       |   6 +
>   block/io_uring.c                        |  12 +-
>   block/linux-aio.c                       |  10 +-
>   hw/core/qdev.c                          |  24 +
>   hw/pci/msi.c                            |  49 +-
>   hw/pci/msix.c                           |  35 +-
>   hw/pci/pci.c                            |  13 +
>   hw/remote/iommu.c                       | 131 ++++
>   hw/remote/machine.c                     |  88 ++-
>   hw/remote/vfio-user-obj.c               | 958 ++++++++++++++++++++++++
>   softmmu/physmem.c                       |   4 +-
>   softmmu/qdev-monitor.c                  |   4 +
>   stubs/vfio-user-obj.c                   |   6 +
>   tests/qtest/fuzz/generic_fuzz.c         |   9 +-
>   .gitlab-ci.d/buildtest.yml              |   1 +
>   .gitmodules                             |   3 +
>   Kconfig.host                            |   4 +
>   hw/remote/Kconfig                       |   4 +
>   hw/remote/meson.build                   |   4 +
>   hw/remote/trace-events                  |  11 +
>   scripts/meson-buildoptions.sh           |   4 +
>   stubs/meson.build                       |   1 +
>   subprojects/libvfio-user                |   1 +
>   tests/docker/dockerfiles/centos8.docker |   2 +
>   37 files changed, 1565 insertions(+), 31 deletions(-)
>   create mode 100644 include/hw/remote/iommu.h
>   create mode 100644 include/hw/remote/vfio-user-obj.h
>   create mode 100644 hw/remote/iommu.c
>   create mode 100644 hw/remote/vfio-user-obj.c
>   create mode 100644 stubs/vfio-user-obj.c
>   create mode 160000 subprojects/libvfio-user
> 

[PULL 00/18] Block patches

[Stefan Hajnoczi]

The following changes since commit 8e6c70b9d4a1b1f3011805947925cfdb31642f7f:

  Merge tag 'kraxel-20220614-pull-request' of git://git.kraxel.org/qemu into staging (2022-06-14 06:21:46 -0700)

are available in the Git repository at:

  https://gitlab.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 99b969fbe105117f5af6060d3afef40ca39cc9c1:

  linux-aio: explain why max batch is checked in laio_io_unplug() (2022-06-15 16:43:42 +0100)

----------------------------------------------------------------
Pull request

This pull request includes an important aio=native I/O stall fix, the
experimental vifo-user server, the io_uring_register_ring_fd() optimization for
aio=io_uring, and an update to Vladimir Sementsov-Ogievskiy's maintainership
details.

----------------------------------------------------------------

Jagannathan Raman (14):
  qdev: unplug blocker for devices
  remote/machine: add HotplugHandler for remote machine
  remote/machine: add vfio-user property
  vfio-user: build library
  vfio-user: define vfio-user-server object
  vfio-user: instantiate vfio-user context
  vfio-user: find and init PCI device
  vfio-user: run vfio-user context
  vfio-user: handle PCI config space accesses
  vfio-user: IOMMU support for remote device
  vfio-user: handle DMA mappings
  vfio-user: handle PCI BAR accesses
  vfio-user: handle device interrupts
  vfio-user: handle reset of remote device

Sam Li (1):
  Use io_uring_register_ring_fd() to skip fd operations

Stefan Hajnoczi (2):
  linux-aio: fix unbalanced plugged counter in laio_io_unplug()
  linux-aio: explain why max batch is checked in laio_io_unplug()

Vladimir Sementsov-Ogievskiy (1):
  MAINTAINERS: update Vladimir's address and repositories

 MAINTAINERS                             |  27 +-
 meson_options.txt                       |   2 +
 qapi/misc.json                          |  31 +
 qapi/qom.json                           |  20 +-
 configure                               |  17 +
 meson.build                             |  24 +-
 include/exec/memory.h                   |   3 +
 include/hw/pci/msi.h                    |   1 +
 include/hw/pci/msix.h                   |   1 +
 include/hw/pci/pci.h                    |  13 +
 include/hw/qdev-core.h                  |  29 +
 include/hw/remote/iommu.h               |  40 +
 include/hw/remote/machine.h             |   4 +
 include/hw/remote/vfio-user-obj.h       |   6 +
 block/io_uring.c                        |  12 +-
 block/linux-aio.c                       |  10 +-
 hw/core/qdev.c                          |  24 +
 hw/pci/msi.c                            |  49 +-
 hw/pci/msix.c                           |  35 +-
 hw/pci/pci.c                            |  13 +
 hw/remote/iommu.c                       | 131 ++++
 hw/remote/machine.c                     |  88 ++-
 hw/remote/vfio-user-obj.c               | 958 ++++++++++++++++++++++++
 softmmu/physmem.c                       |   4 +-
 softmmu/qdev-monitor.c                  |   4 +
 stubs/vfio-user-obj.c                   |   6 +
 tests/qtest/fuzz/generic_fuzz.c         |   9 +-
 .gitlab-ci.d/buildtest.yml              |   1 +
 .gitmodules                             |   3 +
 Kconfig.host                            |   4 +
 hw/remote/Kconfig                       |   4 +
 hw/remote/meson.build                   |   4 +
 hw/remote/trace-events                  |  11 +
 scripts/meson-buildoptions.sh           |   4 +
 stubs/meson.build                       |   1 +
 subprojects/libvfio-user                |   1 +
 tests/docker/dockerfiles/centos8.docker |   2 +
 37 files changed, 1565 insertions(+), 31 deletions(-)
 create mode 100644 include/hw/remote/iommu.h
 create mode 100644 include/hw/remote/vfio-user-obj.h
 create mode 100644 hw/remote/iommu.c
 create mode 100644 hw/remote/vfio-user-obj.c
 create mode 100644 stubs/vfio-user-obj.c
 create mode 160000 subprojects/libvfio-user

-- 
2.36.1

Re: [PULL 00/18] Block patches

[Thomas Huth]

On 24/06/2020 09.27, Max Reitz wrote:
> On 23.06.20 14:55, Peter Maydell wrote:
>> On Mon, 22 Jun 2020 at 16:11, Max Reitz <mreitz@redhat.com> wrote:
>>>
>>> The following changes since commit bae31bfa48b9caecee25da3d5333901a126a06b4:
>>>
>>>    Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200619-pull-request' into staging (2020-06-19 22:56:59 +0100)
>>>
>>> are available in the Git repository at:
>>>
>>>    https://github.com/XanClic/qemu.git tags/pull-block-2020-06-22
>>>
>>> for you to fetch changes up to 74c55e4142a7bb835c38d3770c74210cbb1e4fab:
>>>
>>>    iotests: don't test qcow2.py inside 291 (2020-06-22 16:05:23 +0200)
>>>
>>> ----------------------------------------------------------------
>>> Block patches:
>>> - Support modifying a LUKS-encrypted image's keyslots
>>> - iotest fixes
>>>
>>> ----------------------------------------------------------------
>>
>> Hi; I see various iotest failures, different things on
>> PPC64 Linux, OpenBSD and FreeBSD, and on an AArch32 build
>> that happens to not have optional crypto libs installed.
> 
> OK.  Sorry.  None of this looks easily fixable, so I suppose I’ll have
> to drop everything but the last two patches for now.

At least the problem with "seq" in test 293 should be easy to fix, since 
we're requiring bash for the iotests. See e.g. commit 30edd9fa50e86fbf 
as an example.

  Thomas

Re: [PULL 00/18] Block patches

[Max Reitz]

[-- Attachment #1.1: Type: text/plain, Size: 15994 bytes --]

On 23.06.20 14:55, Peter Maydell wrote:
> On Mon, 22 Jun 2020 at 16:11, Max Reitz <mreitz@redhat.com> wrote:
>>
>> The following changes since commit bae31bfa48b9caecee25da3d5333901a126a06b4:
>>
>>   Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200619-pull-request' into staging (2020-06-19 22:56:59 +0100)
>>
>> are available in the Git repository at:
>>
>>   https://github.com/XanClic/qemu.git tags/pull-block-2020-06-22
>>
>> for you to fetch changes up to 74c55e4142a7bb835c38d3770c74210cbb1e4fab:
>>
>>   iotests: don't test qcow2.py inside 291 (2020-06-22 16:05:23 +0200)
>>
>> ----------------------------------------------------------------
>> Block patches:
>> - Support modifying a LUKS-encrypted image's keyslots
>> - iotest fixes
>>
>> ----------------------------------------------------------------
> 
> Hi; I see various iotest failures, different things on
> PPC64 Linux, OpenBSD and FreeBSD, and on an AArch32 build
> that happens to not have optional crypto libs installed.

OK.  Sorry.  None of this looks easily fixable, so I suppose I’ll have
to drop everything but the last two patches for now.

> On PPC64 Linux, lots of iotests fail like this:
> 
>   TEST    iotest-qcow2: 001 [fail]
> QEMU          --
> "/home/pm215/qemu/build/all/tests/qemu-iotests/../../ppc64-softmmu/qemu-system-ppc64"
> -nodefaults -display none -accel qtest
> QEMU_IMG      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-io"  --cache
> writeback --aio threads -f qcow2
> QEMU_NBD      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- Linux/ppc64 gcc1-power7 3.10.0-862.14.4.el7.ppc64
> TEST_DIR      -- /home/pm215/qemu/build/all/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.vvBdnkatyZ
> SOCKET_SCM_HELPER --
> /home/pm215/qemu/build/all/tests/qemu-iotests/socket_scm_helper
> 
> --- /home/pm215/qemu/tests/qemu-iotests/001.out 2015-04-08
> 18:43:24.908449234 +0000
> +++ /home/pm215/qemu/build/all/tests/qemu-iotests/001.out.bad
> 2020-06-23 10:42:29.262626202 +0000
> @@ -1,5 +1,6 @@
>  QA output created by 001
> -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
> +./common.filter: line 128: readarray: -d: invalid option
> +readarray: usage: readarray [-n count] [-O origin] [-s count] [-t]
> [-u fd] [-C callback] [-c quantum] [array]
> 
>  == reading whole image ==
>  read 134217728/134217728 bytes at offset 0
>   TEST    iotest-qcow2: 002 [fail]
> QEMU          --
> "/home/pm215/qemu/build/all/tests/qemu-iotests/../../ppc64-softmmu/qemu-system-ppc64"
> -nodefaults -display none -accel qtest
> QEMU_IMG      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-io"  --cache
> writeback --aio threads -f qcow2
> QEMU_NBD      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- Linux/ppc64 gcc1-power7 3.10.0-862.14.4.el7.ppc64
> TEST_DIR      -- /home/pm215/qemu/build/all/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.vvBdnkatyZ
> SOCKET_SCM_HELPER --
> /home/pm215/qemu/build/all/tests/qemu-iotests/socket_scm_helper
> 
> Looks like you're trying to use a readarray option that doesn't
> exist (maybe only exists in newer shells?)

Yes, I am.  Well, that’s a real shame. :/

I wasn’t aware that readarray’s -d option was only introduced at some
later point.  Looks like that happened in 4.4, which was only released
2016.  So that’s probably too new indeed.

> iotests failures on aarch32 which happens to not have some
> optional crypto lib dependency installed I guess; these
> iotests ought to be made to skip if the functionality they're
> testing isn't compiled into this QEMU:

Right.  Maxim, can you do that?

>   TEST    iotest-qcow2: 293 [fail]
> QEMU          --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
> -nodefaults -display none -machine virt -accel qtest
> QEMU_IMG      --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-io"
>  --cache writeback --aio threads -f qcow2
> QEMU_NBD      --
> "/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- Linux/aarch64 mustang-maydell 4.15.0-101-generic
> TEST_DIR      --
> /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.tjBWiNDvED
> SOCKET_SCM_HELPER --
> /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/socket_scm_helper
> 
> --- /home/peter.maydell/qemu/tests/qemu-iotests/293.out 2020-06-23
> 10:38:50.091867725 +0000
> +++ /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/293.out.bad
>       2020-06-23 10:59:09.027627162 +0000
> @@ -1,5 +1,6 @@
>  QA output created by 293
>  == creating a test image ==
> +qemu-img: TEST_DIR/t.IMGFMT: No crypto library supporting PBKDF in
> this build: Function not implemented
>  Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432
> 
>  == test that key 0 opens the image ==
> @@ -7,10 +8,15 @@
>  4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> 
>  == adding a password to slot 4 ==
> +qemu-img: Can't amend encryption options - encryption not present
>  == adding a password to slot 1 ==
> +qemu-img: Can't amend encryption options - encryption not present
>  == adding a password to slot 3 ==
> +qemu-img: Can't amend encryption options - encryption not present
>  == adding a password to slot 2 ==
> +qemu-img: Can't amend encryption options - encryption not present
>  == erase slot 4 ==
> +qemu-img: Can't amend encryption options - encryption not present
> 
>  == all secrets should work ==
>  read 4096/4096 bytes at offset 0
> @@ -23,19 +29,29 @@
>  4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> 
>  == erase slot 0 and try it ==
> -qemu-io: can't open: Invalid password, cannot unlock any keyslot
> +qemu-img: Can't amend encryption options - encryption not present
> +read 4096/4096 bytes at offset 0
> +4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> [etc]
> 
> 
> On OpenBSD: iotest 293 fails trying to use "seq" when it doesn't exist:

Well that’s nice.  Maxim, I think we’ll need a for (()) loop then (like
e.g. tests/qemu-iotests/common.pattern:25 does it).

>   TEST    iotest-qcow2: 293 [fail]
> QEMU          --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
> -nodef
> aults -display none -machine virt -accel qtest
> QEMU_IMG      --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-io"
> --cache writeback --aio threads -
> f qcow2
> QEMU_NBD      --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- OpenBSD/amd64 openbsd 6.6
> TEST_DIR      -- /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.o4s3SCLIsx
> SOCKET_SCM_HELPER --
> 
> --- /home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/293.out  Tue
> Jun 23 10:38:33 2020
> +++ /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/293.out.bad
>  Tue Jun 23 11:06:19 2020
> @@ -1,6 +1,6 @@
>  QA output created by 293
>  == creating a test image ==
> -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432
> +Formatting 'TEST_DIR/t.IMGFMT'x0IMGFMT encrypt.format=luks
> encrypt.key-secret=sec0 encrypt.iter-time=10 cluster_size=65
> 536 compression_type=zlib size=33554432 compat=1.1 lazy_refcounts=off
> refcount_bits=16
> 
>  == test that key 0 opens the image ==
>  read 4096/4096 bytes at offset 0
> @@ -29,25 +29,26 @@
>  qemu-io: can't open: Invalid password, cannot unlock any keyslot
> 
>  == filling  4 slots with secret 2 ==
> +./293: line 111: seq: command not found
> 
>  == adding secret 0 ==
> 
>  == adding secret 3 (last slot) ==
> 
>  == trying to add another slot (should fail) ==
> -qemu-img: Can't add a keyslot - all keyslots are in use
> +qemu-img: Could not open
> 'driver=qcow2,file.filename=/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/scratch/t.qco
> w2,encrypt.key-secret=sec2': Invalid password, cannot unlock any keyslot
> 
>  == all secrets should work again ==
>  read 4096/4096 bytes at offset 0
>  4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>  read 4096/4096 bytes at offset 0
>  4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +qemu-io: can't open: Invalid password, cannot unlock any keyslot
>  read 4096/4096 bytes at offset 0
>  4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> -read 4096/4096 bytes at offset 0
> -4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> 
>  == erase all keys of secret 2==
> +qemu-img: No keyslots match given (old) password for erase operation
>  == erase all keys of secret 1==
>  == erase all keys of secret 0==
>  == erasing secret3 will fail now since it is the only secret (in 3 slots) ==
> 
> Also OpenBSD, iotest 295 failed because the QEMU dumped core:

I assume the SIGKILL occurred because 295 somewhere waits on some QMP
event that never happens (probably because the encryption is again not
supported), so the iotest kills it on timeout.

(Maxim, I think that we’ll need something similar to what we need in 293
to skip the test if there’s no crypto library.)

>   TEST    iotest-qcow2: 295 [fail]
> QEMU          --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
> -nodef
> aults -display none -machine virt -accel qtest
> QEMU_IMG      --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-io"
> --cache writeback --aio threads -
> f qcow2
> QEMU_NBD      --
> "/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- OpenBSD/amd64 openbsd 6.6
> TEST_DIR      -- /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.o4s3SCLIsx
> SOCKET_SCM_HELPER --
> 
> --- /home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/295.out  Tue
> Jun 23 10:38:33 2020
> +++ /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/295.out.bad
>  Tue Jun 23 11:09:01 2020
> @@ -4,19 +4,11 @@
>  {"return": {}}
>  {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
>  {"return": {}}
> +WARNING:qemu.machine:qemu received signal 9:
> /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64
> -display none -vga none -chardev
> socket,id=mon,path=/tmp/tmp.o4s3SCLIsx/qemu-36824-monitor.sock -mon
> chardev=mon,mode=control -qtest
> unix:path=/tmp/tmp.o4s3SCLIsx/qemu-36824-qtest.sock -accel qtest
> -nodefaults -display none -machine virt -accel qtest
>  {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
>  {"return": {}}
> -{"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
> -{"return": {}}
> -Job failed: Invalid password, cannot unlock any keyslot
>  {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
>  {"return": {}}
> -{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
> -{"return": {}}
> -{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
> -{"return": {}}
> -{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
> -{"return": {}}
>  Job failed: Refusing to overwrite active keyslot 2 - please erase it first
>  {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
>  {"return": {}}
> @@ -33,8 +25,37 @@
>  {"return": {}}
>  {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
>  {"return": {}}
> -...
> +.E.
> +======================================================================
> +ERROR: testOldPassword (__main__.EncryptionSetupTestCase)
>  ----------------------------------------------------------------------
> +Traceback (most recent call last):
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
> line 139, in __get_events
> +    ret = self.__json_read(only_event=True)
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
> line 99, in __json_read
> +    data = self.__sockfile.readline()
> +  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
> +    return self._sock.recv_into(b)
> +socket.timeout: timed out
> +
> +During handling of the above exception, another exception occurred:
> +
> +Traceback (most recent call last):
> +  File "295", line 234, in testOldPassword
> +    self.addKeyQmp("testdev", new_secret = self.secrets[1])
> +  File "295", line 164, in addKeyQmp
> +    self.vm.run_job('job_add_key')
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/iotests.py",
> line 648, in run_job
> +    ev = filter_qmp_event(self.events_wait(events, timeout=wait))
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/machine.py",
> line 518, in events_wait
> +    event = self._qmp.pull_event(wait=timeout)
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
> line 254, in pull_event
> +    self.__get_events(wait)
> +  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
> line 141, in __get_events
> +    raise QMPTimeoutError("Timeout waiting for event")
> +qemu.qmp.QMPTimeoutError: Timeout waiting for event
> +
> +----------------------------------------------------------------------
>  Ran 3 tests
> 
> -OK
> +FAILED (errors=1)
> 
> FreeBSD: lots of iotest failures, looks like some sed or other
> command trying to sanitize output is broken on this host OS
> (assuming gnu sed or whatever, maybe?)

It’s assuming that \x0 produces a NUL character.  It’s a shame it
doesn’t, but I suppose it doesn’t even really matter without readarray
supporting -d.

>   TEST    iotest-qcow2: 001 [fail]
> QEMU          --
> "/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
> -nodefaults -display none -machine virt -accel qtest
> QEMU_IMG      --
> "/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../qemu-img"
> QEMU_IO       --
> "/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../qemu-io"
> --cache writeback --aio threads -f qcow2
> QEMU_NBD      --
> "/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../qemu-nbd"
> IMGFMT        -- qcow2 (compat=1.1)
> IMGPROTO      -- file
> PLATFORM      -- FreeBSD/amd64 freebsd 12.1-RELEASE
> TEST_DIR      -- /home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/scratch
> SOCK_DIR      -- /tmp/tmp.l5ksaQ9N
> SOCKET_SCM_HELPER --
> 
> --- /home/qemu/qemu-test.RNYtim/src/tests/qemu-iotests/001.out
> 2020-06-23 10:38:33.000000000 +0000
> +++ /home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/001.out.bad
>  2020-06-23 10:56:47.833020000 +0000
> @@ -1,5 +1,5 @@
>  QA output created by 001
> -Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
> +Formatting 'TEST_DIR/t.IMGFMT'x0IMGFMT cluster_size=65536
> compression_type=zlib size=134217728 compat=1.1 lazy_refcounts=off
> refcount_bits=16
> 
>  == reading whole image ==
>  read 134217728/134217728 bytes at offset 0
> 
> 
> thanks

Thank you!

Max


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]

Re: [PULL 00/18] Block patches

[Peter Maydell]

On Mon, 22 Jun 2020 at 16:11, Max Reitz <mreitz@redhat.com> wrote:
>
> The following changes since commit bae31bfa48b9caecee25da3d5333901a126a06b4:
>
>   Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200619-pull-request' into staging (2020-06-19 22:56:59 +0100)
>
> are available in the Git repository at:
>
>   https://github.com/XanClic/qemu.git tags/pull-block-2020-06-22
>
> for you to fetch changes up to 74c55e4142a7bb835c38d3770c74210cbb1e4fab:
>
>   iotests: don't test qcow2.py inside 291 (2020-06-22 16:05:23 +0200)
>
> ----------------------------------------------------------------
> Block patches:
> - Support modifying a LUKS-encrypted image's keyslots
> - iotest fixes
>
> ----------------------------------------------------------------

Hi; I see various iotest failures, different things on
PPC64 Linux, OpenBSD and FreeBSD, and on an AArch32 build
that happens to not have optional crypto libs installed.

On PPC64 Linux, lots of iotests fail like this:

  TEST    iotest-qcow2: 001 [fail]
QEMU          --
"/home/pm215/qemu/build/all/tests/qemu-iotests/../../ppc64-softmmu/qemu-system-ppc64"
-nodefaults -display none -accel qtest
QEMU_IMG      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-io"  --cache
writeback --aio threads -f qcow2
QEMU_NBD      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- Linux/ppc64 gcc1-power7 3.10.0-862.14.4.el7.ppc64
TEST_DIR      -- /home/pm215/qemu/build/all/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.vvBdnkatyZ
SOCKET_SCM_HELPER --
/home/pm215/qemu/build/all/tests/qemu-iotests/socket_scm_helper

--- /home/pm215/qemu/tests/qemu-iotests/001.out 2015-04-08
18:43:24.908449234 +0000
+++ /home/pm215/qemu/build/all/tests/qemu-iotests/001.out.bad
2020-06-23 10:42:29.262626202 +0000
@@ -1,5 +1,6 @@
 QA output created by 001
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
+./common.filter: line 128: readarray: -d: invalid option
+readarray: usage: readarray [-n count] [-O origin] [-s count] [-t]
[-u fd] [-C callback] [-c quantum] [array]

 == reading whole image ==
 read 134217728/134217728 bytes at offset 0
  TEST    iotest-qcow2: 002 [fail]
QEMU          --
"/home/pm215/qemu/build/all/tests/qemu-iotests/../../ppc64-softmmu/qemu-system-ppc64"
-nodefaults -display none -accel qtest
QEMU_IMG      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-io"  --cache
writeback --aio threads -f qcow2
QEMU_NBD      -- "/home/pm215/qemu/build/all/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- Linux/ppc64 gcc1-power7 3.10.0-862.14.4.el7.ppc64
TEST_DIR      -- /home/pm215/qemu/build/all/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.vvBdnkatyZ
SOCKET_SCM_HELPER --
/home/pm215/qemu/build/all/tests/qemu-iotests/socket_scm_helper

Looks like you're trying to use a readarray option that doesn't
exist (maybe only exists in newer shells?)

iotests failures on aarch32 which happens to not have some
optional crypto lib dependency installed I guess; these
iotests ought to be made to skip if the functionality they're
testing isn't compiled into this QEMU:

  TEST    iotest-qcow2: 293 [fail]
QEMU          --
"/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
-nodefaults -display none -machine virt -accel qtest
QEMU_IMG      --
"/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-io"
 --cache writeback --aio threads -f qcow2
QEMU_NBD      --
"/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- Linux/aarch64 mustang-maydell 4.15.0-101-generic
TEST_DIR      --
/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.tjBWiNDvED
SOCKET_SCM_HELPER --
/home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/socket_scm_helper

--- /home/peter.maydell/qemu/tests/qemu-iotests/293.out 2020-06-23
10:38:50.091867725 +0000
+++ /home/peter.maydell/qemu/build/all-a32/tests/qemu-iotests/293.out.bad
      2020-06-23 10:59:09.027627162 +0000
@@ -1,5 +1,6 @@
 QA output created by 293
 == creating a test image ==
+qemu-img: TEST_DIR/t.IMGFMT: No crypto library supporting PBKDF in
this build: Function not implemented
 Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432

 == test that key 0 opens the image ==
@@ -7,10 +8,15 @@
 4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

 == adding a password to slot 4 ==
+qemu-img: Can't amend encryption options - encryption not present
 == adding a password to slot 1 ==
+qemu-img: Can't amend encryption options - encryption not present
 == adding a password to slot 3 ==
+qemu-img: Can't amend encryption options - encryption not present
 == adding a password to slot 2 ==
+qemu-img: Can't amend encryption options - encryption not present
 == erase slot 4 ==
+qemu-img: Can't amend encryption options - encryption not present

 == all secrets should work ==
 read 4096/4096 bytes at offset 0
@@ -23,19 +29,29 @@
 4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

 == erase slot 0 and try it ==
-qemu-io: can't open: Invalid password, cannot unlock any keyslot
+qemu-img: Can't amend encryption options - encryption not present
+read 4096/4096 bytes at offset 0
+4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
[etc]


On OpenBSD: iotest 293 fails trying to use "seq" when it doesn't exist:
  TEST    iotest-qcow2: 293 [fail]
QEMU          --
"/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
-nodef
aults -display none -machine virt -accel qtest
QEMU_IMG      --
"/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-io"
--cache writeback --aio threads -
f qcow2
QEMU_NBD      --
"/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- OpenBSD/amd64 openbsd 6.6
TEST_DIR      -- /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.o4s3SCLIsx
SOCKET_SCM_HELPER --

--- /home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/293.out  Tue
Jun 23 10:38:33 2020
+++ /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/293.out.bad
 Tue Jun 23 11:06:19 2020
@@ -1,6 +1,6 @@
 QA output created by 293
 == creating a test image ==
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=33554432
+Formatting 'TEST_DIR/t.IMGFMT'x0IMGFMT encrypt.format=luks
encrypt.key-secret=sec0 encrypt.iter-time=10 cluster_size=65
536 compression_type=zlib size=33554432 compat=1.1 lazy_refcounts=off
refcount_bits=16

 == test that key 0 opens the image ==
 read 4096/4096 bytes at offset 0
@@ -29,25 +29,26 @@
 qemu-io: can't open: Invalid password, cannot unlock any keyslot

 == filling  4 slots with secret 2 ==
+./293: line 111: seq: command not found

 == adding secret 0 ==

 == adding secret 3 (last slot) ==

 == trying to add another slot (should fail) ==
-qemu-img: Can't add a keyslot - all keyslots are in use
+qemu-img: Could not open
'driver=qcow2,file.filename=/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/scratch/t.qco
w2,encrypt.key-secret=sec2': Invalid password, cannot unlock any keyslot

 == all secrets should work again ==
 read 4096/4096 bytes at offset 0
 4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
 read 4096/4096 bytes at offset 0
 4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+qemu-io: can't open: Invalid password, cannot unlock any keyslot
 read 4096/4096 bytes at offset 0
 4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-read 4096/4096 bytes at offset 0
-4 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

 == erase all keys of secret 2==
+qemu-img: No keyslots match given (old) password for erase operation
 == erase all keys of secret 1==
 == erase all keys of secret 0==
 == erasing secret3 will fail now since it is the only secret (in 3 slots) ==

Also OpenBSD, iotest 295 failed because the QEMU dumped core:
  TEST    iotest-qcow2: 295 [fail]
QEMU          --
"/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
-nodef
aults -display none -machine virt -accel qtest
QEMU_IMG      --
"/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-io"
--cache writeback --aio threads -
f qcow2
QEMU_NBD      --
"/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- OpenBSD/amd64 openbsd 6.6
TEST_DIR      -- /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.o4s3SCLIsx
SOCKET_SCM_HELPER --

--- /home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/295.out  Tue
Jun 23 10:38:33 2020
+++ /home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/295.out.bad
 Tue Jun 23 11:09:01 2020
@@ -4,19 +4,11 @@
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
 {"return": {}}
+WARNING:qemu.machine:qemu received signal 9:
/home/qemu/qemu-test.DyTmtb/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64
-display none -vga none -chardev
socket,id=mon,path=/tmp/tmp.o4s3SCLIsx/qemu-36824-monitor.sock -mon
chardev=mon,mode=control -qtest
unix:path=/tmp/tmp.o4s3SCLIsx/qemu-36824-qtest.sock -accel qtest
-nodefaults -display none -machine virt -accel qtest
 {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
 {"return": {}}
-{"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
-{"return": {}}
-Job failed: Invalid password, cannot unlock any keyslot
 {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
 {"return": {}}
-{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
-{"return": {}}
-{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
-{"return": {}}
-{"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
-{"return": {}}
 Job failed: Refusing to overwrite active keyslot 2 - please erase it first
 {"execute": "job-dismiss", "arguments": {"id": "job_add_key"}}
 {"return": {}}
@@ -33,8 +25,37 @@
 {"return": {}}
 {"execute": "job-dismiss", "arguments": {"id": "job_erase_key"}}
 {"return": {}}
-...
+.E.
+======================================================================
+ERROR: testOldPassword (__main__.EncryptionSetupTestCase)
 ----------------------------------------------------------------------
+Traceback (most recent call last):
+  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
line 139, in __get_events
+    ret = self.__json_read(only_event=True)
+  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
line 99, in __json_read
+    data = self.__sockfile.readline()
+  File "/usr/local/lib/python3.7/socket.py", line 589, in readinto
+    return self._sock.recv_into(b)
+socket.timeout: timed out
+
+During handling of the above exception, another exception occurred:
+
+Traceback (most recent call last):
+  File "295", line 234, in testOldPassword
+    self.addKeyQmp("testdev", new_secret = self.secrets[1])
+  File "295", line 164, in addKeyQmp
+    self.vm.run_job('job_add_key')
+  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/iotests.py",
line 648, in run_job
+    ev = filter_qmp_event(self.events_wait(events, timeout=wait))
+  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/machine.py",
line 518, in events_wait
+    event = self._qmp.pull_event(wait=timeout)
+  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
line 254, in pull_event
+    self.__get_events(wait)
+  File "/home/qemu/qemu-test.DyTmtb/src/tests/qemu-iotests/../../python/qemu/qmp.py",
line 141, in __get_events
+    raise QMPTimeoutError("Timeout waiting for event")
+qemu.qmp.QMPTimeoutError: Timeout waiting for event
+
+----------------------------------------------------------------------
 Ran 3 tests

-OK
+FAILED (errors=1)

FreeBSD: lots of iotest failures, looks like some sed or other
command trying to sanitize output is broken on this host OS
(assuming gnu sed or whatever, maybe?)

  TEST    iotest-qcow2: 001 [fail]
QEMU          --
"/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../aarch64-softmmu/qemu-system-aarch64"
-nodefaults -display none -machine virt -accel qtest
QEMU_IMG      --
"/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../qemu-img"
QEMU_IO       --
"/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../qemu-io"
--cache writeback --aio threads -f qcow2
QEMU_NBD      --
"/home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/../../qemu-nbd"
IMGFMT        -- qcow2 (compat=1.1)
IMGPROTO      -- file
PLATFORM      -- FreeBSD/amd64 freebsd 12.1-RELEASE
TEST_DIR      -- /home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/scratch
SOCK_DIR      -- /tmp/tmp.l5ksaQ9N
SOCKET_SCM_HELPER --

--- /home/qemu/qemu-test.RNYtim/src/tests/qemu-iotests/001.out
2020-06-23 10:38:33.000000000 +0000
+++ /home/qemu/qemu-test.RNYtim/build/tests/qemu-iotests/001.out.bad
 2020-06-23 10:56:47.833020000 +0000
@@ -1,5 +1,5 @@
 QA output created by 001
-Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=134217728
+Formatting 'TEST_DIR/t.IMGFMT'x0IMGFMT cluster_size=65536
compression_type=zlib size=134217728 compat=1.1 lazy_refcounts=off
refcount_bits=16

 == reading whole image ==
 read 134217728/134217728 bytes at offset 0


thanks
-- PMM

Re: [PULL 00/18] Block patches

[no-reply]

Patchew URL: https://patchew.org/QEMU/20200622151059.921191-1-mreitz@redhat.com/



Hi,

This series seems to have some coding style problems. See output below for
more information:

Subject: [PULL 00/18] Block patches
Type: series
Message-id: 20200622151059.921191-1-mreitz@redhat.com

=== TEST SCRIPT BEGIN ===
#!/bin/bash
git rev-parse base > /dev/null || exit 0
git config --local diff.renamelimit 0
git config --local diff.renames True
git config --local diff.algorithm histogram
./scripts/checkpatch.pl --mailback base..
=== TEST SCRIPT END ===

Switched to a new branch 'test'
3170989 iotests: don't test qcow2.py inside 291
04ade1c iotests: Fix 051 output after qdev_init_nofail() removal
e126517 iotests: add tests for blockdev-amend
04e2dc4 block/qcow2: implement blockdev-amend
4e43879 block/crypto: implement blockdev-amend
9207247 block/core: add generic infrastructure for x-blockdev-amend qmp command
5da1c75 iotests: qemu-img tests for luks key management
7a61c0c block/qcow2: extend qemu-img amend interface with crypto options
91d3523 block/crypto: implement the encryption key management
b87419a block/crypto: rename two functions
204e824 block/amend: refactor qcow2 amend options
6264ccb block/amend: separate amend and create options for qemu-img
52f8343 block/amend: add 'force' option
61029e2 qcrypto/luks: implement encryption key management
4ce359d qcrypto/core: add generic infrastructure for crypto options amendment
555e09e block/block-copy: block_copy_dirty_clusters: fix failure check
3de5cab iotests: filter few more luks specific create options
ad75eea iotests: Make _filter_img_create more active

=== OUTPUT BEGIN ===
1/18 Checking commit ad75eea3d2b2 (iotests: Make _filter_img_create more active)
2/18 Checking commit 3de5cab58b52 (iotests: filter few more luks specific create options)
3/18 Checking commit 555e09eaabd5 (block/block-copy: block_copy_dirty_clusters: fix failure check)
4/18 Checking commit 4ce359d028ff (qcrypto/core: add generic infrastructure for crypto options amendment)
5/18 Checking commit 61029e241398 (qcrypto/luks: implement encryption key management)
6/18 Checking commit 52f83435e0e8 (block/amend: add 'force' option)
7/18 Checking commit 6264ccb96333 (block/amend: separate amend and create options for qemu-img)
ERROR: Macros with multiple statements should be enclosed in a do - while loop
#34: FILE: block/qcow2.c:5652:
+#define QCOW_COMMON_OPTIONS                                         \
+    {                                                               \
+        .name = BLOCK_OPT_SIZE,                                     \
+        .type = QEMU_OPT_SIZE,                                      \
+        .help = "Virtual disk size"                                 \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_COMPAT_LEVEL,                             \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "Compatibility level (v2 [0.10] or v3 [1.1])"       \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_BACKING_FILE,                             \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "File name of a base image"                         \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_BACKING_FMT,                              \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "Image format of the base image"                    \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_DATA_FILE,                                \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "File name of an external data file"                \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_DATA_FILE_RAW,                            \
+        .type = QEMU_OPT_BOOL,                                      \
+        .help = "The external data file must stay valid "           \
+                "as a raw image"                                    \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_ENCRYPT,                                  \
+        .type = QEMU_OPT_BOOL,                                      \
+        .help = "Encrypt the image with format 'aes'. (Deprecated " \
+                "in favor of " BLOCK_OPT_ENCRYPT_FORMAT "=aes)",    \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_ENCRYPT_FORMAT,                           \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "Encrypt the image, format choices: 'aes', 'luks'", \
+    },                                                              \
+    BLOCK_CRYPTO_OPT_DEF_KEY_SECRET("encrypt.",                     \
+        "ID of secret providing qcow AES key or LUKS passphrase"),  \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_CIPHER_ALG("encrypt."),               \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_CIPHER_MODE("encrypt."),              \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_IVGEN_ALG("encrypt."),                \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_IVGEN_HASH_ALG("encrypt."),           \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_HASH_ALG("encrypt."),                 \
+    BLOCK_CRYPTO_OPT_DEF_LUKS_ITER_TIME("encrypt."),                \
+    {                                                               \
+        .name = BLOCK_OPT_CLUSTER_SIZE,                             \
+        .type = QEMU_OPT_SIZE,                                      \
+        .help = "qcow2 cluster size",                               \
+        .def_value_str = stringify(DEFAULT_CLUSTER_SIZE)            \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_PREALLOC,                                 \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "Preallocation mode (allowed values: off, "         \
+                "metadata, falloc, full)"                           \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_LAZY_REFCOUNTS,                           \
+        .type = QEMU_OPT_BOOL,                                      \
+        .help = "Postpone refcount updates",                        \
+        .def_value_str = "off"                                      \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_REFCOUNT_BITS,                            \
+        .type = QEMU_OPT_NUMBER,                                    \
+        .help = "Width of a reference count entry in bits",         \
+        .def_value_str = "16"                                       \
+    },                                                              \
+    {                                                               \
+        .name = BLOCK_OPT_COMPRESSION_TYPE,                         \
+        .type = QEMU_OPT_STRING,                                    \
+        .help = "Compression method used for image cluster "        \
+                "compression",                                      \
+        .def_value_str = "zlib"                                     \
+    }

total: 1 errors, 0 warnings, 244 lines checked

Patch 7/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.

8/18 Checking commit 204e8241c4cf (block/amend: refactor qcow2 amend options)
9/18 Checking commit b87419aa40da (block/crypto: rename two functions)
10/18 Checking commit 91d35230beea (block/crypto: implement the encryption key management)
11/18 Checking commit 7a61c0c033a5 (block/qcow2: extend qemu-img amend interface with crypto options)
12/18 Checking commit 5da1c755835e (iotests: qemu-img tests for luks key management)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#17: 
new file mode 100755

total: 0 errors, 1 warnings, 432 lines checked

Patch 12/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
13/18 Checking commit 92072473fcd1 (block/core: add generic infrastructure for x-blockdev-amend qmp command)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#33: 
new file mode 100644

total: 0 errors, 1 warnings, 221 lines checked

Patch 13/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
14/18 Checking commit 4e4387949bcc (block/crypto: implement blockdev-amend)
15/18 Checking commit 04e2dc46dfb4 (block/qcow2: implement blockdev-amend)
16/18 Checking commit e126517e9737 (iotests: add tests for blockdev-amend)
WARNING: added, moved or deleted file(s), does MAINTAINERS need updating?
#17: 
new file mode 100755

total: 0 errors, 1 warnings, 592 lines checked

Patch 16/18 has style problems, please review.  If any of these errors
are false positives report them to the maintainer, see
CHECKPATCH in MAINTAINERS.
17/18 Checking commit 04ade1c09357 (iotests: Fix 051 output after qdev_init_nofail() removal)
18/18 Checking commit 31709895351b (iotests: don't test qcow2.py inside 291)
=== OUTPUT END ===

Test command exited with code: 1


The full log is available at
http://patchew.org/logs/20200622151059.921191-1-mreitz@redhat.com/testing.checkpatch/?type=message.
---
Email generated automatically by Patchew [https://patchew.org/].
Please send your feedback to patchew-devel@redhat.com

[PULL 00/18] Block patches

[Max Reitz]

The following changes since commit bae31bfa48b9caecee25da3d5333901a126a06b4:

  Merge remote-tracking branch 'remotes/kraxel/tags/audio-20200619-pull-request' into staging (2020-06-19 22:56:59 +0100)

are available in the Git repository at:

  https://github.com/XanClic/qemu.git tags/pull-block-2020-06-22

for you to fetch changes up to 74c55e4142a7bb835c38d3770c74210cbb1e4fab:

  iotests: don't test qcow2.py inside 291 (2020-06-22 16:05:23 +0200)

----------------------------------------------------------------
Block patches:
- Support modifying a LUKS-encrypted image's keyslots
- iotest fixes

----------------------------------------------------------------
Max Reitz (1):
  iotests: Make _filter_img_create more active

Maxim Levitsky (14):
  iotests: filter few more luks specific create options
  qcrypto/core: add generic infrastructure for crypto options amendment
  qcrypto/luks: implement encryption key management
  block/amend: add 'force' option
  block/amend: separate amend and create options for qemu-img
  block/amend: refactor qcow2 amend options
  block/crypto: rename two functions
  block/crypto: implement the encryption key management
  block/qcow2: extend qemu-img amend interface with crypto options
  iotests: qemu-img tests for luks key management
  block/core: add generic infrastructure for x-blockdev-amend qmp
    command
  block/crypto: implement blockdev-amend
  block/qcow2: implement blockdev-amend
  iotests: add tests for blockdev-amend

Philippe Mathieu-Daudé (1):
  iotests: Fix 051 output after qdev_init_nofail() removal

Vladimir Sementsov-Ogievskiy (2):
  block/block-copy: block_copy_dirty_clusters: fix failure check
  iotests: don't test qcow2.py inside 291

 docs/tools/qemu-img.rst          |   5 +-
 qapi/block-core.json             |  68 +++++
 qapi/crypto.json                 |  73 +++++-
 qapi/job.json                    |   4 +-
 block/crypto.h                   |  37 +++
 crypto/blockpriv.h               |   8 +
 include/block/block.h            |   1 +
 include/block/block_int.h        |  24 +-
 include/crypto/block.h           |  22 ++
 block.c                          |   4 +-
 block/amend.c                    | 113 +++++++++
 block/block-copy.c               |   4 +-
 block/crypto.c                   | 206 +++++++++++++--
 block/qcow2.c                    | 332 +++++++++++++-----------
 crypto/block-luks.c              | 416 ++++++++++++++++++++++++++++++-
 crypto/block.c                   |  29 +++
 qemu-img.c                       |  44 +++-
 block/Makefile.objs              |   2 +-
 qemu-img-cmds.hx                 |   4 +-
 tests/qemu-iotests/049.out       | 102 ++++----
 tests/qemu-iotests/051.pc.out    |   4 +-
 tests/qemu-iotests/061.out       |  12 +-
 tests/qemu-iotests/082.out       | 185 ++++----------
 tests/qemu-iotests/085.out       |  38 +--
 tests/qemu-iotests/087.out       |   6 +-
 tests/qemu-iotests/112.out       |   2 +-
 tests/qemu-iotests/134.out       |   2 +-
 tests/qemu-iotests/141           |   2 +-
 tests/qemu-iotests/144.out       |   4 +-
 tests/qemu-iotests/153           |   9 +-
 tests/qemu-iotests/158.out       |   4 +-
 tests/qemu-iotests/182.out       |   2 +-
 tests/qemu-iotests/185.out       |   8 +-
 tests/qemu-iotests/188.out       |   2 +-
 tests/qemu-iotests/189.out       |   4 +-
 tests/qemu-iotests/198.out       |   4 +-
 tests/qemu-iotests/255.out       |   8 +-
 tests/qemu-iotests/263.out       |   4 +-
 tests/qemu-iotests/274.out       |  46 ++--
 tests/qemu-iotests/280.out       |   2 +-
 tests/qemu-iotests/284.out       |   6 +-
 tests/qemu-iotests/291           |   4 -
 tests/qemu-iotests/291.out       |  33 ---
 tests/qemu-iotests/293           | 207 +++++++++++++++
 tests/qemu-iotests/293.out       |  99 ++++++++
 tests/qemu-iotests/294           |  90 +++++++
 tests/qemu-iotests/294.out       |  30 +++
 tests/qemu-iotests/295           | 279 +++++++++++++++++++++
 tests/qemu-iotests/295.out       |  40 +++
 tests/qemu-iotests/296           | 234 +++++++++++++++++
 tests/qemu-iotests/296.out       |  33 +++
 tests/qemu-iotests/common.filter |  93 +++++--
 tests/qemu-iotests/group         |   4 +
 53 files changed, 2482 insertions(+), 516 deletions(-)
 create mode 100644 block/amend.c
 create mode 100755 tests/qemu-iotests/293
 create mode 100644 tests/qemu-iotests/293.out
 create mode 100755 tests/qemu-iotests/294
 create mode 100644 tests/qemu-iotests/294.out
 create mode 100755 tests/qemu-iotests/295
 create mode 100644 tests/qemu-iotests/295.out
 create mode 100755 tests/qemu-iotests/296
 create mode 100644 tests/qemu-iotests/296.out

-- 
2.26.2

Re: [PULL 00/18] Block patches

[Peter Maydell]

On Thu, 30 Jan 2020 at 21:31, Stefan Hajnoczi <stefanha@redhat.com> wrote:
>
> The following changes since commit 928173659d6e5dc368284f73f90ea1d129e1f57d:
>
>   Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20200130' into staging (2020-01-30 16:19:04 +0000)
>
> are available in the Git repository at:
>
>   https://github.com/stefanha/qemu.git tags/block-pull-request
>
> for you to fetch changes up to 8dff69b9415b4287e900358744b732195e1ab2e2:
>
>   tests/qemu-iotests: use AIOMODE with various tests (2020-01-30 21:01:40 +0000)
>
> ----------------------------------------------------------------
> Pull request
>
> ----------------------------------------------------------------
>
> Aarushi Mehta (15):
>   configure: permit use of io_uring
>   qapi/block-core: add option for io_uring
>   block/block: add BDRV flag for io_uring
>   block/io_uring: implements interfaces for io_uring
>   stubs: add stubs for io_uring interface
>   util/async: add aio interfaces for io_uring
>   blockdev: adds bdrv_parse_aio to use io_uring
>   block/file-posix.c: extend to use io_uring
>   block: add trace events for io_uring
>   block/io_uring: adds userspace completion polling
>   qemu-io: adds option to use aio engine
>   qemu-img: adds option to use aio engine for benchmarking
>   qemu-nbd: adds option for aio engines
>   tests/qemu-iotests: enable testing with aio options
>   tests/qemu-iotests: use AIOMODE with various tests
>
> Paolo Bonzini (3):
>   block: eliminate BDRV_REQ_NO_SERIALISING
>   block/io: wait for serialising requests when a request becomes
>     serialising
>   block/io: take bs->reqs_lock in bdrv_mark_request_serialising


Applied, thanks.

Please update the changelog at https://wiki.qemu.org/ChangeLog/5.0
for any user-visible changes.

-- PMM

[PULL 00/18] Block patches

[Stefan Hajnoczi]

The following changes since commit 928173659d6e5dc368284f73f90ea1d129e1f57d:

  Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20200130' into staging (2020-01-30 16:19:04 +0000)

are available in the Git repository at:

  https://github.com/stefanha/qemu.git tags/block-pull-request

for you to fetch changes up to 8dff69b9415b4287e900358744b732195e1ab2e2:

  tests/qemu-iotests: use AIOMODE with various tests (2020-01-30 21:01:40 +0000)

----------------------------------------------------------------
Pull request

----------------------------------------------------------------

Aarushi Mehta (15):
  configure: permit use of io_uring
  qapi/block-core: add option for io_uring
  block/block: add BDRV flag for io_uring
  block/io_uring: implements interfaces for io_uring
  stubs: add stubs for io_uring interface
  util/async: add aio interfaces for io_uring
  blockdev: adds bdrv_parse_aio to use io_uring
  block/file-posix.c: extend to use io_uring
  block: add trace events for io_uring
  block/io_uring: adds userspace completion polling
  qemu-io: adds option to use aio engine
  qemu-img: adds option to use aio engine for benchmarking
  qemu-nbd: adds option for aio engines
  tests/qemu-iotests: enable testing with aio options
  tests/qemu-iotests: use AIOMODE with various tests

Paolo Bonzini (3):
  block: eliminate BDRV_REQ_NO_SERIALISING
  block/io: wait for serialising requests when a request becomes
    serialising
  block/io: take bs->reqs_lock in bdrv_mark_request_serialising

 MAINTAINERS                   |   9 +
 block.c                       |  22 ++
 block/Makefile.objs           |   3 +
 block/file-posix.c            |  99 ++++++--
 block/io.c                    | 162 +++++++------
 block/io_uring.c              | 433 ++++++++++++++++++++++++++++++++++
 block/trace-events            |  12 +
 blockdev.c                    |  12 +-
 configure                     |  27 +++
 docs/interop/qemu-nbd.rst     |   4 +-
 include/block/aio.h           |  16 +-
 include/block/block.h         |  14 +-
 include/block/block_int.h     |   3 +-
 include/block/raw-aio.h       |  12 +
 qapi/block-core.json          |   4 +-
 qemu-img-cmds.hx              |   4 +-
 qemu-img.c                    |  11 +-
 qemu-img.texi                 |   5 +-
 qemu-io.c                     |  25 +-
 qemu-nbd.c                    |  12 +-
 stubs/Makefile.objs           |   1 +
 stubs/io_uring.c              |  32 +++
 tests/qemu-iotests/028        |   2 +-
 tests/qemu-iotests/058        |   2 +-
 tests/qemu-iotests/089        |   4 +-
 tests/qemu-iotests/091        |   4 +-
 tests/qemu-iotests/109        |   2 +-
 tests/qemu-iotests/147        |   5 +-
 tests/qemu-iotests/181        |   8 +-
 tests/qemu-iotests/183        |   4 +-
 tests/qemu-iotests/185        |  10 +-
 tests/qemu-iotests/200        |   2 +-
 tests/qemu-iotests/201        |   8 +-
 tests/qemu-iotests/check      |  15 +-
 tests/qemu-iotests/common.rc  |  14 ++
 tests/qemu-iotests/iotests.py |  12 +-
 util/async.c                  |  36 +++
 37 files changed, 878 insertions(+), 172 deletions(-)
 create mode 100644 block/io_uring.c
 create mode 100644 stubs/io_uring.c

-- 
2.24.1