* [PATCH v20 0/4] qcow2: Implement zstd cluster compression methodi
@ 2020-04-21 8:11 Denis Plotnikov
2020-04-21 8:11 ` [PATCH v20 1/4] qcow2: introduce compression type feature Denis Plotnikov
` (3 more replies)
0 siblings, 4 replies; 19+ messages in thread
From: Denis Plotnikov @ 2020-04-21 8:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, berto, qemu-block, armbru, mreitz, den
v20:
04: fix a number of flaws [Vladimir]
* don't use $RAND_FILE passing to qemu-io,
so check $TEST_DIR is redundant
* re-arrage $RAND_FILE writing
* fix a typo
v19:
04: fix a number of flaws [Eric]
* remove rudundant test case descriptions
* fix stdout redirect
* don't use (())
* use peek_file_be instead of od
* check $TEST_DIR for spaces and other before using
* use $RAND_FILE safer
v18:
* 04: add quotes to all file name variables [Vladimir]
* 04: add Vladimir's comment according to "qemu-io write -s"
option issue.
v17:
* 03: remove incorrect comment in zstd decompress [Vladimir]
* 03: remove "paraniod" and rewrite the comment on decompress [Vladimir]
* 03: fix dead assignment [Vladimir]
* 04: add and remove quotes [Vladimir]
* 04: replace long offset form with the short one [Vladimir]
v16:
* 03: ssize_t for ret, size_t for zstd_ret [Vladimir]
* 04: small fixes according to the comments [Vladimir]
v15:
* 01: aiming qemu 5.1 [Eric]
* 03: change zstd_res definition place [Vladimir]
* 04: add two new test cases [Eric]
1. test adjacent cluster compression with zstd
2. test incompressible cluster processing
* 03, 04: many rewording and gramma fixing [Eric]
v14:
* fix bug on compression - looping until compress == 0 [Me]
* apply reworked Vladimir's suggestions:
1. not mixing ssize_t with size_t
2. safe check for ENOMEM in compression part - avoid overflow
3. tolerate sanity check allow zstd to make progress only
on one of the buffers
v13:
* 03: add progress sanity check to decompression loop [Vladimir]
03: add successful decompression check [Me]
v12:
* 03: again, rework compression and decompression loops
to make them more correct [Vladimir]
03: move assert in compression to more appropriate place
[Vladimir]
v11:
* 03: the loops don't need "do{}while" form anymore and
the they were buggy (missed "do" in the beginning)
replace them with usual "while(){}" loops [Vladimir]
v10:
* 03: fix zstd (de)compressed loops for multi-frame
cases [Vladimir]
v9:
* 01: fix error checking and reporting in qcow2_amend compression type part [Vladimir]
* 03: replace asserts with -EIO in qcow2_zstd_decompression [Vladimir, Alberto]
* 03: reword/amend/add comments, fix typos [Vladimir]
v8:
* 03: switch zstd API from simple to stream [Eric]
No need to state a special cluster layout for zstd
compressed clusters.
v7:
* use qapi_enum_parse instead of the open-coding [Eric]
* fix wording, typos and spelling [Eric]
v6:
* "block/qcow2-threads: fix qcow2_decompress" is removed from the series
since it has been accepted by Max already
* add compile time checking for Qcow2Header to be a multiple of 8 [Max, Alberto]
* report error on qcow2 amending when the compression type is actually chnged [Max]
* remove the extra space and the extra new line [Max]
* re-arrange acks and signed-off-s [Vladimir]
v5:
* replace -ENOTSUP with abort in qcow2_co_decompress [Vladimir]
* set cluster size for all test cases in the beginning of the 287 test
v4:
* the series is rebased on top of 01 "block/qcow2-threads: fix qcow2_decompress"
* 01 is just a no-change resend to avoid extra dependencies. Still, it may be merged in separate
v3:
* remove redundant max compression type value check [Vladimir, Eric]
(the switch below checks everything)
* prevent compression type changing on "qemu-img amend" [Vladimir]
* remove zstd config setting, since it has been added already by
"migration" patches [Vladimir]
* change the compression type error message [Vladimir]
* fix alignment and 80-chars exceeding [Vladimir]
v2:
* rework compression type setting [Vladimir]
* squash iotest changes to the compression type introduction patch [Vladimir, Eric]
* fix zstd availability checking in zstd iotest [Vladimir]
* remove unnecessry casting [Eric]
* remove rudundant checks [Eric]
* fix compressed cluster layout in qcow2 spec [Vladimir]
* fix wording [Eric, Vladimir]
* fix compression type filtering in iotests [Eric]
v1:
the initial series
Denis Plotnikov (4):
qcow2: introduce compression type feature
qcow2: rework the cluster compression routine
qcow2: add zstd cluster compression
iotests: 287: add qcow2 compression type test
docs/interop/qcow2.txt | 1 +
configure | 2 +-
qapi/block-core.json | 23 +++-
block/qcow2.h | 20 ++-
include/block/block_int.h | 1 +
block/qcow2-threads.c | 228 +++++++++++++++++++++++++++++--
block/qcow2.c | 120 ++++++++++++++++
tests/qemu-iotests/031.out | 14 +-
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 +++++++-------
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 +++--
tests/qemu-iotests/065 | 28 ++--
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 +
tests/qemu-iotests/255.out | 8 +-
tests/qemu-iotests/287 | 146 ++++++++++++++++++++
tests/qemu-iotests/287.out | 67 +++++++++
tests/qemu-iotests/common.filter | 3 +-
tests/qemu-iotests/group | 1 +
22 files changed, 708 insertions(+), 108 deletions(-)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
--
2.17.0
^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH v20 1/4] qcow2: introduce compression type feature
2020-04-21 8:11 [PATCH v20 0/4] qcow2: Implement zstd cluster compression methodi Denis Plotnikov
@ 2020-04-21 8:11 ` Denis Plotnikov
2020-04-21 10:40 ` Alberto Garcia
2020-04-27 12:35 ` Max Reitz
2020-04-21 8:11 ` [PATCH v20 2/4] qcow2: rework the cluster compression routine Denis Plotnikov
` (2 subsequent siblings)
3 siblings, 2 replies; 19+ messages in thread
From: Denis Plotnikov @ 2020-04-21 8:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, berto, qemu-block, armbru, mreitz, den
The patch adds some preparation parts for incompatible compression type
feature to qcow2 allowing the use different compression methods for
image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Adding of the compression type breaks a number of tests because now the
compression type is reported on image creation and there are some changes
in the qcow2 header in size and offsets.
The tests are fixed in the following ways:
* filter out compression_type for many tests
* fix header size, feature table size and backing file offset
affected tests: 031, 036, 061, 080
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feature compression type
backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
* add "compression type" for test output matching when it isn't filtered
affected tests: 049, 060, 061, 065, 144, 182, 242, 255
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
---
qapi/block-core.json | 22 +++++-
block/qcow2.h | 20 +++++-
include/block/block_int.h | 1 +
block/qcow2.c | 113 +++++++++++++++++++++++++++++++
tests/qemu-iotests/031.out | 14 ++--
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 ++++++++++++++--------------
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 ++++++----
tests/qemu-iotests/065 | 28 +++++---
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +--
tests/qemu-iotests/common.filter | 3 +-
16 files changed, 267 insertions(+), 96 deletions(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 943df1926a..1522e2983f 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
#
# @bitmaps: A list of qcow2 bitmap details (since 4.0)
#
+# @compression-type: the image cluster compression method (since 5.1)
+#
# Since: 1.7
##
{ 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
'*corrupt': 'bool',
'refcount-bits': 'int',
'*encrypt': 'ImageInfoSpecificQCow2Encryption',
- '*bitmaps': ['Qcow2BitmapInfo']
+ '*bitmaps': ['Qcow2BitmapInfo'],
+ 'compression-type': 'Qcow2CompressionType'
} }
##
@@ -4284,6 +4287,18 @@
'data': [ 'v2', 'v3' ] }
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.1
+##
+{ 'enum': 'Qcow2CompressionType',
+ 'data': [ 'zlib' ] }
+
##
# @BlockdevCreateOptionsQcow2:
#
@@ -4307,6 +4322,8 @@
# allowed values: off, falloc, full, metadata)
# @lazy-refcounts: True if refcounts may be updated lazily (default: off)
# @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+# (default: zlib, since 5.1)
#
# Since: 2.12
##
@@ -4322,7 +4339,8 @@
'*cluster-size': 'size',
'*preallocation': 'PreallocMode',
'*lazy-refcounts': 'bool',
- '*refcount-bits': 'int' } }
+ '*refcount-bits': 'int',
+ '*compression-type':'Qcow2CompressionType' } }
##
# @BlockdevCreateOptionsQed:
diff --git a/block/qcow2.h b/block/qcow2.h
index f4de0a27d5..6a8b82e6cc 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -146,8 +146,16 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+ /* Additional fields */
+ uint8_t compression_type;
+
+ /* header must be a multiple of 8 */
+ uint8_t padding[7];
} QEMU_PACKED QCowHeader;
+QEMU_BUILD_BUG_ON(!QEMU_IS_ALIGNED(sizeof(QCowHeader), 8));
+
typedef struct QEMU_PACKED QCowSnapshotHeader {
/* header is 8 byte aligned */
uint64_t l1_table_offset;
@@ -216,13 +224,16 @@ enum {
QCOW2_INCOMPAT_DIRTY_BITNR = 0,
QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+ QCOW2_INCOMPAT_COMPRESSION_BITNR = 3,
QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+ QCOW2_INCOMPAT_COMPRESSION = 1 << QCOW2_INCOMPAT_COMPRESSION_BITNR,
QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
| QCOW2_INCOMPAT_CORRUPT
- | QCOW2_INCOMPAT_DATA_FILE,
+ | QCOW2_INCOMPAT_DATA_FILE
+ | QCOW2_INCOMPAT_COMPRESSION,
};
/* Compatible feature bits */
@@ -366,6 +377,13 @@ typedef struct BDRVQcow2State {
bool metadata_preallocation_checked;
bool metadata_preallocation;
+ /*
+ * Compression type used for the image. Default: 0 - ZLIB
+ * The image compression type is set on image creation.
+ * For now, the only way to change the compression type
+ * is to convert the image with the desired compression type set.
+ */
+ Qcow2CompressionType compression_type;
} BDRVQcow2State;
typedef struct Qcow2COWRegion {
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 4c3587ea19..bc91b69f46 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -57,6 +57,7 @@
#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"
#define BLOCK_OPT_DATA_FILE "data_file"
#define BLOCK_OPT_DATA_FILE_RAW "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE "compression_type"
#define BLOCK_PROBE_BUF_SIZE 512
diff --git a/block/qcow2.c b/block/qcow2.c
index b524b0c53f..9c0b20c912 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1242,6 +1242,39 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
return ret;
}
+static int validate_compression_type(BDRVQcow2State *s, Error **errp)
+{
+ switch (s->compression_type) {
+ case QCOW2_COMPRESSION_TYPE_ZLIB:
+ break;
+
+ default:
+ error_setg(errp, "qcow2: unknown compression type: %u",
+ s->compression_type);
+ return -ENOTSUP;
+ }
+
+ /*
+ * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
+ * the incompatible feature flag must be set
+ */
+ if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
+ if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION) {
+ error_setg(errp, "qcow2: Compression type incompatible feature "
+ "bit must not be set");
+ return -EINVAL;
+ }
+ } else {
+ if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION)) {
+ error_setg(errp, "qcow2: Compression type incompatible feature "
+ "bit must be set");
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
/* Called with s->lock held. */
static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
int flags, Error **errp)
@@ -1357,6 +1390,23 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
s->compatible_features = header.compatible_features;
s->autoclear_features = header.autoclear_features;
+ /*
+ * Handle compression type
+ * Older qcow2 images don't contain the compression type header.
+ * Distinguish them by the header length and use
+ * the only valid (default) compression type in that case
+ */
+ if (header.header_length > offsetof(QCowHeader, compression_type)) {
+ s->compression_type = header.compression_type;
+ } else {
+ s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+ }
+
+ ret = validate_compression_type(s, errp);
+ if (ret) {
+ goto fail;
+ }
+
if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
void *feature_table = NULL;
qcow2_read_extensions(bs, header.header_length, ext_end,
@@ -2727,6 +2777,11 @@ int qcow2_update_header(BlockDriverState *bs)
total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
+ ret = validate_compression_type(s, NULL);
+ if (ret) {
+ goto fail;
+ }
+
*header = (QCowHeader) {
/* Version 2 fields */
.magic = cpu_to_be32(QCOW_MAGIC),
@@ -2749,6 +2804,7 @@ int qcow2_update_header(BlockDriverState *bs)
.autoclear_features = cpu_to_be64(s->autoclear_features),
.refcount_order = cpu_to_be32(s->refcount_order),
.header_length = cpu_to_be32(header_length),
+ .compression_type = s->compression_type,
};
/* For older versions, write a shorter header */
@@ -2848,6 +2904,11 @@ int qcow2_update_header(BlockDriverState *bs)
.bit = QCOW2_INCOMPAT_DATA_FILE_BITNR,
.name = "external data file",
},
+ {
+ .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
+ .bit = QCOW2_INCOMPAT_COMPRESSION_BITNR,
+ .name = "compression type",
+ },
{
.type = QCOW2_FEAT_TYPE_COMPATIBLE,
.bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
@@ -3286,6 +3347,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
uint64_t* refcount_table;
Error *local_err = NULL;
int ret;
+ uint8_t compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
assert(create_options->driver == BLOCKDEV_DRIVER_QCOW2);
qcow2_opts = &create_options->u.qcow2;
@@ -3403,6 +3465,27 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
}
}
+ if (qcow2_opts->has_compression_type &&
+ qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
+
+ ret = -EINVAL;
+
+ if (version < 3) {
+ error_setg(errp, "Non-zlib compression type is only supported with "
+ "compatibility level 1.1 and above (use version=v3 or "
+ "greater)");
+ goto out;
+ }
+
+ switch (qcow2_opts->compression_type) {
+ default:
+ error_setg(errp, "Unknown compression type");
+ goto out;
+ }
+
+ compression_type = qcow2_opts->compression_type;
+ }
+
/* Create BlockBackend to write to the image */
blk = blk_new(bdrv_get_aio_context(bs),
BLK_PERM_WRITE | BLK_PERM_RESIZE, BLK_PERM_ALL);
@@ -3425,6 +3508,8 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
.refcount_table_offset = cpu_to_be64(cluster_size),
.refcount_table_clusters = cpu_to_be32(1),
.refcount_order = cpu_to_be32(refcount_order),
+ /* don't deal with endianness since compression_type is 1 byte long */
+ .compression_type = compression_type,
.header_length = cpu_to_be32(sizeof(*header)),
};
@@ -3443,6 +3528,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
header->autoclear_features |=
cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
}
+ if (compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
+ header->incompatible_features |=
+ cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION);
+ }
ret = blk_pwrite(blk, 0, header, cluster_size, 0);
g_free(header);
@@ -3628,6 +3717,7 @@ static int coroutine_fn qcow2_co_create_opts(BlockDriver *drv,
{ BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT },
{ BLOCK_OPT_COMPAT_LEVEL, "version" },
{ BLOCK_OPT_DATA_FILE_RAW, "data-file-raw" },
+ { BLOCK_OPT_COMPRESSION_TYPE, "compression-type" },
{ NULL, NULL },
};
@@ -4861,6 +4951,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs,
.data_file = g_strdup(s->image_data_file),
.has_data_file_raw = has_data_file(bs),
.data_file_raw = data_file_is_raw(bs),
+ .compression_type = s->compression_type,
};
} else {
/* if this assertion fails, this probably means a new version was
@@ -5250,6 +5341,22 @@ static int qcow2_amend_options(BlockDriverState *bs, QemuOpts *opts,
"images");
return -EINVAL;
}
+ } else if (!strcmp(desc->name, BLOCK_OPT_COMPRESSION_TYPE)) {
+ const char *ct_name =
+ qemu_opt_get(opts, BLOCK_OPT_COMPRESSION_TYPE);
+ int compression_type =
+ qapi_enum_parse(&Qcow2CompressionType_lookup, ct_name, -1,
+ NULL);
+ if (compression_type == -1) {
+ error_setg(errp, "Unknown compression type: %s", ct_name);
+ return -ENOTSUP;
+ }
+
+ if (compression_type != s->compression_type) {
+ error_setg(errp, "Changing the compression type "
+ "is not supported");
+ return -ENOTSUP;
+ }
} else {
/* if this point is reached, this probably means a new option was
* added without having it covered here */
@@ -5518,6 +5625,12 @@ static QemuOptsList qcow2_create_opts = {
.help = "Width of a reference count entry in bits",
.def_value_str = "16"
},
+ {
+ .name = BLOCK_OPT_COMPRESSION_TYPE,
+ .type = QEMU_OPT_STRING,
+ .help = "Compression method used for image cluster compression",
+ .def_value_str = "zlib"
+ },
{ /* end of list */ }
}
};
diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out
index 46f97c5a4e..5a4beda6a2 100644
--- a/tests/qemu-iotests/031.out
+++ b/tests/qemu-iotests/031.out
@@ -113,11 +113,11 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
Header extension:
@@ -146,11 +146,11 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
Header extension:
@@ -164,7 +164,7 @@ No errors were found on the image.
magic 0x514649fb
version 3
-backing_file_offset 0x1d8
+backing_file_offset 0x210
backing_file_size 0x17
cluster_bits 16
size 67108864
@@ -179,7 +179,7 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0xe2792aca
@@ -188,7 +188,7 @@ data 'host_device'
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
Header extension:
diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out
index 23b699ce06..e409acf60e 100644
--- a/tests/qemu-iotests/036.out
+++ b/tests/qemu-iotests/036.out
@@ -26,7 +26,7 @@ compatible_features []
autoclear_features [63]
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
@@ -38,7 +38,7 @@ compatible_features []
autoclear_features []
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
*** done
diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out
index affa55b341..a5cfba1756 100644
--- a/tests/qemu-iotests/049.out
+++ b/tests/qemu-iotests/049.out
@@ -4,90 +4,90 @@ QA output created by 049
== 1. Traditional size parameter ==
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== 2. Specifying size via -o ==
qemu-img create -f qcow2 -o size=1024 TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1024b TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1k TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1K TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1M TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1G TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1T TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1024.0 TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1024.0b TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5k TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5K TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5M TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5G TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5T TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== 3. Invalid sizes ==
@@ -129,84 +129,84 @@ qemu-img: TEST_DIR/t.qcow2: The image size must be specified only once
== Check correct interpretation of suffixes for cluster size ==
qemu-img create -f qcow2 -o cluster_size=1024 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024b TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1k TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1K TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1M TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024.0 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024.0b TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5k TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5K TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5M TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check compat level option ==
qemu-img create -f qcow2 -o compat=0.10 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check preallocation option ==
qemu-img create -f qcow2 -o preallocation=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check encryption option ==
qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o encryption=on,encrypt.key-secret=sec0 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check lazy_refcounts option (only with v3) ==
qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=on TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater)
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
*** done
diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out
index 09caaea865..be5f8707a3 100644
--- a/tests/qemu-iotests/060.out
+++ b/tests/qemu-iotests/060.out
@@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: true
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index 413cc4e0f4..45c029228c 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -22,11 +22,11 @@ incompatible_features []
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
magic 0x514649fb
@@ -80,11 +80,11 @@ incompatible_features []
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
magic 0x514649fb
@@ -136,11 +136,11 @@ incompatible_features [0]
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
ERROR cluster 5 refcount=0 reference=1
@@ -191,11 +191,11 @@ incompatible_features []
compatible_features [42]
autoclear_features [42]
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
magic 0x514649fb
@@ -260,11 +260,11 @@ incompatible_features []
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
read 65536/65536 bytes at offset 44040192
@@ -294,11 +294,11 @@ incompatible_features [0]
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
ERROR cluster 5 refcount=0 reference=1
@@ -323,11 +323,11 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 288
+length 336
data <binary>
read 131072/131072 bytes at offset 0
@@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: foo
@@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file raw: false
@@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 6426474271..18dc488c7a 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -88,24 +88,30 @@ class TestQMP(TestImageInfoSpecific):
class TestQCow2(TestQemuImgInfo):
'''Testing a qcow2 version 2 image'''
img_options = 'compat=0.10'
- json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
- human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
+ json_compare = { 'compat': '0.10', 'refcount-bits': 16,
+ 'compression-type': 'zlib' }
+ human_compare = [ 'compat: 0.10', 'compression type: zlib',
+ 'refcount bits: 16' ]
class TestQCow3NotLazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts disabled'''
img_options = 'compat=1.1,lazy_refcounts=off'
json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
- 'refcount-bits': 16, 'corrupt': False }
- human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
- 'refcount bits: 16', 'corrupt: false' ]
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
+ human_compare = [ 'compat: 1.1', 'compression type: zlib',
+ 'lazy refcounts: false', 'refcount bits: 16',
+ 'corrupt: false' ]
class TestQCow3Lazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts enabled'''
img_options = 'compat=1.1,lazy_refcounts=on'
json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
- 'refcount-bits': 16, 'corrupt': False }
- human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
- 'refcount bits: 16', 'corrupt: false' ]
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
+ human_compare = [ 'compat: 1.1', 'compression type: zlib',
+ 'lazy refcounts: true', 'refcount bits: 16',
+ 'corrupt: false' ]
class TestQCow3NotLazyQMP(TestQMP):
'''Testing a qcow2 version 3 image with lazy refcounts disabled, opening
@@ -113,7 +119,8 @@ class TestQCow3NotLazyQMP(TestQMP):
img_options = 'compat=1.1,lazy_refcounts=off'
qemu_options = 'lazy-refcounts=on'
compare = { 'compat': '1.1', 'lazy-refcounts': False,
- 'refcount-bits': 16, 'corrupt': False }
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
class TestQCow3LazyQMP(TestQMP):
@@ -122,7 +129,8 @@ class TestQCow3LazyQMP(TestQMP):
img_options = 'compat=1.1,lazy_refcounts=on'
qemu_options = 'lazy-refcounts=off'
compare = { 'compat': '1.1', 'lazy-refcounts': True,
- 'refcount-bits': 16, 'corrupt': False }
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
TestImageInfoSpecific = None
TestQemuImgInfo = None
diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080
index a3d13c414e..7588c63b6c 100755
--- a/tests/qemu-iotests/080
+++ b/tests/qemu-iotests/080
@@ -45,7 +45,7 @@ _supported_os Linux
# - This is generally a test for compat=1.1 images
_unsupported_imgopts 'refcount_bits=1[^0-9]' data_file 'compat=0.10'
-header_size=104
+header_size=112
offset_backing_file_offset=8
offset_backing_file_size=16
diff --git a/tests/qemu-iotests/144.out b/tests/qemu-iotests/144.out
index c7aa2e4820..885a8874a5 100644
--- a/tests/qemu-iotests/144.out
+++ b/tests/qemu-iotests/144.out
@@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912
{ 'execute': 'qmp_capabilities' }
{"return": {}}
{ 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } }
-Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
{"return": {}}
=== Performing block-commit on active layer ===
@@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/
=== Performing Live Snapshot 2 ===
{ 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } }
-Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
{"return": {}}
*** done
diff --git a/tests/qemu-iotests/182.out b/tests/qemu-iotests/182.out
index a8eea166c3..ae43654d32 100644
--- a/tests/qemu-iotests/182.out
+++ b/tests/qemu-iotests/182.out
@@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]?
{'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
{"return": {}}
{'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } }
-Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
{"return": {}}
{'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
{"return": {}}
diff --git a/tests/qemu-iotests/242.out b/tests/qemu-iotests/242.out
index 7ac8404d11..091b9126ce 100644
--- a/tests/qemu-iotests/242.out
+++ b/tests/qemu-iotests/242.out
@@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
@@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
@@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
@@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
diff --git a/tests/qemu-iotests/255.out b/tests/qemu-iotests/255.out
index 348909fdef..a3c99fd62e 100644
--- a/tests/qemu-iotests/255.out
+++ b/tests/qemu-iotests/255.out
@@ -3,9 +3,9 @@ Finishing a commit job with background reads
=== Create backing chain and start VM ===
-Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
-Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
=== Start background read requests ===
@@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled
=== Create images and start VM ===
-Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
-Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
wrote 1048576/1048576 bytes at offset 0
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index 3f8ee3e5f7..279e0bbb0d 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -152,7 +152,8 @@ _filter_img_create()
-e "s# refcount_bits=[0-9]\\+##g" \
-e "s# key-secret=[a-zA-Z0-9]\\+##g" \
-e "s# iter-time=[0-9]\\+##g" \
- -e "s# force_size=\\(on\\|off\\)##g"
+ -e "s# force_size=\\(on\\|off\\)##g" \
+ -e "s# compression_type=[a-zA-Z0-9]\\+##g"
}
_filter_img_info()
--
2.17.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v20 2/4] qcow2: rework the cluster compression routine
2020-04-21 8:11 [PATCH v20 0/4] qcow2: Implement zstd cluster compression methodi Denis Plotnikov
2020-04-21 8:11 ` [PATCH v20 1/4] qcow2: introduce compression type feature Denis Plotnikov
@ 2020-04-21 8:11 ` Denis Plotnikov
2020-04-27 12:36 ` Max Reitz
2020-04-21 8:11 ` [PATCH v20 3/4] qcow2: add zstd cluster compression Denis Plotnikov
2020-04-21 8:11 ` [PATCH v20 4/4] iotests: 287: add qcow2 compression type test Denis Plotnikov
3 siblings, 1 reply; 19+ messages in thread
From: Denis Plotnikov @ 2020-04-21 8:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, berto, qemu-block, armbru, mreitz, den
The patch enables processing the image compression type defined
for the image and chooses an appropriate method for image clusters
(de)compression.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
---
block/qcow2-threads.c | 71 ++++++++++++++++++++++++++++++++++++-------
1 file changed, 60 insertions(+), 11 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index a68126f291..7dbaf53489 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
} Qcow2CompressData;
/*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
* -ENOMEM destination buffer is not enough to store compressed data
* -EIO on any other error
*/
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
ssize_t ret;
z_stream strm;
@@ -119,10 +121,10 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
}
/*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
*
* Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -130,8 +132,8 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
* Returns: 0 on success
* -EIO on fail
*/
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
int ret;
z_stream strm;
@@ -191,20 +193,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,
return arg.ret;
}
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
- return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
- qcow2_compress);
+ BDRVQcow2State *s = bs->opaque;
+ Qcow2CompressFunc fn;
+
+ switch (s->compression_type) {
+ case QCOW2_COMPRESSION_TYPE_ZLIB:
+ fn = qcow2_zlib_compress;
+ break;
+
+ default:
+ abort();
+ }
+
+ return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
- return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
- qcow2_decompress);
+ BDRVQcow2State *s = bs->opaque;
+ Qcow2CompressFunc fn;
+
+ switch (s->compression_type) {
+ case QCOW2_COMPRESSION_TYPE_ZLIB:
+ fn = qcow2_zlib_decompress;
+ break;
+
+ default:
+ abort();
+ }
+
+ return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
--
2.17.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v20 3/4] qcow2: add zstd cluster compression
2020-04-21 8:11 [PATCH v20 0/4] qcow2: Implement zstd cluster compression methodi Denis Plotnikov
2020-04-21 8:11 ` [PATCH v20 1/4] qcow2: introduce compression type feature Denis Plotnikov
2020-04-21 8:11 ` [PATCH v20 2/4] qcow2: rework the cluster compression routine Denis Plotnikov
@ 2020-04-21 8:11 ` Denis Plotnikov
2020-04-27 12:35 ` Max Reitz
2020-04-21 8:11 ` [PATCH v20 4/4] iotests: 287: add qcow2 compression type test Denis Plotnikov
3 siblings, 1 reply; 19+ messages in thread
From: Denis Plotnikov @ 2020-04-21 8:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, berto, qemu-block, armbru, mreitz, den
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
user 65.0 15.8 5.3 2.5
sys 3.3 0.2 2.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
QAPI part:
Acked-by: Markus Armbruster <armbru@redhat.com>
---
docs/interop/qcow2.txt | 1 +
configure | 2 +-
qapi/block-core.json | 3 +-
block/qcow2-threads.c | 157 +++++++++++++++++++++++++++++++++++++++++
block/qcow2.c | 7 ++
5 files changed, 168 insertions(+), 2 deletions(-)
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 640e0eca40..18a77f737e 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -209,6 +209,7 @@ version 2.
Available compression type values:
0: zlib <https://www.zlib.net/>
+ 1: zstd <http://github.com/facebook/zstd>
=== Header padding ===
diff --git a/configure b/configure
index 23b5e93752..4e3a1690ea 100755
--- a/configure
+++ b/configure
@@ -1861,7 +1861,7 @@ disabled with --disable-FEATURE, default is enabled if available:
lzfse support of lzfse compression library
(for reading lzfse-compressed dmg images)
zstd support for zstd compression library
- (for migration compression)
+ (for migration compression and qcow2 cluster compression)
seccomp seccomp support
coroutine-pool coroutine freelist (better performance)
glusterfs GlusterFS backend
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 1522e2983f..6fbacddab2 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4293,11 +4293,12 @@
# Compression type used in qcow2 image file
#
# @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
#
# Since: 5.1
##
{ 'enum': 'Qcow2CompressionType',
- 'data': [ 'zlib' ] }
+ 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
##
# @BlockdevCreateOptionsQcow2:
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 7dbaf53489..0525718704 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include <zlib.h>
+#ifdef CONFIG_ZSTD
+#include <zstd.h>
+#include <zstd_errors.h>
+#endif
+
#include "qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"
@@ -166,6 +171,148 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
return ret;
}
+#ifdef CONFIG_ZSTD
+
+/*
+ * qcow2_zstd_compress()
+ *
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * -ENOMEM destination buffer is not enough to store compressed data
+ * -EIO on any other error
+ */
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+ ssize_t ret;
+ ZSTD_outBuffer output = { dest, dest_size, 0 };
+ ZSTD_inBuffer input = { src, src_size, 0 };
+ ZSTD_CCtx *cctx = ZSTD_createCCtx();
+
+ if (!cctx) {
+ return -EIO;
+ }
+ /*
+ * Use the zstd streamed interface for symmetry with decompression,
+ * where streaming is essential since we don't record the exact
+ * compressed size.
+ *
+ * In the loop, we try to compress all the data into one zstd frame.
+ * ZSTD_compressStream2 potentially can finish a frame earlier
+ * than the full input data is consumed. That's why we are looping
+ * until all the input data is consumed.
+ */
+ while (input.pos < input.size) {
+ size_t zstd_ret;
+ /*
+ * ZSTD spec: "You must continue calling ZSTD_compressStream2()
+ * with ZSTD_e_end until it returns 0, at which point you are
+ * free to start a new frame". We assume that "start a new frame"
+ * means call ZSTD_compressStream2 in the very beginning or when
+ * ZSTD_compressStream2 has returned with 0.
+ */
+ do {
+ zstd_ret = ZSTD_compressStream2(cctx, &output, &input, ZSTD_e_end);
+
+ if (ZSTD_isError(zstd_ret)) {
+ ret = -EIO;
+ goto out;
+ }
+ /* Dest buffer isn't big enough to store compressed content */
+ if (zstd_ret > output.size - output.pos) {
+ ret = -ENOMEM;
+ goto out;
+ }
+ } while (zstd_ret);
+ }
+ /* make sure we can safely return compressed buffer size with ssize_t */
+ assert(output.pos <= SSIZE_MAX);
+ ret = output.pos;
+out:
+ ZSTD_freeCCtx(cctx);
+ return ret;
+}
+
+/*
+ * qcow2_zstd_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * -EIO on any error
+ */
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+ size_t zstd_ret = 0;
+ ssize_t ret = 0;
+ ZSTD_outBuffer output = { dest, dest_size, 0 };
+ ZSTD_inBuffer input = { src, src_size, 0 };
+ ZSTD_DCtx *dctx = ZSTD_createDCtx();
+
+ if (!dctx) {
+ return -EIO;
+ }
+
+ /*
+ * The compressed stream from the input buffer may consist of more
+ * than one zstd frame. So we iterate until we get a fully
+ * uncompressed cluster.
+ * From zstd docs related to ZSTD_decompressStream:
+ * "return : 0 when a frame is completely decoded and fully flushed"
+ * We suppose that this means: each time ZSTD_decompressStream reads
+ * only ONE full frame and returns 0 if and only if that frame
+ * is completely decoded and flushed. Only after returning 0,
+ * ZSTD_decompressStream reads another ONE full frame.
+ */
+ while (output.pos < output.size) {
+ size_t last_in_pos = input.pos;
+ size_t last_out_pos = output.pos;
+ zstd_ret = ZSTD_decompressStream(dctx, &output, &input);
+
+ if (ZSTD_isError(zstd_ret)) {
+ ret = -EIO;
+ break;
+ }
+
+ /*
+ * The ZSTD manual is vague about what to do if it reads
+ * the buffer partially, and we don't want to get stuck
+ * in an infinite loop where ZSTD_decompressStream
+ * returns > 0 waiting for another input chunk. So, we add
+ * a check which ensures that the loop makes some progress
+ * on each step.
+ */
+ if (last_in_pos >= input.pos &&
+ last_out_pos >= output.pos) {
+ ret = -EIO;
+ break;
+ }
+ }
+ /*
+ * Make sure that we have the frame fully flushed here
+ * if not, we somehow managed to get uncompressed cluster
+ * greater then the cluster size, possibly because of its
+ * damage.
+ */
+ if (zstd_ret > 0) {
+ ret = -EIO;
+ }
+
+ ZSTD_freeDCtx(dctx);
+ assert(ret == 0 || ret == -EIO);
+ return ret;
+}
+#endif
+
static int qcow2_compress_pool_func(void *opaque)
{
Qcow2CompressData *data = opaque;
@@ -217,6 +364,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
fn = qcow2_zlib_compress;
break;
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+ fn = qcow2_zstd_compress;
+ break;
+#endif
default:
abort();
}
@@ -249,6 +401,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
fn = qcow2_zlib_decompress;
break;
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+ fn = qcow2_zstd_decompress;
+ break;
+#endif
default:
abort();
}
diff --git a/block/qcow2.c b/block/qcow2.c
index 9c0b20c912..21f281b7f8 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1246,6 +1246,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp)
{
switch (s->compression_type) {
case QCOW2_COMPRESSION_TYPE_ZLIB:
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+#endif
break;
default:
@@ -3478,6 +3481,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
}
switch (qcow2_opts->compression_type) {
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+ break;
+#endif
default:
error_setg(errp, "Unknown compression type");
goto out;
--
2.17.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* [PATCH v20 4/4] iotests: 287: add qcow2 compression type test
2020-04-21 8:11 [PATCH v20 0/4] qcow2: Implement zstd cluster compression methodi Denis Plotnikov
` (2 preceding siblings ...)
2020-04-21 8:11 ` [PATCH v20 3/4] qcow2: add zstd cluster compression Denis Plotnikov
@ 2020-04-21 8:11 ` Denis Plotnikov
2020-04-21 12:06 ` Vladimir Sementsov-Ogievskiy
2020-04-27 13:29 ` Max Reitz
3 siblings, 2 replies; 19+ messages in thread
From: Denis Plotnikov @ 2020-04-21 8:11 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, berto, qemu-block, armbru, mreitz, den
The test checks fulfilling qcow2 requirements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
tests/qemu-iotests/287 | 146 +++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/287.out | 67 +++++++++++++++++
tests/qemu-iotests/group | 1 +
3 files changed, 214 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 0000000000..156acc40ad
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,146 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotnikov@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+COMPR_IMG="$TEST_IMG.compressed"
+RAND_FILE="$TEST_DIR/rand_data"
+
+_cleanup()
+{
+ _cleanup_test_img
+ rm -f "$COMPR_IMG"
+ rm -f "$RAND_FILE"
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# for all the cases
+CLUSTER_SIZE=65536
+
+# Check if we can run this test.
+if IMGOPTS='compression_type=zstd' _make_test_img 64M |
+ grep "Invalid parameter 'zstd'"; then
+ _notrun "ZSTD is disabled"
+fi
+
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+if $QEMU_IMG info "$TEST_IMG" >/dev/null 2>&1 ; then
+ echo "Error: The image opened successfully. The image must not be opened."
+fi
+
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+if $QEMU_IMG info "$TEST_IMG" >/dev/null 2>&1 ; then
+ echo "Error: The image opened successfully. The image must not be opened."
+fi
+
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+peek_file_be "$TEST_IMG" 104 1
+echo
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+peek_file_be "$TEST_IMG" 104 1
+echo
+
+echo
+echo "=== Testing simple reading and writing with zstd ==="
+echo
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c -P 0xAC 64K 64K " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 64K 64K " "$TEST_IMG" | _filter_qemu_io
+# read on the cluster boundaries
+$QEMU_IO -c "read -v 131070 8 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65534 8" "$TEST_IMG" | _filter_qemu_io
+
+echo
+echo "=== Testing adjacent clusters reading and writing with zstd ==="
+echo
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c -P 0xAB 0 64K " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "write -c -P 0xAC 64K 64K " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "write -c -P 0xAD 128K 64K " "$TEST_IMG" | _filter_qemu_io
+
+$QEMU_IO -c "read -P 0xAB 0 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAC 64K 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -P 0xAD 128K 64k " "$TEST_IMG" | _filter_qemu_io
+
+echo
+echo "=== Testing incompressible cluster processing with zstd ==="
+echo
+# create a 2M image and fill it with 1M likely incompressible data
+# and 1M compressible data
+dd if=/dev/urandom of="$RAND_FILE" bs=1M count=1 seek=1
+QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS_NO_FMT" \
+$QEMU_IO -f raw -c "write -P 0xFA 0 1M" "$RAND_FILE" | _filter_qemu_io
+$QEMU_IMG convert -f raw -O $IMGFMT -c "$RAND_FILE" "$TEST_IMG" | _filter_qemu_io
+
+$QEMU_IMG convert -O $IMGFMT -c -o compression_type=zstd \
+ "$TEST_IMG" "$COMPR_IMG"
+$QEMU_IMG compare "$TEST_IMG" "$COMPR_IMG"
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/287.out b/tests/qemu-iotests/287.out
new file mode 100644
index 0000000000..6b9dfb4af0
--- /dev/null
+++ b/tests/qemu-iotests/287.out
@@ -0,0 +1,67 @@
+QA output created by 287
+
+=== Testing compression type incompatible bit setting for zlib ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+incompatible_features []
+
+=== Testing compression type incompatible bit setting for zstd ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+incompatible_features [3]
+
+=== Testing zlib with incompatible bit set ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+incompatible_features [3]
+
+=== Testing zstd with incompatible bit unset ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+incompatible_features []
+
+=== Testing compression type values ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+ 0
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+ 1
+
+=== Testing simple reading and writing with zstd ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+wrote 65536/65536 bytes at offset 65536
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 65536/65536 bytes at offset 65536
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+0001fffe: ac ac 00 00 00 00 00 00 ........
+read 8/8 bytes at offset 131070
+8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+0000fffe: 00 00 ac ac ac ac ac ac ........
+read 8/8 bytes at offset 65534
+8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+=== Testing adjacent clusters reading and writing with zstd ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+wrote 65536/65536 bytes at offset 0
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 65536/65536 bytes at offset 65536
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 65536/65536 bytes at offset 131072
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 65536/65536 bytes at offset 0
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 65536/65536 bytes at offset 65536
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+read 65536/65536 bytes at offset 131072
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+
+=== Testing incompressible cluster processing with zstd ===
+
+1+0 records in
+1+0 records out
+wrote 1048576/1048576 bytes at offset 0
+1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+Images are identical.
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 435dccd5af..3bbe02c88d 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -294,6 +294,7 @@
283 auto quick
284 rw
286 rw quick
+287 auto quick
288 quick
289 rw quick
290 rw auto quick
--
2.17.0
^ permalink raw reply related [flat|nested] 19+ messages in thread
* Re: [PATCH v20 1/4] qcow2: introduce compression type feature
2020-04-21 8:11 ` [PATCH v20 1/4] qcow2: introduce compression type feature Denis Plotnikov
@ 2020-04-21 10:40 ` Alberto Garcia
2020-04-27 12:35 ` Max Reitz
1 sibling, 0 replies; 19+ messages in thread
From: Alberto Garcia @ 2020-04-21 10:40 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel
Cc: kwolf, vsementsov, qemu-block, armbru, mreitz, den
On Tue 21 Apr 2020 10:11:14 AM CEST, Denis Plotnikov <dplotnikov@virtuozzo.com> wrote:
> The patch adds some preparation parts for incompatible compression type
> feature to qcow2 allowing the use different compression methods for
> image clusters (de)compressing.
>
> It is implied that the compression type is set on the image creation and
> can be changed only later by image conversion, thus compression type
> defines the only compression algorithm used for the image, and thus,
> for all image clusters.
>
> The goal of the feature is to add support of other compression methods
> to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
>
> The default compression is ZLIB. Images created with ZLIB compression type
> are backward compatible with older qemu versions.
>
> Adding of the compression type breaks a number of tests because now the
> compression type is reported on image creation and there are some changes
> in the qcow2 header in size and offsets.
>
> The tests are fixed in the following ways:
> * filter out compression_type for many tests
> * fix header size, feature table size and backing file offset
> affected tests: 031, 036, 061, 080
> header_size +=8: 1 byte compression type
> 7 bytes padding
> feature_table += 48: incompatible feature compression type
> backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
> * add "compression type" for test output matching when it isn't filtered
> affected tests: 049, 060, 061, 065, 144, 182, 242, 255
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Reviewed-by: Eric Blake <eblake@redhat.com>
> QAPI part:
> Acked-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Berto
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 4/4] iotests: 287: add qcow2 compression type test
2020-04-21 8:11 ` [PATCH v20 4/4] iotests: 287: add qcow2 compression type test Denis Plotnikov
@ 2020-04-21 12:06 ` Vladimir Sementsov-Ogievskiy
2020-04-27 13:29 ` Max Reitz
1 sibling, 0 replies; 19+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-04-21 12:06 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, berto, qemu-block, armbru, mreitz, den
21.04.2020 11:11, Denis Plotnikov wrote:
> The test checks fulfilling qcow2 requirements for the compression
> type feature and zstd compression type operability.
>
> Signed-off-by: Denis Plotnikov<dplotnikov@virtuozzo.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Tested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 3/4] qcow2: add zstd cluster compression
2020-04-21 8:11 ` [PATCH v20 3/4] qcow2: add zstd cluster compression Denis Plotnikov
@ 2020-04-27 12:35 ` Max Reitz
2020-04-27 19:26 ` Denis Plotnikov
0 siblings, 1 reply; 19+ messages in thread
From: Max Reitz @ 2020-04-27 12:35 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel
Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
[-- Attachment #1.1: Type: text/plain, Size: 7238 bytes --]
On 21.04.20 10:11, Denis Plotnikov wrote:
> zstd significantly reduces cluster compression time.
> It provides better compression performance maintaining
> the same level of the compression ratio in comparison with
> zlib, which, at the moment, is the only compression
> method available.
>
> The performance test results:
> Test compresses and decompresses qemu qcow2 image with just
> installed rhel-7.6 guest.
> Image cluster size: 64K. Image on disk size: 2.2G
>
> The test was conducted with brd disk to reduce the influence
> of disk subsystem to the test results.
> The results is given in seconds.
>
> compress cmd:
> time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
> src.img [zlib|zstd]_compressed.img
> decompress cmd
> time ./qemu-img convert -O qcow2
> [zlib|zstd]_compressed.img uncompressed.img
>
> compression decompression
> zlib zstd zlib zstd
> ------------------------------------------------------------
> real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
> user 65.0 15.8 5.3 2.5
> sys 3.3 0.2 2.0 2.0
>
> Both ZLIB and ZSTD gave the same compression ratio: 1.57
> compressed image size in both cases: 1.4G
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Reviewed-by: Alberto Garcia <berto@igalia.com>
> QAPI part:
> Acked-by: Markus Armbruster <armbru@redhat.com>
> ---
> docs/interop/qcow2.txt | 1 +
> configure | 2 +-
> qapi/block-core.json | 3 +-
> block/qcow2-threads.c | 157 +++++++++++++++++++++++++++++++++++++++++
> block/qcow2.c | 7 ++
> 5 files changed, 168 insertions(+), 2 deletions(-)
[...]
> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
> index 7dbaf53489..0525718704 100644
> --- a/block/qcow2-threads.c
> +++ b/block/qcow2-threads.c
> @@ -28,6 +28,11 @@
> #define ZLIB_CONST
> #include <zlib.h>
>
> +#ifdef CONFIG_ZSTD
> +#include <zstd.h>
> +#include <zstd_errors.h>
> +#endif
> +
> #include "qcow2.h"
> #include "block/thread-pool.h"
> #include "crypto.h"
> @@ -166,6 +171,148 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
> return ret;
> }
>
> +#ifdef CONFIG_ZSTD
> +
> +/*
> + * qcow2_zstd_compress()
> + *
> + * Compress @src_size bytes of data using zstd compression method
> + *
> + * @dest - destination buffer, @dest_size bytes
> + * @src - source buffer, @src_size bytes
> + *
> + * Returns: compressed size on success
> + * -ENOMEM destination buffer is not enough to store compressed data
> + * -EIO on any other error
> + */
> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
> + const void *src, size_t src_size)
> +{
> + ssize_t ret;
> + ZSTD_outBuffer output = { dest, dest_size, 0 };
> + ZSTD_inBuffer input = { src, src_size, 0 };
Minor style note: I think it’d be nicer to use designated initializers here.
> + ZSTD_CCtx *cctx = ZSTD_createCCtx();
> +
> + if (!cctx) {
> + return -EIO;
> + }
> + /*
> + * Use the zstd streamed interface for symmetry with decompression,
> + * where streaming is essential since we don't record the exact
> + * compressed size.
> + *
> + * In the loop, we try to compress all the data into one zstd frame.
> + * ZSTD_compressStream2 potentially can finish a frame earlier
> + * than the full input data is consumed. That's why we are looping
> + * until all the input data is consumed.
> + */
> + while (input.pos < input.size) {
> + size_t zstd_ret;
> + /*
> + * ZSTD spec: "You must continue calling ZSTD_compressStream2()
> + * with ZSTD_e_end until it returns 0, at which point you are
> + * free to start a new frame". We assume that "start a new frame"
> + * means call ZSTD_compressStream2 in the very beginning or when
> + * ZSTD_compressStream2 has returned with 0.
> + */
> + do {
> + zstd_ret = ZSTD_compressStream2(cctx, &output, &input, ZSTD_e_end);
The spec makes it sound to me like ZSTD_e_end will always complete in a
single call if there’s enough space in the output buffer. So the only
team we have to loop would be when there isn’t enough space anyway:
It says this about ZSTD_e_end:
> flush operation is the same, and follows same rules as calling
> ZSTD_compressStream2() with ZSTD_e_flush.
Those rules being:
> Note that, if `output->size` is too small, a single invocation with
> ZSTD_e_flush might not be enough (return code > 0).
So it seems like it will only return a value > 0 if the output buffer is
definitely too small.
The spec also notes that the return value is greater than 0 if:
> >0 if some data still present within internal buffer (the value is
> minimal estimation of remaining size),
So it’s a minimum estimate. That’s another point that heavily implies
to me that if the return value were less than what’s left in the buffer,
the function wouldn’t return but still try to write it out, until it
realizes that there isn’t enough space in the output buffer, and then
return a value that exceeds the remaining output buffer size.
(Because if the function just played it safe, I would expect it to
return a maximum estimate.)
OTOH, if it were actually possible for ZSTD_e_end to finish a frame
earlier than the end of the input, I think it would make more sense to
use ZSTD_e_continue until the input is done and then finish with
ZSTD_e_end, like the spec seems to propose. That way, we’d always end
up with a single frame to make decompression simpler (and I think it
would also make more sense overall).
But anyway. From how I understand the spec, this code simply always
ends up creating a single frame or erroring out, without looping ever.
So it isn’t exactly wrong, it just seems overly complicated. (Again,
assuming I understand the spec correctly. Which seems like a tough
thing to assume, because the spec is not exactly obvious to read...)
(Running some quick tests by converting some images with zstd
compression seems to confirm that whenever ZSTD_compressStream2()
returns, either zstd_ret > output.size - output.pos, or zstd_ret == 0
and input.pos == input.size. So none of the loops ever loop.)
Max
> +
> + if (ZSTD_isError(zstd_ret)) {
> + ret = -EIO;
> + goto out;
> + }
> + /* Dest buffer isn't big enough to store compressed content */
> + if (zstd_ret > output.size - output.pos) {
> + ret = -ENOMEM;
> + goto out;
> + }
> + } while (zstd_ret);
> + }
> + /* make sure we can safely return compressed buffer size with ssize_t */
> + assert(output.pos <= SSIZE_MAX);
> + ret = output.pos;
> +out:
> + ZSTD_freeCCtx(cctx);
> + return ret;
> +}
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 1/4] qcow2: introduce compression type feature
2020-04-21 8:11 ` [PATCH v20 1/4] qcow2: introduce compression type feature Denis Plotnikov
2020-04-21 10:40 ` Alberto Garcia
@ 2020-04-27 12:35 ` Max Reitz
1 sibling, 0 replies; 19+ messages in thread
From: Max Reitz @ 2020-04-27 12:35 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel
Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
[-- Attachment #1.1: Type: text/plain, Size: 2747 bytes --]
On 21.04.20 10:11, Denis Plotnikov wrote:
> The patch adds some preparation parts for incompatible compression type
> feature to qcow2 allowing the use different compression methods for
> image clusters (de)compressing.
>
> It is implied that the compression type is set on the image creation and
> can be changed only later by image conversion, thus compression type
> defines the only compression algorithm used for the image, and thus,
> for all image clusters.
>
> The goal of the feature is to add support of other compression methods
> to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
>
> The default compression is ZLIB. Images created with ZLIB compression type
> are backward compatible with older qemu versions.
>
> Adding of the compression type breaks a number of tests because now the
> compression type is reported on image creation and there are some changes
> in the qcow2 header in size and offsets.
>
> The tests are fixed in the following ways:
> * filter out compression_type for many tests
> * fix header size, feature table size and backing file offset
> affected tests: 031, 036, 061, 080
> header_size +=8: 1 byte compression type
> 7 bytes padding
> feature_table += 48: incompatible feature compression type
> backing_file_offset += 56 (8 + 48 -> header_change + feature_table_change)
> * add "compression type" for test output matching when it isn't filtered
> affected tests: 049, 060, 061, 065, 144, 182, 242, 255
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Reviewed-by: Eric Blake <eblake@redhat.com>
> QAPI part:
> Acked-by: Markus Armbruster <armbru@redhat.com>
> ---
> qapi/block-core.json | 22 +++++-
> block/qcow2.h | 20 +++++-
> include/block/block_int.h | 1 +
> block/qcow2.c | 113 +++++++++++++++++++++++++++++++
> tests/qemu-iotests/031.out | 14 ++--
> tests/qemu-iotests/036.out | 4 +-
> tests/qemu-iotests/049.out | 102 ++++++++++++++--------------
> tests/qemu-iotests/060.out | 1 +
> tests/qemu-iotests/061.out | 34 ++++++----
> tests/qemu-iotests/065 | 28 +++++---
> tests/qemu-iotests/080 | 2 +-
> tests/qemu-iotests/144.out | 4 +-
> tests/qemu-iotests/182.out | 2 +-
> tests/qemu-iotests/242.out | 5 ++
> tests/qemu-iotests/255.out | 8 +--
> tests/qemu-iotests/common.filter | 3 +-
> 16 files changed, 267 insertions(+), 96 deletions(-)
Reviewed-by: Max Reitz <mreitz@redhat.com>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 2/4] qcow2: rework the cluster compression routine
2020-04-21 8:11 ` [PATCH v20 2/4] qcow2: rework the cluster compression routine Denis Plotnikov
@ 2020-04-27 12:36 ` Max Reitz
0 siblings, 0 replies; 19+ messages in thread
From: Max Reitz @ 2020-04-27 12:36 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel
Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
[-- Attachment #1.1: Type: text/plain, Size: 571 bytes --]
On 21.04.20 10:11, Denis Plotnikov wrote:
> The patch enables processing the image compression type defined
> for the image and chooses an appropriate method for image clusters
> (de)compression.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Reviewed-by: Alberto Garcia <berto@igalia.com>
> ---
> block/qcow2-threads.c | 71 ++++++++++++++++++++++++++++++++++++-------
> 1 file changed, 60 insertions(+), 11 deletions(-)
Reviewed-by: Max Reitz <mreitz@redhat.com>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 4/4] iotests: 287: add qcow2 compression type test
2020-04-21 8:11 ` [PATCH v20 4/4] iotests: 287: add qcow2 compression type test Denis Plotnikov
2020-04-21 12:06 ` Vladimir Sementsov-Ogievskiy
@ 2020-04-27 13:29 ` Max Reitz
2020-04-28 11:41 ` Denis Plotnikov
1 sibling, 1 reply; 19+ messages in thread
From: Max Reitz @ 2020-04-27 13:29 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel
Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
[-- Attachment #1.1: Type: text/plain, Size: 4101 bytes --]
On 21.04.20 10:11, Denis Plotnikov wrote:
> The test checks fulfilling qcow2 requirements for the compression
> type feature and zstd compression type operability.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> tests/qemu-iotests/287 | 146 +++++++++++++++++++++++++++++++++++++
> tests/qemu-iotests/287.out | 67 +++++++++++++++++
> tests/qemu-iotests/group | 1 +
> 3 files changed, 214 insertions(+)
> create mode 100755 tests/qemu-iotests/287
> create mode 100644 tests/qemu-iotests/287.out
>
> diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
> new file mode 100755
> index 0000000000..156acc40ad
> --- /dev/null
> +++ b/tests/qemu-iotests/287
> @@ -0,0 +1,146 @@
> +#!/usr/bin/env bash
> +#
> +# Test case for an image using zstd compression
> +#
> +# Copyright (c) 2020 Virtuozzo International GmbH
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +# creator
> +owner=dplotnikov@virtuozzo.com
> +
> +seq="$(basename $0)"
> +echo "QA output created by $seq"
> +
> +status=1 # failure is the default!
> +
> +# standard environment
> +. ./common.rc
> +. ./common.filter
> +
> +# This tests qocw2-specific low-level functionality
> +_supported_fmt qcow2
> +_supported_proto file
> +_supported_os Linux
This test doesn’t work with compat=0.10 (because we can’t store a
non-default compression type there) or data_file (because those don’t
support compression), so those options should be marked as unsupported.
(It does seem to work with any refcount_bits, though.)
> +
> +COMPR_IMG="$TEST_IMG.compressed"
> +RAND_FILE="$TEST_DIR/rand_data"
> +
> +_cleanup()
> +{
> + _cleanup_test_img
> + rm -f "$COMPR_IMG"
Using _rm_test_img() would be nicer. There shouldn’t be a functional
difference here because there’d only be one with external data files (I
think), which won’t work with this test, but still.
> + rm -f "$RAND_FILE"
> +}
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# for all the cases
> +CLUSTER_SIZE=65536
> +
> +# Check if we can run this test.
> +if IMGOPTS='compression_type=zstd' _make_test_img 64M |
> + grep "Invalid parameter 'zstd'"; then
> + _notrun "ZSTD is disabled"
> +fi
> +
> +echo
> +echo "=== Testing compression type incompatible bit setting for zlib ==="
> +echo
> +IMGOPTS='compression_type=zlib' _make_test_img 64M
Please use -o so user options are still considered.
(i.e., _make_test_img -o compression_type=zlib)
[...]
> +echo
> +echo "=== Testing incompressible cluster processing with zstd ==="
> +echo
> +# create a 2M image and fill it with 1M likely incompressible data
> +# and 1M compressible data
> +dd if=/dev/urandom of="$RAND_FILE" bs=1M count=1 seek=1
> +QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS_NO_FMT" \
> +$QEMU_IO -f raw -c "write -P 0xFA 0 1M" "$RAND_FILE" | _filter_qemu_io
> +$QEMU_IMG convert -f raw -O $IMGFMT -c "$RAND_FILE" "$TEST_IMG" | _filter_qemu_io
> +
> +$QEMU_IMG convert -O $IMGFMT -c -o compression_type=zstd \
> + "$TEST_IMG" "$COMPR_IMG"
Again, it would be nice to not discard the user-supplied options here,
and maybe it would also be nicer to explicitly pass the compression type
for the first convert, too. So we’d use
-o "$(_optstr_add "$IMGOPTS" "compression_type=zlib")"
for the first convert, and
-o "$(_optstr_add "$IMGOPTS" "compression_type=zstd")"
for the second one.
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 3/4] qcow2: add zstd cluster compression
2020-04-27 12:35 ` Max Reitz
@ 2020-04-27 19:26 ` Denis Plotnikov
2020-04-28 6:16 ` Max Reitz
0 siblings, 1 reply; 19+ messages in thread
From: Denis Plotnikov @ 2020-04-27 19:26 UTC (permalink / raw)
To: Max Reitz, qemu-devel; +Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
On 27.04.2020 15:35, Max Reitz wrote:
> On 21.04.20 10:11, Denis Plotnikov wrote:
>> zstd significantly reduces cluster compression time.
>> It provides better compression performance maintaining
>> the same level of the compression ratio in comparison with
>> zlib, which, at the moment, is the only compression
>> method available.
>>
>> The performance test results:
>> Test compresses and decompresses qemu qcow2 image with just
>> installed rhel-7.6 guest.
>> Image cluster size: 64K. Image on disk size: 2.2G
>>
>> The test was conducted with brd disk to reduce the influence
>> of disk subsystem to the test results.
>> The results is given in seconds.
>>
>> compress cmd:
>> time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
>> src.img [zlib|zstd]_compressed.img
>> decompress cmd
>> time ./qemu-img convert -O qcow2
>> [zlib|zstd]_compressed.img uncompressed.img
>>
>> compression decompression
>> zlib zstd zlib zstd
>> ------------------------------------------------------------
>> real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
>> user 65.0 15.8 5.3 2.5
>> sys 3.3 0.2 2.0 2.0
>>
>> Both ZLIB and ZSTD gave the same compression ratio: 1.57
>> compressed image size in both cases: 1.4G
>>
>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> Reviewed-by: Alberto Garcia <berto@igalia.com>
>> QAPI part:
>> Acked-by: Markus Armbruster <armbru@redhat.com>
>> ---
>> docs/interop/qcow2.txt | 1 +
>> configure | 2 +-
>> qapi/block-core.json | 3 +-
>> block/qcow2-threads.c | 157 +++++++++++++++++++++++++++++++++++++++++
>> block/qcow2.c | 7 ++
>> 5 files changed, 168 insertions(+), 2 deletions(-)
> [...]
>
>> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
>> index 7dbaf53489..0525718704 100644
>> --- a/block/qcow2-threads.c
>> +++ b/block/qcow2-threads.c
>> @@ -28,6 +28,11 @@
>> #define ZLIB_CONST
>> #include <zlib.h>
>>
>> +#ifdef CONFIG_ZSTD
>> +#include <zstd.h>
>> +#include <zstd_errors.h>
>> +#endif
>> +
>> #include "qcow2.h"
>> #include "block/thread-pool.h"
>> #include "crypto.h"
>> @@ -166,6 +171,148 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
>> return ret;
>> }
>>
>> +#ifdef CONFIG_ZSTD
>> +
>> +/*
>> + * qcow2_zstd_compress()
>> + *
>> + * Compress @src_size bytes of data using zstd compression method
>> + *
>> + * @dest - destination buffer, @dest_size bytes
>> + * @src - source buffer, @src_size bytes
>> + *
>> + * Returns: compressed size on success
>> + * -ENOMEM destination buffer is not enough to store compressed data
>> + * -EIO on any other error
>> + */
>> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
>> + const void *src, size_t src_size)
>> +{
>> + ssize_t ret;
>> + ZSTD_outBuffer output = { dest, dest_size, 0 };
>> + ZSTD_inBuffer input = { src, src_size, 0 };
> Minor style note: I think it’d be nicer to use designated initializers here.
>
>> + ZSTD_CCtx *cctx = ZSTD_createCCtx();
>> +
>> + if (!cctx) {
>> + return -EIO;
>> + }
>> + /*
>> + * Use the zstd streamed interface for symmetry with decompression,
>> + * where streaming is essential since we don't record the exact
>> + * compressed size.
>> + *
>> + * In the loop, we try to compress all the data into one zstd frame.
>> + * ZSTD_compressStream2 potentially can finish a frame earlier
>> + * than the full input data is consumed. That's why we are looping
>> + * until all the input data is consumed.
>> + */
>> + while (input.pos < input.size) {
>> + size_t zstd_ret;
>> + /*
>> + * ZSTD spec: "You must continue calling ZSTD_compressStream2()
>> + * with ZSTD_e_end until it returns 0, at which point you are
>> + * free to start a new frame". We assume that "start a new frame"
>> + * means call ZSTD_compressStream2 in the very beginning or when
>> + * ZSTD_compressStream2 has returned with 0.
>> + */
>> + do {
>> + zstd_ret = ZSTD_compressStream2(cctx, &output, &input, ZSTD_e_end);
> The spec makes it sound to me like ZSTD_e_end will always complete in a
> single call if there’s enough space in the output buffer. So the only
> team we have to loop would be when there isn’t enough space anyway:
>
> It says this about ZSTD_e_end:
>> flush operation is the same, and follows same rules as calling
>> ZSTD_compressStream2() with ZSTD_e_flush.
> Those rules being:
>> Note that, if `output->size` is too small, a single invocation with
>> ZSTD_e_flush might not be enough (return code > 0).
> So it seems like it will only return a value > 0 if the output buffer is
> definitely too small.
>
> The spec also notes that the return value is greater than 0 if:
>>> 0 if some data still present within internal buffer (the value is
>> minimal estimation of remaining size),
> So it’s a minimum estimate. That’s another point that heavily implies
> to me that if the return value were less than what’s left in the buffer,
> the function wouldn’t return but still try to write it out, until it
> realizes that there isn’t enough space in the output buffer, and then
> return a value that exceeds the remaining output buffer size.
>
> (Because if the function just played it safe, I would expect it to
> return a maximum estimate.)
>
>
> OTOH, if it were actually possible for ZSTD_e_end to finish a frame
> earlier than the end of the input, I think it would make more sense to
> use ZSTD_e_continue until the input is done and then finish with
> ZSTD_e_end, like the spec seems to propose. That way, we’d always end
> up with a single frame to make decompression simpler (and I think it
> would also make more sense overall).
>
>
> But anyway. From how I understand the spec, this code simply always
> ends up creating a single frame or erroring out, without looping ever.
> So it isn’t exactly wrong, it just seems overly complicated. (Again,
> assuming I understand the spec correctly. Which seems like a tough
> thing to assume, because the spec is not exactly obvious to read...)
>
> (Running some quick tests by converting some images with zstd
> compression seems to confirm that whenever ZSTD_compressStream2()
> returns, either zstd_ret > output.size - output.pos, or zstd_ret == 0
> and input.pos == input.size. So none of the loops ever loop.)
>
> Max
So, what should we do?
1. Rely on the test that there's no need for the loop:
* make one ZSTD_compressStream2() call
* make sure it returned with zstd_ret == 0 and
input.pos == input.size.
if so, return with the size
* if not, check that zstd_ret > output.size - output.pos
if so, return with -ENOMEM
* if none above return with -EIO
This should cover the majority of the compressing cases
2. Leave the loop as is, because of the documentation:
"You *must* continue calling ZSTD_compressStream2() with ZSTD_e_end
until it returns 0,
at which point you are free to start a new frame."
This fulfills the documentation requirements.
3. Any other option?
Denis
>
>> +
>> + if (ZSTD_isError(zstd_ret)) {
>> + ret = -EIO;
>> + goto out;
>> + }
>> + /* Dest buffer isn't big enough to store compressed content */
>> + if (zstd_ret > output.size - output.pos) {
>> + ret = -ENOMEM;
>> + goto out;
>> + }
>> + } while (zstd_ret);
>> + }
>> + /* make sure we can safely return compressed buffer size with ssize_t */
>> + assert(output.pos <= SSIZE_MAX);
>> + ret = output.pos;
>> +out:
>> + ZSTD_freeCCtx(cctx);
>> + return ret;
>> +}
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 3/4] qcow2: add zstd cluster compression
2020-04-27 19:26 ` Denis Plotnikov
@ 2020-04-28 6:16 ` Max Reitz
2020-04-28 7:23 ` Denis Plotnikov
0 siblings, 1 reply; 19+ messages in thread
From: Max Reitz @ 2020-04-28 6:16 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel
Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
[-- Attachment #1.1: Type: text/plain, Size: 8821 bytes --]
On 27.04.20 21:26, Denis Plotnikov wrote:
>
>
> On 27.04.2020 15:35, Max Reitz wrote:
>> On 21.04.20 10:11, Denis Plotnikov wrote:
>>> zstd significantly reduces cluster compression time.
>>> It provides better compression performance maintaining
>>> the same level of the compression ratio in comparison with
>>> zlib, which, at the moment, is the only compression
>>> method available.
>>>
>>> The performance test results:
>>> Test compresses and decompresses qemu qcow2 image with just
>>> installed rhel-7.6 guest.
>>> Image cluster size: 64K. Image on disk size: 2.2G
>>>
>>> The test was conducted with brd disk to reduce the influence
>>> of disk subsystem to the test results.
>>> The results is given in seconds.
>>>
>>> compress cmd:
>>> time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
>>> src.img [zlib|zstd]_compressed.img
>>> decompress cmd
>>> time ./qemu-img convert -O qcow2
>>> [zlib|zstd]_compressed.img uncompressed.img
>>>
>>> compression decompression
>>> zlib zstd zlib zstd
>>> ------------------------------------------------------------
>>> real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
>>> user 65.0 15.8 5.3 2.5
>>> sys 3.3 0.2 2.0 2.0
>>>
>>> Both ZLIB and ZSTD gave the same compression ratio: 1.57
>>> compressed image size in both cases: 1.4G
>>>
>>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>> Reviewed-by: Alberto Garcia <berto@igalia.com>
>>> QAPI part:
>>> Acked-by: Markus Armbruster <armbru@redhat.com>
>>> ---
>>> docs/interop/qcow2.txt | 1 +
>>> configure | 2 +-
>>> qapi/block-core.json | 3 +-
>>> block/qcow2-threads.c | 157 +++++++++++++++++++++++++++++++++++++++++
>>> block/qcow2.c | 7 ++
>>> 5 files changed, 168 insertions(+), 2 deletions(-)
>> [...]
>>
>>> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
>>> index 7dbaf53489..0525718704 100644
>>> --- a/block/qcow2-threads.c
>>> +++ b/block/qcow2-threads.c
>>> @@ -28,6 +28,11 @@
>>> #define ZLIB_CONST
>>> #include <zlib.h>
>>> +#ifdef CONFIG_ZSTD
>>> +#include <zstd.h>
>>> +#include <zstd_errors.h>
>>> +#endif
>>> +
>>> #include "qcow2.h"
>>> #include "block/thread-pool.h"
>>> #include "crypto.h"
>>> @@ -166,6 +171,148 @@ static ssize_t qcow2_zlib_decompress(void
>>> *dest, size_t dest_size,
>>> return ret;
>>> }
>>> +#ifdef CONFIG_ZSTD
>>> +
>>> +/*
>>> + * qcow2_zstd_compress()
>>> + *
>>> + * Compress @src_size bytes of data using zstd compression method
>>> + *
>>> + * @dest - destination buffer, @dest_size bytes
>>> + * @src - source buffer, @src_size bytes
>>> + *
>>> + * Returns: compressed size on success
>>> + * -ENOMEM destination buffer is not enough to store
>>> compressed data
>>> + * -EIO on any other error
>>> + */
>>> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
>>> + const void *src, size_t src_size)
>>> +{
>>> + ssize_t ret;
>>> + ZSTD_outBuffer output = { dest, dest_size, 0 };
>>> + ZSTD_inBuffer input = { src, src_size, 0 };
>> Minor style note: I think it’d be nicer to use designated initializers
>> here.
>>
>>> + ZSTD_CCtx *cctx = ZSTD_createCCtx();
>>> +
>>> + if (!cctx) {
>>> + return -EIO;
>>> + }
>>> + /*
>>> + * Use the zstd streamed interface for symmetry with decompression,
>>> + * where streaming is essential since we don't record the exact
>>> + * compressed size.
>>> + *
>>> + * In the loop, we try to compress all the data into one zstd
>>> frame.
>>> + * ZSTD_compressStream2 potentially can finish a frame earlier
>>> + * than the full input data is consumed. That's why we are looping
>>> + * until all the input data is consumed.
>>> + */
>>> + while (input.pos < input.size) {
>>> + size_t zstd_ret;
>>> + /*
>>> + * ZSTD spec: "You must continue calling ZSTD_compressStream2()
>>> + * with ZSTD_e_end until it returns 0, at which point you are
>>> + * free to start a new frame". We assume that "start a new
>>> frame"
>>> + * means call ZSTD_compressStream2 in the very beginning or
>>> when
>>> + * ZSTD_compressStream2 has returned with 0.
>>> + */
>>> + do {
>>> + zstd_ret = ZSTD_compressStream2(cctx, &output, &input,
>>> ZSTD_e_end);
>> The spec makes it sound to me like ZSTD_e_end will always complete in a
>> single call if there’s enough space in the output buffer. So the only
>> team we have to loop would be when there isn’t enough space anyway:
>>
>> It says this about ZSTD_e_end:
>>> flush operation is the same, and follows same rules as calling
>>> ZSTD_compressStream2() with ZSTD_e_flush.
>> Those rules being:
>>> Note that, if `output->size` is too small, a single invocation with
>>> ZSTD_e_flush might not be enough (return code > 0).
>> So it seems like it will only return a value > 0 if the output buffer is
>> definitely too small.
>>
>> The spec also notes that the return value is greater than 0 if:
>>>> 0 if some data still present within internal buffer (the value is
>>> minimal estimation of remaining size),
>> So it’s a minimum estimate. That’s another point that heavily implies
>> to me that if the return value were less than what’s left in the buffer,
>> the function wouldn’t return but still try to write it out, until it
>> realizes that there isn’t enough space in the output buffer, and then
>> return a value that exceeds the remaining output buffer size.
>>
>> (Because if the function just played it safe, I would expect it to
>> return a maximum estimate.)
>>
>>
>> OTOH, if it were actually possible for ZSTD_e_end to finish a frame
>> earlier than the end of the input, I think it would make more sense to
>> use ZSTD_e_continue until the input is done and then finish with
>> ZSTD_e_end, like the spec seems to propose. That way, we’d always end
>> up with a single frame to make decompression simpler (and I think it
>> would also make more sense overall).
>>
>>
>> But anyway. From how I understand the spec, this code simply always
>> ends up creating a single frame or erroring out, without looping ever.
>> So it isn’t exactly wrong, it just seems overly complicated. (Again,
>> assuming I understand the spec correctly. Which seems like a tough
>> thing to assume, because the spec is not exactly obvious to read...)
>>
>> (Running some quick tests by converting some images with zstd
>> compression seems to confirm that whenever ZSTD_compressStream2()
>> returns, either zstd_ret > output.size - output.pos, or zstd_ret == 0
>> and input.pos == input.size. So none of the loops ever loop.)
>>
>> Max
>
> So, what should we do?
>
> 1. Rely on the test that there's no need for the loop:
> * make one ZSTD_compressStream2() call
> * make sure it returned with zstd_ret == 0 and
> input.pos == input.size.
> if so, return with the size
> * if not, check that zstd_ret > output.size - output.pos
> if so, return with -ENOMEM
> * if none above return with -EIO
>
> This should cover the majority of the compressing cases
According to how I interpret the spec, “none of the above” should never
happen except for ZSTD_isError(zstd_ret), so this should cover all
compressing cases, actually.
> 2. Leave the loop as is, because of the documentation:
> "You *must* continue calling ZSTD_compressStream2() with ZSTD_e_end
> until it returns 0,
> at which point you are free to start a new frame."
As far as I can see, the return value is always 0 or greater than the
remaining buffer space, so this will always be satisfied even without a
loop. (We will always have one of three cases: (1) Success and all
input has been consumed, (2) ZSTD_isError(zstd_ret), so we return -EIO,
(3) zstd_ret > output.size - output.pos, so we return -ENOMEM.
I interpret the “You *must* continue until it returns 0” as “If there is
no sufficient space in the output buffer, this function will return a
value greater than 0 indicating how much space is at least still
required. The caller is free to supply a greater output buffer for the
next call (by supplying a different ZSTD_outBuffer structure), and then
we’ll try again.”
(I.e., retrying with the same ZSTD_outBuffer will make the function
return immediately because it knows that it’s insufficient.)
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 3/4] qcow2: add zstd cluster compression
2020-04-28 6:16 ` Max Reitz
@ 2020-04-28 7:23 ` Denis Plotnikov
2020-04-28 10:17 ` Max Reitz
0 siblings, 1 reply; 19+ messages in thread
From: Denis Plotnikov @ 2020-04-28 7:23 UTC (permalink / raw)
To: Max Reitz, qemu-devel; +Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
On 28.04.2020 09:16, Max Reitz wrote:
> On 27.04.20 21:26, Denis Plotnikov wrote:
>>
>> On 27.04.2020 15:35, Max Reitz wrote:
>>> On 21.04.20 10:11, Denis Plotnikov wrote:
>>>> zstd significantly reduces cluster compression time.
>>>> It provides better compression performance maintaining
>>>> the same level of the compression ratio in comparison with
>>>> zlib, which, at the moment, is the only compression
>>>> method available.
>>>>
>>>> The performance test results:
>>>> Test compresses and decompresses qemu qcow2 image with just
>>>> installed rhel-7.6 guest.
>>>> Image cluster size: 64K. Image on disk size: 2.2G
>>>>
>>>> The test was conducted with brd disk to reduce the influence
>>>> of disk subsystem to the test results.
>>>> The results is given in seconds.
>>>>
>>>> compress cmd:
>>>> time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
>>>> src.img [zlib|zstd]_compressed.img
>>>> decompress cmd
>>>> time ./qemu-img convert -O qcow2
>>>> [zlib|zstd]_compressed.img uncompressed.img
>>>>
>>>> compression decompression
>>>> zlib zstd zlib zstd
>>>> ------------------------------------------------------------
>>>> real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
>>>> user 65.0 15.8 5.3 2.5
>>>> sys 3.3 0.2 2.0 2.0
>>>>
>>>> Both ZLIB and ZSTD gave the same compression ratio: 1.57
>>>> compressed image size in both cases: 1.4G
>>>>
>>>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>>>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>>> Reviewed-by: Alberto Garcia <berto@igalia.com>
>>>> QAPI part:
>>>> Acked-by: Markus Armbruster <armbru@redhat.com>
>>>> ---
>>>> docs/interop/qcow2.txt | 1 +
>>>> configure | 2 +-
>>>> qapi/block-core.json | 3 +-
>>>> block/qcow2-threads.c | 157 +++++++++++++++++++++++++++++++++++++++++
>>>> block/qcow2.c | 7 ++
>>>> 5 files changed, 168 insertions(+), 2 deletions(-)
>>> [...]
>>>
>>>> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
>>>> index 7dbaf53489..0525718704 100644
>>>> --- a/block/qcow2-threads.c
>>>> +++ b/block/qcow2-threads.c
>>>> @@ -28,6 +28,11 @@
>>>> #define ZLIB_CONST
>>>> #include <zlib.h>
>>>> +#ifdef CONFIG_ZSTD
>>>> +#include <zstd.h>
>>>> +#include <zstd_errors.h>
>>>> +#endif
>>>> +
>>>> #include "qcow2.h"
>>>> #include "block/thread-pool.h"
>>>> #include "crypto.h"
>>>> @@ -166,6 +171,148 @@ static ssize_t qcow2_zlib_decompress(void
>>>> *dest, size_t dest_size,
>>>> return ret;
>>>> }
>>>> +#ifdef CONFIG_ZSTD
>>>> +
>>>> +/*
>>>> + * qcow2_zstd_compress()
>>>> + *
>>>> + * Compress @src_size bytes of data using zstd compression method
>>>> + *
>>>> + * @dest - destination buffer, @dest_size bytes
>>>> + * @src - source buffer, @src_size bytes
>>>> + *
>>>> + * Returns: compressed size on success
>>>> + * -ENOMEM destination buffer is not enough to store
>>>> compressed data
>>>> + * -EIO on any other error
>>>> + */
>>>> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
>>>> + const void *src, size_t src_size)
>>>> +{
>>>> + ssize_t ret;
>>>> + ZSTD_outBuffer output = { dest, dest_size, 0 };
>>>> + ZSTD_inBuffer input = { src, src_size, 0 };
>>> Minor style note: I think it’d be nicer to use designated initializers
>>> here.
>>>
>>>> + ZSTD_CCtx *cctx = ZSTD_createCCtx();
>>>> +
>>>> + if (!cctx) {
>>>> + return -EIO;
>>>> + }
>>>> + /*
>>>> + * Use the zstd streamed interface for symmetry with decompression,
>>>> + * where streaming is essential since we don't record the exact
>>>> + * compressed size.
>>>> + *
>>>> + * In the loop, we try to compress all the data into one zstd
>>>> frame.
>>>> + * ZSTD_compressStream2 potentially can finish a frame earlier
>>>> + * than the full input data is consumed. That's why we are looping
>>>> + * until all the input data is consumed.
>>>> + */
>>>> + while (input.pos < input.size) {
>>>> + size_t zstd_ret;
>>>> + /*
>>>> + * ZSTD spec: "You must continue calling ZSTD_compressStream2()
>>>> + * with ZSTD_e_end until it returns 0, at which point you are
>>>> + * free to start a new frame". We assume that "start a new
>>>> frame"
>>>> + * means call ZSTD_compressStream2 in the very beginning or
>>>> when
>>>> + * ZSTD_compressStream2 has returned with 0.
>>>> + */
>>>> + do {
>>>> + zstd_ret = ZSTD_compressStream2(cctx, &output, &input,
>>>> ZSTD_e_end);
>>> The spec makes it sound to me like ZSTD_e_end will always complete in a
>>> single call if there’s enough space in the output buffer. So the only
>>> team we have to loop would be when there isn’t enough space anyway:
>>>
>>> It says this about ZSTD_e_end:
>>>> flush operation is the same, and follows same rules as calling
>>>> ZSTD_compressStream2() with ZSTD_e_flush.
>>> Those rules being:
>>>> Note that, if `output->size` is too small, a single invocation with
>>>> ZSTD_e_flush might not be enough (return code > 0).
>>> So it seems like it will only return a value > 0 if the output buffer is
>>> definitely too small.
>>>
>>> The spec also notes that the return value is greater than 0 if:
>>>>> 0 if some data still present within internal buffer (the value is
>>>> minimal estimation of remaining size),
>>> So it’s a minimum estimate. That’s another point that heavily implies
>>> to me that if the return value were less than what’s left in the buffer,
>>> the function wouldn’t return but still try to write it out, until it
>>> realizes that there isn’t enough space in the output buffer, and then
>>> return a value that exceeds the remaining output buffer size.
>>>
>>> (Because if the function just played it safe, I would expect it to
>>> return a maximum estimate.)
>>>
>>>
>>> OTOH, if it were actually possible for ZSTD_e_end to finish a frame
>>> earlier than the end of the input, I think it would make more sense to
>>> use ZSTD_e_continue until the input is done and then finish with
>>> ZSTD_e_end, like the spec seems to propose. That way, we’d always end
>>> up with a single frame to make decompression simpler (and I think it
>>> would also make more sense overall).
>>>
>>>
>>> But anyway. From how I understand the spec, this code simply always
>>> ends up creating a single frame or erroring out, without looping ever.
>>> So it isn’t exactly wrong, it just seems overly complicated. (Again,
>>> assuming I understand the spec correctly. Which seems like a tough
>>> thing to assume, because the spec is not exactly obvious to read...)
>>>
>>> (Running some quick tests by converting some images with zstd
>>> compression seems to confirm that whenever ZSTD_compressStream2()
>>> returns, either zstd_ret > output.size - output.pos, or zstd_ret == 0
>>> and input.pos == input.size. So none of the loops ever loop.)
>>>
>>> Max
>> So, what should we do?
>>
>> 1. Rely on the test that there's no need for the loop:
>> * make one ZSTD_compressStream2() call
>> * make sure it returned with zstd_ret == 0 and
>> input.pos == input.size.
>> if so, return with the size
>> * if not, check that zstd_ret > output.size - output.pos
>> if so, return with -ENOMEM
>> * if none above return with -EIO
>>
>> This should cover the majority of the compressing cases
> According to how I interpret the spec, “none of the above” should never
> happen except for ZSTD_isError(zstd_ret), so this should cover all
> compressing cases, actually.
>
>> 2. Leave the loop as is, because of the documentation:
>> "You *must* continue calling ZSTD_compressStream2() with ZSTD_e_end
>> until it returns 0,
>> at which point you are free to start a new frame."
> As far as I can see, the return value is always 0 or greater than the
> remaining buffer space, so this will always be satisfied even without a
> loop. (We will always have one of three cases: (1) Success and all
> input has been consumed, (2) ZSTD_isError(zstd_ret), so we return -EIO,
> (3) zstd_ret > output.size - output.pos, so we return -ENOMEM.
>
> I interpret the “You *must* continue until it returns 0” as “If there is
> no sufficient space in the output buffer, this function will return a
> value greater than 0 indicating how much space is at least still
> required. The caller is free to supply a greater output buffer for the
> next call (by supplying a different ZSTD_outBuffer structure), and then
> we’ll try again.”
> (I.e., retrying with the same ZSTD_outBuffer will make the function
> return immediately because it knows that it’s insufficient.)
>
> Max
ok, removing the loop sounds reasonable.
My only concern is that *must* in the doc.
Could ZSTD-lib change the logic in the future relying on the fact
that they make users use ZSTD_compressStream() in a loop.
Honestly, I can't imagine the case when they would want to do that,
but still.
Without the loop we're protected even in this case. The worst thing
could happen because of that is qcow2_zstd_compress() would return
with -EIO more frequently.
So, if I understand correctly, you are ok with removing the loop.
Denis
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 3/4] qcow2: add zstd cluster compression
2020-04-28 7:23 ` Denis Plotnikov
@ 2020-04-28 10:17 ` Max Reitz
0 siblings, 0 replies; 19+ messages in thread
From: Max Reitz @ 2020-04-28 10:17 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel
Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
[-- Attachment #1.1: Type: text/plain, Size: 10615 bytes --]
On 28.04.20 09:23, Denis Plotnikov wrote:
>
>
> On 28.04.2020 09:16, Max Reitz wrote:
>> On 27.04.20 21:26, Denis Plotnikov wrote:
>>>
>>> On 27.04.2020 15:35, Max Reitz wrote:
>>>> On 21.04.20 10:11, Denis Plotnikov wrote:
>>>>> zstd significantly reduces cluster compression time.
>>>>> It provides better compression performance maintaining
>>>>> the same level of the compression ratio in comparison with
>>>>> zlib, which, at the moment, is the only compression
>>>>> method available.
>>>>>
>>>>> The performance test results:
>>>>> Test compresses and decompresses qemu qcow2 image with just
>>>>> installed rhel-7.6 guest.
>>>>> Image cluster size: 64K. Image on disk size: 2.2G
>>>>>
>>>>> The test was conducted with brd disk to reduce the influence
>>>>> of disk subsystem to the test results.
>>>>> The results is given in seconds.
>>>>>
>>>>> compress cmd:
>>>>> time ./qemu-img convert -O qcow2 -c -o
>>>>> compression_type=[zlib|zstd]
>>>>> src.img [zlib|zstd]_compressed.img
>>>>> decompress cmd
>>>>> time ./qemu-img convert -O qcow2
>>>>> [zlib|zstd]_compressed.img uncompressed.img
>>>>>
>>>>> compression decompression
>>>>> zlib zstd zlib zstd
>>>>> ------------------------------------------------------------
>>>>> real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
>>>>> user 65.0 15.8 5.3 2.5
>>>>> sys 3.3 0.2 2.0 2.0
>>>>>
>>>>> Both ZLIB and ZSTD gave the same compression ratio: 1.57
>>>>> compressed image size in both cases: 1.4G
>>>>>
>>>>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>>>>> Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>>>> Reviewed-by: Alberto Garcia <berto@igalia.com>
>>>>> QAPI part:
>>>>> Acked-by: Markus Armbruster <armbru@redhat.com>
>>>>> ---
>>>>> docs/interop/qcow2.txt | 1 +
>>>>> configure | 2 +-
>>>>> qapi/block-core.json | 3 +-
>>>>> block/qcow2-threads.c | 157
>>>>> +++++++++++++++++++++++++++++++++++++++++
>>>>> block/qcow2.c | 7 ++
>>>>> 5 files changed, 168 insertions(+), 2 deletions(-)
>>>> [...]
>>>>
>>>>> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
>>>>> index 7dbaf53489..0525718704 100644
>>>>> --- a/block/qcow2-threads.c
>>>>> +++ b/block/qcow2-threads.c
>>>>> @@ -28,6 +28,11 @@
>>>>> #define ZLIB_CONST
>>>>> #include <zlib.h>
>>>>> +#ifdef CONFIG_ZSTD
>>>>> +#include <zstd.h>
>>>>> +#include <zstd_errors.h>
>>>>> +#endif
>>>>> +
>>>>> #include "qcow2.h"
>>>>> #include "block/thread-pool.h"
>>>>> #include "crypto.h"
>>>>> @@ -166,6 +171,148 @@ static ssize_t qcow2_zlib_decompress(void
>>>>> *dest, size_t dest_size,
>>>>> return ret;
>>>>> }
>>>>> +#ifdef CONFIG_ZSTD
>>>>> +
>>>>> +/*
>>>>> + * qcow2_zstd_compress()
>>>>> + *
>>>>> + * Compress @src_size bytes of data using zstd compression method
>>>>> + *
>>>>> + * @dest - destination buffer, @dest_size bytes
>>>>> + * @src - source buffer, @src_size bytes
>>>>> + *
>>>>> + * Returns: compressed size on success
>>>>> + * -ENOMEM destination buffer is not enough to store
>>>>> compressed data
>>>>> + * -EIO on any other error
>>>>> + */
>>>>> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
>>>>> + const void *src, size_t src_size)
>>>>> +{
>>>>> + ssize_t ret;
>>>>> + ZSTD_outBuffer output = { dest, dest_size, 0 };
>>>>> + ZSTD_inBuffer input = { src, src_size, 0 };
>>>> Minor style note: I think it’d be nicer to use designated initializers
>>>> here.
>>>>
>>>>> + ZSTD_CCtx *cctx = ZSTD_createCCtx();
>>>>> +
>>>>> + if (!cctx) {
>>>>> + return -EIO;
>>>>> + }
>>>>> + /*
>>>>> + * Use the zstd streamed interface for symmetry with
>>>>> decompression,
>>>>> + * where streaming is essential since we don't record the exact
>>>>> + * compressed size.
>>>>> + *
>>>>> + * In the loop, we try to compress all the data into one zstd
>>>>> frame.
>>>>> + * ZSTD_compressStream2 potentially can finish a frame earlier
>>>>> + * than the full input data is consumed. That's why we are
>>>>> looping
>>>>> + * until all the input data is consumed.
>>>>> + */
>>>>> + while (input.pos < input.size) {
>>>>> + size_t zstd_ret;
>>>>> + /*
>>>>> + * ZSTD spec: "You must continue calling
>>>>> ZSTD_compressStream2()
>>>>> + * with ZSTD_e_end until it returns 0, at which point you are
>>>>> + * free to start a new frame". We assume that "start a new
>>>>> frame"
>>>>> + * means call ZSTD_compressStream2 in the very beginning or
>>>>> when
>>>>> + * ZSTD_compressStream2 has returned with 0.
>>>>> + */
>>>>> + do {
>>>>> + zstd_ret = ZSTD_compressStream2(cctx, &output, &input,
>>>>> ZSTD_e_end);
>>>> The spec makes it sound to me like ZSTD_e_end will always complete in a
>>>> single call if there’s enough space in the output buffer. So the only
>>>> team we have to loop would be when there isn’t enough space anyway:
>>>>
>>>> It says this about ZSTD_e_end:
>>>>> flush operation is the same, and follows same rules as calling
>>>>> ZSTD_compressStream2() with ZSTD_e_flush.
>>>> Those rules being:
>>>>> Note that, if `output->size` is too small, a single invocation with
>>>>> ZSTD_e_flush might not be enough (return code > 0).
>>>> So it seems like it will only return a value > 0 if the output
>>>> buffer is
>>>> definitely too small.
>>>>
>>>> The spec also notes that the return value is greater than 0 if:
>>>>>> 0 if some data still present within internal buffer (the value is
>>>>> minimal estimation of remaining size),
>>>> So it’s a minimum estimate. That’s another point that heavily implies
>>>> to me that if the return value were less than what’s left in the
>>>> buffer,
>>>> the function wouldn’t return but still try to write it out, until it
>>>> realizes that there isn’t enough space in the output buffer, and then
>>>> return a value that exceeds the remaining output buffer size.
>>>>
>>>> (Because if the function just played it safe, I would expect it to
>>>> return a maximum estimate.)
>>>>
>>>>
>>>> OTOH, if it were actually possible for ZSTD_e_end to finish a frame
>>>> earlier than the end of the input, I think it would make more sense to
>>>> use ZSTD_e_continue until the input is done and then finish with
>>>> ZSTD_e_end, like the spec seems to propose. That way, we’d always end
>>>> up with a single frame to make decompression simpler (and I think it
>>>> would also make more sense overall).
>>>>
>>>>
>>>> But anyway. From how I understand the spec, this code simply always
>>>> ends up creating a single frame or erroring out, without looping ever.
>>>> So it isn’t exactly wrong, it just seems overly complicated. (Again,
>>>> assuming I understand the spec correctly. Which seems like a tough
>>>> thing to assume, because the spec is not exactly obvious to read...)
>>>>
>>>> (Running some quick tests by converting some images with zstd
>>>> compression seems to confirm that whenever ZSTD_compressStream2()
>>>> returns, either zstd_ret > output.size - output.pos, or zstd_ret == 0
>>>> and input.pos == input.size. So none of the loops ever loop.)
>>>>
>>>> Max
>>> So, what should we do?
>>>
>>> 1. Rely on the test that there's no need for the loop:
>>> * make one ZSTD_compressStream2() call
>>> * make sure it returned with zstd_ret == 0 and
>>> input.pos == input.size.
>>> if so, return with the size
>>> * if not, check that zstd_ret > output.size - output.pos
>>> if so, return with -ENOMEM
>>> * if none above return with -EIO
>>>
>>> This should cover the majority of the compressing cases
>> According to how I interpret the spec, “none of the above” should never
>> happen except for ZSTD_isError(zstd_ret), so this should cover all
>> compressing cases, actually.
>>
>>> 2. Leave the loop as is, because of the documentation:
>>> "You *must* continue calling ZSTD_compressStream2() with ZSTD_e_end
>>> until it returns 0,
>>> at which point you are free to start a new frame."
>> As far as I can see, the return value is always 0 or greater than the
>> remaining buffer space, so this will always be satisfied even without a
>> loop. (We will always have one of three cases: (1) Success and all
>> input has been consumed, (2) ZSTD_isError(zstd_ret), so we return -EIO,
>> (3) zstd_ret > output.size - output.pos, so we return -ENOMEM.
>>
>> I interpret the “You *must* continue until it returns 0” as “If there is
>> no sufficient space in the output buffer, this function will return a
>> value greater than 0 indicating how much space is at least still
>> required. The caller is free to supply a greater output buffer for the
>> next call (by supplying a different ZSTD_outBuffer structure), and then
>> we’ll try again.”
>> (I.e., retrying with the same ZSTD_outBuffer will make the function
>> return immediately because it knows that it’s insufficient.)
>>
>> Max
>
> ok, removing the loop sounds reasonable.
> My only concern is that *must* in the doc.
Well, if we just return an error whenever we get a return value != 0,
then we shouldn’t have to care what we must and mustn’t do, because
we’ll just abort the compression process then.
> Could ZSTD-lib change the logic in the future relying on the fact
> that they make users use ZSTD_compressStream() in a loop.
It isn’t like I just wondered whether the loop is necessary and saw that
with the current implementation, it didn’t seem necessary for any of the
test images I have.
My reasoning is based on the specification, which says for ZSTD_e_flush
that it will only return a value > 0 if output->size is too small; and
that ZSTD_e_end follows the same rules.
So I think if they were to change behavior, they’d violate the spec.
> Honestly, I can't imagine the case when they would want to do that,
> but still.
> Without the loop we're protected even in this case. The worst thing
> could happen because of that is qcow2_zstd_compress() would return
> with -EIO more frequently.
I think so, too.
> So, if I understand correctly, you are ok with removing the loop.
Yes.
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 4/4] iotests: 287: add qcow2 compression type test
2020-04-27 13:29 ` Max Reitz
@ 2020-04-28 11:41 ` Denis Plotnikov
2020-04-28 12:55 ` Max Reitz
0 siblings, 1 reply; 19+ messages in thread
From: Denis Plotnikov @ 2020-04-28 11:41 UTC (permalink / raw)
To: Max Reitz, qemu-devel; +Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
On 27.04.2020 16:29, Max Reitz wrote:
> On 21.04.20 10:11, Denis Plotnikov wrote:
>> The test checks fulfilling qcow2 requirements for the compression
>> type feature and zstd compression type operability.
>>
>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>> ---
>> tests/qemu-iotests/287 | 146 +++++++++++++++++++++++++++++++++++++
>> tests/qemu-iotests/287.out | 67 +++++++++++++++++
>> tests/qemu-iotests/group | 1 +
>> 3 files changed, 214 insertions(+)
>> create mode 100755 tests/qemu-iotests/287
>> create mode 100644 tests/qemu-iotests/287.out
>>
>> diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
>> new file mode 100755
>> index 0000000000..156acc40ad
>> --- /dev/null
>> +++ b/tests/qemu-iotests/287
>> @@ -0,0 +1,146 @@
>> +#!/usr/bin/env bash
>> +#
>> +# Test case for an image using zstd compression
>> +#
>> +# Copyright (c) 2020 Virtuozzo International GmbH
>> +#
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation; either version 2 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
>> +#
>> +
>> +# creator
>> +owner=dplotnikov@virtuozzo.com
>> +
>> +seq="$(basename $0)"
>> +echo "QA output created by $seq"
>> +
>> +status=1 # failure is the default!
>> +
>> +# standard environment
>> +. ./common.rc
>> +. ./common.filter
>> +
>> +# This tests qocw2-specific low-level functionality
>> +_supported_fmt qcow2
>> +_supported_proto file
>> +_supported_os Linux
> This test doesn’t work with compat=0.10 (because we can’t store a
> non-default compression type there) or data_file (because those don’t
> support compression), so those options should be marked as unsupported.
>
> (It does seem to work with any refcount_bits, though.)
Could I ask how to achieve that?
I can't find any _supported_* related.
Denis
>
>> +
>> +COMPR_IMG="$TEST_IMG.compressed"
>> +RAND_FILE="$TEST_DIR/rand_data"
>> +
>> +_cleanup()
>> +{
>> + _cleanup_test_img
>> + rm -f "$COMPR_IMG"
> Using _rm_test_img() would be nicer. There shouldn’t be a functional
> difference here because there’d only be one with external data files (I
> think), which won’t work with this test, but still.
>
>> + rm -f "$RAND_FILE"
>> +}
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +# for all the cases
>> +CLUSTER_SIZE=65536
>> +
>> +# Check if we can run this test.
>> +if IMGOPTS='compression_type=zstd' _make_test_img 64M |
>> + grep "Invalid parameter 'zstd'"; then
>> + _notrun "ZSTD is disabled"
>> +fi
>> +
>> +echo
>> +echo "=== Testing compression type incompatible bit setting for zlib ==="
>> +echo
>> +IMGOPTS='compression_type=zlib' _make_test_img 64M
> Please use -o so user options are still considered.
>
> (i.e., _make_test_img -o compression_type=zlib)
>
> [...]
>
>> +echo
>> +echo "=== Testing incompressible cluster processing with zstd ==="
>> +echo
>> +# create a 2M image and fill it with 1M likely incompressible data
>> +# and 1M compressible data
>> +dd if=/dev/urandom of="$RAND_FILE" bs=1M count=1 seek=1
>> +QEMU_IO_OPTIONS="$QEMU_IO_OPTIONS_NO_FMT" \
>> +$QEMU_IO -f raw -c "write -P 0xFA 0 1M" "$RAND_FILE" | _filter_qemu_io
>> +$QEMU_IMG convert -f raw -O $IMGFMT -c "$RAND_FILE" "$TEST_IMG" | _filter_qemu_io
>> +
>> +$QEMU_IMG convert -O $IMGFMT -c -o compression_type=zstd \
>> + "$TEST_IMG" "$COMPR_IMG"
> Again, it would be nice to not discard the user-supplied options here,
> and maybe it would also be nicer to explicitly pass the compression type
> for the first convert, too. So we’d use
> -o "$(_optstr_add "$IMGOPTS" "compression_type=zlib")"
> for the first convert, and
> -o "$(_optstr_add "$IMGOPTS" "compression_type=zstd")"
> for the second one.
>
> Max
>
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 4/4] iotests: 287: add qcow2 compression type test
2020-04-28 11:41 ` Denis Plotnikov
@ 2020-04-28 12:55 ` Max Reitz
2020-04-28 13:01 ` Eric Blake
0 siblings, 1 reply; 19+ messages in thread
From: Max Reitz @ 2020-04-28 12:55 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel
Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
[-- Attachment #1.1: Type: text/plain, Size: 2562 bytes --]
On 28.04.20 13:41, Denis Plotnikov wrote:
>
>
> On 27.04.2020 16:29, Max Reitz wrote:
>> On 21.04.20 10:11, Denis Plotnikov wrote:
>>> The test checks fulfilling qcow2 requirements for the compression
>>> type feature and zstd compression type operability.
>>>
>>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>>> ---
>>> tests/qemu-iotests/287 | 146 +++++++++++++++++++++++++++++++++++++
>>> tests/qemu-iotests/287.out | 67 +++++++++++++++++
>>> tests/qemu-iotests/group | 1 +
>>> 3 files changed, 214 insertions(+)
>>> create mode 100755 tests/qemu-iotests/287
>>> create mode 100644 tests/qemu-iotests/287.out
>>>
>>> diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
>>> new file mode 100755
>>> index 0000000000..156acc40ad
>>> --- /dev/null
>>> +++ b/tests/qemu-iotests/287
>>> @@ -0,0 +1,146 @@
>>> +#!/usr/bin/env bash
>>> +#
>>> +# Test case for an image using zstd compression
>>> +#
>>> +# Copyright (c) 2020 Virtuozzo International GmbH
>>> +#
>>> +# This program is free software; you can redistribute it and/or modify
>>> +# it under the terms of the GNU General Public License as published by
>>> +# the Free Software Foundation; either version 2 of the License, or
>>> +# (at your option) any later version.
>>> +#
>>> +# This program is distributed in the hope that it will be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> +# GNU General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU General Public License
>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
>>> +#
>>> +
>>> +# creator
>>> +owner=dplotnikov@virtuozzo.com
>>> +
>>> +seq="$(basename $0)"
>>> +echo "QA output created by $seq"
>>> +
>>> +status=1 # failure is the default!
>>> +
>>> +# standard environment
>>> +. ./common.rc
>>> +. ./common.filter
>>> +
>>> +# This tests qocw2-specific low-level functionality
>>> +_supported_fmt qcow2
>>> +_supported_proto file
>>> +_supported_os Linux
>> This test doesn’t work with compat=0.10 (because we can’t store a
>> non-default compression type there) or data_file (because those don’t
>> support compression), so those options should be marked as unsupported.
>>
>> (It does seem to work with any refcount_bits, though.)
>
> Could I ask how to achieve that?
> I can't find any _supported_* related.
It’s _unsupported_imgopts.
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 4/4] iotests: 287: add qcow2 compression type test
2020-04-28 12:55 ` Max Reitz
@ 2020-04-28 13:01 ` Eric Blake
2020-04-28 13:34 ` Denis Plotnikov
0 siblings, 1 reply; 19+ messages in thread
From: Eric Blake @ 2020-04-28 13:01 UTC (permalink / raw)
To: Max Reitz, Denis Plotnikov, qemu-devel
Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
On 4/28/20 7:55 AM, Max Reitz wrote:
>>>> +# This tests qocw2-specific low-level functionality
>>>> +_supported_fmt qcow2
>>>> +_supported_proto file
>>>> +_supported_os Linux
>>> This test doesn’t work with compat=0.10 (because we can’t store a
>>> non-default compression type there) or data_file (because those don’t
>>> support compression), so those options should be marked as unsupported.
>>>
>>> (It does seem to work with any refcount_bits, though.)
>>
>> Could I ask how to achieve that?
>> I can't find any _supported_* related.
>
>
> It’s _unsupported_imgopts.
Test 036 is an example of this.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v20 4/4] iotests: 287: add qcow2 compression type test
2020-04-28 13:01 ` Eric Blake
@ 2020-04-28 13:34 ` Denis Plotnikov
0 siblings, 0 replies; 19+ messages in thread
From: Denis Plotnikov @ 2020-04-28 13:34 UTC (permalink / raw)
To: Eric Blake, Max Reitz, qemu-devel
Cc: kwolf, vsementsov, berto, qemu-block, armbru, den
On 28.04.2020 16:01, Eric Blake wrote:
> On 4/28/20 7:55 AM, Max Reitz wrote:
>
>>>>> +# This tests qocw2-specific low-level functionality
>>>>> +_supported_fmt qcow2
>>>>> +_supported_proto file
>>>>> +_supported_os Linux
>>>> This test doesn’t work with compat=0.10 (because we can’t store a
>>>> non-default compression type there) or data_file (because those don’t
>>>> support compression), so those options should be marked as
>>>> unsupported.
>>>>
>>>> (It does seem to work with any refcount_bits, though.)
>>>
>>> Could I ask how to achieve that?
>>> I can't find any _supported_* related.
>>
>>
>> It’s _unsupported_imgopts.
>
> Test 036 is an example of this.
Max, Eric
Thanks!
Denis
>
>
^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2020-04-28 13:37 UTC | newest]
Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-04-21 8:11 [PATCH v20 0/4] qcow2: Implement zstd cluster compression methodi Denis Plotnikov
2020-04-21 8:11 ` [PATCH v20 1/4] qcow2: introduce compression type feature Denis Plotnikov
2020-04-21 10:40 ` Alberto Garcia
2020-04-27 12:35 ` Max Reitz
2020-04-21 8:11 ` [PATCH v20 2/4] qcow2: rework the cluster compression routine Denis Plotnikov
2020-04-27 12:36 ` Max Reitz
2020-04-21 8:11 ` [PATCH v20 3/4] qcow2: add zstd cluster compression Denis Plotnikov
2020-04-27 12:35 ` Max Reitz
2020-04-27 19:26 ` Denis Plotnikov
2020-04-28 6:16 ` Max Reitz
2020-04-28 7:23 ` Denis Plotnikov
2020-04-28 10:17 ` Max Reitz
2020-04-21 8:11 ` [PATCH v20 4/4] iotests: 287: add qcow2 compression type test Denis Plotnikov
2020-04-21 12:06 ` Vladimir Sementsov-Ogievskiy
2020-04-27 13:29 ` Max Reitz
2020-04-28 11:41 ` Denis Plotnikov
2020-04-28 12:55 ` Max Reitz
2020-04-28 13:01 ` Eric Blake
2020-04-28 13:34 ` Denis Plotnikov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.