* [PATCH v1 0/8] qcow2: Implement zstd cluster compression method
@ 2020-02-27 7:29 Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 1/8] qcow2: introduce compression type feature Denis Plotnikov
` (7 more replies)
0 siblings, 8 replies; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-27 7:29 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, qemu-block, den, armbru, mreitz
zstd comression method is faster than the only available zlib.
The series adds zstd to the methods available for clusters compression.
The implementation is done with respect to the recently added compression
type additional header to the qcow2 specification.
Denis Plotnikov (8):
qcow2: introduce compression type feature
qcow2: rework the cluster compression routine
qcow2: add zstd cluster compression
iotests: filter out compression_type
iotests: fix header size, feature table size and backing file offset
iotests: add "compression type" for test output matching
iotests: 080: update header size value because of adding compression
type
iotests: 287: add qcow2 compression type test
block/qcow2-threads.c | 199 ++++++++++++++++++++++++++++---
block/qcow2.c | 112 +++++++++++++++++
block/qcow2.h | 31 +++--
configure | 29 +++++
docs/interop/qcow2.txt | 18 +++
include/block/block_int.h | 1 +
qapi/block-core.json | 23 +++-
tests/qemu-iotests/031.out | 14 +--
tests/qemu-iotests/036.out | 4 +-
tests/qemu-iotests/049.out | 102 ++++++++--------
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 34 +++---
tests/qemu-iotests/065 | 20 ++--
tests/qemu-iotests/080 | 2 +-
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 +
tests/qemu-iotests/255.out | 8 +-
tests/qemu-iotests/287 | 123 +++++++++++++++++++
tests/qemu-iotests/287.out | 41 +++++++
tests/qemu-iotests/common.filter | 3 +-
tests/qemu-iotests/group | 1 +
22 files changed, 663 insertions(+), 114 deletions(-)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
--
2.17.0
^ permalink raw reply [flat|nested] 35+ messages in thread
* [PATCH v1 1/8] qcow2: introduce compression type feature
2020-02-27 7:29 [PATCH v1 0/8] qcow2: Implement zstd cluster compression method Denis Plotnikov
@ 2020-02-27 7:29 ` Denis Plotnikov
2020-02-27 8:21 ` Vladimir Sementsov-Ogievskiy
2020-02-27 13:48 ` Eric Blake
2020-02-27 7:29 ` [PATCH v1 2/8] qcow2: rework the cluster compression routine Denis Plotnikov
` (6 subsequent siblings)
7 siblings, 2 replies; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-27 7:29 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, qemu-block, den, armbru, mreitz
The patch adds some preparation parts for incompatible compression type
feature to Qcow2 that indicates which allow to use different compression
methods for image clusters (de)compressing.
It is implied that the compression type is set on the image creation and
can be changed only later by image conversion, thus compression type
defines the only compression algorithm used for the image, and thus,
for all image clusters.
The goal of the feature is to add support of other compression methods
to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
The default compression is ZLIB. Images created with ZLIB compression type
are backward compatible with older qemu versions.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++
block/qcow2.h | 31 ++++++++---
include/block/block_int.h | 1 +
qapi/block-core.json | 22 +++++++-
4 files changed, 150 insertions(+), 9 deletions(-)
diff --git a/block/qcow2.c b/block/qcow2.c
index 3c754f616b..2ccb2cabd1 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
return ret;
}
+static int validate_compression_type(BDRVQcow2State *s, Error **errp)
+{
+ /*
+ * Sanity check
+ * according to qcow2 spec, the compression type is 1-byte field
+ * but in BDRVQcow2State the compression_type is enum sizeof(int)
+ * so, the max compression_type value is 255.
+ */
+ if (s->compression_type > 0xff) {
+ error_setg(errp, "qcow2: compression type value is too big");
+ return -EINVAL;
+ }
+
+ switch (s->compression_type) {
+ case QCOW2_COMPRESSION_TYPE_ZLIB:
+ break;
+
+ default:
+ error_setg(errp, "qcow2: unknown compression type: %u",
+ s->compression_type);
+ return -ENOTSUP;
+ }
+
+ /*
+ * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
+ * the incompatible feature flag must be set
+ */
+ if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
+ if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE) {
+ error_setg(errp, "qcow2: Compression type incompatible feature "
+ "bit must not be set");
+ return -EINVAL;
+ }
+ } else {
+ if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
+ error_setg(errp, "qcow2: Compression type incompatible feature "
+ "bit must be set");
+ return -EINVAL;
+ }
+ }
+
+ return 0;
+}
+
/* Called with s->lock held. */
static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
int flags, Error **errp)
@@ -1357,6 +1401,26 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
s->compatible_features = header.compatible_features;
s->autoclear_features = header.autoclear_features;
+ /*
+ * Handle compression type
+ * Older qcow2 images don't contain the compression type header.
+ * Distinguish them by the header length and use
+ * the only valid (default) compression type in that case
+ */
+ if (header.header_length > offsetof(QCowHeader, compression_type)) {
+ /*
+ * don't deal with endians since compression_type is 1 byte long
+ */
+ s->compression_type = header.compression_type;
+ } else {
+ s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
+ }
+
+ ret = validate_compression_type(s, errp);
+ if (ret) {
+ goto fail;
+ }
+
if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
void *feature_table = NULL;
qcow2_read_extensions(bs, header.header_length, ext_end,
@@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs)
total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
+ ret = validate_compression_type(s, NULL);
+
+ if (ret) {
+ goto fail;
+ }
+
*header = (QCowHeader) {
/* Version 2 fields */
.magic = cpu_to_be32(QCOW_MAGIC),
@@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs)
.autoclear_features = cpu_to_be64(s->autoclear_features),
.refcount_order = cpu_to_be32(s->refcount_order),
.header_length = cpu_to_be32(header_length),
+ .compression_type = (uint8_t) s->compression_type,
};
/* For older versions, write a shorter header */
@@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs)
.bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
.name = "lazy refcounts",
},
+ {
+ .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
+ .bit = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
+ .name = "compression type",
+ },
};
ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
@@ -3401,6 +3477,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
.refcount_table_offset = cpu_to_be64(cluster_size),
.refcount_table_clusters = cpu_to_be32(1),
.refcount_order = cpu_to_be32(refcount_order),
+ .compression_type = (uint8_t) QCOW2_COMPRESSION_TYPE_ZLIB,
.header_length = cpu_to_be32(sizeof(*header)),
};
@@ -3420,6 +3497,26 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
}
+ if (qcow2_opts->has_compression_type &&
+ qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
+
+ if (qcow2_opts->compression_type > 0xff) {
+ error_setg_errno(errp, -EINVAL, "Too big compression type value");
+ goto out;
+ }
+
+ switch (qcow2_opts->compression_type) {
+ default:
+ error_setg_errno(errp, -EINVAL, "Unknown compression type");
+ goto out;
+ }
+
+ header->compression_type = (uint8_t) qcow2_opts->compression_type;
+
+ header->incompatible_features |=
+ cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE);
+ }
+
ret = blk_pwrite(blk, 0, header, cluster_size, 0);
g_free(header);
if (ret < 0) {
@@ -3602,6 +3699,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
{ BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT },
{ BLOCK_OPT_COMPAT_LEVEL, "version" },
{ BLOCK_OPT_DATA_FILE_RAW, "data-file-raw" },
+ { BLOCK_OPT_COMPRESSION_TYPE, "compression-type" },
{ NULL, NULL },
};
@@ -4859,6 +4957,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs,
.data_file = g_strdup(s->image_data_file),
.has_data_file_raw = has_data_file(bs),
.data_file_raw = data_file_is_raw(bs),
+ .compression_type = s->compression_type,
};
} else {
/* if this assertion fails, this probably means a new version was
@@ -5516,6 +5615,12 @@ static QemuOptsList qcow2_create_opts = {
.help = "Width of a reference count entry in bits",
.def_value_str = "16"
},
+ {
+ .name = BLOCK_OPT_COMPRESSION_TYPE,
+ .type = QEMU_OPT_STRING,
+ .help = "Compression method used for image clusters compression",
+ .def_value_str = "zlib"
+ },
{ /* end of list */ }
}
};
diff --git a/block/qcow2.h b/block/qcow2.h
index 0942126232..f92412ed5e 100644
--- a/block/qcow2.h
+++ b/block/qcow2.h
@@ -146,6 +146,12 @@ typedef struct QCowHeader {
uint32_t refcount_order;
uint32_t header_length;
+
+ /* Additional fields */
+ uint8_t compression_type;
+
+ /* header must be a multiple of 8 */
+ uint8_t padding[7];
} QEMU_PACKED QCowHeader;
typedef struct QEMU_PACKED QCowSnapshotHeader {
@@ -213,16 +219,20 @@ enum {
/* Incompatible feature bits */
enum {
- QCOW2_INCOMPAT_DIRTY_BITNR = 0,
- QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
- QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
- QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
- QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
- QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+ QCOW2_INCOMPAT_DIRTY_BITNR = 0,
+ QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
+ QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
+ QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3,
+ QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
+ QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
+ QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
+ QCOW2_INCOMPAT_COMPRESSION_TYPE =
+ 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
| QCOW2_INCOMPAT_CORRUPT
- | QCOW2_INCOMPAT_DATA_FILE,
+ | QCOW2_INCOMPAT_DATA_FILE
+ | QCOW2_INCOMPAT_COMPRESSION_TYPE,
};
/* Compatible feature bits */
@@ -369,6 +379,13 @@ typedef struct BDRVQcow2State {
bool metadata_preallocation_checked;
bool metadata_preallocation;
+ /*
+ * Compression type used for the image. Default: 0 - ZLIB
+ * The image compression type is set on image creation.
+ * The only way to change the compression type is to convert the image
+ * with the desired compression type set
+ */
+ Qcow2CompressionType compression_type;
} BDRVQcow2State;
typedef struct Qcow2COWRegion {
diff --git a/include/block/block_int.h b/include/block/block_int.h
index 6f9fd5e20e..2c6bb9dc99 100644
--- a/include/block/block_int.h
+++ b/include/block/block_int.h
@@ -57,6 +57,7 @@
#define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"
#define BLOCK_OPT_DATA_FILE "data_file"
#define BLOCK_OPT_DATA_FILE_RAW "data_file_raw"
+#define BLOCK_OPT_COMPRESSION_TYPE "compression_type"
#define BLOCK_PROBE_BUF_SIZE 512
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 85e27bb61f..873fbef3b5 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -78,6 +78,8 @@
#
# @bitmaps: A list of qcow2 bitmap details (since 4.0)
#
+# @compression-type: the image cluster compression method (since 5.0)
+#
# Since: 1.7
##
{ 'struct': 'ImageInfoSpecificQCow2',
@@ -89,7 +91,8 @@
'*corrupt': 'bool',
'refcount-bits': 'int',
'*encrypt': 'ImageInfoSpecificQCow2Encryption',
- '*bitmaps': ['Qcow2BitmapInfo']
+ '*bitmaps': ['Qcow2BitmapInfo'],
+ 'compression-type': 'Qcow2CompressionType'
} }
##
@@ -4392,6 +4395,18 @@
'data': [ 'v2', 'v3' ] }
+##
+# @Qcow2CompressionType:
+#
+# Compression type used in qcow2 image file
+#
+# @zlib: zlib compression, see <http://zlib.net/>
+#
+# Since: 5.0
+##
+{ 'enum': 'Qcow2CompressionType',
+ 'data': [ 'zlib' ] }
+
##
# @BlockdevCreateOptionsQcow2:
#
@@ -4415,6 +4430,8 @@
# allowed values: off, falloc, full, metadata)
# @lazy-refcounts: True if refcounts may be updated lazily (default: off)
# @refcount-bits: Width of reference counts in bits (default: 16)
+# @compression-type: The image cluster compression method
+# (default: zlib, since 5.0)
#
# Since: 2.12
##
@@ -4430,7 +4447,8 @@
'*cluster-size': 'size',
'*preallocation': 'PreallocMode',
'*lazy-refcounts': 'bool',
- '*refcount-bits': 'int' } }
+ '*refcount-bits': 'int',
+ '*compression-type': 'Qcow2CompressionType' } }
##
# @BlockdevCreateOptionsQed:
--
2.17.0
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v1 2/8] qcow2: rework the cluster compression routine
2020-02-27 7:29 [PATCH v1 0/8] qcow2: Implement zstd cluster compression method Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 1/8] qcow2: introduce compression type feature Denis Plotnikov
@ 2020-02-27 7:29 ` Denis Plotnikov
2020-02-27 8:54 ` Vladimir Sementsov-Ogievskiy
2020-02-27 7:29 ` [PATCH v1 3/8] qcow2: add zstd cluster compression Denis Plotnikov
` (5 subsequent siblings)
7 siblings, 1 reply; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-27 7:29 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, qemu-block, den, armbru, mreitz
The patch enables processing the image compression type defined
for the image and chooses an appropriate method for image clusters
(de)compression.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
block/qcow2-threads.c | 77 +++++++++++++++++++++++++++++++++++--------
1 file changed, 63 insertions(+), 14 deletions(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 77bb578cdf..1c128e9840 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
} Qcow2CompressData;
/*
- * qcow2_compress()
+ * qcow2_zlib_compress()
+ *
+ * Compress @src_size bytes of data using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
@@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
* -ENOMEM destination buffer is not enough to store compressed data
* -EIO on any other error
*/
-static ssize_t qcow2_compress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
ssize_t ret;
z_stream strm;
@@ -119,19 +121,19 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
}
/*
- * qcow2_decompress()
+ * qcow2_zlib_decompress()
*
* Decompress some data (not more than @src_size bytes) to produce exactly
- * @dest_size bytes.
+ * @dest_size bytes using zlib compression method
*
* @dest - destination buffer, @dest_size bytes
* @src - source buffer, @src_size bytes
*
* Returns: 0 on success
- * -1 on fail
+ * -EIO on failure
*/
-static ssize_t qcow2_decompress(void *dest, size_t dest_size,
- const void *src, size_t src_size)
+static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
{
int ret = 0;
z_stream strm;
@@ -144,7 +146,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
ret = inflateInit2(&strm, -12);
if (ret != Z_OK) {
- return -1;
+ return -EIO;
}
ret = inflate(&strm, Z_FINISH);
@@ -154,7 +156,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
* @src buffer may be processed partly (because in qcow2 we know size of
* compressed data with precision of one sector)
*/
- ret = -1;
+ ret = -EIO;
}
inflateEnd(&strm);
@@ -189,20 +191,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,
return arg.ret;
}
+/*
+ * qcow2_co_compress()
+ *
+ * Compress @src_size bytes of data using the compression
+ * method defined by the image compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
- return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
- qcow2_compress);
+ BDRVQcow2State *s = bs->opaque;
+ Qcow2CompressFunc fn;
+
+ switch (s->compression_type) {
+ case QCOW2_COMPRESSION_TYPE_ZLIB:
+ fn = qcow2_zlib_compress;
+ break;
+
+ default:
+ return -ENOTSUP;
+ }
+
+ return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
+/*
+ * qcow2_co_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using the compression method defined by the image
+ * compression type
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * a negative error code on failure
+ */
ssize_t coroutine_fn
qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
const void *src, size_t src_size)
{
- return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
- qcow2_decompress);
+ BDRVQcow2State *s = bs->opaque;
+ Qcow2CompressFunc fn;
+
+ switch (s->compression_type) {
+ case QCOW2_COMPRESSION_TYPE_ZLIB:
+ fn = qcow2_zlib_decompress;
+ break;
+
+ default:
+ return -ENOTSUP;
+ }
+
+ return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
}
--
2.17.0
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v1 3/8] qcow2: add zstd cluster compression
2020-02-27 7:29 [PATCH v1 0/8] qcow2: Implement zstd cluster compression method Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 1/8] qcow2: introduce compression type feature Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 2/8] qcow2: rework the cluster compression routine Denis Plotnikov
@ 2020-02-27 7:29 ` Denis Plotnikov
2020-02-27 9:55 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:01 ` Eric Blake
2020-02-27 7:29 ` [PATCH v1 4/8] iotests: filter out compression_type Denis Plotnikov
` (4 subsequent siblings)
7 siblings, 2 replies; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-27 7:29 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, qemu-block, den, armbru, mreitz
zstd significantly reduces cluster compression time.
It provides better compression performance maintaining
the same level of the compression ratio in comparison with
zlib, which, at the moment, is the only compression
method available.
The performance test results:
Test compresses and decompresses qemu qcow2 image with just
installed rhel-7.6 guest.
Image cluster size: 64K. Image on disk size: 2.2G
The test was conducted with brd disk to reduce the influence
of disk subsystem to the test results.
The results is given in seconds.
compress cmd:
time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
src.img [zlib|zstd]_compressed.img
decompress cmd
time ./qemu-img convert -O qcow2
[zlib|zstd]_compressed.img uncompressed.img
compression decompression
zlib zstd zlib zstd
------------------------------------------------------------
real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
user 65.0 15.8 5.3 2.5
sys 3.3 0.2 2.0 2.0
Both ZLIB and ZSTD gave the same compression ratio: 1.57
compressed image size in both cases: 1.4G
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
block/qcow2-threads.c | 122 +++++++++++++++++++++++++++++++++++++++++
block/qcow2.c | 7 +++
configure | 29 ++++++++++
docs/interop/qcow2.txt | 18 ++++++
qapi/block-core.json | 3 +-
5 files changed, 178 insertions(+), 1 deletion(-)
diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
index 1c128e9840..e942c4d7e5 100644
--- a/block/qcow2-threads.c
+++ b/block/qcow2-threads.c
@@ -28,6 +28,11 @@
#define ZLIB_CONST
#include <zlib.h>
+#ifdef CONFIG_ZSTD
+#include <zstd.h>
+#include <zstd_errors.h>
+#endif
+
#include "qcow2.h"
#include "block/thread-pool.h"
#include "crypto.h"
@@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
return ret;
}
+#ifdef CONFIG_ZSTD
+
+#define ZSTD_LEN_BUF 4
+
+/*
+ * qcow2_zstd_compress()
+ *
+ * Compress @src_size bytes of data using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: compressed size on success
+ * -ENOMEM destination buffer is not enough to store compressed data
+ * -EIO on any other error
+ */
+
+static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+ size_t ret;
+
+ /*
+ * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
+ * to store compressed chunk size
+ */
+ char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
+
+ /*
+ * sanity check that we can store the compressed data length,
+ * and there is some space left for the compressor buffer
+ */
+ if (dest_size <= ZSTD_LEN_BUF) {
+ return -ENOMEM;
+ }
+
+ dest_size -= ZSTD_LEN_BUF;
+
+ ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
+
+ if (ZSTD_isError(ret)) {
+ if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
+ return -ENOMEM;
+ } else {
+ return -EIO;
+ }
+ }
+
+ /* paraniod sanity check that we can store the commpressed size */
+ if (ret > UINT_MAX) {
+ return -ENOMEM;
+ }
+
+ /* store the compressed chunk size in the very beginning of the buffer */
+ stl_be_p(dest, ret);
+
+ return ret + ZSTD_LEN_BUF;
+}
+
+/*
+ * qcow2_zstd_decompress()
+ *
+ * Decompress some data (not more than @src_size bytes) to produce exactly
+ * @dest_size bytes using zstd compression method
+ *
+ * @dest - destination buffer, @dest_size bytes
+ * @src - source buffer, @src_size bytes
+ *
+ * Returns: 0 on success
+ * -EIO on any error
+ */
+
+static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
+ const void *src, size_t src_size)
+{
+ /*
+ * zstd decompress wants to know the exact length of the data.
+ * For that purpose, on compression, the length is stored in
+ * the very beginning of the compressed buffer
+ */
+ size_t s_size;
+ const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF;
+
+ /*
+ * sanity check that we can read 4 byte the content length and
+ * and there is some content to decompress
+ */
+ if (src_size <= ZSTD_LEN_BUF) {
+ return -EIO;
+ }
+
+ s_size = ldl_be_p(src);
+
+ /* sanity check that the buffer is big enough to read the content from */
+ if (src_size - ZSTD_LEN_BUF < s_size) {
+ return -EIO;
+ }
+
+ if (ZSTD_isError(
+ ZSTD_decompress(dest, dest_size, s_buf, s_size))) {
+ return -EIO;
+ }
+
+ return 0;
+}
+#endif
+
static int qcow2_compress_pool_func(void *opaque)
{
Qcow2CompressData *data = opaque;
@@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
fn = qcow2_zlib_compress;
break;
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+ fn = qcow2_zstd_compress;
+ break;
+#endif
default:
return -ENOTSUP;
}
@@ -247,6 +364,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
fn = qcow2_zlib_decompress;
break;
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+ fn = qcow2_zstd_decompress;
+ break;
+#endif
default:
return -ENOTSUP;
}
diff --git a/block/qcow2.c b/block/qcow2.c
index 2ccb2cabd1..9c8ad9d580 100644
--- a/block/qcow2.c
+++ b/block/qcow2.c
@@ -1257,6 +1257,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp)
switch (s->compression_type) {
case QCOW2_COMPRESSION_TYPE_ZLIB:
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+#endif
break;
default:
@@ -3506,6 +3509,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
}
switch (qcow2_opts->compression_type) {
+#ifdef CONFIG_ZSTD
+ case QCOW2_COMPRESSION_TYPE_ZSTD:
+ break;
+#endif
default:
error_setg_errno(errp, -EINVAL, "Unknown compression type");
goto out;
diff --git a/configure b/configure
index 48d6f89d57..4690a7ea9f 100755
--- a/configure
+++ b/configure
@@ -444,6 +444,7 @@ opengl_dmabuf="no"
cpuid_h="no"
avx2_opt=""
zlib="yes"
+zstd=""
capstone=""
lzo=""
snappy=""
@@ -1371,6 +1372,10 @@ for opt do
;;
--disable-lzfse) lzfse="no"
;;
+ --enable-zstd) zstd="yes"
+ ;;
+ --disable-zstd) zstd="no"
+ ;;
--enable-guest-agent) guest_agent="yes"
;;
--disable-guest-agent) guest_agent="no"
@@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default is enabled if available:
(for reading bzip2-compressed dmg images)
lzfse support of lzfse compression library
(for reading lzfse-compressed dmg images)
+ zstd support of zstd compression library
seccomp seccomp support
coroutine-pool coroutine freelist (better performance)
glusterfs GlusterFS backend
@@ -2453,6 +2459,25 @@ EOF
fi
fi
+#########################################
+# zstd check
+
+if test "$zstd" != "no" ; then
+ cat > $TMPC << EOF
+#include <zstd.h>
+int main(void) { ZSTD_versionNumber(); return 0; }
+EOF
+ if compile_prog "" "-lzstd" ; then
+ LIBS="$LIBS -lzstd"
+ zstd="yes"
+ else
+ if test "$zstd" = "yes"; then
+ feature_not_found "zstd" "Install libzstd-devel"
+ fi
+ zstd="no"
+ fi
+fi
+
##########################################
# libseccomp check
@@ -6668,6 +6693,7 @@ echo "lzo support $lzo"
echo "snappy support $snappy"
echo "bzip2 support $bzip2"
echo "lzfse support $lzfse"
+echo "zstd support $zstd"
echo "NUMA host support $numa"
echo "libxml2 $libxml2"
echo "tcmalloc support $tcmalloc"
@@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then
"\$ld_exported_symbols_list should have been set to 'yes'."
fi
fi
+if test "$zstd" = "yes" ; then
+ echo "CONFIG_ZSTD=y" >> $config_host_mak
+fi
if test "$tcg_interpreter" = "yes"; then
QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
index 5597e24474..aeca2ddebd 100644
--- a/docs/interop/qcow2.txt
+++ b/docs/interop/qcow2.txt
@@ -208,6 +208,7 @@ version 2.
Available compression type values:
0: zlib <https://www.zlib.net/>
+ 1: zstd <http://github.com/facebook/zstd>
=== Header padding ===
@@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)):
Another compressed cluster may map to the tail of the final
sector used by this compressed cluster.
+ The layout of the compressed data depends on the compression
+ type used for the image (see compressed cluster layout).
+
If a cluster is unallocated, read requests shall read the data from the backing
file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
no backing file or the backing file is smaller than the image, they shall read
zeros for all parts that are not covered by the backing file.
+=== Compressed Cluster Layout ===
+
+The compressed cluster data has a layout depending on the compression
+type used for the image, as follows:
+
+Compressed data layout for the available compression types:
+(x = data_space_length - 1)
+
+ 0: (default) zlib <http://zlib.net/>:
+ Byte 0 - x: the compressed data content
+ all the space provided used for compressed data
+ 1: zstd <http://github.com/facebook/zstd>:
+ Byte 0 - 3: the length of compressed data in bytes
+ 4 - x: the compressed data content
== Snapshots ==
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 873fbef3b5..4b6e576c44 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -4401,11 +4401,12 @@
# Compression type used in qcow2 image file
#
# @zlib: zlib compression, see <http://zlib.net/>
+# @zstd: zstd compression, see <http://github.com/facebook/zstd>
#
# Since: 5.0
##
{ 'enum': 'Qcow2CompressionType',
- 'data': [ 'zlib' ] }
+ 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
##
# @BlockdevCreateOptionsQcow2:
--
2.17.0
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v1 4/8] iotests: filter out compression_type
2020-02-27 7:29 [PATCH v1 0/8] qcow2: Implement zstd cluster compression method Denis Plotnikov
` (2 preceding siblings ...)
2020-02-27 7:29 ` [PATCH v1 3/8] qcow2: add zstd cluster compression Denis Plotnikov
@ 2020-02-27 7:29 ` Denis Plotnikov
2020-02-27 9:57 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:03 ` Eric Blake
2020-02-27 7:29 ` [PATCH v1 5/8] iotests: fix header size, feature table size and backing file offset Denis Plotnikov
` (3 subsequent siblings)
7 siblings, 2 replies; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-27 7:29 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, qemu-block, den, armbru, mreitz
After adding compression type feature to qcow2 format, qemu framework
commands reporting the image settingd, e.g. "qemu-img create", started
reporting the compression type for the image which breaks the iotests
output matching.
To fix it, add compression_type=zlib to the list of filtered image parameters.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
tests/qemu-iotests/common.filter | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
index 3f8ee3e5f7..c6962d199c 100644
--- a/tests/qemu-iotests/common.filter
+++ b/tests/qemu-iotests/common.filter
@@ -152,7 +152,8 @@ _filter_img_create()
-e "s# refcount_bits=[0-9]\\+##g" \
-e "s# key-secret=[a-zA-Z0-9]\\+##g" \
-e "s# iter-time=[0-9]\\+##g" \
- -e "s# force_size=\\(on\\|off\\)##g"
+ -e "s# force_size=\\(on\\|off\\)##g" \
+ -e "s# compression_type=zlib##g"
}
_filter_img_info()
--
2.17.0
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v1 5/8] iotests: fix header size, feature table size and backing file offset
2020-02-27 7:29 [PATCH v1 0/8] qcow2: Implement zstd cluster compression method Denis Plotnikov
` (3 preceding siblings ...)
2020-02-27 7:29 ` [PATCH v1 4/8] iotests: filter out compression_type Denis Plotnikov
@ 2020-02-27 7:29 ` Denis Plotnikov
2020-02-27 9:59 ` Vladimir Sementsov-Ogievskiy
2020-02-27 7:29 ` [PATCH v1 6/8] iotests: add "compression type" for test output matching Denis Plotnikov
` (2 subsequent siblings)
7 siblings, 1 reply; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-27 7:29 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, qemu-block, den, armbru, mreitz
Affected tests: 031, 036, 061
Because of adding the compression type feature, some size values in the
qcow2 v3 header are changed:
header_size +=8: 1 byte compression type
7 bytes padding
feature_table += 48: incompatible feture compression type
backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
Change the values for the test output comparison accordingly.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
tests/qemu-iotests/031.out | 14 +++++++-------
tests/qemu-iotests/036.out | 4 ++--
tests/qemu-iotests/061.out | 28 ++++++++++++++--------------
3 files changed, 23 insertions(+), 23 deletions(-)
diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out
index d535e407bc..ed51afe9ce 100644
--- a/tests/qemu-iotests/031.out
+++ b/tests/qemu-iotests/031.out
@@ -113,11 +113,11 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
Header extension:
@@ -146,11 +146,11 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
Header extension:
@@ -164,7 +164,7 @@ No errors were found on the image.
magic 0x514649fb
version 3
-backing_file_offset 0x178
+backing_file_offset 0x1b0
backing_file_size 0x17
cluster_bits 16
size 67108864
@@ -179,7 +179,7 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0xe2792aca
@@ -188,7 +188,7 @@ data 'host_device'
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
Header extension:
diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out
index 0b52b934e1..fb509f6357 100644
--- a/tests/qemu-iotests/036.out
+++ b/tests/qemu-iotests/036.out
@@ -26,7 +26,7 @@ compatible_features []
autoclear_features [63]
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
@@ -38,7 +38,7 @@ compatible_features []
autoclear_features []
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
*** done
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index 8b3091a412..cea7fedfdc 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -22,11 +22,11 @@ incompatible_features []
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
magic 0x514649fb
@@ -80,11 +80,11 @@ incompatible_features []
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
magic 0x514649fb
@@ -136,11 +136,11 @@ incompatible_features [0]
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
ERROR cluster 5 refcount=0 reference=1
@@ -191,11 +191,11 @@ incompatible_features []
compatible_features [42]
autoclear_features [42]
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
magic 0x514649fb
@@ -260,11 +260,11 @@ incompatible_features []
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
read 65536/65536 bytes at offset 44040192
@@ -294,11 +294,11 @@ incompatible_features [0]
compatible_features [0]
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
ERROR cluster 5 refcount=0 reference=1
@@ -323,11 +323,11 @@ incompatible_features []
compatible_features []
autoclear_features []
refcount_order 4
-header_length 104
+header_length 112
Header extension:
magic 0x6803f857
-length 192
+length 240
data <binary>
read 131072/131072 bytes at offset 0
--
2.17.0
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v1 6/8] iotests: add "compression type" for test output matching
2020-02-27 7:29 [PATCH v1 0/8] qcow2: Implement zstd cluster compression method Denis Plotnikov
` (4 preceding siblings ...)
2020-02-27 7:29 ` [PATCH v1 5/8] iotests: fix header size, feature table size and backing file offset Denis Plotnikov
@ 2020-02-27 7:29 ` Denis Plotnikov
2020-02-27 10:04 ` Vladimir Sementsov-Ogievskiy
2020-02-27 7:29 ` [PATCH v1 7/8] iotests: 080: update header size value because of adding compression type Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 8/8] iotests: 287: add qcow2 compression type test Denis Plotnikov
7 siblings, 1 reply; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-27 7:29 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, qemu-block, den, armbru, mreitz
Affected tests: 049, 060, 061, 065, 144, 182, 242, 255
After adding the compression type feature for qcow2, the compression type
is reported on image quering.
Add the corresponding values of the "compression type" for the tests' output
matching.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
tests/qemu-iotests/049.out | 102 ++++++++++++++++++-------------------
tests/qemu-iotests/060.out | 1 +
tests/qemu-iotests/061.out | 6 +++
tests/qemu-iotests/065 | 20 +++++---
tests/qemu-iotests/144.out | 4 +-
tests/qemu-iotests/182.out | 2 +-
tests/qemu-iotests/242.out | 5 ++
tests/qemu-iotests/255.out | 8 +--
8 files changed, 82 insertions(+), 66 deletions(-)
diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out
index affa55b341..a5cfba1756 100644
--- a/tests/qemu-iotests/049.out
+++ b/tests/qemu-iotests/049.out
@@ -4,90 +4,90 @@ QA output created by 049
== 1. Traditional size parameter ==
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== 2. Specifying size via -o ==
qemu-img create -f qcow2 -o size=1024 TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1024b TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1k TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1K TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1M TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1G TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1T TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1024.0 TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1024.0b TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5k TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5K TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5M TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5G TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o size=1.5T TEST_DIR/t.qcow2
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== 3. Invalid sizes ==
@@ -129,84 +129,84 @@ qemu-img: TEST_DIR/t.qcow2: The image size must be specified only once
== Check correct interpretation of suffixes for cluster size ==
qemu-img create -f qcow2 -o cluster_size=1024 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024b TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1k TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1K TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1M TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024.0 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=1024.0b TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5k TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5K TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o cluster_size=0.5M TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check compat level option ==
qemu-img create -f qcow2 -o compat=0.10 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check preallocation option ==
qemu-img create -f qcow2 -o preallocation=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234'
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check encryption option ==
qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o encryption=on,encrypt.key-secret=sec0 TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
== Check lazy_refcounts option (only with v3) ==
qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=on TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M
qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater)
-Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
*** done
diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out
index d27692a33c..3e47f537e8 100644
--- a/tests/qemu-iotests/060.out
+++ b/tests/qemu-iotests/060.out
@@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: true
diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
index cea7fedfdc..c913f02ad6 100644
--- a/tests/qemu-iotests/061.out
+++ b/tests/qemu-iotests/061.out
@@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: foo
@@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file raw: false
@@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
@@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
data file: TEST_DIR/t.IMGFMT.data
diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
index 6426474271..106303b5a5 100755
--- a/tests/qemu-iotests/065
+++ b/tests/qemu-iotests/065
@@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific):
class TestQCow2(TestQemuImgInfo):
'''Testing a qcow2 version 2 image'''
img_options = 'compat=0.10'
- json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
- human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
+ json_compare = { 'compat': '0.10', 'refcount-bits': 16, 'compression-type': 'zlib' }
+ human_compare = [ 'compat: 0.10', 'compression type: zlib', 'refcount bits: 16' ]
class TestQCow3NotLazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts disabled'''
img_options = 'compat=1.1,lazy_refcounts=off'
json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
- 'refcount-bits': 16, 'corrupt': False }
- human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
+ human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: false',
'refcount bits: 16', 'corrupt: false' ]
class TestQCow3Lazy(TestQemuImgInfo):
'''Testing a qcow2 version 3 image with lazy refcounts enabled'''
img_options = 'compat=1.1,lazy_refcounts=on'
json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
- 'refcount-bits': 16, 'corrupt': False }
- human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
+ human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: true',
'refcount bits: 16', 'corrupt: false' ]
class TestQCow3NotLazyQMP(TestQMP):
@@ -113,7 +115,8 @@ class TestQCow3NotLazyQMP(TestQMP):
img_options = 'compat=1.1,lazy_refcounts=off'
qemu_options = 'lazy-refcounts=on'
compare = { 'compat': '1.1', 'lazy-refcounts': False,
- 'refcount-bits': 16, 'corrupt': False }
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
class TestQCow3LazyQMP(TestQMP):
@@ -122,7 +125,8 @@ class TestQCow3LazyQMP(TestQMP):
img_options = 'compat=1.1,lazy_refcounts=on'
qemu_options = 'lazy-refcounts=off'
compare = { 'compat': '1.1', 'lazy-refcounts': True,
- 'refcount-bits': 16, 'corrupt': False }
+ 'refcount-bits': 16, 'corrupt': False,
+ 'compression-type': 'zlib' }
TestImageInfoSpecific = None
TestQemuImgInfo = None
diff --git a/tests/qemu-iotests/144.out b/tests/qemu-iotests/144.out
index c7aa2e4820..885a8874a5 100644
--- a/tests/qemu-iotests/144.out
+++ b/tests/qemu-iotests/144.out
@@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912
{ 'execute': 'qmp_capabilities' }
{"return": {}}
{ 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } }
-Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
{"return": {}}
=== Performing block-commit on active layer ===
@@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/
=== Performing Live Snapshot 2 ===
{ 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } }
-Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
{"return": {}}
*** done
diff --git a/tests/qemu-iotests/182.out b/tests/qemu-iotests/182.out
index a8eea166c3..ae43654d32 100644
--- a/tests/qemu-iotests/182.out
+++ b/tests/qemu-iotests/182.out
@@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]?
{'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
{"return": {}}
{'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } }
-Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
{"return": {}}
{'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
{"return": {}}
diff --git a/tests/qemu-iotests/242.out b/tests/qemu-iotests/242.out
index 7ac8404d11..091b9126ce 100644
--- a/tests/qemu-iotests/242.out
+++ b/tests/qemu-iotests/242.out
@@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
refcount bits: 16
corrupt: false
@@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
@@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
@@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
@@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes)
cluster_size: 65536
Format specific information:
compat: 1.1
+ compression type: zlib
lazy refcounts: false
bitmaps:
[0]:
diff --git a/tests/qemu-iotests/255.out b/tests/qemu-iotests/255.out
index 348909fdef..a3c99fd62e 100644
--- a/tests/qemu-iotests/255.out
+++ b/tests/qemu-iotests/255.out
@@ -3,9 +3,9 @@ Finishing a commit job with background reads
=== Create backing chain and start VM ===
-Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
-Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
=== Start background read requests ===
@@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled
=== Create images and start VM ===
-Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
-Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
+Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
wrote 1048576/1048576 bytes at offset 0
1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
--
2.17.0
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v1 7/8] iotests: 080: update header size value because of adding compression type
2020-02-27 7:29 [PATCH v1 0/8] qcow2: Implement zstd cluster compression method Denis Plotnikov
` (5 preceding siblings ...)
2020-02-27 7:29 ` [PATCH v1 6/8] iotests: add "compression type" for test output matching Denis Plotnikov
@ 2020-02-27 7:29 ` Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 8/8] iotests: 287: add qcow2 compression type test Denis Plotnikov
7 siblings, 0 replies; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-27 7:29 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, qemu-block, den, armbru, mreitz
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
tests/qemu-iotests/080 | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/tests/qemu-iotests/080 b/tests/qemu-iotests/080
index a3d13c414e..7588c63b6c 100755
--- a/tests/qemu-iotests/080
+++ b/tests/qemu-iotests/080
@@ -45,7 +45,7 @@ _supported_os Linux
# - This is generally a test for compat=1.1 images
_unsupported_imgopts 'refcount_bits=1[^0-9]' data_file 'compat=0.10'
-header_size=104
+header_size=112
offset_backing_file_offset=8
offset_backing_file_size=16
--
2.17.0
^ permalink raw reply related [flat|nested] 35+ messages in thread
* [PATCH v1 8/8] iotests: 287: add qcow2 compression type test
2020-02-27 7:29 [PATCH v1 0/8] qcow2: Implement zstd cluster compression method Denis Plotnikov
` (6 preceding siblings ...)
2020-02-27 7:29 ` [PATCH v1 7/8] iotests: 080: update header size value because of adding compression type Denis Plotnikov
@ 2020-02-27 7:29 ` Denis Plotnikov
2020-02-27 10:29 ` Vladimir Sementsov-Ogievskiy
7 siblings, 1 reply; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-27 7:29 UTC (permalink / raw)
To: qemu-devel; +Cc: kwolf, vsementsov, qemu-block, den, armbru, mreitz
The test checks fulfilling qcow2 requiriements for the compression
type feature and zstd compression type operability.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
tests/qemu-iotests/287 | 123 +++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/287.out | 41 +++++++++++++
tests/qemu-iotests/group | 1 +
3 files changed, 165 insertions(+)
create mode 100755 tests/qemu-iotests/287
create mode 100644 tests/qemu-iotests/287.out
diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
new file mode 100755
index 0000000000..41b916f690
--- /dev/null
+++ b/tests/qemu-iotests/287
@@ -0,0 +1,123 @@
+#!/usr/bin/env bash
+#
+# Test case for an image using zstd compression
+#
+# Copyright (c) 2020 Virtuozzo International GmbH
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+# creator
+owner=dplotnikov@virtuozzo.com
+
+seq="$(basename $0)"
+echo "QA output created by $seq"
+
+status=1 # failure is the default!
+
+_cleanup()
+{
+ _cleanup_test_img
+}
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+# standard environment
+. ./common.rc
+. ./common.filter
+
+# This tests qocw2-specific low-level functionality
+_supported_fmt qcow2
+_supported_proto file
+_supported_os Linux
+
+P=`echo "$QEMU_PROG" | sed "s/qemu-system-x86_64//"`
+
+grep "CONFIG_ZSTD=y" "$P"../config-host.mak >/dev/null
+RES=$?
+if (($RES)); then
+ _notrun "ZSTD is disabled in the current configuration"
+fi
+
+# Test: when compression is zlib the incompatible is unset
+echo
+echo "=== Testing compression type incompatible bit setting for zlib ==="
+echo
+
+_make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: when compression differs from zlib the incompatible bit is set
+echo
+echo "=== Testing compression type incompatible bit setting for zstd ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+
+# Test: an image can't be openned if compression type is zlib and
+# incompatible feature compression type is set
+echo
+echo "=== Testing zlib with incompatible bit set ==="
+echo
+
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
+# to make sure the bit was actually set
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+ echo "Error: The image openned successfully. The image must not be openned"
+fi
+
+# Test: an image can't be openned if compression type is NOT zlib and
+# incompatible feature compression type is UNSET
+echo
+echo "=== Testing zstd with incompatible bit unset ==="
+echo
+
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
+# to make sure the bit was actually unset
+$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
+$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
+if (($?==0)); then
+ echo "Error: The image openned successfully. The image must not be openned"
+fi
+# Test: check compression type values
+echo
+echo "=== Testing compression type values ==="
+echo
+# zlib=0
+IMGOPTS='compression_type=zlib' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# zstd=1
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+od -j104 -N1 -An -vtu1 "$TEST_IMG"
+
+# Test: using zstd compression, write to and read from an image
+echo
+echo "=== Testing reading and writing with zstd ==="
+echo
+
+CLUSTER_SIZE=65536
+IMGOPTS='compression_type=zstd' _make_test_img 64M
+$QEMU_IO -c "write -c 0 64k " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 0 10 " "$TEST_IMG" | _filter_qemu_io
+$QEMU_IO -c "read -v 65530 8" "$TEST_IMG" | _filter_qemu_io
+
+# success, all done
+echo "*** done"
+rm -f $seq.full
+status=0
diff --git a/tests/qemu-iotests/287.out b/tests/qemu-iotests/287.out
new file mode 100644
index 0000000000..4218254ce0
--- /dev/null
+++ b/tests/qemu-iotests/287.out
@@ -0,0 +1,41 @@
+QA output created by 287
+
+=== Testing compression type incompatible bit setting for zlib ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+incompatible_features []
+
+=== Testing compression type incompatible bit setting for zstd ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
+incompatible_features [3]
+
+=== Testing zlib with incompatible bit set ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+incompatible_features [3]
+
+=== Testing zstd with incompatible bit unset ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
+incompatible_features []
+
+=== Testing compression type values ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
+ 0
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
+ 1
+
+=== Testing reading and writing with zstd ===
+
+Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
+wrote 65536/65536 bytes at offset 0
+64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+00000000: cd cd cd cd cd cd cd cd cd cd ..........
+read 10/10 bytes at offset 0
+10 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+0000fffa: cd cd cd cd cd cd 00 00 ........
+read 8/8 bytes at offset 65530
+8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+*** done
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 0317667695..5edbadef40 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -293,3 +293,4 @@
283 auto quick
284 rw
286 rw quick
+287 auto quick
--
2.17.0
^ permalink raw reply related [flat|nested] 35+ messages in thread
* Re: [PATCH v1 1/8] qcow2: introduce compression type feature
2020-02-27 7:29 ` [PATCH v1 1/8] qcow2: introduce compression type feature Denis Plotnikov
@ 2020-02-27 8:21 ` Vladimir Sementsov-Ogievskiy
2020-02-27 13:24 ` Eric Blake
2020-02-27 13:48 ` Eric Blake
1 sibling, 1 reply; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 8:21 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, qemu-block, den, armbru, mreitz
27.02.2020 10:29, Denis Plotnikov wrote:
> The patch adds some preparation parts for incompatible compression type
> feature to Qcow2 that indicates which allow to use different compression
> methods for image clusters (de)compressing.
>
> It is implied that the compression type is set on the image creation and
> can be changed only later by image conversion, thus compression type
> defines the only compression algorithm used for the image, and thus,
> for all image clusters.
>
> The goal of the feature is to add support of other compression methods
> to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
>
> The default compression is ZLIB. Images created with ZLIB compression type
> are backward compatible with older qemu versions.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++
> block/qcow2.h | 31 ++++++++---
> include/block/block_int.h | 1 +
> qapi/block-core.json | 22 +++++++-
> 4 files changed, 150 insertions(+), 9 deletions(-)
>
> diff --git a/block/qcow2.c b/block/qcow2.c
> index 3c754f616b..2ccb2cabd1 100644
> --- a/block/qcow2.c
> +++ b/block/qcow2.c
Please, add to .git/config:
[diff]
orderFile = /path/to/qemu/scripts/git.orderfile
This will force git format-patch to sort files in more comfortable order (header changes first, etc).
> @@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
> return ret;
> }
>
> +static int validate_compression_type(BDRVQcow2State *s, Error **errp)
> +{
> + /*
> + * Sanity check
> + * according to qcow2 spec, the compression type is 1-byte field
> + * but in BDRVQcow2State the compression_type is enum sizeof(int)
> + * so, the max compression_type value is 255.
> + */
> + if (s->compression_type > 0xff) {
> + error_setg(errp, "qcow2: compression type value is too big");
> + return -EINVAL;
> + }
> +
> + switch (s->compression_type) {
> + case QCOW2_COMPRESSION_TYPE_ZLIB:
> + break;
> +
> + default:
> + error_setg(errp, "qcow2: unknown compression type: %u",
> + s->compression_type);
> + return -ENOTSUP;
> + }
honestly, I think that just
if (s->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
error out
}
is enough, and don't see why to check > 0xff in separate..
But it works as is.
> +
> + /*
> + * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
> + * the incompatible feature flag must be set
> + */
> + if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
> + if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE) {
> + error_setg(errp, "qcow2: Compression type incompatible feature "
> + "bit must not be set");
> + return -EINVAL;
> + }
> + } else {
This is unreachable now.. But it's OK as preparation for further patches I think.
> + if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
> + error_setg(errp, "qcow2: Compression type incompatible feature "
> + "bit must be set");
> + return -EINVAL;
> + }
> + }
> +
> + return 0;
> +}
> +
> /* Called with s->lock held. */
> static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
> int flags, Error **errp)
> @@ -1357,6 +1401,26 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
> s->compatible_features = header.compatible_features;
> s->autoclear_features = header.autoclear_features;
>
> + /*
> + * Handle compression type
> + * Older qcow2 images don't contain the compression type header.
> + * Distinguish them by the header length and use
> + * the only valid (default) compression type in that case
> + */
> + if (header.header_length > offsetof(QCowHeader, compression_type)) {
> + /*
> + * don't deal with endians since compression_type is 1 byte long
> + */
this comment would be more appropriate above, where be-to-cpu transformation is done,
as actually previous "s->autoclear_features = header.autoclear_features" doesn't
deal with endians too.
> + s->compression_type = header.compression_type;
> + } else {
> + s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
> + }
> +
> + ret = validate_compression_type(s, errp);
> + if (ret) {
> + goto fail;
> + }
> +
> if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
> void *feature_table = NULL;
> qcow2_read_extensions(bs, header.header_length, ext_end,
> @@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs)
> total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
> refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
>
> + ret = validate_compression_type(s, NULL);
> +
> + if (ret) {
> + goto fail;
> + }
> +
> *header = (QCowHeader) {
> /* Version 2 fields */
> .magic = cpu_to_be32(QCOW_MAGIC),
> @@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs)
> .autoclear_features = cpu_to_be64(s->autoclear_features),
> .refcount_order = cpu_to_be32(s->refcount_order),
> .header_length = cpu_to_be32(header_length),
> + .compression_type = (uint8_t) s->compression_type,
> };
>
> /* For older versions, write a shorter header */
> @@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs)
> .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
> .name = "lazy refcounts",
> },
> + {
> + .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
> + .bit = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
> + .name = "compression type",
> + },
> };
>
> ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
> @@ -3401,6 +3477,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
> .refcount_table_offset = cpu_to_be64(cluster_size),
> .refcount_table_clusters = cpu_to_be32(1),
> .refcount_order = cpu_to_be32(refcount_order),
> + .compression_type = (uint8_t) QCOW2_COMPRESSION_TYPE_ZLIB,
Hmm. Following qcow2_co_create existing code, you'd better
1. do check compression_type correctnes before header initialization, may be add local variable
compression_type, don't forget to check version < 3, and here you'll have only
.compression_type = compression_type
> .header_length = cpu_to_be32(sizeof(*header)),
> };
>
> @@ -3420,6 +3497,26 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
> cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
> }
>
2. Here you'll need only set incompatible_features if needed.
> + if (qcow2_opts->has_compression_type &&
> + qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
> +
> + if (qcow2_opts->compression_type > 0xff) {
> + error_setg_errno(errp, -EINVAL, "Too big compression type value");
I still doubt that we need seperate check for this thing. Moreover, here we can get
only element of enum, so we can do assertion, not if.
> + goto out;
going out here, you leak header
> + }
> +
> + switch (qcow2_opts->compression_type) {
> + default:
> + error_setg_errno(errp, -EINVAL, "Unknown compression type");
> + goto out;
> + }
> +
> + header->compression_type = (uint8_t) qcow2_opts->compression_type;
> +
> + header->incompatible_features |=
> + cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE);
> + }
> +
> ret = blk_pwrite(blk, 0, header, cluster_size, 0);
> g_free(header);
> if (ret < 0) {
> @@ -3602,6 +3699,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
> { BLOCK_OPT_ENCRYPT, BLOCK_OPT_ENCRYPT_FORMAT },
> { BLOCK_OPT_COMPAT_LEVEL, "version" },
> { BLOCK_OPT_DATA_FILE_RAW, "data-file-raw" },
> + { BLOCK_OPT_COMPRESSION_TYPE, "compression-type" },
> { NULL, NULL },
> };
>
> @@ -4859,6 +4957,7 @@ static ImageInfoSpecific *qcow2_get_specific_info(BlockDriverState *bs,
> .data_file = g_strdup(s->image_data_file),
> .has_data_file_raw = has_data_file(bs),
> .data_file_raw = data_file_is_raw(bs),
> + .compression_type = s->compression_type,
> };
> } else {
> /* if this assertion fails, this probably means a new version was
> @@ -5516,6 +5615,12 @@ static QemuOptsList qcow2_create_opts = {
> .help = "Width of a reference count entry in bits",
> .def_value_str = "16"
> },
> + {
> + .name = BLOCK_OPT_COMPRESSION_TYPE,
> + .type = QEMU_OPT_STRING,
> + .help = "Compression method used for image clusters compression",
> + .def_value_str = "zlib"
> + },
> { /* end of list */ }
> }
> };
> diff --git a/block/qcow2.h b/block/qcow2.h
> index 0942126232..f92412ed5e 100644
> --- a/block/qcow2.h
> +++ b/block/qcow2.h
> @@ -146,6 +146,12 @@ typedef struct QCowHeader {
>
> uint32_t refcount_order;
> uint32_t header_length;
> +
> + /* Additional fields */
> + uint8_t compression_type;
> +
> + /* header must be a multiple of 8 */
> + uint8_t padding[7];
> } QEMU_PACKED QCowHeader;
>
> typedef struct QEMU_PACKED QCowSnapshotHeader {
> @@ -213,16 +219,20 @@ enum {
>
> /* Incompatible feature bits */
> enum {
> - QCOW2_INCOMPAT_DIRTY_BITNR = 0,
> - QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
> - QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
> - QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
> - QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
> - QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
> + QCOW2_INCOMPAT_DIRTY_BITNR = 0,
> + QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
> + QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
> + QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3,
> + QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
> + QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
> + QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
> + QCOW2_INCOMPAT_COMPRESSION_TYPE =
> + 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
>
> QCOW2_INCOMPAT_MASK = QCOW2_INCOMPAT_DIRTY
> | QCOW2_INCOMPAT_CORRUPT
> - | QCOW2_INCOMPAT_DATA_FILE,
> + | QCOW2_INCOMPAT_DATA_FILE
> + | QCOW2_INCOMPAT_COMPRESSION_TYPE,
> };
>
> /* Compatible feature bits */
> @@ -369,6 +379,13 @@ typedef struct BDRVQcow2State {
>
> bool metadata_preallocation_checked;
> bool metadata_preallocation;
> + /*
> + * Compression type used for the image. Default: 0 - ZLIB
> + * The image compression type is set on image creation.
> + * The only way to change the compression type is to convert the image
> + * with the desired compression type set
> + */
> + Qcow2CompressionType compression_type;
> } BDRVQcow2State;
>
> typedef struct Qcow2COWRegion {
> diff --git a/include/block/block_int.h b/include/block/block_int.h
> index 6f9fd5e20e..2c6bb9dc99 100644
> --- a/include/block/block_int.h
> +++ b/include/block/block_int.h
> @@ -57,6 +57,7 @@
> #define BLOCK_OPT_REFCOUNT_BITS "refcount_bits"
> #define BLOCK_OPT_DATA_FILE "data_file"
> #define BLOCK_OPT_DATA_FILE_RAW "data_file_raw"
> +#define BLOCK_OPT_COMPRESSION_TYPE "compression_type"
>
> #define BLOCK_PROBE_BUF_SIZE 512
>
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index 85e27bb61f..873fbef3b5 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -78,6 +78,8 @@
> #
> # @bitmaps: A list of qcow2 bitmap details (since 4.0)
> #
> +# @compression-type: the image cluster compression method (since 5.0)
> +#
> # Since: 1.7
> ##
> { 'struct': 'ImageInfoSpecificQCow2',
> @@ -89,7 +91,8 @@
> '*corrupt': 'bool',
> 'refcount-bits': 'int',
> '*encrypt': 'ImageInfoSpecificQCow2Encryption',
> - '*bitmaps': ['Qcow2BitmapInfo']
> + '*bitmaps': ['Qcow2BitmapInfo'],
> + 'compression-type': 'Qcow2CompressionType'
> } }
>
> ##
> @@ -4392,6 +4395,18 @@
> 'data': [ 'v2', 'v3' ] }
>
>
> +##
> +# @Qcow2CompressionType:
> +#
> +# Compression type used in qcow2 image file
> +#
> +# @zlib: zlib compression, see <http://zlib.net/>
> +#
> +# Since: 5.0
> +##
> +{ 'enum': 'Qcow2CompressionType',
> + 'data': [ 'zlib' ] }
> +
> ##
> # @BlockdevCreateOptionsQcow2:
> #
> @@ -4415,6 +4430,8 @@
> # allowed values: off, falloc, full, metadata)
> # @lazy-refcounts: True if refcounts may be updated lazily (default: off)
> # @refcount-bits: Width of reference counts in bits (default: 16)
> +# @compression-type: The image cluster compression method
> +# (default: zlib, since 5.0)
> #
> # Since: 2.12
> ##
> @@ -4430,7 +4447,8 @@
> '*cluster-size': 'size',
> '*preallocation': 'PreallocMode',
> '*lazy-refcounts': 'bool',
> - '*refcount-bits': 'int' } }
> + '*refcount-bits': 'int',
> + '*compression-type': 'Qcow2CompressionType' } }
>
> ##
> # @BlockdevCreateOptionsQed:
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 2/8] qcow2: rework the cluster compression routine
2020-02-27 7:29 ` [PATCH v1 2/8] qcow2: rework the cluster compression routine Denis Plotnikov
@ 2020-02-27 8:54 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 8:54 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, qemu-block, den, armbru, mreitz
27.02.2020 10:29, Denis Plotnikov wrote:
> The patch enables processing the image compression type defined
> for the image and chooses an appropriate method for image clusters
> (de)compression.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> block/qcow2-threads.c | 77 +++++++++++++++++++++++++++++++++++--------
> 1 file changed, 63 insertions(+), 14 deletions(-)
>
> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
> index 77bb578cdf..1c128e9840 100644
> --- a/block/qcow2-threads.c
> +++ b/block/qcow2-threads.c
> @@ -74,7 +74,9 @@ typedef struct Qcow2CompressData {
> } Qcow2CompressData;
>
> /*
> - * qcow2_compress()
> + * qcow2_zlib_compress()
> + *
> + * Compress @src_size bytes of data using zlib compression method
> *
> * @dest - destination buffer, @dest_size bytes
> * @src - source buffer, @src_size bytes
> @@ -83,8 +85,8 @@ typedef struct Qcow2CompressData {
> * -ENOMEM destination buffer is not enough to store compressed data
> * -EIO on any other error
> */
> -static ssize_t qcow2_compress(void *dest, size_t dest_size,
> - const void *src, size_t src_size)
> +static ssize_t qcow2_zlib_compress(void *dest, size_t dest_size,
> + const void *src, size_t src_size)
> {
> ssize_t ret;
> z_stream strm;
> @@ -119,19 +121,19 @@ static ssize_t qcow2_compress(void *dest, size_t dest_size,
> }
>
> /*
> - * qcow2_decompress()
> + * qcow2_zlib_decompress()
> *
> * Decompress some data (not more than @src_size bytes) to produce exactly
> - * @dest_size bytes.
> + * @dest_size bytes using zlib compression method
> *
> * @dest - destination buffer, @dest_size bytes
> * @src - source buffer, @src_size bytes
> *
> * Returns: 0 on success
> - * -1 on fail
> + * -EIO on failure
> */
> -static ssize_t qcow2_decompress(void *dest, size_t dest_size,
> - const void *src, size_t src_size)
> +static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
> + const void *src, size_t src_size)
> {
> int ret = 0;
> z_stream strm;
> @@ -144,7 +146,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
>
> ret = inflateInit2(&strm, -12);
> if (ret != Z_OK) {
> - return -1;
> + return -EIO;
> }
>
> ret = inflate(&strm, Z_FINISH);
> @@ -154,7 +156,7 @@ static ssize_t qcow2_decompress(void *dest, size_t dest_size,
> * @src buffer may be processed partly (because in qcow2 we know size of
> * compressed data with precision of one sector)
> */
> - ret = -1;
> + ret = -EIO;
> }
>
> inflateEnd(&strm);
> @@ -189,20 +191,67 @@ qcow2_co_do_compress(BlockDriverState *bs, void *dest, size_t dest_size,
> return arg.ret;
> }
>
> +/*
> + * qcow2_co_compress()
> + *
> + * Compress @src_size bytes of data using the compression
> + * method defined by the image compression type
> + *
> + * @dest - destination buffer, @dest_size bytes
> + * @src - source buffer, @src_size bytes
> + *
> + * Returns: 0 on success
> + * a negative error code on failure
Hmm, it's default semantics and it used without any comment for most of C
functions, so I don't think we need the comment. As well as dest/src
argument names are obvious enough. Still, I'm not against too.
> + */
> ssize_t coroutine_fn
> qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
> const void *src, size_t src_size)
> {
> - return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
> - qcow2_compress);
> + BDRVQcow2State *s = bs->opaque;
> + Qcow2CompressFunc fn;
> +
> + switch (s->compression_type) {
> + case QCOW2_COMPRESSION_TYPE_ZLIB:
> + fn = qcow2_zlib_compress;
> + break;
> +
> + default:
> + return -ENOTSUP;
it can't be anything other. Maybe, better abort() ?
> + }
> +
> + return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
> }
>
> +/*
> + * qcow2_co_decompress()
> + *
> + * Decompress some data (not more than @src_size bytes) to produce exactly
> + * @dest_size bytes using the compression method defined by the image
> + * compression type
> + *
> + * @dest - destination buffer, @dest_size bytes
> + * @src - source buffer, @src_size bytes
> + *
> + * Returns: 0 on success
> + * a negative error code on failure
> + */
> ssize_t coroutine_fn
> qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
> const void *src, size_t src_size)
> {
> - return qcow2_co_do_compress(bs, dest, dest_size, src, src_size,
> - qcow2_decompress);
> + BDRVQcow2State *s = bs->opaque;
> + Qcow2CompressFunc fn;
> +
> + switch (s->compression_type) {
> + case QCOW2_COMPRESSION_TYPE_ZLIB:
> + fn = qcow2_zlib_decompress;
> + break;
> +
> + default:
> + return -ENOTSUP;
> + }
> +
> + return qcow2_co_do_compress(bs, dest, dest_size, src, src_size, fn);
> }
>
>
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
2020-02-27 7:29 ` [PATCH v1 3/8] qcow2: add zstd cluster compression Denis Plotnikov
@ 2020-02-27 9:55 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:11 ` Denis Plotnikov
2020-02-27 14:01 ` Eric Blake
1 sibling, 1 reply; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 9:55 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, qemu-block, den, armbru, mreitz
27.02.2020 10:29, Denis Plotnikov wrote:
> zstd significantly reduces cluster compression time.
> It provides better compression performance maintaining
> the same level of the compression ratio in comparison with
> zlib, which, at the moment, is the only compression
> method available.
>
> The performance test results:
> Test compresses and decompresses qemu qcow2 image with just
> installed rhel-7.6 guest.
> Image cluster size: 64K. Image on disk size: 2.2G
>
> The test was conducted with brd disk to reduce the influence
> of disk subsystem to the test results.
> The results is given in seconds.
>
> compress cmd:
> time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
> src.img [zlib|zstd]_compressed.img
> decompress cmd
> time ./qemu-img convert -O qcow2
> [zlib|zstd]_compressed.img uncompressed.img
>
> compression decompression
> zlib zstd zlib zstd
> ------------------------------------------------------------
> real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
> user 65.0 15.8 5.3 2.5
> sys 3.3 0.2 2.0 2.0
>
> Both ZLIB and ZSTD gave the same compression ratio: 1.57
> compressed image size in both cases: 1.4G
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> block/qcow2-threads.c | 122 +++++++++++++++++++++++++++++++++++++++++
> block/qcow2.c | 7 +++
> configure | 29 ++++++++++
> docs/interop/qcow2.txt | 18 ++++++
> qapi/block-core.json | 3 +-
> 5 files changed, 178 insertions(+), 1 deletion(-)
>
> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
> index 1c128e9840..e942c4d7e5 100644
> --- a/block/qcow2-threads.c
> +++ b/block/qcow2-threads.c
> @@ -28,6 +28,11 @@
> #define ZLIB_CONST
> #include <zlib.h>
>
> +#ifdef CONFIG_ZSTD
> +#include <zstd.h>
> +#include <zstd_errors.h>
> +#endif
> +
> #include "qcow2.h"
> #include "block/thread-pool.h"
> #include "crypto.h"
> @@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
> return ret;
> }
>
> +#ifdef CONFIG_ZSTD
> +
> +#define ZSTD_LEN_BUF 4
> +
> +/*
> + * qcow2_zstd_compress()
> + *
> + * Compress @src_size bytes of data using zstd compression method
> + *
> + * @dest - destination buffer, @dest_size bytes
> + * @src - source buffer, @src_size bytes
> + *
> + * Returns: compressed size on success
This doesn't match qcow2_co_compress definition. You should return 0 on success.
> + * -ENOMEM destination buffer is not enough to store compressed data
> + * -EIO on any other error
> + */
> +
> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
> + const void *src, size_t src_size)
> +{
> + size_t ret;
> +
> + /*
> + * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
beginning
> + * to store compressed chunk size
> + */
> + char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
> +
> + /*
> + * sanity check that we can store the compressed data length,
> + * and there is some space left for the compressor buffer
> + */
> + if (dest_size <= ZSTD_LEN_BUF) {
> + return -ENOMEM;
> + }
> +
> + dest_size -= ZSTD_LEN_BUF;
> +
> + ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
> +
> + if (ZSTD_isError(ret)) {
> + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
> + return -ENOMEM;
> + } else {
> + return -EIO;
> + }
> + }
> +
> + /* paraniod sanity check that we can store the commpressed size */
> + if (ret > UINT_MAX) {
> + return -ENOMEM;
> + }
I'd use UINT32_MAX, possibly even more paranoid)
> +
> + /* store the compressed chunk size in the very beginning of the buffer */
> + stl_be_p(dest, ret);
> +
> + return ret + ZSTD_LEN_BUF;
return 0;
> +}
> +
> +/*
> + * qcow2_zstd_decompress()
> + *
> + * Decompress some data (not more than @src_size bytes) to produce exactly
> + * @dest_size bytes using zstd compression method
> + *
> + * @dest - destination buffer, @dest_size bytes
> + * @src - source buffer, @src_size bytes
> + *
> + * Returns: 0 on success
> + * -EIO on any error
> + */
> +
extra empty line.
> +static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
> + const void *src, size_t src_size)
> +{
> + /*
> + * zstd decompress wants to know the exact length of the data.
> + * For that purpose, on compression, the length is stored in
> + * the very beginning of the compressed buffer
> + */
> + size_t s_size;
> + const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF;
> +
> + /*
> + * sanity check that we can read 4 byte the content length and
> + * and there is some content to decompress
> + */
> + if (src_size <= ZSTD_LEN_BUF) {
> + return -EIO;
> + }
> +
> + s_size = ldl_be_p(src);
> +
> + /* sanity check that the buffer is big enough to read the content from */
> + if (src_size - ZSTD_LEN_BUF < s_size) {
> + return -EIO;
> + }
> +
> + if (ZSTD_isError(
> + ZSTD_decompress(dest, dest_size, s_buf, s_size))) {
hmm, it fit into one line actually
> + return -EIO;
> + }
> +
> + return 0;
> +}
> +#endif
> +
> static int qcow2_compress_pool_func(void *opaque)
> {
> Qcow2CompressData *data = opaque;
> @@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
> fn = qcow2_zlib_compress;
> break;
>
> +#ifdef CONFIG_ZSTD
> + case QCOW2_COMPRESSION_TYPE_ZSTD:
> + fn = qcow2_zstd_compress;
> + break;
> +#endif
> default:
> return -ENOTSUP;
> }
> @@ -247,6 +364,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
> fn = qcow2_zlib_decompress;
> break;
>
> +#ifdef CONFIG_ZSTD
> + case QCOW2_COMPRESSION_TYPE_ZSTD:
> + fn = qcow2_zstd_decompress;
> + break;
> +#endif
> default:
> return -ENOTSUP;
> }
> diff --git a/block/qcow2.c b/block/qcow2.c
> index 2ccb2cabd1..9c8ad9d580 100644
> --- a/block/qcow2.c
> +++ b/block/qcow2.c
> @@ -1257,6 +1257,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp)
>
> switch (s->compression_type) {
> case QCOW2_COMPRESSION_TYPE_ZLIB:
> +#ifdef CONFIG_ZSTD
> + case QCOW2_COMPRESSION_TYPE_ZSTD:
> +#endif
> break;
>
> default:
> @@ -3506,6 +3509,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
> }
>
> switch (qcow2_opts->compression_type) {
> +#ifdef CONFIG_ZSTD
> + case QCOW2_COMPRESSION_TYPE_ZSTD:
> + break;
> +#endif
> default:
> error_setg_errno(errp, -EINVAL, "Unknown compression type");
> goto out;
> diff --git a/configure b/configure
> index 48d6f89d57..4690a7ea9f 100755
> --- a/configure
> +++ b/configure
> @@ -444,6 +444,7 @@ opengl_dmabuf="no"
> cpuid_h="no"
> avx2_opt=""
> zlib="yes"
> +zstd=""
> capstone=""
> lzo=""
> snappy=""
> @@ -1371,6 +1372,10 @@ for opt do
> ;;
> --disable-lzfse) lzfse="no"
> ;;
> + --enable-zstd) zstd="yes"
> + ;;
> + --disable-zstd) zstd="no"
> + ;;
> --enable-guest-agent) guest_agent="yes"
> ;;
> --disable-guest-agent) guest_agent="no"
> @@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default is enabled if available:
> (for reading bzip2-compressed dmg images)
> lzfse support of lzfse compression library
> (for reading lzfse-compressed dmg images)
> + zstd support of zstd compression library
> seccomp seccomp support
> coroutine-pool coroutine freelist (better performance)
> glusterfs GlusterFS backend
> @@ -2453,6 +2459,25 @@ EOF
> fi
> fi
>
> +#########################################
> +# zstd check
> +
> +if test "$zstd" != "no" ; then
> + cat > $TMPC << EOF
> +#include <zstd.h>
> +int main(void) { ZSTD_versionNumber(); return 0; }
> +EOF
> + if compile_prog "" "-lzstd" ; then
> + LIBS="$LIBS -lzstd"
> + zstd="yes"
> + else
> + if test "$zstd" = "yes"; then
> + feature_not_found "zstd" "Install libzstd-devel"
to correspond to style used around: s/libzstd-devel/libzstd devel/
> + fi
> + zstd="no"
> + fi
> +fi
> +
> ##########################################
> # libseccomp check
>
> @@ -6668,6 +6693,7 @@ echo "lzo support $lzo"
> echo "snappy support $snappy"
> echo "bzip2 support $bzip2"
> echo "lzfse support $lzfse"
> +echo "zstd support $zstd"
> echo "NUMA host support $numa"
> echo "libxml2 $libxml2"
> echo "tcmalloc support $tcmalloc"
> @@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then
> "\$ld_exported_symbols_list should have been set to 'yes'."
> fi
> fi
> +if test "$zstd" = "yes" ; then
> + echo "CONFIG_ZSTD=y" >> $config_host_mak
> +fi
>
> if test "$tcg_interpreter" = "yes"; then
> QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
> diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
> index 5597e24474..aeca2ddebd 100644
> --- a/docs/interop/qcow2.txt
> +++ b/docs/interop/qcow2.txt
> @@ -208,6 +208,7 @@ version 2.
>
> Available compression type values:
> 0: zlib <https://www.zlib.net/>
> + 1: zstd <http://github.com/facebook/zstd>
>
>
> === Header padding ===
> @@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)):
> Another compressed cluster may map to the tail of the final
> sector used by this compressed cluster.
>
> + The layout of the compressed data depends on the compression
> + type used for the image (see compressed cluster layout).
> +
> If a cluster is unallocated, read requests shall read the data from the backing
> file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
> no backing file or the backing file is smaller than the image, they shall read
> zeros for all parts that are not covered by the backing file.
>
> +=== Compressed Cluster Layout ===
> +
> +The compressed cluster data has a layout depending on the compression
> +type used for the image, as follows:
> +
> +Compressed data layout for the available compression types:
> +(x = data_space_length - 1)
Note, that term 'data_space_lenght' is not defined in the spec..
Hmm. And it's not trivial to define it correctly, all we have is offset and
"number of additional sectors", and may be not all bytes of final sector are
occupied... What about something like this:
In the following layout description byte 0 corresponds to byte at host cluster offset,
as defined by "Compressed Clusters Descriptor" paragraph above. The whole layout occupy
space starting from this offset, using additional 512-byte sectors defined by
"Compressed Clusters Descriptor" paragraph, not necessarily occupying all of the bytes
in the final sector. Let x be number of last byte of the layout.
> +
> + 0: (default) zlib <http://zlib.net/>:
> + Byte 0 - x: the compressed data content
> + all the space provided used for compressed data
> + 1: zstd <http://github.com/facebook/zstd>:
> + Byte 0 - 3: the length of compressed data in bytes
> + 4 - x: the compressed data content
Maybe, note that x+! == 4 + length, where length is value of first field of the layout.
>
> == Snapshots ==
>
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index 873fbef3b5..4b6e576c44 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -4401,11 +4401,12 @@
> # Compression type used in qcow2 image file
> #
> # @zlib: zlib compression, see <http://zlib.net/>
> +# @zstd: zstd compression, see <http://github.com/facebook/zstd>
> #
> # Since: 5.0
> ##
> { 'enum': 'Qcow2CompressionType',
> - 'data': [ 'zlib' ] }
> + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
>
> ##
> # @BlockdevCreateOptionsQcow2:
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 4/8] iotests: filter out compression_type
2020-02-27 7:29 ` [PATCH v1 4/8] iotests: filter out compression_type Denis Plotnikov
@ 2020-02-27 9:57 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:03 ` Eric Blake
1 sibling, 0 replies; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 9:57 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, qemu-block, den, armbru, mreitz
27.02.2020 10:29, Denis Plotnikov wrote:
> After adding compression type feature to qcow2 format, qemu framework
> commands reporting the image settingd, e.g. "qemu-img create", started
> reporting the compression type for the image which breaks the iotests
> output matching.
>
> To fix it, add compression_type=zlib to the list of filtered image parameters.
So, the first patch breaks iotests? Than it should be merged into first patch or
moved before it, to not break git bisect.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> tests/qemu-iotests/common.filter | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
> index 3f8ee3e5f7..c6962d199c 100644
> --- a/tests/qemu-iotests/common.filter
> +++ b/tests/qemu-iotests/common.filter
> @@ -152,7 +152,8 @@ _filter_img_create()
> -e "s# refcount_bits=[0-9]\\+##g" \
> -e "s# key-secret=[a-zA-Z0-9]\\+##g" \
> -e "s# iter-time=[0-9]\\+##g" \
> - -e "s# force_size=\\(on\\|off\\)##g"
> + -e "s# force_size=\\(on\\|off\\)##g" \
> + -e "s# compression_type=zlib##g"
> }
>
> _filter_img_info()
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 5/8] iotests: fix header size, feature table size and backing file offset
2020-02-27 7:29 ` [PATCH v1 5/8] iotests: fix header size, feature table size and backing file offset Denis Plotnikov
@ 2020-02-27 9:59 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 9:59 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, qemu-block, den, armbru, mreitz
27.02.2020 10:29, Denis Plotnikov wrote:
> Affected tests: 031, 036, 061
>
> Because of adding the compression type feature, some size values in the
> qcow2 v3 header are changed:
>
> header_size +=8: 1 byte compression type
> 7 bytes padding
> feature_table += 48: incompatible feture compression type
>
> backing_file_offset += 56 (8 + 48 -> header_change + fature_table_change)
>
> Change the values for the test output comparison accordingly.
Again, this should be merged to the patch, which actually break iotests.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> tests/qemu-iotests/031.out | 14 +++++++-------
> tests/qemu-iotests/036.out | 4 ++--
> tests/qemu-iotests/061.out | 28 ++++++++++++++--------------
> 3 files changed, 23 insertions(+), 23 deletions(-)
>
> diff --git a/tests/qemu-iotests/031.out b/tests/qemu-iotests/031.out
> index d535e407bc..ed51afe9ce 100644
> --- a/tests/qemu-iotests/031.out
> +++ b/tests/qemu-iotests/031.out
> @@ -113,11 +113,11 @@ incompatible_features []
> compatible_features []
> autoclear_features []
> refcount_order 4
> -header_length 104
> +header_length 112
>
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> Header extension:
> @@ -146,11 +146,11 @@ incompatible_features []
> compatible_features []
> autoclear_features []
> refcount_order 4
> -header_length 104
> +header_length 112
>
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> Header extension:
> @@ -164,7 +164,7 @@ No errors were found on the image.
>
> magic 0x514649fb
> version 3
> -backing_file_offset 0x178
> +backing_file_offset 0x1b0
> backing_file_size 0x17
> cluster_bits 16
> size 67108864
> @@ -179,7 +179,7 @@ incompatible_features []
> compatible_features []
> autoclear_features []
> refcount_order 4
> -header_length 104
> +header_length 112
>
> Header extension:
> magic 0xe2792aca
> @@ -188,7 +188,7 @@ data 'host_device'
>
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> Header extension:
> diff --git a/tests/qemu-iotests/036.out b/tests/qemu-iotests/036.out
> index 0b52b934e1..fb509f6357 100644
> --- a/tests/qemu-iotests/036.out
> +++ b/tests/qemu-iotests/036.out
> @@ -26,7 +26,7 @@ compatible_features []
> autoclear_features [63]
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
>
> @@ -38,7 +38,7 @@ compatible_features []
> autoclear_features []
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> *** done
> diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
> index 8b3091a412..cea7fedfdc 100644
> --- a/tests/qemu-iotests/061.out
> +++ b/tests/qemu-iotests/061.out
> @@ -22,11 +22,11 @@ incompatible_features []
> compatible_features [0]
> autoclear_features []
> refcount_order 4
> -header_length 104
> +header_length 112
>
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> magic 0x514649fb
> @@ -80,11 +80,11 @@ incompatible_features []
> compatible_features [0]
> autoclear_features []
> refcount_order 4
> -header_length 104
> +header_length 112
>
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> magic 0x514649fb
> @@ -136,11 +136,11 @@ incompatible_features [0]
> compatible_features [0]
> autoclear_features []
> refcount_order 4
> -header_length 104
> +header_length 112
>
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> ERROR cluster 5 refcount=0 reference=1
> @@ -191,11 +191,11 @@ incompatible_features []
> compatible_features [42]
> autoclear_features [42]
> refcount_order 4
> -header_length 104
> +header_length 112
>
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> magic 0x514649fb
> @@ -260,11 +260,11 @@ incompatible_features []
> compatible_features [0]
> autoclear_features []
> refcount_order 4
> -header_length 104
> +header_length 112
>
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> read 65536/65536 bytes at offset 44040192
> @@ -294,11 +294,11 @@ incompatible_features [0]
> compatible_features [0]
> autoclear_features []
> refcount_order 4
> -header_length 104
> +header_length 112
>
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> ERROR cluster 5 refcount=0 reference=1
> @@ -323,11 +323,11 @@ incompatible_features []
> compatible_features []
> autoclear_features []
> refcount_order 4
> -header_length 104
> +header_length 112
>
> Header extension:
> magic 0x6803f857
> -length 192
> +length 240
> data <binary>
>
> read 131072/131072 bytes at offset 0
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 6/8] iotests: add "compression type" for test output matching
2020-02-27 7:29 ` [PATCH v1 6/8] iotests: add "compression type" for test output matching Denis Plotnikov
@ 2020-02-27 10:04 ` Vladimir Sementsov-Ogievskiy
2020-02-27 10:09 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 10:04 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, qemu-block, den, armbru, mreitz
27.02.2020 10:29, Denis Plotnikov wrote:
> Affected tests: 049, 060, 061, 065, 144, 182, 242, 255
>
> After adding the compression type feature for qcow2, the compression type
> is reported on image quering.
>
> Add the corresponding values of the "compression type" for the tests' output
> matching.
And this and the following patch.
Ideally, patch should not break any iotests. This means that all iotest updates
should be merged to the patch which changes their output. Probably, you can split
behavior-changing patch, to reduce iotest-updates per patch, but anyway, big patch
with a lot of iotests updates is better than patch which breaks iotests.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> tests/qemu-iotests/049.out | 102 ++++++++++++++++++-------------------
> tests/qemu-iotests/060.out | 1 +
> tests/qemu-iotests/061.out | 6 +++
> tests/qemu-iotests/065 | 20 +++++---
> tests/qemu-iotests/144.out | 4 +-
> tests/qemu-iotests/182.out | 2 +-
> tests/qemu-iotests/242.out | 5 ++
> tests/qemu-iotests/255.out | 8 +--
> 8 files changed, 82 insertions(+), 66 deletions(-)
>
> diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out
> index affa55b341..a5cfba1756 100644
> --- a/tests/qemu-iotests/049.out
> +++ b/tests/qemu-iotests/049.out
> @@ -4,90 +4,90 @@ QA output created by 049
> == 1. Traditional size parameter ==
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> == 2. Specifying size via -o ==
>
> qemu-img create -f qcow2 -o size=1024 TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1024b TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1k TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1K TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1M TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1G TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1T TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1024.0 TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1024.0b TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1.5k TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1.5K TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1.5M TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1.5G TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o size=1.5T TEST_DIR/t.qcow2
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> == 3. Invalid sizes ==
>
> @@ -129,84 +129,84 @@ qemu-img: TEST_DIR/t.qcow2: The image size must be specified only once
> == Check correct interpretation of suffixes for cluster size ==
>
> qemu-img create -f qcow2 -o cluster_size=1024 TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o cluster_size=1024b TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o cluster_size=1k TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o cluster_size=1K TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o cluster_size=1M TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o cluster_size=1024.0 TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o cluster_size=1024.0b TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o cluster_size=0.5k TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o cluster_size=0.5K TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o cluster_size=0.5M TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> == Check compat level option ==
>
> qemu-img create -f qcow2 -o compat=0.10 TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M
> qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42'
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M
> qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar'
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> == Check preallocation option ==
>
> qemu-img create -f qcow2 -o preallocation=off TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
> qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234'
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> == Check encryption option ==
>
> qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o encryption=on,encrypt.key-secret=sec0 TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> == Check lazy_refcounts option (only with v3) ==
>
> qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=off TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=on TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M
> qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater)
> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
>
> *** done
> diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out
> index d27692a33c..3e47f537e8 100644
> --- a/tests/qemu-iotests/060.out
> +++ b/tests/qemu-iotests/060.out
> @@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> refcount bits: 16
> corrupt: true
> diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
> index cea7fedfdc..c913f02ad6 100644
> --- a/tests/qemu-iotests/061.out
> +++ b/tests/qemu-iotests/061.out
> @@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> refcount bits: 16
> data file: TEST_DIR/t.IMGFMT.data
> @@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> refcount bits: 16
> data file: foo
> @@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> refcount bits: 16
> data file raw: false
> @@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> refcount bits: 16
> data file: TEST_DIR/t.IMGFMT.data
> @@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> refcount bits: 16
> data file: TEST_DIR/t.IMGFMT.data
> @@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> refcount bits: 16
> data file: TEST_DIR/t.IMGFMT.data
> diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
> index 6426474271..106303b5a5 100755
> --- a/tests/qemu-iotests/065
> +++ b/tests/qemu-iotests/065
> @@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific):
> class TestQCow2(TestQemuImgInfo):
> '''Testing a qcow2 version 2 image'''
> img_options = 'compat=0.10'
> - json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
> - human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
> + json_compare = { 'compat': '0.10', 'refcount-bits': 16, 'compression-type': 'zlib' }
> + human_compare = [ 'compat: 0.10', 'compression type: zlib', 'refcount bits: 16' ]
>
> class TestQCow3NotLazy(TestQemuImgInfo):
> '''Testing a qcow2 version 3 image with lazy refcounts disabled'''
> img_options = 'compat=1.1,lazy_refcounts=off'
> json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
> - 'refcount-bits': 16, 'corrupt': False }
> - human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
> + 'refcount-bits': 16, 'corrupt': False,
> + 'compression-type': 'zlib' }
> + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: false',
> 'refcount bits: 16', 'corrupt: false' ]
>
> class TestQCow3Lazy(TestQemuImgInfo):
> '''Testing a qcow2 version 3 image with lazy refcounts enabled'''
> img_options = 'compat=1.1,lazy_refcounts=on'
> json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
> - 'refcount-bits': 16, 'corrupt': False }
> - human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
> + 'refcount-bits': 16, 'corrupt': False,
> + 'compression-type': 'zlib' }
> + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: true',
> 'refcount bits: 16', 'corrupt: false' ]
>
> class TestQCow3NotLazyQMP(TestQMP):
> @@ -113,7 +115,8 @@ class TestQCow3NotLazyQMP(TestQMP):
> img_options = 'compat=1.1,lazy_refcounts=off'
> qemu_options = 'lazy-refcounts=on'
> compare = { 'compat': '1.1', 'lazy-refcounts': False,
> - 'refcount-bits': 16, 'corrupt': False }
> + 'refcount-bits': 16, 'corrupt': False,
> + 'compression-type': 'zlib' }
>
>
> class TestQCow3LazyQMP(TestQMP):
> @@ -122,7 +125,8 @@ class TestQCow3LazyQMP(TestQMP):
> img_options = 'compat=1.1,lazy_refcounts=on'
> qemu_options = 'lazy-refcounts=off'
> compare = { 'compat': '1.1', 'lazy-refcounts': True,
> - 'refcount-bits': 16, 'corrupt': False }
> + 'refcount-bits': 16, 'corrupt': False,
> + 'compression-type': 'zlib' }
>
> TestImageInfoSpecific = None
> TestQemuImgInfo = None
> diff --git a/tests/qemu-iotests/144.out b/tests/qemu-iotests/144.out
> index c7aa2e4820..885a8874a5 100644
> --- a/tests/qemu-iotests/144.out
> +++ b/tests/qemu-iotests/144.out
> @@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912
> { 'execute': 'qmp_capabilities' }
> {"return": {}}
> { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } }
> -Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
> {"return": {}}
>
> === Performing block-commit on active layer ===
> @@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/
> === Performing Live Snapshot 2 ===
>
> { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } }
> -Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
> {"return": {}}
> *** done
> diff --git a/tests/qemu-iotests/182.out b/tests/qemu-iotests/182.out
> index a8eea166c3..ae43654d32 100644
> --- a/tests/qemu-iotests/182.out
> +++ b/tests/qemu-iotests/182.out
> @@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]?
> {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
> {"return": {}}
> {'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } }
> -Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
> {"return": {}}
> {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
> {"return": {}}
> diff --git a/tests/qemu-iotests/242.out b/tests/qemu-iotests/242.out
> index 7ac8404d11..091b9126ce 100644
> --- a/tests/qemu-iotests/242.out
> +++ b/tests/qemu-iotests/242.out
> @@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> refcount bits: 16
> corrupt: false
> @@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> bitmaps:
> [0]:
> @@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> bitmaps:
> [0]:
> @@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> bitmaps:
> [0]:
> @@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes)
> cluster_size: 65536
> Format specific information:
> compat: 1.1
> + compression type: zlib
> lazy refcounts: false
> bitmaps:
> [0]:
> diff --git a/tests/qemu-iotests/255.out b/tests/qemu-iotests/255.out
> index 348909fdef..a3c99fd62e 100644
> --- a/tests/qemu-iotests/255.out
> +++ b/tests/qemu-iotests/255.out
> @@ -3,9 +3,9 @@ Finishing a commit job with background reads
>
> === Create backing chain and start VM ===
>
> -Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> -Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> === Start background read requests ===
>
> @@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled
>
> === Create images and start VM ===
>
> -Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> -Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
> +Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>
> wrote 1048576/1048576 bytes at offset 0
> 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 6/8] iotests: add "compression type" for test output matching
2020-02-27 10:04 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-27 10:09 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:06 ` Eric Blake
2020-02-28 8:13 ` Denis Plotnikov
0 siblings, 2 replies; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 10:09 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, qemu-block, den, armbru, mreitz
27.02.2020 13:04, Vladimir Sementsov-Ogievskiy wrote:
> 27.02.2020 10:29, Denis Plotnikov wrote:
>> Affected tests: 049, 060, 061, 065, 144, 182, 242, 255
>>
>> After adding the compression type feature for qcow2, the compression type
>> is reported on image quering.
>>
>> Add the corresponding values of the "compression type" for the tests' output
>> matching.
>
> And this and the following patch.
>
> Ideally, patch should not break any iotests. This means that all iotest updates
> should be merged to the patch which changes their output. Probably, you can split
> behavior-changing patch, to reduce iotest-updates per patch, but anyway, big patch
> with a lot of iotests updates is better than patch which breaks iotests.
Or we can try to reduce behavior changes in case of zlib:
- keep header small in case of zlib, not adding zero field
- don't add feature table entry, if compress type is zlib
- don't report compression type on quering, if it is zlib
- then, all iotests output will be saved. And, then, if we need, we can change
these theree points one-by-one, updating iotests outputs. But I doubt that we
really need it, compatible behavior seems good enough.
>
>>
>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>> ---
>> tests/qemu-iotests/049.out | 102 ++++++++++++++++++-------------------
>> tests/qemu-iotests/060.out | 1 +
>> tests/qemu-iotests/061.out | 6 +++
>> tests/qemu-iotests/065 | 20 +++++---
>> tests/qemu-iotests/144.out | 4 +-
>> tests/qemu-iotests/182.out | 2 +-
>> tests/qemu-iotests/242.out | 5 ++
>> tests/qemu-iotests/255.out | 8 +--
>> 8 files changed, 82 insertions(+), 66 deletions(-)
>>
>> diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out
>> index affa55b341..a5cfba1756 100644
>> --- a/tests/qemu-iotests/049.out
>> +++ b/tests/qemu-iotests/049.out
>> @@ -4,90 +4,90 @@ QA output created by 049
>> == 1. Traditional size parameter ==
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> == 2. Specifying size via -o ==
>> qemu-img create -f qcow2 -o size=1024 TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1024b TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1k TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1K TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1M TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1G TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1T TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1024.0 TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1024.0b TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1.5k TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1.5K TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1.5M TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1.5G TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o size=1.5T TEST_DIR/t.qcow2
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> == 3. Invalid sizes ==
>> @@ -129,84 +129,84 @@ qemu-img: TEST_DIR/t.qcow2: The image size must be specified only once
>> == Check correct interpretation of suffixes for cluster size ==
>> qemu-img create -f qcow2 -o cluster_size=1024 TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o cluster_size=1024b TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o cluster_size=1k TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o cluster_size=1K TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o cluster_size=1M TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1048576 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o cluster_size=1024.0 TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o cluster_size=1024.0b TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=1024 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o cluster_size=0.5k TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o cluster_size=0.5K TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=512 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o cluster_size=0.5M TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=524288 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> == Check compat level option ==
>> qemu-img create -f qcow2 -o compat=0.10 TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M
>> qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42'
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M
>> qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar'
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> == Check preallocation option ==
>> qemu-img create -f qcow2 -o preallocation=off TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=off lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o preallocation=metadata TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=metadata lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
>> qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234'
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 cluster_size=65536 preallocation=1234 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> == Check encryption option ==
>> qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o encryption=on,encrypt.key-secret=sec0 TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 encryption=on encrypt.key-secret=sec0 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> == Check lazy_refcounts option (only with v3) ==
>> qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=off TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=on TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off TEST_DIR/t.qcow2 64M
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on TEST_DIR/t.qcow2 64M
>> qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with compatibility level 1.1 and above (use version=v3 or greater)
>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10 cluster_size=65536 lazy_refcounts=on refcount_bits=16 compression_type=zlib
>> *** done
>> diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out
>> index d27692a33c..3e47f537e8 100644
>> --- a/tests/qemu-iotests/060.out
>> +++ b/tests/qemu-iotests/060.out
>> @@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> refcount bits: 16
>> corrupt: true
>> diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
>> index cea7fedfdc..c913f02ad6 100644
>> --- a/tests/qemu-iotests/061.out
>> +++ b/tests/qemu-iotests/061.out
>> @@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> refcount bits: 16
>> data file: TEST_DIR/t.IMGFMT.data
>> @@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> refcount bits: 16
>> data file: foo
>> @@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> refcount bits: 16
>> data file raw: false
>> @@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> refcount bits: 16
>> data file: TEST_DIR/t.IMGFMT.data
>> @@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> refcount bits: 16
>> data file: TEST_DIR/t.IMGFMT.data
>> @@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> refcount bits: 16
>> data file: TEST_DIR/t.IMGFMT.data
>> diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
>> index 6426474271..106303b5a5 100755
>> --- a/tests/qemu-iotests/065
>> +++ b/tests/qemu-iotests/065
>> @@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific):
>> class TestQCow2(TestQemuImgInfo):
>> '''Testing a qcow2 version 2 image'''
>> img_options = 'compat=0.10'
>> - json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
>> - human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
>> + json_compare = { 'compat': '0.10', 'refcount-bits': 16, 'compression-type': 'zlib' }
>> + human_compare = [ 'compat: 0.10', 'compression type: zlib', 'refcount bits: 16' ]
>> class TestQCow3NotLazy(TestQemuImgInfo):
>> '''Testing a qcow2 version 3 image with lazy refcounts disabled'''
>> img_options = 'compat=1.1,lazy_refcounts=off'
>> json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
>> - 'refcount-bits': 16, 'corrupt': False }
>> - human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
>> + 'refcount-bits': 16, 'corrupt': False,
>> + 'compression-type': 'zlib' }
>> + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: false',
>> 'refcount bits: 16', 'corrupt: false' ]
>> class TestQCow3Lazy(TestQemuImgInfo):
>> '''Testing a qcow2 version 3 image with lazy refcounts enabled'''
>> img_options = 'compat=1.1,lazy_refcounts=on'
>> json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
>> - 'refcount-bits': 16, 'corrupt': False }
>> - human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
>> + 'refcount-bits': 16, 'corrupt': False,
>> + 'compression-type': 'zlib' }
>> + human_compare = [ 'compat: 1.1', 'compression type: zlib', 'lazy refcounts: true',
>> 'refcount bits: 16', 'corrupt: false' ]
>> class TestQCow3NotLazyQMP(TestQMP):
>> @@ -113,7 +115,8 @@ class TestQCow3NotLazyQMP(TestQMP):
>> img_options = 'compat=1.1,lazy_refcounts=off'
>> qemu_options = 'lazy-refcounts=on'
>> compare = { 'compat': '1.1', 'lazy-refcounts': False,
>> - 'refcount-bits': 16, 'corrupt': False }
>> + 'refcount-bits': 16, 'corrupt': False,
>> + 'compression-type': 'zlib' }
>> class TestQCow3LazyQMP(TestQMP):
>> @@ -122,7 +125,8 @@ class TestQCow3LazyQMP(TestQMP):
>> img_options = 'compat=1.1,lazy_refcounts=on'
>> qemu_options = 'lazy-refcounts=off'
>> compare = { 'compat': '1.1', 'lazy-refcounts': True,
>> - 'refcount-bits': 16, 'corrupt': False }
>> + 'refcount-bits': 16, 'corrupt': False,
>> + 'compression-type': 'zlib' }
>> TestImageInfoSpecific = None
>> TestQemuImgInfo = None
>> diff --git a/tests/qemu-iotests/144.out b/tests/qemu-iotests/144.out
>> index c7aa2e4820..885a8874a5 100644
>> --- a/tests/qemu-iotests/144.out
>> +++ b/tests/qemu-iotests/144.out
>> @@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=536870912
>> { 'execute': 'qmp_capabilities' }
>> {"return": {}}
>> { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT' } }
>> -Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> {"return": {}}
>> === Performing block-commit on active layer ===
>> @@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/
>> === Performing Live Snapshot 2 ===
>> { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device': 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format': 'IMGFMT' } }
>> -Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912 backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> {"return": {}}
>> *** done
>> diff --git a/tests/qemu-iotests/182.out b/tests/qemu-iotests/182.out
>> index a8eea166c3..ae43654d32 100644
>> --- a/tests/qemu-iotests/182.out
>> +++ b/tests/qemu-iotests/182.out
>> @@ -13,7 +13,7 @@ Is another process using the image [TEST_DIR/t.qcow2]?
>> {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
>> {"return": {}}
>> {'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name': 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay', 'snapshot-node-name': 'node1' } }
>> -Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120 backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> {"return": {}}
>> {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1', 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
>> {"return": {}}
>> diff --git a/tests/qemu-iotests/242.out b/tests/qemu-iotests/242.out
>> index 7ac8404d11..091b9126ce 100644
>> --- a/tests/qemu-iotests/242.out
>> +++ b/tests/qemu-iotests/242.out
>> @@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> refcount bits: 16
>> corrupt: false
>> @@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> bitmaps:
>> [0]:
>> @@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> bitmaps:
>> [0]:
>> @@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> bitmaps:
>> [0]:
>> @@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes)
>> cluster_size: 65536
>> Format specific information:
>> compat: 1.1
>> + compression type: zlib
>> lazy refcounts: false
>> bitmaps:
>> [0]:
>> diff --git a/tests/qemu-iotests/255.out b/tests/qemu-iotests/255.out
>> index 348909fdef..a3c99fd62e 100644
>> --- a/tests/qemu-iotests/255.out
>> +++ b/tests/qemu-iotests/255.out
>> @@ -3,9 +3,9 @@ Finishing a commit job with background reads
>> === Create backing chain and start VM ===
>> -Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> -Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> === Start background read requests ===
>> @@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled
>> === Create images and start VM ===
>> -Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> -Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16
>> +Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728 cluster_size=65536 lazy_refcounts=off refcount_bits=16 compression_type=zlib
>> wrote 1048576/1048576 bytes at offset 0
>> 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>
>
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 8/8] iotests: 287: add qcow2 compression type test
2020-02-27 7:29 ` [PATCH v1 8/8] iotests: 287: add qcow2 compression type test Denis Plotnikov
@ 2020-02-27 10:29 ` Vladimir Sementsov-Ogievskiy
2020-02-28 8:23 ` Denis Plotnikov
0 siblings, 1 reply; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 10:29 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, qemu-block, den, armbru, mreitz
27.02.2020 10:29, Denis Plotnikov wrote:
> The test checks fulfilling qcow2 requiriements for the compression
> type feature and zstd compression type operability.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> tests/qemu-iotests/287 | 123 +++++++++++++++++++++++++++++++++++++
> tests/qemu-iotests/287.out | 41 +++++++++++++
> tests/qemu-iotests/group | 1 +
> 3 files changed, 165 insertions(+)
> create mode 100755 tests/qemu-iotests/287
> create mode 100644 tests/qemu-iotests/287.out
>
> diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
> new file mode 100755
> index 0000000000..41b916f690
> --- /dev/null
> +++ b/tests/qemu-iotests/287
> @@ -0,0 +1,123 @@
> +#!/usr/bin/env bash
> +#
> +# Test case for an image using zstd compression
> +#
> +# Copyright (c) 2020 Virtuozzo International GmbH
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +# creator
> +owner=dplotnikov@virtuozzo.com
> +
> +seq="$(basename $0)"
> +echo "QA output created by $seq"
> +
> +status=1 # failure is the default!
> +
> +_cleanup()
> +{
> + _cleanup_test_img
> +}
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +# standard environment
> +. ./common.rc
> +. ./common.filter
> +
> +# This tests qocw2-specific low-level functionality
> +_supported_fmt qcow2
> +_supported_proto file
> +_supported_os Linux
> +
> +P=`echo "$QEMU_PROG" | sed "s/qemu-system-x86_64//"`
> +
> +grep "CONFIG_ZSTD=y" "$P"../config-host.mak >/dev/null
> +RES=$?
Hmm. This will not work for other architectures and for
out of tree builds. Also, it checks config but not current
binary (they may be out of sync, or even unrelated).
Probably better try to create image with zstd compression type
and handle expected error.
> +if (($RES)); then
> + _notrun "ZSTD is disabled in the current configuration"
> +fi
> +
> +# Test: when compression is zlib the incompatible is unset
> +echo
> +echo "=== Testing compression type incompatible bit setting for zlib ==="
> +echo
> +
> +_make_test_img 64M
> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
> +
> +# Test: when compression differs from zlib the incompatible bit is set
> +echo
> +echo "=== Testing compression type incompatible bit setting for zstd ==="
> +echo
> +
> +IMGOPTS='compression_type=zstd' _make_test_img 64M
> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
> +
> +# Test: an image can't be openned if compression type is zlib and
opened
> +# incompatible feature compression type is set
> +echo
> +echo "=== Testing zlib with incompatible bit set ==="
> +echo
> +
> +IMGOPTS='compression_type=zlib' _make_test_img 64M
> +$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
> +# to make sure the bit was actually set
> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
> +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
> +if (($?==0)); then
> + echo "Error: The image openned successfully. The image must not be openned"
> +fi
may be better to instead keep error output and just check it..
> +
> +# Test: an image can't be openned if compression type is NOT zlib and
> +# incompatible feature compression type is UNSET
> +echo
> +echo "=== Testing zstd with incompatible bit unset ==="
> +echo
> +
> +IMGOPTS='compression_type=zstd' _make_test_img 64M
> +$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
> +# to make sure the bit was actually unset
> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
> +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
> +if (($?==0)); then
> + echo "Error: The image openned successfully. The image must not be openned"
> +fi
> +# Test: check compression type values
> +echo
> +echo "=== Testing compression type values ==="
> +echo
> +# zlib=0
> +IMGOPTS='compression_type=zlib' _make_test_img 64M
> +od -j104 -N1 -An -vtu1 "$TEST_IMG"
> +
> +# zstd=1
> +IMGOPTS='compression_type=zstd' _make_test_img 64M
> +od -j104 -N1 -An -vtu1 "$TEST_IMG"
> +
> +# Test: using zstd compression, write to and read from an image
> +echo
> +echo "=== Testing reading and writing with zstd ==="
> +echo
> +
> +CLUSTER_SIZE=65536
> +IMGOPTS='compression_type=zstd' _make_test_img 64M
> +$QEMU_IO -c "write -c 0 64k " "$TEST_IMG" | _filter_qemu_io
> +$QEMU_IO -c "read -v 0 10 " "$TEST_IMG" | _filter_qemu_io
> +$QEMU_IO -c "read -v 65530 8" "$TEST_IMG" | _filter_qemu_io
Hmm output depends on default pattern. Better use "write -c -P 0x11 0 64k"
(or any pattern you want), to make it explicit.
> +
> +# success, all done
> +echo "*** done"
> +rm -f $seq.full
> +status=0
> diff --git a/tests/qemu-iotests/287.out b/tests/qemu-iotests/287.out
> new file mode 100644
> index 0000000000..4218254ce0
> --- /dev/null
> +++ b/tests/qemu-iotests/287.out
> @@ -0,0 +1,41 @@
> +QA output created by 287
> +
> +=== Testing compression type incompatible bit setting for zlib ===
> +
> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
> +incompatible_features []
> +
> +=== Testing compression type incompatible bit setting for zstd ===
> +
> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
> +incompatible_features [3]
> +
> +=== Testing zlib with incompatible bit set ===
> +
> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
> +incompatible_features [3]
> +
> +=== Testing zstd with incompatible bit unset ===
> +
> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
> +incompatible_features []
> +
> +=== Testing compression type values ===
> +
> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
> + 0
> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
> + 1
> +
> +=== Testing reading and writing with zstd ===
> +
> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
> +wrote 65536/65536 bytes at offset 0
> +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +00000000: cd cd cd cd cd cd cd cd cd cd ..........
> +read 10/10 bytes at offset 0
> +10 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +0000fffa: cd cd cd cd cd cd 00 00 ........
> +read 8/8 bytes at offset 65530
> +8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +*** done
> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> index 0317667695..5edbadef40 100644
> --- a/tests/qemu-iotests/group
> +++ b/tests/qemu-iotests/group
> @@ -293,3 +293,4 @@
> 283 auto quick
> 284 rw
> 286 rw quick
> +287 auto quick
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 1/8] qcow2: introduce compression type feature
2020-02-27 8:21 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-27 13:24 ` Eric Blake
0 siblings, 0 replies; 35+ messages in thread
From: Eric Blake @ 2020-02-27 13:24 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, Denis Plotnikov, qemu-devel
Cc: kwolf, armbru, qemu-block, mreitz
On 2/27/20 2:21 AM, Vladimir Sementsov-Ogievskiy wrote:
> 27.02.2020 10:29, Denis Plotnikov wrote:
>> The patch adds some preparation parts for incompatible compression type
>> feature to Qcow2 that indicates which allow to use different compression
>> methods for image clusters (de)compressing.
>>
>> It is implied that the compression type is set on the image creation and
>> can be changed only later by image conversion, thus compression type
>> defines the only compression algorithm used for the image, and thus,
>> for all image clusters.
>>
>> The goal of the feature is to add support of other compression methods
>> to qcow2. For example, ZSTD which is more effective on compression
>> than ZLIB.
>>
>> The default compression is ZLIB. Images created with ZLIB compression
>> type
>> are backward compatible with older qemu versions.
>>
>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>> ---
>> block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++
>> block/qcow2.h | 31 ++++++++---
>> include/block/block_int.h | 1 +
>> qapi/block-core.json | 22 +++++++-
>> 4 files changed, 150 insertions(+), 9 deletions(-)
>>
>> diff --git a/block/qcow2.c b/block/qcow2.c
>> index 3c754f616b..2ccb2cabd1 100644
>> --- a/block/qcow2.c
>> +++ b/block/qcow2.c
>
> Please, add to .git/config:
>
> [diff]
> orderFile = /path/to/qemu/scripts/git.orderfile
>
> This will force git format-patch to sort files in more comfortable order
> (header changes first, etc).
As I learned yesterday, git 2.23 and 2.24 have a bug where git
format-patch fails to honor diff.orderfile (fixed in 2.25):
https://bugzilla.redhat.com/show_bug.cgi?id=1807681
(and that explains why some of my recent patches have not been ordered
the way I wanted, as Fedora 31 currently has a git from the broken
window in time)
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 1/8] qcow2: introduce compression type feature
2020-02-27 7:29 ` [PATCH v1 1/8] qcow2: introduce compression type feature Denis Plotnikov
2020-02-27 8:21 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-27 13:48 ` Eric Blake
2020-02-27 13:59 ` Vladimir Sementsov-Ogievskiy
2020-02-28 8:34 ` Denis Plotnikov
1 sibling, 2 replies; 35+ messages in thread
From: Eric Blake @ 2020-02-27 13:48 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, vsementsov, armbru, qemu-block, mreitz
On 2/27/20 1:29 AM, Denis Plotnikov wrote:
> The patch adds some preparation parts for incompatible compression type
> feature to Qcow2 that indicates which allow to use different compression
to qcow2, allowing the use of different
> methods for image clusters (de)compressing.
>
> It is implied that the compression type is set on the image creation and
> can be changed only later by image conversion, thus compression type
> defines the only compression algorithm used for the image, and thus,
> for all image clusters.
>
> The goal of the feature is to add support of other compression methods
> to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
>
> The default compression is ZLIB. Images created with ZLIB compression type
> are backward compatible with older qemu versions.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++
> block/qcow2.h | 31 ++++++++---
> include/block/block_int.h | 1 +
> qapi/block-core.json | 22 +++++++-
> 4 files changed, 150 insertions(+), 9 deletions(-)
>
> diff --git a/block/qcow2.c b/block/qcow2.c
> index 3c754f616b..2ccb2cabd1 100644
> --- a/block/qcow2.c
> +++ b/block/qcow2.c
> @@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
> return ret;
> }
>
> +static int validate_compression_type(BDRVQcow2State *s, Error **errp)
> +{
> + /*
> + * Sanity check
> + * according to qcow2 spec, the compression type is 1-byte field
> + * but in BDRVQcow2State the compression_type is enum sizeof(int)
> + * so, the max compression_type value is 255.
> + */
> + if (s->compression_type > 0xff) {
> + error_setg(errp, "qcow2: compression type value is too big");
> + return -EINVAL;
> + }
Hmm - I think it may be worth a tweak to qcow2.txt to call out:
104: compression_type
105 - 111: padding, must be 0
or else call out:
104-111: compression type
and just blindly use all 8 bytes for the value even though really only 1
or two values will ever be defined. Of course, that moves the byte in
question from 104 to 111, thanks to our big endian encoding, but as this
series is the first one installing a non-zero value in those 8 bytes,
and as we just finished documenting that the header length must be a
multiple of 8, there is no real impact - we can make such tweaks up
until the 5.0 release.
> +
> + switch (s->compression_type) {
> + case QCOW2_COMPRESSION_TYPE_ZLIB:
> + break;
> +
> + default:
> + error_setg(errp, "qcow2: unknown compression type: %u",
> + s->compression_type);
> + return -ENOTSUP;
> + }
Having two checks feels redundant, compared to just letting the default
catch all unrecognized values in that field.
> +
> + /*
> + * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
> + * the incompatible feature flag must be set
> + */
> + if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
> + if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE) {
> + error_setg(errp, "qcow2: Compression type incompatible feature "
> + "bit must not be set");
> + return -EINVAL;
> + }
> + } else {
> + if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
> + error_setg(errp, "qcow2: Compression type incompatible feature "
> + "bit must be set");
> + return -EINVAL;
> + }
> + }
Matches what we documented in the spec.
> +
> + return 0;
> +}
> +
> /* Called with s->lock held. */
> static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
> int flags, Error **errp)
> @@ -1357,6 +1401,26 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
> s->compatible_features = header.compatible_features;
> s->autoclear_features = header.autoclear_features;
>
> + /*
> + * Handle compression type
> + * Older qcow2 images don't contain the compression type header.
> + * Distinguish them by the header length and use
> + * the only valid (default) compression type in that case
> + */
> + if (header.header_length > offsetof(QCowHeader, compression_type)) {
> + /*
> + * don't deal with endians since compression_type is 1 byte long
> + */
> + s->compression_type = header.compression_type;
Changes if you go with my suggestion of just making the compression_type
field occupy 8 bytes in the qcow2 header. (And if you want to keep it 1
byte, I still think the spec should call out explicit padding bytes).
> + } else {
> + s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
> + }
> +
> + ret = validate_compression_type(s, errp);
> + if (ret) {
> + goto fail;
> + }
> +
> if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
> void *feature_table = NULL;
> qcow2_read_extensions(bs, header.header_length, ext_end,
> @@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs)
> total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
> refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
>
> + ret = validate_compression_type(s, NULL);
> +
> + if (ret) {
> + goto fail;
> + }
> +
> *header = (QCowHeader) {
> /* Version 2 fields */
> .magic = cpu_to_be32(QCOW_MAGIC),
> @@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs)
> .autoclear_features = cpu_to_be64(s->autoclear_features),
> .refcount_order = cpu_to_be32(s->refcount_order),
> .header_length = cpu_to_be32(header_length),
> + .compression_type = (uint8_t) s->compression_type,
Is the cast necessary?
> };
>
> /* For older versions, write a shorter header */
> @@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs)
> .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
> .name = "lazy refcounts",
> },
> + {
> + .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
Ordering: please group all the incompatible bits side-by-side (this
should come before the lazy refcount bit).
> + .bit = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
> + .name = "compression type",
This change breaks iotests; at least 31, 36, and 61 need updates (I've
got a similar patch pending which fixes the fact that we forgot the
autoclear bit [1]). You'll need to squash in fixes for those at the
same time.
[1] https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg08069.html
> + },
> };
>
> ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
> @@ -3401,6 +3477,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
> .refcount_table_offset = cpu_to_be64(cluster_size),
> .refcount_table_clusters = cpu_to_be32(1),
> .refcount_order = cpu_to_be32(refcount_order),
> + .compression_type = (uint8_t) QCOW2_COMPRESSION_TYPE_ZLIB,
Is the cast necessary?
> .header_length = cpu_to_be32(sizeof(*header)),
> };
>
> @@ -3420,6 +3497,26 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
> cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
> }
>
> + if (qcow2_opts->has_compression_type &&
> + qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
> +
> + if (qcow2_opts->compression_type > 0xff) {
> + error_setg_errno(errp, -EINVAL, "Too big compression type value");
> + goto out;
> + }
> +
> + switch (qcow2_opts->compression_type) {
> + default:
> + error_setg_errno(errp, -EINVAL, "Unknown compression type");
> + goto out;
> + }
This should probably be an assert that qcow2_opts->compression_type is
in range, rather than a switch statement and error_setg. Callers of
qcow2_co_create should not be handing us unknown values.
> +
> + header->compression_type = (uint8_t) qcow2_opts->compression_type;
Why the cast?
> +
> + header->incompatible_features |=
> + cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE);
> + }
> +
> ret = blk_pwrite(blk, 0, header, cluster_size, 0);
> g_free(header);
> if (ret < 0) {
> @@ -3602,6 +3699,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
> +++ b/block/qcow2.h
> @@ -146,6 +146,12 @@ typedef struct QCowHeader {
>
> uint32_t refcount_order;
> uint32_t header_length;
> +
> + /* Additional fields */
> + uint8_t compression_type;
> +
> + /* header must be a multiple of 8 */
> + uint8_t padding[7];
> } QEMU_PACKED QCowHeader;
You're changing the size of this struct, which WILL break iotests (and
even more than just the 3 I pointed out above for the feature name table).
/me looks ahead
Aha - you even noticed it: patch 7/8 fixes test 80. That fix needs to
be squashed in here, where the change is made.
>
> typedef struct QEMU_PACKED QCowSnapshotHeader {
> @@ -213,16 +219,20 @@ enum {
>
> /* Incompatible feature bits */
> enum {
> - QCOW2_INCOMPAT_DIRTY_BITNR = 0,
> - QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
> - QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
> - QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
> - QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
> - QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
> + QCOW2_INCOMPAT_DIRTY_BITNR = 0,
> + QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
> + QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
> + QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3,
> + QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
> + QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
> + QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
> + QCOW2_INCOMPAT_COMPRESSION_TYPE =
> + 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
Uggh. I hate realigning = just because we added a longer name,
especially when you then can't even keep things on one line because of
length. If it were me, I'd leave the existing lines alone, and/or
switch everything to just use 'BITNR = ' rather than trying to align =.
Bikeshedding - since the new name is so long, can you get by with the
shorter QCOW2_INCOMPAT_COMPRESSION_BITNR (drop the _TYPE)?
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 1/8] qcow2: introduce compression type feature
2020-02-27 13:48 ` Eric Blake
@ 2020-02-27 13:59 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:13 ` Eric Blake
2020-02-28 8:34 ` Denis Plotnikov
1 sibling, 1 reply; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 13:59 UTC (permalink / raw)
To: Eric Blake, Denis Plotnikov, qemu-devel; +Cc: kwolf, armbru, qemu-block, mreitz
27.02.2020 16:48, Eric Blake wrote:
> On 2/27/20 1:29 AM, Denis Plotnikov wrote:
>> The patch adds some preparation parts for incompatible compression type
>> feature to Qcow2 that indicates which allow to use different compression
>
> to qcow2, allowing the use of different
>
>> methods for image clusters (de)compressing.
>>
>> It is implied that the compression type is set on the image creation and
>> can be changed only later by image conversion, thus compression type
>> defines the only compression algorithm used for the image, and thus,
>> for all image clusters.
>>
>> The goal of the feature is to add support of other compression methods
>> to qcow2. For example, ZSTD which is more effective on compression than ZLIB.
>>
>> The default compression is ZLIB. Images created with ZLIB compression type
>> are backward compatible with older qemu versions.
>>
>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>> ---
>> block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++
>> block/qcow2.h | 31 ++++++++---
>> include/block/block_int.h | 1 +
>> qapi/block-core.json | 22 +++++++-
>> 4 files changed, 150 insertions(+), 9 deletions(-)
>>
>> diff --git a/block/qcow2.c b/block/qcow2.c
>> index 3c754f616b..2ccb2cabd1 100644
>> --- a/block/qcow2.c
>> +++ b/block/qcow2.c
>> @@ -1242,6 +1242,50 @@ static int qcow2_update_options(BlockDriverState *bs, QDict *options,
>> return ret;
>> }
>> +static int validate_compression_type(BDRVQcow2State *s, Error **errp)
>> +{
>> + /*
>> + * Sanity check
>> + * according to qcow2 spec, the compression type is 1-byte field
>> + * but in BDRVQcow2State the compression_type is enum sizeof(int)
>> + * so, the max compression_type value is 255.
>> + */
>> + if (s->compression_type > 0xff) {
>> + error_setg(errp, "qcow2: compression type value is too big");
>> + return -EINVAL;
>> + }
>
> Hmm - I think it may be worth a tweak to qcow2.txt to call out:
>
> 104: compression_type
> 105 - 111: padding, must be 0
>
> or else call out:
>
> 104-111: compression type
>
> and just blindly use all 8 bytes for the value even though really only 1 or two values will ever be defined. Of course, that moves the byte in question from 104 to 111, thanks to our big endian encoding, but as this series is the first one installing a non-zero value in those 8 bytes, and as we just finished documenting that the header length must be a multiple of 8, there is no real impact - we can make such tweaks up until the 5.0 release.
But what is the benefit? We have already documented padding in the spec, and discussed it so much time... What is the problem with padding? And why to add 8 bytes for every new feature which needs only one byte?
>
>> +
>> + switch (s->compression_type) {
>> + case QCOW2_COMPRESSION_TYPE_ZLIB:
>> + break;
>> +
>> + default:
>> + error_setg(errp, "qcow2: unknown compression type: %u",
>> + s->compression_type);
>> + return -ENOTSUP;
>> + }
>
> Having two checks feels redundant, compared to just letting the default catch all unrecognized values in that field.
>
>> +
>> + /*
>> + * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
>> + * the incompatible feature flag must be set
>> + */
>> + if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
>> + if (s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE) {
>> + error_setg(errp, "qcow2: Compression type incompatible feature "
>> + "bit must not be set");
>> + return -EINVAL;
>> + }
>> + } else {
>> + if (!(s->incompatible_features & QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
>> + error_setg(errp, "qcow2: Compression type incompatible feature "
>> + "bit must be set");
>> + return -EINVAL;
>> + }
>> + }
>
> Matches what we documented in the spec.
>
>> +
>> + return 0;
>> +}
>> +
>> /* Called with s->lock held. */
>> static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
>> int flags, Error **errp)
>> @@ -1357,6 +1401,26 @@ static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict *options,
>> s->compatible_features = header.compatible_features;
>> s->autoclear_features = header.autoclear_features;
>> + /*
>> + * Handle compression type
>> + * Older qcow2 images don't contain the compression type header.
>> + * Distinguish them by the header length and use
>> + * the only valid (default) compression type in that case
>> + */
>> + if (header.header_length > offsetof(QCowHeader, compression_type)) {
>> + /*
>> + * don't deal with endians since compression_type is 1 byte long
>> + */
>> + s->compression_type = header.compression_type;
>
> Changes if you go with my suggestion of just making the compression_type field occupy 8 bytes in the qcow2 header. (And if you want to keep it 1 byte, I still think the spec should call out explicit padding bytes).
>
>> + } else {
>> + s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
>> + }
>> +
>> + ret = validate_compression_type(s, errp);
>> + if (ret) {
>> + goto fail;
>> + }
>> +
>> if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
>> void *feature_table = NULL;
>> qcow2_read_extensions(bs, header.header_length, ext_end,
>> @@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs)
>> total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
>> refcount_table_clusters = s->refcount_table_size >> (s->cluster_bits - 3);
>> + ret = validate_compression_type(s, NULL);
>> +
>> + if (ret) {
>> + goto fail;
>> + }
>> +
>> *header = (QCowHeader) {
>> /* Version 2 fields */
>> .magic = cpu_to_be32(QCOW_MAGIC),
>> @@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs)
>> .autoclear_features = cpu_to_be64(s->autoclear_features),
>> .refcount_order = cpu_to_be32(s->refcount_order),
>> .header_length = cpu_to_be32(header_length),
>> + .compression_type = (uint8_t) s->compression_type,
>
> Is the cast necessary?
>
>> };
>> /* For older versions, write a shorter header */
>> @@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs)
>> .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
>> .name = "lazy refcounts",
>> },
>> + {
>> + .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
>
> Ordering: please group all the incompatible bits side-by-side (this should come before the lazy refcount bit).
>
>> + .bit = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
>> + .name = "compression type",
>
> This change breaks iotests; at least 31, 36, and 61 need updates (I've got a similar patch pending which fixes the fact that we forgot the autoclear bit [1]). You'll need to squash in fixes for those at the same time.
>
> [1] https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg08069.html
>
>> + },
>> };
>> ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
>> @@ -3401,6 +3477,7 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
>> .refcount_table_offset = cpu_to_be64(cluster_size),
>> .refcount_table_clusters = cpu_to_be32(1),
>> .refcount_order = cpu_to_be32(refcount_order),
>> + .compression_type = (uint8_t) QCOW2_COMPRESSION_TYPE_ZLIB,
>
> Is the cast necessary?
>
>> .header_length = cpu_to_be32(sizeof(*header)),
>> };
>> @@ -3420,6 +3497,26 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
>> cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
>> }
>> + if (qcow2_opts->has_compression_type &&
>> + qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
>> +
>> + if (qcow2_opts->compression_type > 0xff) {
>> + error_setg_errno(errp, -EINVAL, "Too big compression type value");
>> + goto out;
>> + }
>> +
>> + switch (qcow2_opts->compression_type) {
>> + default:
>> + error_setg_errno(errp, -EINVAL, "Unknown compression type");
>> + goto out;
>> + }
>
> This should probably be an assert that qcow2_opts->compression_type is in range, rather than a switch statement and error_setg. Callers of qcow2_co_create should not be handing us unknown values.
>
>> +
>> + header->compression_type = (uint8_t) qcow2_opts->compression_type;
>
> Why the cast?
>
>> +
>> + header->incompatible_features |=
>> + cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE);
>> + }
>> +
>> ret = blk_pwrite(blk, 0, header, cluster_size, 0);
>> g_free(header);
>> if (ret < 0) {
>> @@ -3602,6 +3699,7 @@ static int coroutine_fn qcow2_co_create_opts(const char *filename, QemuOpts *opt
>
>> +++ b/block/qcow2.h
>> @@ -146,6 +146,12 @@ typedef struct QCowHeader {
>> uint32_t refcount_order;
>> uint32_t header_length;
>> +
>> + /* Additional fields */
>> + uint8_t compression_type;
>> +
>> + /* header must be a multiple of 8 */
>> + uint8_t padding[7];
>> } QEMU_PACKED QCowHeader;
>
> You're changing the size of this struct, which WILL break iotests (and even more than just the 3 I pointed out above for the feature name table).
>
> /me looks ahead
>
> Aha - you even noticed it: patch 7/8 fixes test 80. That fix needs to be squashed in here, where the change is made.
>
>> typedef struct QEMU_PACKED QCowSnapshotHeader {
>> @@ -213,16 +219,20 @@ enum {
>> /* Incompatible feature bits */
>> enum {
>> - QCOW2_INCOMPAT_DIRTY_BITNR = 0,
>> - QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
>> - QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
>> - QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
>> - QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
>> - QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
>> + QCOW2_INCOMPAT_DIRTY_BITNR = 0,
>> + QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
>> + QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
>> + QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3,
>> + QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
>> + QCOW2_INCOMPAT_CORRUPT = 1 << QCOW2_INCOMPAT_CORRUPT_BITNR,
>> + QCOW2_INCOMPAT_DATA_FILE = 1 << QCOW2_INCOMPAT_DATA_FILE_BITNR,
>> + QCOW2_INCOMPAT_COMPRESSION_TYPE =
>> + 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
>
> Uggh. I hate realigning = just because we added a longer name, especially when you then can't even keep things on one line because of length. If it were me, I'd leave the existing lines alone, and/or switch everything to just use 'BITNR = ' rather than trying to align =.
>
> Bikeshedding - since the new name is so long, can you get by with the shorter QCOW2_INCOMPAT_COMPRESSION_BITNR (drop the _TYPE)?
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
2020-02-27 7:29 ` [PATCH v1 3/8] qcow2: add zstd cluster compression Denis Plotnikov
2020-02-27 9:55 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-27 14:01 ` Eric Blake
2020-02-28 11:40 ` Denis Plotnikov
1 sibling, 1 reply; 35+ messages in thread
From: Eric Blake @ 2020-02-27 14:01 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, vsementsov, armbru, qemu-block, mreitz
On 2/27/20 1:29 AM, Denis Plotnikov wrote:
> zstd significantly reduces cluster compression time.
> It provides better compression performance maintaining
> the same level of the compression ratio in comparison with
> zlib, which, at the moment, is the only compression
> method available.
>
> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
> + const void *src, size_t src_size)
> +{
> + size_t ret;
> +
> + /*
> + * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
beginning
> + * to store compressed chunk size
> + */
> + char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
> +
> + /*
> + * sanity check that we can store the compressed data length,
> + * and there is some space left for the compressor buffer
> + */
> + if (dest_size <= ZSTD_LEN_BUF) {
> + return -ENOMEM;
> + }
> +
> + dest_size -= ZSTD_LEN_BUF;
> +
> + ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
> +
> + if (ZSTD_isError(ret)) {
> + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
> + return -ENOMEM;
> + } else {
> + return -EIO;
> + }
> + }
> +
> + /* paraniod sanity check that we can store the commpressed size */
paranoid, compressed
> + if (ret > UINT_MAX) {
> + return -ENOMEM;
> + }
This is pointless. Better is to ensure that we actually compressed data
(the pigeonhole principle states that there are some inputs that MUST
result in inflation, in order for most other inputs to result in
compression). But that check was satisfied by checking for
ZSTD_error_dstSize_tooSmall, which is what happens for one of those
uncompressible inputs. Namely, zstd will never return a result larger
than dest_size, and since dest_size is smaller than UINT_MAX on entry,
this check is pointless. But if you want something, I'd be okay with:
assert(ret <= dest_size).
> +++ b/docs/interop/qcow2.txt
> @@ -208,6 +208,7 @@ version 2.
>
> Available compression type values:
> 0: zlib <https://www.zlib.net/>
> + 1: zstd <http://github.com/facebook/zstd>
>
>
> === Header padding ===
> @@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)):
> Another compressed cluster may map to the tail of the final
> sector used by this compressed cluster.
>
> + The layout of the compressed data depends on the compression
> + type used for the image (see compressed cluster layout).
> +
> If a cluster is unallocated, read requests shall read the data from the backing
> file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
> no backing file or the backing file is smaller than the image, they shall read
> zeros for all parts that are not covered by the backing file.
>
> +=== Compressed Cluster Layout ===
> +
> +The compressed cluster data has a layout depending on the compression
> +type used for the image, as follows:
> +
> +Compressed data layout for the available compression types:
> +(x = data_space_length - 1)
> +
> + 0: (default) zlib <http://zlib.net/>:
> + Byte 0 - x: the compressed data content
> + all the space provided used for compressed data
> + 1: zstd <http://github.com/facebook/zstd>:
> + Byte 0 - 3: the length of compressed data in bytes
> + 4 - x: the compressed data content
>
> == Snapshots ==
>
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index 873fbef3b5..4b6e576c44 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -4401,11 +4401,12 @@
> # Compression type used in qcow2 image file
> #
> # @zlib: zlib compression, see <http://zlib.net/>
> +# @zstd: zstd compression, see <http://github.com/facebook/zstd>
> #
> # Since: 5.0
> ##
> { 'enum': 'Qcow2CompressionType',
> - 'data': [ 'zlib' ] }
> + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
>
The spec and UI changes are okay.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 4/8] iotests: filter out compression_type
2020-02-27 7:29 ` [PATCH v1 4/8] iotests: filter out compression_type Denis Plotnikov
2020-02-27 9:57 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-27 14:03 ` Eric Blake
2020-02-28 11:53 ` Denis Plotnikov
1 sibling, 1 reply; 35+ messages in thread
From: Eric Blake @ 2020-02-27 14:03 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, vsementsov, armbru, qemu-block, mreitz
On 2/27/20 1:29 AM, Denis Plotnikov wrote:
> After adding compression type feature to qcow2 format, qemu framework
> commands reporting the image settingd, e.g. "qemu-img create", started
settings
> reporting the compression type for the image which breaks the iotests
> output matching.
>
> To fix it, add compression_type=zlib to the list of filtered image parameters.
>
> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
> ---
> tests/qemu-iotests/common.filter | 3 ++-
> 1 file changed, 2 insertions(+), 1 deletion(-)
This should be squashed in to the patch that caused the breakage (3/8,
if I'm right).
>
> diff --git a/tests/qemu-iotests/common.filter b/tests/qemu-iotests/common.filter
> index 3f8ee3e5f7..c6962d199c 100644
> --- a/tests/qemu-iotests/common.filter
> +++ b/tests/qemu-iotests/common.filter
> @@ -152,7 +152,8 @@ _filter_img_create()
> -e "s# refcount_bits=[0-9]\\+##g" \
> -e "s# key-secret=[a-zA-Z0-9]\\+##g" \
> -e "s# iter-time=[0-9]\\+##g" \
> - -e "s# force_size=\\(on\\|off\\)##g"
> + -e "s# force_size=\\(on\\|off\\)##g" \
> + -e "s# compression_type=zlib##g"
Do you really want to hard-code just zlib, or should this be more
generic as compression_type=[a-zA-Z0-9]\\+ as done on other lines like
key-secret?
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 6/8] iotests: add "compression type" for test output matching
2020-02-27 10:09 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-27 14:06 ` Eric Blake
2020-02-28 8:13 ` Denis Plotnikov
1 sibling, 0 replies; 35+ messages in thread
From: Eric Blake @ 2020-02-27 14:06 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, Denis Plotnikov, qemu-devel
Cc: kwolf, armbru, qemu-block, mreitz
On 2/27/20 4:09 AM, Vladimir Sementsov-Ogievskiy wrote:
> 27.02.2020 13:04, Vladimir Sementsov-Ogievskiy wrote:
>> 27.02.2020 10:29, Denis Plotnikov wrote:
>>> Affected tests: 049, 060, 061, 065, 144, 182, 242, 255
>>>
>>> After adding the compression type feature for qcow2, the compression
>>> type
>>> is reported on image quering.
>>>
>>> Add the corresponding values of the "compression type" for the tests'
>>> output
>>> matching.
>>
>> And this and the following patch.
>>
>> Ideally, patch should not break any iotests. This means that all
>> iotest updates
>> should be merged to the patch which changes their output. Probably,
>> you can split
>> behavior-changing patch, to reduce iotest-updates per patch, but
>> anyway, big patch
>> with a lot of iotests updates is better than patch which breaks iotests.
>
> Or we can try to reduce behavior changes in case of zlib:
>
> - keep header small in case of zlib, not adding zero field
No, I don't see the point in doing that. It makes maintenance harder to
try to minimize the header, for no gain (older images parse the larger
header just fine).
> - don't add feature table entry, if compress type is zlib
> - don't report compression type on quering, if it is zlib
>
> - then, all iotests output will be saved. And, then, if we need, we can
> change
> these theree points one-by-one, updating iotests outputs. But I doubt
> that we
> really need it, compatible behavior seems good enough.
>
[by the way, replying to this series is painful, since den@vrtuozzo.com
is not a valid email address]
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
2020-02-27 9:55 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-27 14:11 ` Denis Plotnikov
2020-02-27 14:18 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-27 14:11 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-devel
Cc: kwolf, qemu-block, den, armbru, mreitz
On 27.02.2020 12:55, Vladimir Sementsov-Ogievskiy wrote:
> 27.02.2020 10:29, Denis Plotnikov wrote:
>> zstd significantly reduces cluster compression time.
>> It provides better compression performance maintaining
>> the same level of the compression ratio in comparison with
>> zlib, which, at the moment, is the only compression
>> method available.
>>
>> The performance test results:
>> Test compresses and decompresses qemu qcow2 image with just
>> installed rhel-7.6 guest.
>> Image cluster size: 64K. Image on disk size: 2.2G
>>
>> The test was conducted with brd disk to reduce the influence
>> of disk subsystem to the test results.
>> The results is given in seconds.
>>
>> compress cmd:
>> time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
>> src.img [zlib|zstd]_compressed.img
>> decompress cmd
>> time ./qemu-img convert -O qcow2
>> [zlib|zstd]_compressed.img uncompressed.img
>>
>> compression decompression
>> zlib zstd zlib zstd
>> ------------------------------------------------------------
>> real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
>> user 65.0 15.8 5.3 2.5
>> sys 3.3 0.2 2.0 2.0
>>
>> Both ZLIB and ZSTD gave the same compression ratio: 1.57
>> compressed image size in both cases: 1.4G
>>
>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>> ---
>> block/qcow2-threads.c | 122 +++++++++++++++++++++++++++++++++++++++++
>> block/qcow2.c | 7 +++
>> configure | 29 ++++++++++
>> docs/interop/qcow2.txt | 18 ++++++
>> qapi/block-core.json | 3 +-
>> 5 files changed, 178 insertions(+), 1 deletion(-)
>>
>> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
>> index 1c128e9840..e942c4d7e5 100644
>> --- a/block/qcow2-threads.c
>> +++ b/block/qcow2-threads.c
>> @@ -28,6 +28,11 @@
>> #define ZLIB_CONST
>> #include <zlib.h>
>> +#ifdef CONFIG_ZSTD
>> +#include <zstd.h>
>> +#include <zstd_errors.h>
>> +#endif
>> +
>> #include "qcow2.h"
>> #include "block/thread-pool.h"
>> #include "crypto.h"
>> @@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void
>> *dest, size_t dest_size,
>> return ret;
>> }
>> +#ifdef CONFIG_ZSTD
>> +
>> +#define ZSTD_LEN_BUF 4
>> +
>> +/*
>> + * qcow2_zstd_compress()
>> + *
>> + * Compress @src_size bytes of data using zstd compression method
>> + *
>> + * @dest - destination buffer, @dest_size bytes
>> + * @src - source buffer, @src_size bytes
>> + *
>> + * Returns: compressed size on success
>
> This doesn't match qcow2_co_compress definition. You should return 0
> on success.
does it? I'd rather say it doesn't match to qcow2_co_compress
description in the function comment, which we can change actually,
because qcow2_co_compress is used like:
block/qcow2.c:
static coroutine_fn int
qcow2_co_pwritev_compressed_task(BlockDriverState *bs,
uint64_t offset, uint64_t bytes,
QEMUIOVector *qiov, size_t qiov_offset)
{
...
out_buf = g_malloc(s->cluster_size);
out_len = qcow2_co_compress(bs, out_buf, s->cluster_size - 1,
buf, s->cluster_size);
if (out_len == -ENOMEM) {
/* could not compress: write normal cluster */
ret = qcow2_co_pwritev_part(bs, offset, bytes, qiov,
qiov_offset, 0);
if (ret < 0) {
goto fail;
}
goto success;
} else if (out_len < 0) {
ret = -EINVAL;
goto fail;
}
qemu_co_mutex_lock(&s->lock);
ret = qcow2_alloc_compressed_cluster_offset(bs, offset, out_len,
<<<<<<<<<<<<
&cluster_offset);
...
}
>
>> + * -ENOMEM destination buffer is not enough to store
>> compressed data
>> + * -EIO on any other error
>> + */
>> +
>> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
>> + const void *src, size_t src_size)
>> +{
>> + size_t ret;
>> +
>> + /*
>> + * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
>
> beginning
>
>> + * to store compressed chunk size
>> + */
>> + char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
>> +
>> + /*
>> + * sanity check that we can store the compressed data length,
>> + * and there is some space left for the compressor buffer
>> + */
>> + if (dest_size <= ZSTD_LEN_BUF) {
>> + return -ENOMEM;
>> + }
>> +
>> + dest_size -= ZSTD_LEN_BUF;
>> +
>> + ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
>> +
>> + if (ZSTD_isError(ret)) {
>> + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
>> + return -ENOMEM;
>> + } else {
>> + return -EIO;
>> + }
>> + }
>> +
>> + /* paraniod sanity check that we can store the commpressed size */
>> + if (ret > UINT_MAX) {
>> + return -ENOMEM;
>> + }
>
> I'd use UINT32_MAX, possibly even more paranoid)
ok
>
>
>> +
>> + /* store the compressed chunk size in the very beginning of the
>> buffer */
>> + stl_be_p(dest, ret);
>> +
>> + return ret + ZSTD_LEN_BUF;
>
> return 0;
>
>> +}
>> +
>> +/*
>> + * qcow2_zstd_decompress()
>> + *
>> + * Decompress some data (not more than @src_size bytes) to produce
>> exactly
>> + * @dest_size bytes using zstd compression method
>> + *
>> + * @dest - destination buffer, @dest_size bytes
>> + * @src - source buffer, @src_size bytes
>> + *
>> + * Returns: 0 on success
>> + * -EIO on any error
>> + */
>> +
>
> extra empty line.
will be removed
>
>
>> +static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
>> + const void *src, size_t src_size)
>> +{
>> + /*
>> + * zstd decompress wants to know the exact length of the data.
>> + * For that purpose, on compression, the length is stored in
>> + * the very beginning of the compressed buffer
>> + */
>> + size_t s_size;
>> + const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF;
>> +
>> + /*
>> + * sanity check that we can read 4 byte the content length and
>> + * and there is some content to decompress
>> + */
>> + if (src_size <= ZSTD_LEN_BUF) {
>> + return -EIO;
>> + }
>> +
>> + s_size = ldl_be_p(src);
>> +
>> + /* sanity check that the buffer is big enough to read the
>> content from */
>> + if (src_size - ZSTD_LEN_BUF < s_size) {
>> + return -EIO;
>> + }
>> +
>> + if (ZSTD_isError(
>> + ZSTD_decompress(dest, dest_size, s_buf, s_size))) {
>
> hmm, it fit into one line actually
will be fixed if so
>
>
>> + return -EIO;
>> + }
>> +
>> + return 0;
>> +}
>> +#endif
>> +
>> static int qcow2_compress_pool_func(void *opaque)
>> {
>> Qcow2CompressData *data = opaque;
>> @@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs, void
>> *dest, size_t dest_size,
>> fn = qcow2_zlib_compress;
>> break;
>> +#ifdef CONFIG_ZSTD
>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>> + fn = qcow2_zstd_compress;
>> + break;
>> +#endif
>> default:
>> return -ENOTSUP;
>> }
>> @@ -247,6 +364,11 @@ qcow2_co_decompress(BlockDriverState *bs, void
>> *dest, size_t dest_size,
>> fn = qcow2_zlib_decompress;
>> break;
>> +#ifdef CONFIG_ZSTD
>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>> + fn = qcow2_zstd_decompress;
>> + break;
>> +#endif
>> default:
>> return -ENOTSUP;
>> }
>> diff --git a/block/qcow2.c b/block/qcow2.c
>> index 2ccb2cabd1..9c8ad9d580 100644
>> --- a/block/qcow2.c
>> +++ b/block/qcow2.c
>> @@ -1257,6 +1257,9 @@ static int
>> validate_compression_type(BDRVQcow2State *s, Error **errp)
>> switch (s->compression_type) {
>> case QCOW2_COMPRESSION_TYPE_ZLIB:
>> +#ifdef CONFIG_ZSTD
>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>> +#endif
>> break;
>> default:
>> @@ -3506,6 +3509,10 @@ qcow2_co_create(BlockdevCreateOptions
>> *create_options, Error **errp)
>> }
>> switch (qcow2_opts->compression_type) {
>> +#ifdef CONFIG_ZSTD
>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>> + break;
>> +#endif
>> default:
>> error_setg_errno(errp, -EINVAL, "Unknown compression
>> type");
>> goto out;
>> diff --git a/configure b/configure
>> index 48d6f89d57..4690a7ea9f 100755
>> --- a/configure
>> +++ b/configure
>> @@ -444,6 +444,7 @@ opengl_dmabuf="no"
>> cpuid_h="no"
>> avx2_opt=""
>> zlib="yes"
>> +zstd=""
>> capstone=""
>> lzo=""
>> snappy=""
>> @@ -1371,6 +1372,10 @@ for opt do
>> ;;
>> --disable-lzfse) lzfse="no"
>> ;;
>> + --enable-zstd) zstd="yes"
>> + ;;
>> + --disable-zstd) zstd="no"
>> + ;;
>> --enable-guest-agent) guest_agent="yes"
>> ;;
>> --disable-guest-agent) guest_agent="no"
>> @@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default is
>> enabled if available:
>> (for reading bzip2-compressed dmg images)
>> lzfse support of lzfse compression library
>> (for reading lzfse-compressed dmg images)
>> + zstd support of zstd compression library
>> seccomp seccomp support
>> coroutine-pool coroutine freelist (better performance)
>> glusterfs GlusterFS backend
>> @@ -2453,6 +2459,25 @@ EOF
>> fi
>> fi
>> +#########################################
>> +# zstd check
>> +
>> +if test "$zstd" != "no" ; then
>> + cat > $TMPC << EOF
>> +#include <zstd.h>
>> +int main(void) { ZSTD_versionNumber(); return 0; }
>> +EOF
>> + if compile_prog "" "-lzstd" ; then
>> + LIBS="$LIBS -lzstd"
>> + zstd="yes"
>> + else
>> + if test "$zstd" = "yes"; then
>> + feature_not_found "zstd" "Install libzstd-devel"
>
> to correspond to style used around: s/libzstd-devel/libzstd devel/
My intention was to give a package name for a user encountered this
message to copy-paste it to "yum install".
If this doesn't make things easier, because the package name can be
changed in the future (although I doubt) I'll fix it.
If the style isn't that important I'd rather leave it as is.
>
>> + fi
>> + zstd="no"
>> + fi
>> +fi
>> +
>> ##########################################
>> # libseccomp check
>> @@ -6668,6 +6693,7 @@ echo "lzo support $lzo"
>> echo "snappy support $snappy"
>> echo "bzip2 support $bzip2"
>> echo "lzfse support $lzfse"
>> +echo "zstd support $zstd"
>> echo "NUMA host support $numa"
>> echo "libxml2 $libxml2"
>> echo "tcmalloc support $tcmalloc"
>> @@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then
>> "\$ld_exported_symbols_list should have been set to 'yes'."
>> fi
>> fi
>> +if test "$zstd" = "yes" ; then
>> + echo "CONFIG_ZSTD=y" >> $config_host_mak
>> +fi
>> if test "$tcg_interpreter" = "yes"; then
>> QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
>> diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
>> index 5597e24474..aeca2ddebd 100644
>> --- a/docs/interop/qcow2.txt
>> +++ b/docs/interop/qcow2.txt
>> @@ -208,6 +208,7 @@ version 2.
>> Available compression type values:
>> 0: zlib <https://www.zlib.net/>
>> + 1: zstd <http://github.com/facebook/zstd>
>> === Header padding ===
>> @@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 -
>> (cluster_bits - 8)):
>> Another compressed cluster may map to the tail
>> of the final
>> sector used by this compressed cluster.
>> + The layout of the compressed data depends on
>> the compression
>> + type used for the image (see compressed cluster
>> layout).
>> +
>> If a cluster is unallocated, read requests shall read the data from
>> the backing
>> file (except if bit 0 in the Standard Cluster Descriptor is set).
>> If there is
>> no backing file or the backing file is smaller than the image, they
>> shall read
>> zeros for all parts that are not covered by the backing file.
>> +=== Compressed Cluster Layout ===
>> +
>> +The compressed cluster data has a layout depending on the compression
>> +type used for the image, as follows:
>> +
>> +Compressed data layout for the available compression types:
>> +(x = data_space_length - 1)
>
> Note, that term 'data_space_lenght' is not defined in the spec..
>
> Hmm. And it's not trivial to define it correctly, all we have is
> offset and
> "number of additional sectors", and may be not all bytes of final
> sector are
> occupied... What about something like this:
>
> In the following layout description byte 0 corresponds to byte at host
> cluster offset,
> as defined by "Compressed Clusters Descriptor" paragraph above. The
> whole layout occupy
> space starting from this offset, using additional 512-byte sectors
> defined by
> "Compressed Clusters Descriptor" paragraph, not necessarily occupying
> all of the bytes
> in the final sector. Let x be number of last byte of the layout.
Sounds too complicated for me, may be
x - data chunk length available to store a compressed cluster. (for more
details see "Compressed Clusters Descriptor")
>
>
>> +
>> + 0: (default) zlib <http://zlib.net/>:
>> + Byte 0 - x: the compressed data content
>> + all the space provided used for
>> compressed data
>> + 1: zstd <http://github.com/facebook/zstd>:
>> + Byte 0 - 3: the length of compressed data in bytes
>> + 4 - x: the compressed data content
>
> Maybe, note that x+! == 4 + length, where length is value of first
> field of the layout.
>
>> == Snapshots ==
>> diff --git a/qapi/block-core.json b/qapi/block-core.json
>> index 873fbef3b5..4b6e576c44 100644
>> --- a/qapi/block-core.json
>> +++ b/qapi/block-core.json
>> @@ -4401,11 +4401,12 @@
>> # Compression type used in qcow2 image file
>> #
>> # @zlib: zlib compression, see <http://zlib.net/>
>> +# @zstd: zstd compression, see <http://github.com/facebook/zstd>
>> #
>> # Since: 5.0
>> ##
>> { 'enum': 'Qcow2CompressionType',
>> - 'data': [ 'zlib' ] }
>> + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' }
>> ] }
>> ##
>> # @BlockdevCreateOptionsQcow2:
>>
>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 1/8] qcow2: introduce compression type feature
2020-02-27 13:59 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-27 14:13 ` Eric Blake
2020-02-27 14:30 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 35+ messages in thread
From: Eric Blake @ 2020-02-27 14:13 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, Denis Plotnikov, qemu-devel
Cc: kwolf, armbru, qemu-block, mreitz
On 2/27/20 7:59 AM, Vladimir Sementsov-Ogievskiy wrote:
>> Hmm - I think it may be worth a tweak to qcow2.txt to call out:
>>
>> 104: compression_type
>> 105 - 111: padding, must be 0
>>
>> or else call out:
>>
>> 104-111: compression type
>>
>> and just blindly use all 8 bytes for the value even though really only
>> 1 or two values will ever be defined. Of course, that moves the byte
>> in question from 104 to 111, thanks to our big endian encoding, but as
>> this series is the first one installing a non-zero value in those 8
>> bytes, and as we just finished documenting that the header length must
>> be a multiple of 8, there is no real impact - we can make such tweaks
>> up until the 5.0 release.
>
> But what is the benefit? We have already documented padding in the spec,
> and discussed it so much time... What is the problem with padding? And
> why to add 8 bytes for every new feature which needs only one byte?
Okay, so requiring an 8-byte field is not necessary. But still, at
least mentioning padding bytes (that may be assigned meanings later) is
consistent with the rest of the document (for example, we have padding
bits documented for the compatible/incompatible/autoclear feature bits),
and reminds implementers to keep size rounded to a multiple of 8.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
2020-02-27 14:11 ` Denis Plotnikov
@ 2020-02-27 14:18 ` Vladimir Sementsov-Ogievskiy
2020-02-28 11:59 ` Denis Plotnikov
0 siblings, 1 reply; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 14:18 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, qemu-block, den, armbru, mreitz
27.02.2020 17:11, Denis Plotnikov wrote:
>
>
> On 27.02.2020 12:55, Vladimir Sementsov-Ogievskiy wrote:
>> 27.02.2020 10:29, Denis Plotnikov wrote:
>>> zstd significantly reduces cluster compression time.
>>> It provides better compression performance maintaining
>>> the same level of the compression ratio in comparison with
>>> zlib, which, at the moment, is the only compression
>>> method available.
>>>
>>> The performance test results:
>>> Test compresses and decompresses qemu qcow2 image with just
>>> installed rhel-7.6 guest.
>>> Image cluster size: 64K. Image on disk size: 2.2G
>>>
>>> The test was conducted with brd disk to reduce the influence
>>> of disk subsystem to the test results.
>>> The results is given in seconds.
>>>
>>> compress cmd:
>>> time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
>>> src.img [zlib|zstd]_compressed.img
>>> decompress cmd
>>> time ./qemu-img convert -O qcow2
>>> [zlib|zstd]_compressed.img uncompressed.img
>>>
>>> compression decompression
>>> zlib zstd zlib zstd
>>> ------------------------------------------------------------
>>> real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
>>> user 65.0 15.8 5.3 2.5
>>> sys 3.3 0.2 2.0 2.0
>>>
>>> Both ZLIB and ZSTD gave the same compression ratio: 1.57
>>> compressed image size in both cases: 1.4G
>>>
>>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>>> ---
>>> block/qcow2-threads.c | 122 +++++++++++++++++++++++++++++++++++++++++
>>> block/qcow2.c | 7 +++
>>> configure | 29 ++++++++++
>>> docs/interop/qcow2.txt | 18 ++++++
>>> qapi/block-core.json | 3 +-
>>> 5 files changed, 178 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
>>> index 1c128e9840..e942c4d7e5 100644
>>> --- a/block/qcow2-threads.c
>>> +++ b/block/qcow2-threads.c
>>> @@ -28,6 +28,11 @@
>>> #define ZLIB_CONST
>>> #include <zlib.h>
>>> +#ifdef CONFIG_ZSTD
>>> +#include <zstd.h>
>>> +#include <zstd_errors.h>
>>> +#endif
>>> +
>>> #include "qcow2.h"
>>> #include "block/thread-pool.h"
>>> #include "crypto.h"
>>> @@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void *dest, size_t dest_size,
>>> return ret;
>>> }
>>> +#ifdef CONFIG_ZSTD
>>> +
>>> +#define ZSTD_LEN_BUF 4
>>> +
>>> +/*
>>> + * qcow2_zstd_compress()
>>> + *
>>> + * Compress @src_size bytes of data using zstd compression method
>>> + *
>>> + * @dest - destination buffer, @dest_size bytes
>>> + * @src - source buffer, @src_size bytes
>>> + *
>>> + * Returns: compressed size on success
>>
>> This doesn't match qcow2_co_compress definition. You should return 0 on success.
> does it? I'd rather say it doesn't match to qcow2_co_compress description in the function comment, which we can change actually,
> because qcow2_co_compress is used like:
Oh, yes, you are right. Then we should change the comment.
>
> block/qcow2.c:
>
> static coroutine_fn int
> qcow2_co_pwritev_compressed_task(BlockDriverState *bs,
> uint64_t offset, uint64_t bytes,
> QEMUIOVector *qiov, size_t qiov_offset)
> {
> ...
> out_buf = g_malloc(s->cluster_size);
>
> out_len = qcow2_co_compress(bs, out_buf, s->cluster_size - 1,
> buf, s->cluster_size);
> if (out_len == -ENOMEM) {
> /* could not compress: write normal cluster */
> ret = qcow2_co_pwritev_part(bs, offset, bytes, qiov, qiov_offset, 0);
> if (ret < 0) {
> goto fail;
> }
> goto success;
> } else if (out_len < 0) {
> ret = -EINVAL;
> goto fail;
> }
>
> qemu_co_mutex_lock(&s->lock);
> ret = qcow2_alloc_compressed_cluster_offset(bs, offset, out_len, <<<<<<<<<<<<
> &cluster_offset);
> ...
> }
>
>>
>>> + * -ENOMEM destination buffer is not enough to store compressed data
>>> + * -EIO on any other error
>>> + */
>>> +
>>> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
>>> + const void *src, size_t src_size)
>>> +{
>>> + size_t ret;
>>> +
>>> + /*
>>> + * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
>>
>> beginning
>>
>>> + * to store compressed chunk size
>>> + */
>>> + char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
>>> +
>>> + /*
>>> + * sanity check that we can store the compressed data length,
>>> + * and there is some space left for the compressor buffer
>>> + */
>>> + if (dest_size <= ZSTD_LEN_BUF) {
>>> + return -ENOMEM;
>>> + }
>>> +
>>> + dest_size -= ZSTD_LEN_BUF;
>>> +
>>> + ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
>>> +
>>> + if (ZSTD_isError(ret)) {
>>> + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
>>> + return -ENOMEM;
>>> + } else {
>>> + return -EIO;
>>> + }
>>> + }
>>> +
>>> + /* paraniod sanity check that we can store the commpressed size */
>>> + if (ret > UINT_MAX) {
>>> + return -ENOMEM;
>>> + }
>>
>> I'd use UINT32_MAX, possibly even more paranoid)
> ok
>>
>>
>>> +
>>> + /* store the compressed chunk size in the very beginning of the buffer */
>>> + stl_be_p(dest, ret);
>>> +
>>> + return ret + ZSTD_LEN_BUF;
>>
>> return 0;
>>
>>> +}
>>> +
>>> +/*
>>> + * qcow2_zstd_decompress()
>>> + *
>>> + * Decompress some data (not more than @src_size bytes) to produce exactly
>>> + * @dest_size bytes using zstd compression method
>>> + *
>>> + * @dest - destination buffer, @dest_size bytes
>>> + * @src - source buffer, @src_size bytes
>>> + *
>>> + * Returns: 0 on success
>>> + * -EIO on any error
>>> + */
>>> +
>>
>> extra empty line.
> will be removed
>>
>>
>>> +static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
>>> + const void *src, size_t src_size)
>>> +{
>>> + /*
>>> + * zstd decompress wants to know the exact length of the data.
>>> + * For that purpose, on compression, the length is stored in
>>> + * the very beginning of the compressed buffer
>>> + */
>>> + size_t s_size;
>>> + const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF;
>>> +
>>> + /*
>>> + * sanity check that we can read 4 byte the content length and
>>> + * and there is some content to decompress
>>> + */
>>> + if (src_size <= ZSTD_LEN_BUF) {
>>> + return -EIO;
>>> + }
>>> +
>>> + s_size = ldl_be_p(src);
>>> +
>>> + /* sanity check that the buffer is big enough to read the content from */
>>> + if (src_size - ZSTD_LEN_BUF < s_size) {
>>> + return -EIO;
>>> + }
>>> +
>>> + if (ZSTD_isError(
>>> + ZSTD_decompress(dest, dest_size, s_buf, s_size))) {
>>
>> hmm, it fit into one line actually
> will be fixed if so
>>
>>
>>> + return -EIO;
>>> + }
>>> +
>>> + return 0;
>>> +}
>>> +#endif
>>> +
>>> static int qcow2_compress_pool_func(void *opaque)
>>> {
>>> Qcow2CompressData *data = opaque;
>>> @@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs, void *dest, size_t dest_size,
>>> fn = qcow2_zlib_compress;
>>> break;
>>> +#ifdef CONFIG_ZSTD
>>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>>> + fn = qcow2_zstd_compress;
>>> + break;
>>> +#endif
>>> default:
>>> return -ENOTSUP;
>>> }
>>> @@ -247,6 +364,11 @@ qcow2_co_decompress(BlockDriverState *bs, void *dest, size_t dest_size,
>>> fn = qcow2_zlib_decompress;
>>> break;
>>> +#ifdef CONFIG_ZSTD
>>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>>> + fn = qcow2_zstd_decompress;
>>> + break;
>>> +#endif
>>> default:
>>> return -ENOTSUP;
>>> }
>>> diff --git a/block/qcow2.c b/block/qcow2.c
>>> index 2ccb2cabd1..9c8ad9d580 100644
>>> --- a/block/qcow2.c
>>> +++ b/block/qcow2.c
>>> @@ -1257,6 +1257,9 @@ static int validate_compression_type(BDRVQcow2State *s, Error **errp)
>>> switch (s->compression_type) {
>>> case QCOW2_COMPRESSION_TYPE_ZLIB:
>>> +#ifdef CONFIG_ZSTD
>>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>>> +#endif
>>> break;
>>> default:
>>> @@ -3506,6 +3509,10 @@ qcow2_co_create(BlockdevCreateOptions *create_options, Error **errp)
>>> }
>>> switch (qcow2_opts->compression_type) {
>>> +#ifdef CONFIG_ZSTD
>>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>>> + break;
>>> +#endif
>>> default:
>>> error_setg_errno(errp, -EINVAL, "Unknown compression type");
>>> goto out;
>>> diff --git a/configure b/configure
>>> index 48d6f89d57..4690a7ea9f 100755
>>> --- a/configure
>>> +++ b/configure
>>> @@ -444,6 +444,7 @@ opengl_dmabuf="no"
>>> cpuid_h="no"
>>> avx2_opt=""
>>> zlib="yes"
>>> +zstd=""
>>> capstone=""
>>> lzo=""
>>> snappy=""
>>> @@ -1371,6 +1372,10 @@ for opt do
>>> ;;
>>> --disable-lzfse) lzfse="no"
>>> ;;
>>> + --enable-zstd) zstd="yes"
>>> + ;;
>>> + --disable-zstd) zstd="no"
>>> + ;;
>>> --enable-guest-agent) guest_agent="yes"
>>> ;;
>>> --disable-guest-agent) guest_agent="no"
>>> @@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default is enabled if available:
>>> (for reading bzip2-compressed dmg images)
>>> lzfse support of lzfse compression library
>>> (for reading lzfse-compressed dmg images)
>>> + zstd support of zstd compression library
>>> seccomp seccomp support
>>> coroutine-pool coroutine freelist (better performance)
>>> glusterfs GlusterFS backend
>>> @@ -2453,6 +2459,25 @@ EOF
>>> fi
>>> fi
>>> +#########################################
>>> +# zstd check
>>> +
>>> +if test "$zstd" != "no" ; then
>>> + cat > $TMPC << EOF
>>> +#include <zstd.h>
>>> +int main(void) { ZSTD_versionNumber(); return 0; }
>>> +EOF
>>> + if compile_prog "" "-lzstd" ; then
>>> + LIBS="$LIBS -lzstd"
>>> + zstd="yes"
>>> + else
>>> + if test "$zstd" = "yes"; then
>>> + feature_not_found "zstd" "Install libzstd-devel"
>>
>> to correspond to style used around: s/libzstd-devel/libzstd devel/
> My intention was to give a package name for a user encountered this message to copy-paste it to "yum install".
What about apt users? :)
> If this doesn't make things easier, because the package name can be changed in the future (although I doubt) I'll fix it.
> If the style isn't that important I'd rather leave it as is.
I don't care, keep it as is.
>>
>>> + fi
>>> + zstd="no"
>>> + fi
>>> +fi
>>> +
>>> ##########################################
>>> # libseccomp check
>>> @@ -6668,6 +6693,7 @@ echo "lzo support $lzo"
>>> echo "snappy support $snappy"
>>> echo "bzip2 support $bzip2"
>>> echo "lzfse support $lzfse"
>>> +echo "zstd support $zstd"
>>> echo "NUMA host support $numa"
>>> echo "libxml2 $libxml2"
>>> echo "tcmalloc support $tcmalloc"
>>> @@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then
>>> "\$ld_exported_symbols_list should have been set to 'yes'."
>>> fi
>>> fi
>>> +if test "$zstd" = "yes" ; then
>>> + echo "CONFIG_ZSTD=y" >> $config_host_mak
>>> +fi
>>> if test "$tcg_interpreter" = "yes"; then
>>> QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
>>> diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
>>> index 5597e24474..aeca2ddebd 100644
>>> --- a/docs/interop/qcow2.txt
>>> +++ b/docs/interop/qcow2.txt
>>> @@ -208,6 +208,7 @@ version 2.
>>> Available compression type values:
>>> 0: zlib <https://www.zlib.net/>
>>> + 1: zstd <http://github.com/facebook/zstd>
>>> === Header padding ===
>>> @@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 - (cluster_bits - 8)):
>>> Another compressed cluster may map to the tail of the final
>>> sector used by this compressed cluster.
>>> + The layout of the compressed data depends on the compression
>>> + type used for the image (see compressed cluster layout).
>>> +
>>> If a cluster is unallocated, read requests shall read the data from the backing
>>> file (except if bit 0 in the Standard Cluster Descriptor is set). If there is
>>> no backing file or the backing file is smaller than the image, they shall read
>>> zeros for all parts that are not covered by the backing file.
>>> +=== Compressed Cluster Layout ===
>>> +
>>> +The compressed cluster data has a layout depending on the compression
>>> +type used for the image, as follows:
>>> +
>>> +Compressed data layout for the available compression types:
>>> +(x = data_space_length - 1)
>>
>> Note, that term 'data_space_lenght' is not defined in the spec..
>>
>> Hmm. And it's not trivial to define it correctly, all we have is offset and
>> "number of additional sectors", and may be not all bytes of final sector are
>> occupied... What about something like this:
>>
>> In the following layout description byte 0 corresponds to byte at host cluster offset,
>> as defined by "Compressed Clusters Descriptor" paragraph above. The whole layout occupy
>> space starting from this offset, using additional 512-byte sectors defined by
>> "Compressed Clusters Descriptor" paragraph, not necessarily occupying all of the bytes
>> in the final sector. Let x be number of last byte of the layout.
> Sounds too complicated for me, may be
> x - data chunk length available to store a compressed cluster. (for more details see "Compressed Clusters Descriptor")
Ok
>>
>>
>>> +
>>> + 0: (default) zlib <http://zlib.net/>:
>>> + Byte 0 - x: the compressed data content
>>> + all the space provided used for compressed data
>>> + 1: zstd <http://github.com/facebook/zstd>:
>>> + Byte 0 - 3: the length of compressed data in bytes
>>> + 4 - x: the compressed data content
>>
>> Maybe, note that x+! == 4 + length, where length is value of first field of the layout.
>>
>>> == Snapshots ==
>>> diff --git a/qapi/block-core.json b/qapi/block-core.json
>>> index 873fbef3b5..4b6e576c44 100644
>>> --- a/qapi/block-core.json
>>> +++ b/qapi/block-core.json
>>> @@ -4401,11 +4401,12 @@
>>> # Compression type used in qcow2 image file
>>> #
>>> # @zlib: zlib compression, see <http://zlib.net/>
>>> +# @zstd: zstd compression, see <http://github.com/facebook/zstd>
>>> #
>>> # Since: 5.0
>>> ##
>>> { 'enum': 'Qcow2CompressionType',
>>> - 'data': [ 'zlib' ] }
>>> + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
>>> ##
>>> # @BlockdevCreateOptionsQcow2:
>>>
>>
>>
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 1/8] qcow2: introduce compression type feature
2020-02-27 14:13 ` Eric Blake
@ 2020-02-27 14:30 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:39 ` Eric Blake
0 siblings, 1 reply; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-27 14:30 UTC (permalink / raw)
To: Eric Blake, Denis Plotnikov, qemu-devel; +Cc: kwolf, armbru, qemu-block, mreitz
27.02.2020 17:13, Eric Blake wrote:
> On 2/27/20 7:59 AM, Vladimir Sementsov-Ogievskiy wrote:
>
>>> Hmm - I think it may be worth a tweak to qcow2.txt to call out:
>>>
>>> 104: compression_type
>>> 105 - 111: padding, must be 0
>>>
>>> or else call out:
>>>
>>> 104-111: compression type
>>>
>>> and just blindly use all 8 bytes for the value even though really only 1 or two values will ever be defined. Of course, that moves the byte in question from 104 to 111, thanks to our big endian encoding, but as this series is the first one installing a non-zero value in those 8 bytes, and as we just finished documenting that the header length must be a multiple of 8, there is no real impact - we can make such tweaks up until the 5.0 release.
>>
>> But what is the benefit? We have already documented padding in the spec, and discussed it so much time... What is the problem with padding? And why to add 8 bytes for every new feature which needs only one byte?
>
> Okay, so requiring an 8-byte field is not necessary. But still, at least mentioning padding bytes (that may be assigned meanings later) is consistent with the rest of the document (for example, we have padding bits documented for the compatible/incompatible/autoclear feature bits), and reminds implementers to keep size rounded to a multiple of 8.
>
Yes, we can add something about it.. But I'm not sure we need, and I can't imaging correct short wording.
We have section about padding:
"
=== Header padding ===
@header_length must be a multiple of 8, which means that if the end of the last
additional field is not aligned, some padding is needed. This padding must be
zeroed, so that if some existing (or future) additional field will fall into
the padding, it will be interpreted accordingly to point [3.] of the previous
paragraph, i.e. in the same manner as when this field is not present.
"
So, if we want to add something about 104-111, it should be added to this section, not to previous "=== Additional fields (version 3 and higher) ===".
And, if we want, to add something, we should consider both cases when compression type field exists and when not... What to write?
"105 - 111: These bytes are padding, if header length > 104. May be turned into new additional fields in future."
Sounds a bit strange... Keeping in mind that different versions of qemu may consider the same bytes as additional field or as padding, and it is correct.
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 1/8] qcow2: introduce compression type feature
2020-02-27 14:30 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-27 14:39 ` Eric Blake
0 siblings, 0 replies; 35+ messages in thread
From: Eric Blake @ 2020-02-27 14:39 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, Denis Plotnikov, qemu-devel
Cc: kwolf, armbru, qemu-block, mreitz
On 2/27/20 8:30 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> But what is the benefit? We have already documented padding in the
>>> spec, and discussed it so much time... What is the problem with
>>> padding? And why to add 8 bytes for every new feature which needs
>>> only one byte?
>>
>> Okay, so requiring an 8-byte field is not necessary. But still, at
>> least mentioning padding bytes (that may be assigned meanings later)
>> is consistent with the rest of the document (for example, we have
>> padding bits documented for the compatible/incompatible/autoclear
>> feature bits), and reminds implementers to keep size rounded to a
>> multiple of 8.
>>
>
> Yes, we can add something about it.. But I'm not sure we need, and I
> can't imaging correct short wording.
>
>
> We have section about padding:
>
> "
> === Header padding ===
>
> @header_length must be a multiple of 8, which means that if the end of
> the last
> additional field is not aligned, some padding is needed. This padding
> must be
> zeroed, so that if some existing (or future) additional field will fall
> into
> the padding, it will be interpreted accordingly to point [3.] of the
> previous
> paragraph, i.e. in the same manner as when this field is not present.
> "
>
>
> So, if we want to add something about 104-111, it should be added to
> this section, not to previous "=== Additional fields (version 3 and
> higher) ===".
>
> And, if we want, to add something, we should consider both cases when
> compression type field exists and when not... What to write?
>
> "105 - 111: These bytes are padding, if header length > 104. May be
> turned into new additional fields in future."
>
> Sounds a bit strange... Keeping in mind that different versions of qemu
> may consider the same bytes as additional field or as padding, and it is
> correct.
Looking at the header extension, it can probably be as simple as:
105 - m: Zero padding to round up the header size to the next
multiple of 8.
I guess I'll propose a patch to make my idea concrete.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 6/8] iotests: add "compression type" for test output matching
2020-02-27 10:09 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:06 ` Eric Blake
@ 2020-02-28 8:13 ` Denis Plotnikov
1 sibling, 0 replies; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-28 8:13 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-devel
Cc: kwolf, qemu-block, den, armbru, mreitz
On 27.02.2020 13:09, Vladimir Sementsov-Ogievskiy wrote:
> 27.02.2020 13:04, Vladimir Sementsov-Ogievskiy wrote:
>> 27.02.2020 10:29, Denis Plotnikov wrote:
>>> Affected tests: 049, 060, 061, 065, 144, 182, 242, 255
>>>
>>> After adding the compression type feature for qcow2, the compression
>>> type
>>> is reported on image quering.
>>>
>>> Add the corresponding values of the "compression type" for the
>>> tests' output
>>> matching.
>>
>> And this and the following patch.
>>
>> Ideally, patch should not break any iotests. This means that all
>> iotest updates
>> should be merged to the patch which changes their output. Probably,
>> you can split
>> behavior-changing patch, to reduce iotest-updates per patch, but
>> anyway, big patch
>> with a lot of iotests updates is better than patch which breaks iotests.
>
> Or we can try to reduce behavior changes in case of zlib:
>
> - keep header small in case of zlib, not adding zero field
> - don't add feature table entry, if compress type is zlib
> - don't report compression type on quering, if it is zlib
>
> - then, all iotests output will be saved. And, then, if we need, we
> can change
> these theree points one-by-one, updating iotests outputs. But I doubt
> that we
> really need it, compatible behavior seems good enough.
I think I would put some efforts in arranging the iotest patches
so they don't break any tests after applying
with the cost of creating a gigantic patch with the compression type
implementation
and iotest fixes
Denis
>
>>
>>>
>>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>>> ---
>>> tests/qemu-iotests/049.out | 102
>>> ++++++++++++++++++-------------------
>>> tests/qemu-iotests/060.out | 1 +
>>> tests/qemu-iotests/061.out | 6 +++
>>> tests/qemu-iotests/065 | 20 +++++---
>>> tests/qemu-iotests/144.out | 4 +-
>>> tests/qemu-iotests/182.out | 2 +-
>>> tests/qemu-iotests/242.out | 5 ++
>>> tests/qemu-iotests/255.out | 8 +--
>>> 8 files changed, 82 insertions(+), 66 deletions(-)
>>>
>>> diff --git a/tests/qemu-iotests/049.out b/tests/qemu-iotests/049.out
>>> index affa55b341..a5cfba1756 100644
>>> --- a/tests/qemu-iotests/049.out
>>> +++ b/tests/qemu-iotests/049.out
>>> @@ -4,90 +4,90 @@ QA output created by 049
>>> == 1. Traditional size parameter ==
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024b
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1k
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1K
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1G
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1T
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1024.0b
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5k
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5K
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5G
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 TEST_DIR/t.qcow2 1.5T
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> == 2. Specifying size via -o ==
>>> qemu-img create -f qcow2 -o size=1024 TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1024b TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1k TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1K TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1M TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1048576
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1G TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1073741824
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1T TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1099511627776
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1024.0 TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1024.0b TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1024
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1.5k TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1.5K TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1536
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1.5M TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1572864
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1.5G TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1610612736
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o size=1.5T TEST_DIR/t.qcow2
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=1649267441664
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> == 3. Invalid sizes ==
>>> @@ -129,84 +129,84 @@ qemu-img: TEST_DIR/t.qcow2: The image size
>>> must be specified only once
>>> == Check correct interpretation of suffixes for cluster size ==
>>> qemu-img create -f qcow2 -o cluster_size=1024 TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o cluster_size=1024b TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o cluster_size=1k TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o cluster_size=1K TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o cluster_size=1M TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1048576 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1048576 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o cluster_size=1024.0 TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o cluster_size=1024.0b TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=1024 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o cluster_size=0.5k TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=512 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=512 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o cluster_size=0.5K TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=512 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=512 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o cluster_size=0.5M TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=524288 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=524288 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> == Check compat level option ==
>>> qemu-img create -f qcow2 -o compat=0.10 TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o compat=1.1 TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o compat=0.42 TEST_DIR/t.qcow2 64M
>>> qemu-img: TEST_DIR/t.qcow2: Invalid parameter '0.42'
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.42
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o compat=foobar TEST_DIR/t.qcow2 64M
>>> qemu-img: TEST_DIR/t.qcow2: Invalid parameter 'foobar'
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> compat=foobar cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> == Check preallocation option ==
>>> qemu-img create -f qcow2 -o preallocation=off TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=65536 preallocation=off lazy_refcounts=off
>>> refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=65536 preallocation=off lazy_refcounts=off
>>> refcount_bits=16 compression_type=zlib
>>> qemu-img create -f qcow2 -o preallocation=metadata
>>> TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=65536 preallocation=metadata lazy_refcounts=off
>>> refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=65536 preallocation=metadata lazy_refcounts=off
>>> refcount_bits=16 compression_type=zlib
>>> qemu-img create -f qcow2 -o preallocation=1234 TEST_DIR/t.qcow2 64M
>>> qemu-img: TEST_DIR/t.qcow2: Invalid parameter '1234'
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=65536 preallocation=1234 lazy_refcounts=off
>>> refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> cluster_size=65536 preallocation=1234 lazy_refcounts=off
>>> refcount_bits=16 compression_type=zlib
>>> == Check encryption option ==
>>> qemu-img create -f qcow2 -o encryption=off TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> encryption=off cluster_size=65536 lazy_refcounts=off
>>> refcount_bits=16 compression_type=zlib
>>> qemu-img create -f qcow2 --object secret,id=sec0,data=123456 -o
>>> encryption=on,encrypt.key-secret=sec0 TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> encryption=on encrypt.key-secret=sec0 cluster_size=65536
>>> lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864
>>> encryption=on encrypt.key-secret=sec0 cluster_size=65536
>>> lazy_refcounts=off refcount_bits=16 compression_type=zlib
>>> == Check lazy_refcounts option (only with v3) ==
>>> qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=off
>>> TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o compat=1.1,lazy_refcounts=on
>>> TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
>>> cluster_size=65536 lazy_refcounts=on refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=1.1
>>> cluster_size=65536 lazy_refcounts=on refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=off
>>> TEST_DIR/t.qcow2 64M
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> qemu-img create -f qcow2 -o compat=0.10,lazy_refcounts=on
>>> TEST_DIR/t.qcow2 64M
>>> qemu-img: TEST_DIR/t.qcow2: Lazy refcounts only supported with
>>> compatibility level 1.1 and above (use version=v3 or greater)
>>> -Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
>>> cluster_size=65536 lazy_refcounts=on refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2', fmt=qcow2 size=67108864 compat=0.10
>>> cluster_size=65536 lazy_refcounts=on refcount_bits=16
>>> compression_type=zlib
>>> *** done
>>> diff --git a/tests/qemu-iotests/060.out b/tests/qemu-iotests/060.out
>>> index d27692a33c..3e47f537e8 100644
>>> --- a/tests/qemu-iotests/060.out
>>> +++ b/tests/qemu-iotests/060.out
>>> @@ -17,6 +17,7 @@ virtual size: 64 MiB (67108864 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> refcount bits: 16
>>> corrupt: true
>>> diff --git a/tests/qemu-iotests/061.out b/tests/qemu-iotests/061.out
>>> index cea7fedfdc..c913f02ad6 100644
>>> --- a/tests/qemu-iotests/061.out
>>> +++ b/tests/qemu-iotests/061.out
>>> @@ -491,6 +491,7 @@ virtual size: 64 MiB (67108864 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> refcount bits: 16
>>> data file: TEST_DIR/t.IMGFMT.data
>>> @@ -511,6 +512,7 @@ virtual size: 64 MiB (67108864 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> refcount bits: 16
>>> data file: foo
>>> @@ -524,6 +526,7 @@ virtual size: 64 MiB (67108864 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> refcount bits: 16
>>> data file raw: false
>>> @@ -538,6 +541,7 @@ virtual size: 64 MiB (67108864 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> refcount bits: 16
>>> data file: TEST_DIR/t.IMGFMT.data
>>> @@ -550,6 +554,7 @@ virtual size: 64 MiB (67108864 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> refcount bits: 16
>>> data file: TEST_DIR/t.IMGFMT.data
>>> @@ -563,6 +568,7 @@ virtual size: 64 MiB (67108864 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> refcount bits: 16
>>> data file: TEST_DIR/t.IMGFMT.data
>>> diff --git a/tests/qemu-iotests/065 b/tests/qemu-iotests/065
>>> index 6426474271..106303b5a5 100755
>>> --- a/tests/qemu-iotests/065
>>> +++ b/tests/qemu-iotests/065
>>> @@ -88,23 +88,25 @@ class TestQMP(TestImageInfoSpecific):
>>> class TestQCow2(TestQemuImgInfo):
>>> '''Testing a qcow2 version 2 image'''
>>> img_options = 'compat=0.10'
>>> - json_compare = { 'compat': '0.10', 'refcount-bits': 16 }
>>> - human_compare = [ 'compat: 0.10', 'refcount bits: 16' ]
>>> + json_compare = { 'compat': '0.10', 'refcount-bits': 16,
>>> 'compression-type': 'zlib' }
>>> + human_compare = [ 'compat: 0.10', 'compression type: zlib',
>>> 'refcount bits: 16' ]
>>> class TestQCow3NotLazy(TestQemuImgInfo):
>>> '''Testing a qcow2 version 3 image with lazy refcounts
>>> disabled'''
>>> img_options = 'compat=1.1,lazy_refcounts=off'
>>> json_compare = { 'compat': '1.1', 'lazy-refcounts': False,
>>> - 'refcount-bits': 16, 'corrupt': False }
>>> - human_compare = [ 'compat: 1.1', 'lazy refcounts: false',
>>> + 'refcount-bits': 16, 'corrupt': False,
>>> + 'compression-type': 'zlib' }
>>> + human_compare = [ 'compat: 1.1', 'compression type: zlib',
>>> 'lazy refcounts: false',
>>> 'refcount bits: 16', 'corrupt: false' ]
>>> class TestQCow3Lazy(TestQemuImgInfo):
>>> '''Testing a qcow2 version 3 image with lazy refcounts enabled'''
>>> img_options = 'compat=1.1,lazy_refcounts=on'
>>> json_compare = { 'compat': '1.1', 'lazy-refcounts': True,
>>> - 'refcount-bits': 16, 'corrupt': False }
>>> - human_compare = [ 'compat: 1.1', 'lazy refcounts: true',
>>> + 'refcount-bits': 16, 'corrupt': False,
>>> + 'compression-type': 'zlib' }
>>> + human_compare = [ 'compat: 1.1', 'compression type: zlib',
>>> 'lazy refcounts: true',
>>> 'refcount bits: 16', 'corrupt: false' ]
>>> class TestQCow3NotLazyQMP(TestQMP):
>>> @@ -113,7 +115,8 @@ class TestQCow3NotLazyQMP(TestQMP):
>>> img_options = 'compat=1.1,lazy_refcounts=off'
>>> qemu_options = 'lazy-refcounts=on'
>>> compare = { 'compat': '1.1', 'lazy-refcounts': False,
>>> - 'refcount-bits': 16, 'corrupt': False }
>>> + 'refcount-bits': 16, 'corrupt': False,
>>> + 'compression-type': 'zlib' }
>>> class TestQCow3LazyQMP(TestQMP):
>>> @@ -122,7 +125,8 @@ class TestQCow3LazyQMP(TestQMP):
>>> img_options = 'compat=1.1,lazy_refcounts=on'
>>> qemu_options = 'lazy-refcounts=off'
>>> compare = { 'compat': '1.1', 'lazy-refcounts': True,
>>> - 'refcount-bits': 16, 'corrupt': False }
>>> + 'refcount-bits': 16, 'corrupt': False,
>>> + 'compression-type': 'zlib' }
>>> TestImageInfoSpecific = None
>>> TestQemuImgInfo = None
>>> diff --git a/tests/qemu-iotests/144.out b/tests/qemu-iotests/144.out
>>> index c7aa2e4820..885a8874a5 100644
>>> --- a/tests/qemu-iotests/144.out
>>> +++ b/tests/qemu-iotests/144.out
>>> @@ -9,7 +9,7 @@ Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT
>>> size=536870912
>>> { 'execute': 'qmp_capabilities' }
>>> {"return": {}}
>>> { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device':
>>> 'virtio0', 'snapshot-file':'TEST_DIR/tmp.IMGFMT', 'format': 'IMGFMT'
>>> } }
>>> -Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912
>>> backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536
>>> lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2 size=536870912
>>> backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536
>>> lazy_refcounts=off refcount_bits=16 compression_type=zlib
>>> {"return": {}}
>>> === Performing block-commit on active layer ===
>>> @@ -31,6 +31,6 @@ Formatting 'TEST_DIR/tmp.qcow2', fmt=qcow2
>>> size=536870912 backing_file=TEST_DIR/
>>> === Performing Live Snapshot 2 ===
>>> { 'execute': 'blockdev-snapshot-sync', 'arguments': { 'device':
>>> 'virtio0', 'snapshot-file':'TEST_DIR/tmp2.IMGFMT', 'format':
>>> 'IMGFMT' } }
>>> -Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912
>>> backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536
>>> lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/tmp2.qcow2', fmt=qcow2 size=536870912
>>> backing_file=TEST_DIR/t.qcow2 backing_fmt=qcow2 cluster_size=65536
>>> lazy_refcounts=off refcount_bits=16 compression_type=zlib
>>> {"return": {}}
>>> *** done
>>> diff --git a/tests/qemu-iotests/182.out b/tests/qemu-iotests/182.out
>>> index a8eea166c3..ae43654d32 100644
>>> --- a/tests/qemu-iotests/182.out
>>> +++ b/tests/qemu-iotests/182.out
>>> @@ -13,7 +13,7 @@ Is another process using the image
>>> [TEST_DIR/t.qcow2]?
>>> {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node0',
>>> 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
>>> {"return": {}}
>>> {'execute': 'blockdev-snapshot-sync', 'arguments': { 'node-name':
>>> 'node0', 'snapshot-file': 'TEST_DIR/t.IMGFMT.overlay',
>>> 'snapshot-node-name': 'node1' } }
>>> -Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120
>>> backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536
>>> lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/t.qcow2.overlay', fmt=qcow2 size=197120
>>> backing_file=TEST_DIR/t.qcow2 backing_fmt=file cluster_size=65536
>>> lazy_refcounts=off refcount_bits=16 compression_type=zlib
>>> {"return": {}}
>>> {'execute': 'blockdev-add', 'arguments': { 'node-name': 'node1',
>>> 'driver': 'file', 'filename': 'TEST_DIR/t.IMGFMT', 'locking': 'on' } }
>>> {"return": {}}
>>> diff --git a/tests/qemu-iotests/242.out b/tests/qemu-iotests/242.out
>>> index 7ac8404d11..091b9126ce 100644
>>> --- a/tests/qemu-iotests/242.out
>>> +++ b/tests/qemu-iotests/242.out
>>> @@ -12,6 +12,7 @@ virtual size: 1 MiB (1048576 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> refcount bits: 16
>>> corrupt: false
>>> @@ -32,6 +33,7 @@ virtual size: 1 MiB (1048576 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> bitmaps:
>>> [0]:
>>> @@ -64,6 +66,7 @@ virtual size: 1 MiB (1048576 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> bitmaps:
>>> [0]:
>>> @@ -104,6 +107,7 @@ virtual size: 1 MiB (1048576 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> bitmaps:
>>> [0]:
>>> @@ -153,6 +157,7 @@ virtual size: 1 MiB (1048576 bytes)
>>> cluster_size: 65536
>>> Format specific information:
>>> compat: 1.1
>>> + compression type: zlib
>>> lazy refcounts: false
>>> bitmaps:
>>> [0]:
>>> diff --git a/tests/qemu-iotests/255.out b/tests/qemu-iotests/255.out
>>> index 348909fdef..a3c99fd62e 100644
>>> --- a/tests/qemu-iotests/255.out
>>> +++ b/tests/qemu-iotests/255.out
>>> @@ -3,9 +3,9 @@ Finishing a commit job with background reads
>>> === Create backing chain and start VM ===
>>> -Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/PID-t.qcow2.mid', fmt=qcow2 size=134217728
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> -Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/PID-t.qcow2', fmt=qcow2 size=134217728
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> === Start background read requests ===
>>> @@ -23,9 +23,9 @@ Closing the VM while a job is being cancelled
>>> === Create images and start VM ===
>>> -Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/PID-src.qcow2', fmt=qcow2 size=134217728
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> -Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> +Formatting 'TEST_DIR/PID-dst.qcow2', fmt=qcow2 size=134217728
>>> cluster_size=65536 lazy_refcounts=off refcount_bits=16
>>> compression_type=zlib
>>> wrote 1048576/1048576 bytes at offset 0
>>> 1 MiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>>
>>
>>
>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 8/8] iotests: 287: add qcow2 compression type test
2020-02-27 10:29 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-28 8:23 ` Denis Plotnikov
2020-02-28 8:32 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-28 8:23 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-devel
Cc: kwolf, qemu-block, den, armbru, mreitz
On 27.02.2020 13:29, Vladimir Sementsov-Ogievskiy wrote:
> 27.02.2020 10:29, Denis Plotnikov wrote:
>> The test checks fulfilling qcow2 requiriements for the compression
>> type feature and zstd compression type operability.
>>
>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>> ---
>> tests/qemu-iotests/287 | 123 +++++++++++++++++++++++++++++++++++++
>> tests/qemu-iotests/287.out | 41 +++++++++++++
>> tests/qemu-iotests/group | 1 +
>> 3 files changed, 165 insertions(+)
>> create mode 100755 tests/qemu-iotests/287
>> create mode 100644 tests/qemu-iotests/287.out
>>
>> diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
>> new file mode 100755
>> index 0000000000..41b916f690
>> --- /dev/null
>> +++ b/tests/qemu-iotests/287
>> @@ -0,0 +1,123 @@
>> +#!/usr/bin/env bash
>> +#
>> +# Test case for an image using zstd compression
>> +#
>> +# Copyright (c) 2020 Virtuozzo International GmbH
>> +#
>> +# This program is free software; you can redistribute it and/or modify
>> +# it under the terms of the GNU General Public License as published by
>> +# the Free Software Foundation; either version 2 of the License, or
>> +# (at your option) any later version.
>> +#
>> +# This program is distributed in the hope that it will be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
>> +#
>> +
>> +# creator
>> +owner=dplotnikov@virtuozzo.com
>> +
>> +seq="$(basename $0)"
>> +echo "QA output created by $seq"
>> +
>> +status=1 # failure is the default!
>> +
>> +_cleanup()
>> +{
>> + _cleanup_test_img
>> +}
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +# standard environment
>> +. ./common.rc
>> +. ./common.filter
>> +
>> +# This tests qocw2-specific low-level functionality
>> +_supported_fmt qcow2
>> +_supported_proto file
>> +_supported_os Linux
>> +
>> +P=`echo "$QEMU_PROG" | sed "s/qemu-system-x86_64//"`
>> +
>> +grep "CONFIG_ZSTD=y" "$P"../config-host.mak >/dev/null
>> +RES=$?
>
> Hmm. This will not work for other architectures and for
> out of tree builds. Also, it checks config but not current
> binary (they may be out of sync, or even unrelated).
>
> Probably better try to create image with zstd compression type
> and handle expected error.
What if the error is "unable to create an image with zstd", although it
has to be?
I think the best way is to ask qemu binary whether it supports zstd, but
it doesn't available by now (should be?)
Is there any other way to make sure that the std compression test has to
be executed?
>
>
>> +if (($RES)); then
>> + _notrun "ZSTD is disabled in the current configuration"
>> +fi
>> +
>> +# Test: when compression is zlib the incompatible is unset
>> +echo
>> +echo "=== Testing compression type incompatible bit setting for zlib
>> ==="
>> +echo
>> +
>> +_make_test_img 64M
>> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
>> +
>> +# Test: when compression differs from zlib the incompatible bit is set
>> +echo
>> +echo "=== Testing compression type incompatible bit setting for zstd
>> ==="
>> +echo
>> +
>> +IMGOPTS='compression_type=zstd' _make_test_img 64M
>> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
>> +
>> +# Test: an image can't be openned if compression type is zlib and
>
> opened
>
>> +# incompatible feature compression type is set
>> +echo
>> +echo "=== Testing zlib with incompatible bit set ==="
>> +echo
>> +
>> +IMGOPTS='compression_type=zlib' _make_test_img 64M
>> +$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
>> +# to make sure the bit was actually set
>> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
>> +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
>> +if (($?==0)); then
>> + echo "Error: The image openned successfully. The image must not
>> be openned"
>> +fi
>
> may be better to instead keep error output and just check it..
I add the explicit message to reduce the investigating time of what
happened and what should it be.
If it isn't that important I'd rather leave it as is.
>
>> +
>> +# Test: an image can't be openned if compression type is NOT zlib and
>> +# incompatible feature compression type is UNSET
>> +echo
>> +echo "=== Testing zstd with incompatible bit unset ==="
>> +echo
>> +
>> +IMGOPTS='compression_type=zstd' _make_test_img 64M
>> +$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
>> +# to make sure the bit was actually unset
>> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
>> +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
>> +if (($?==0)); then
>> + echo "Error: The image openned successfully. The image must not
>> be openned"
>> +fi
>> +# Test: check compression type values
>> +echo
>> +echo "=== Testing compression type values ==="
>> +echo
>> +# zlib=0
>> +IMGOPTS='compression_type=zlib' _make_test_img 64M
>> +od -j104 -N1 -An -vtu1 "$TEST_IMG"
>> +
>> +# zstd=1
>> +IMGOPTS='compression_type=zstd' _make_test_img 64M
>> +od -j104 -N1 -An -vtu1 "$TEST_IMG"
>> +
>> +# Test: using zstd compression, write to and read from an image
>> +echo
>> +echo "=== Testing reading and writing with zstd ==="
>> +echo
>> +
>> +CLUSTER_SIZE=65536
>> +IMGOPTS='compression_type=zstd' _make_test_img 64M
>> +$QEMU_IO -c "write -c 0 64k " "$TEST_IMG" | _filter_qemu_io
>> +$QEMU_IO -c "read -v 0 10 " "$TEST_IMG" | _filter_qemu_io
>> +$QEMU_IO -c "read -v 65530 8" "$TEST_IMG" | _filter_qemu_io
>
> Hmm output depends on default pattern. Better use "write -c -P 0x11 0
> 64k"
> (or any pattern you want), to make it explicit.
Good suggestion, I'll change that
>
>> +
>> +# success, all done
>> +echo "*** done"
>> +rm -f $seq.full
>> +status=0
>> diff --git a/tests/qemu-iotests/287.out b/tests/qemu-iotests/287.out
>> new file mode 100644
>> index 0000000000..4218254ce0
>> --- /dev/null
>> +++ b/tests/qemu-iotests/287.out
>> @@ -0,0 +1,41 @@
>> +QA output created by 287
>> +
>> +=== Testing compression type incompatible bit setting for zlib ===
>> +
>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
>> +incompatible_features []
>> +
>> +=== Testing compression type incompatible bit setting for zstd ===
>> +
>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
>> compression_type=zstd
>> +incompatible_features [3]
>> +
>> +=== Testing zlib with incompatible bit set ===
>> +
>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
>> +incompatible_features [3]
>> +
>> +=== Testing zstd with incompatible bit unset ===
>> +
>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
>> compression_type=zstd
>> +incompatible_features []
>> +
>> +=== Testing compression type values ===
>> +
>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
>> + 0
>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
>> compression_type=zstd
>> + 1
>> +
>> +=== Testing reading and writing with zstd ===
>> +
>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
>> compression_type=zstd
>> +wrote 65536/65536 bytes at offset 0
>> +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +00000000: cd cd cd cd cd cd cd cd cd cd ..........
>> +read 10/10 bytes at offset 0
>> +10 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +0000fffa: cd cd cd cd cd cd 00 00 ........
>> +read 8/8 bytes at offset 65530
>> +8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>> +*** done
>> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
>> index 0317667695..5edbadef40 100644
>> --- a/tests/qemu-iotests/group
>> +++ b/tests/qemu-iotests/group
>> @@ -293,3 +293,4 @@
>> 283 auto quick
>> 284 rw
>> 286 rw quick
>> +287 auto quick
>>
>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 8/8] iotests: 287: add qcow2 compression type test
2020-02-28 8:23 ` Denis Plotnikov
@ 2020-02-28 8:32 ` Vladimir Sementsov-Ogievskiy
0 siblings, 0 replies; 35+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-02-28 8:32 UTC (permalink / raw)
To: Denis Plotnikov, qemu-devel; +Cc: kwolf, qemu-block, den, armbru, mreitz
28.02.2020 11:23, Denis Plotnikov wrote:
>
>
> On 27.02.2020 13:29, Vladimir Sementsov-Ogievskiy wrote:
>> 27.02.2020 10:29, Denis Plotnikov wrote:
>>> The test checks fulfilling qcow2 requiriements for the compression
>>> type feature and zstd compression type operability.
>>>
>>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>>> ---
>>> tests/qemu-iotests/287 | 123 +++++++++++++++++++++++++++++++++++++
>>> tests/qemu-iotests/287.out | 41 +++++++++++++
>>> tests/qemu-iotests/group | 1 +
>>> 3 files changed, 165 insertions(+)
>>> create mode 100755 tests/qemu-iotests/287
>>> create mode 100644 tests/qemu-iotests/287.out
>>>
>>> diff --git a/tests/qemu-iotests/287 b/tests/qemu-iotests/287
>>> new file mode 100755
>>> index 0000000000..41b916f690
>>> --- /dev/null
>>> +++ b/tests/qemu-iotests/287
>>> @@ -0,0 +1,123 @@
>>> +#!/usr/bin/env bash
>>> +#
>>> +# Test case for an image using zstd compression
>>> +#
>>> +# Copyright (c) 2020 Virtuozzo International GmbH
>>> +#
>>> +# This program is free software; you can redistribute it and/or modify
>>> +# it under the terms of the GNU General Public License as published by
>>> +# the Free Software Foundation; either version 2 of the License, or
>>> +# (at your option) any later version.
>>> +#
>>> +# This program is distributed in the hope that it will be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> +# GNU General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU General Public License
>>> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
>>> +#
>>> +
>>> +# creator
>>> +owner=dplotnikov@virtuozzo.com
>>> +
>>> +seq="$(basename $0)"
>>> +echo "QA output created by $seq"
>>> +
>>> +status=1 # failure is the default!
>>> +
>>> +_cleanup()
>>> +{
>>> + _cleanup_test_img
>>> +}
>>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>>> +
>>> +# standard environment
>>> +. ./common.rc
>>> +. ./common.filter
>>> +
>>> +# This tests qocw2-specific low-level functionality
>>> +_supported_fmt qcow2
>>> +_supported_proto file
>>> +_supported_os Linux
>>> +
>>> +P=`echo "$QEMU_PROG" | sed "s/qemu-system-x86_64//"`
>>> +
>>> +grep "CONFIG_ZSTD=y" "$P"../config-host.mak >/dev/null
>>> +RES=$?
>>
>> Hmm. This will not work for other architectures and for
>> out of tree builds. Also, it checks config but not current
>> binary (they may be out of sync, or even unrelated).
>>
>> Probably better try to create image with zstd compression type
>> and handle expected error.
> What if the error is "unable to create an image with zstd", although it has to be?
With such message test should fail. I expect something like "unknown option" error in case of not built in zstd, and I think it's OK to check such message and skip the test.
> I think the best way is to ask qemu binary whether it supports zstd, but it doesn't available by now (should be?)
And in this case the problem is the same: what if it will say that it doesn't support it, although it should support? We can't check such things anyway.
>
> Is there any other way to make sure that the std compression test has to be executed?
>>
>>
>>> +if (($RES)); then
>>> + _notrun "ZSTD is disabled in the current configuration"
>>> +fi
>>> +
>>> +# Test: when compression is zlib the incompatible is unset
>>> +echo
>>> +echo "=== Testing compression type incompatible bit setting for zlib ==="
>>> +echo
>>> +
>>> +_make_test_img 64M
>>> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
>>> +
>>> +# Test: when compression differs from zlib the incompatible bit is set
>>> +echo
>>> +echo "=== Testing compression type incompatible bit setting for zstd ==="
>>> +echo
>>> +
>>> +IMGOPTS='compression_type=zstd' _make_test_img 64M
>>> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
>>> +
>>> +# Test: an image can't be openned if compression type is zlib and
>>
>> opened
>>
>>> +# incompatible feature compression type is set
>>> +echo
>>> +echo "=== Testing zlib with incompatible bit set ==="
>>> +echo
>>> +
>>> +IMGOPTS='compression_type=zlib' _make_test_img 64M
>>> +$PYTHON qcow2.py "$TEST_IMG" set-feature-bit incompatible 3
>>> +# to make sure the bit was actually set
>>> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
>>> +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
>>> +if (($?==0)); then
>>> + echo "Error: The image openned successfully. The image must not be openned"
>>> +fi
>>
>> may be better to instead keep error output and just check it..
> I add the explicit message to reduce the investigating time of what happened and what should it be.
> If it isn't that important I'd rather leave it as is.
of course, up to you
>>
>>> +
>>> +# Test: an image can't be openned if compression type is NOT zlib and
>>> +# incompatible feature compression type is UNSET
>>> +echo
>>> +echo "=== Testing zstd with incompatible bit unset ==="
>>> +echo
>>> +
>>> +IMGOPTS='compression_type=zstd' _make_test_img 64M
>>> +$PYTHON qcow2.py "$TEST_IMG" set-header incompatible_features 0
>>> +# to make sure the bit was actually unset
>>> +$PYTHON qcow2.py "$TEST_IMG" dump-header | grep incompatible_features
>>> +$QEMU_IMG info "$TEST_IMG" 2>1 1>/dev/null
>>> +if (($?==0)); then
>>> + echo "Error: The image openned successfully. The image must not be openned"
>>> +fi
>>> +# Test: check compression type values
>>> +echo
>>> +echo "=== Testing compression type values ==="
>>> +echo
>>> +# zlib=0
>>> +IMGOPTS='compression_type=zlib' _make_test_img 64M
>>> +od -j104 -N1 -An -vtu1 "$TEST_IMG"
>>> +
>>> +# zstd=1
>>> +IMGOPTS='compression_type=zstd' _make_test_img 64M
>>> +od -j104 -N1 -An -vtu1 "$TEST_IMG"
>>> +
>>> +# Test: using zstd compression, write to and read from an image
>>> +echo
>>> +echo "=== Testing reading and writing with zstd ==="
>>> +echo
>>> +
>>> +CLUSTER_SIZE=65536
>>> +IMGOPTS='compression_type=zstd' _make_test_img 64M
>>> +$QEMU_IO -c "write -c 0 64k " "$TEST_IMG" | _filter_qemu_io
>>> +$QEMU_IO -c "read -v 0 10 " "$TEST_IMG" | _filter_qemu_io
>>> +$QEMU_IO -c "read -v 65530 8" "$TEST_IMG" | _filter_qemu_io
>>
>> Hmm output depends on default pattern. Better use "write -c -P 0x11 0 64k"
>> (or any pattern you want), to make it explicit.
> Good suggestion, I'll change that
>>
>>> +
>>> +# success, all done
>>> +echo "*** done"
>>> +rm -f $seq.full
>>> +status=0
>>> diff --git a/tests/qemu-iotests/287.out b/tests/qemu-iotests/287.out
>>> new file mode 100644
>>> index 0000000000..4218254ce0
>>> --- /dev/null
>>> +++ b/tests/qemu-iotests/287.out
>>> @@ -0,0 +1,41 @@
>>> +QA output created by 287
>>> +
>>> +=== Testing compression type incompatible bit setting for zlib ===
>>> +
>>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
>>> +incompatible_features []
>>> +
>>> +=== Testing compression type incompatible bit setting for zstd ===
>>> +
>>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
>>> +incompatible_features [3]
>>> +
>>> +=== Testing zlib with incompatible bit set ===
>>> +
>>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
>>> +incompatible_features [3]
>>> +
>>> +=== Testing zstd with incompatible bit unset ===
>>> +
>>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
>>> +incompatible_features []
>>> +
>>> +=== Testing compression type values ===
>>> +
>>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864
>>> + 0
>>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
>>> + 1
>>> +
>>> +=== Testing reading and writing with zstd ===
>>> +
>>> +Formatting 'TEST_DIR/t.IMGFMT', fmt=IMGFMT size=67108864 compression_type=zstd
>>> +wrote 65536/65536 bytes at offset 0
>>> +64 KiB, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>> +00000000: cd cd cd cd cd cd cd cd cd cd ..........
>>> +read 10/10 bytes at offset 0
>>> +10 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>> +0000fffa: cd cd cd cd cd cd 00 00 ........
>>> +read 8/8 bytes at offset 65530
>>> +8 bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
>>> +*** done
>>> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
>>> index 0317667695..5edbadef40 100644
>>> --- a/tests/qemu-iotests/group
>>> +++ b/tests/qemu-iotests/group
>>> @@ -293,3 +293,4 @@
>>> 283 auto quick
>>> 284 rw
>>> 286 rw quick
>>> +287 auto quick
>>>
>>
>>
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 1/8] qcow2: introduce compression type feature
2020-02-27 13:48 ` Eric Blake
2020-02-27 13:59 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-28 8:34 ` Denis Plotnikov
1 sibling, 0 replies; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-28 8:34 UTC (permalink / raw)
To: Eric Blake, qemu-devel; +Cc: kwolf, vsementsov, armbru, qemu-block, mreitz
On 27.02.2020 16:48, Eric Blake wrote:
> On 2/27/20 1:29 AM, Denis Plotnikov wrote:
>> The patch adds some preparation parts for incompatible compression type
>> feature to Qcow2 that indicates which allow to use different compression
>
> to qcow2, allowing the use of different
>
>> methods for image clusters (de)compressing.
>>
>> It is implied that the compression type is set on the image creation and
>> can be changed only later by image conversion, thus compression type
>> defines the only compression algorithm used for the image, and thus,
>> for all image clusters.
>>
>> The goal of the feature is to add support of other compression methods
>> to qcow2. For example, ZSTD which is more effective on compression
>> than ZLIB.
>>
>> The default compression is ZLIB. Images created with ZLIB compression
>> type
>> are backward compatible with older qemu versions.
>>
>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>> ---
>> block/qcow2.c | 105 ++++++++++++++++++++++++++++++++++++++
>> block/qcow2.h | 31 ++++++++---
>> include/block/block_int.h | 1 +
>> qapi/block-core.json | 22 +++++++-
>> 4 files changed, 150 insertions(+), 9 deletions(-)
>>
>> diff --git a/block/qcow2.c b/block/qcow2.c
>> index 3c754f616b..2ccb2cabd1 100644
>> --- a/block/qcow2.c
>> +++ b/block/qcow2.c
>> @@ -1242,6 +1242,50 @@ static int
>> qcow2_update_options(BlockDriverState *bs, QDict *options,
>> return ret;
>> }
>> +static int validate_compression_type(BDRVQcow2State *s, Error **errp)
>> +{
>> + /*
>> + * Sanity check
>> + * according to qcow2 spec, the compression type is 1-byte field
>> + * but in BDRVQcow2State the compression_type is enum sizeof(int)
>> + * so, the max compression_type value is 255.
>> + */
>> + if (s->compression_type > 0xff) {
>> + error_setg(errp, "qcow2: compression type value is too big");
>> + return -EINVAL;
>> + }
>
> Hmm - I think it may be worth a tweak to qcow2.txt to call out:
>
> 104: compression_type
> 105 - 111: padding, must be 0
>
> or else call out:
>
> 104-111: compression type
>
> and just blindly use all 8 bytes for the value even though really only
> 1 or two values will ever be defined. Of course, that moves the byte
> in question from 104 to 111, thanks to our big endian encoding, but as
> this series is the first one installing a non-zero value in those 8
> bytes, and as we just finished documenting that the header length must
> be a multiple of 8, there is no real impact - we can make such tweaks
> up until the 5.0 release.
>
>> +
>> + switch (s->compression_type) {
>> + case QCOW2_COMPRESSION_TYPE_ZLIB:
>> + break;
>> +
>> + default:
>> + error_setg(errp, "qcow2: unknown compression type: %u",
>> + s->compression_type);
>> + return -ENOTSUP;
>> + }
>
> Having two checks feels redundant, compared to just letting the
> default catch all unrecognized values in that field.
Looks like it is.
>
>> +
>> + /*
>> + * if the compression type differs from QCOW2_COMPRESSION_TYPE_ZLIB
>> + * the incompatible feature flag must be set
>> + */
>> + if (s->compression_type == QCOW2_COMPRESSION_TYPE_ZLIB) {
>> + if (s->incompatible_features &
>> QCOW2_INCOMPAT_COMPRESSION_TYPE) {
>> + error_setg(errp, "qcow2: Compression type incompatible
>> feature "
>> + "bit must not be set");
>> + return -EINVAL;
>> + }
>> + } else {
>> + if (!(s->incompatible_features &
>> QCOW2_INCOMPAT_COMPRESSION_TYPE)) {
>> + error_setg(errp, "qcow2: Compression type incompatible
>> feature "
>> + "bit must be set");
>> + return -EINVAL;
>> + }
>> + }
>
> Matches what we documented in the spec.
>
>> +
>> + return 0;
>> +}
>> +
>> /* Called with s->lock held. */
>> static int coroutine_fn qcow2_do_open(BlockDriverState *bs, QDict
>> *options,
>> int flags, Error **errp)
>> @@ -1357,6 +1401,26 @@ static int coroutine_fn
>> qcow2_do_open(BlockDriverState *bs, QDict *options,
>> s->compatible_features = header.compatible_features;
>> s->autoclear_features = header.autoclear_features;
>> + /*
>> + * Handle compression type
>> + * Older qcow2 images don't contain the compression type header.
>> + * Distinguish them by the header length and use
>> + * the only valid (default) compression type in that case
>> + */
>> + if (header.header_length > offsetof(QCowHeader,
>> compression_type)) {
>> + /*
>> + * don't deal with endians since compression_type is 1 byte
>> long
>> + */
>> + s->compression_type = header.compression_type;
>
> Changes if you go with my suggestion of just making the
> compression_type field occupy 8 bytes in the qcow2 header. (And if
> you want to keep it 1 byte, I still think the spec should call out
> explicit padding bytes).
>
>> + } else {
>> + s->compression_type = QCOW2_COMPRESSION_TYPE_ZLIB;
>> + }
>> +
>> + ret = validate_compression_type(s, errp);
>> + if (ret) {
>> + goto fail;
>> + }
>> +
>> if (s->incompatible_features & ~QCOW2_INCOMPAT_MASK) {
>> void *feature_table = NULL;
>> qcow2_read_extensions(bs, header.header_length, ext_end,
>> @@ -2720,6 +2784,12 @@ int qcow2_update_header(BlockDriverState *bs)
>> total_size = bs->total_sectors * BDRV_SECTOR_SIZE;
>> refcount_table_clusters = s->refcount_table_size >>
>> (s->cluster_bits - 3);
>> + ret = validate_compression_type(s, NULL);
>> +
>> + if (ret) {
>> + goto fail;
>> + }
>> +
>> *header = (QCowHeader) {
>> /* Version 2 fields */
>> .magic = cpu_to_be32(QCOW_MAGIC),
>> @@ -2742,6 +2812,7 @@ int qcow2_update_header(BlockDriverState *bs)
>> .autoclear_features = cpu_to_be64(s->autoclear_features),
>> .refcount_order = cpu_to_be32(s->refcount_order),
>> .header_length = cpu_to_be32(header_length),
>> + .compression_type = (uint8_t) s->compression_type,
>
> Is the cast necessary?
s->compression_type is enum, I thought it would be good to explicitly
show that in the code.
If it's totally useless I'll remove it.
>
>> };
>> /* For older versions, write a shorter header */
>> @@ -2839,6 +2910,11 @@ int qcow2_update_header(BlockDriverState *bs)
>> .bit = QCOW2_COMPAT_LAZY_REFCOUNTS_BITNR,
>> .name = "lazy refcounts",
>> },
>> + {
>> + .type = QCOW2_FEAT_TYPE_INCOMPATIBLE,
>
> Ordering: please group all the incompatible bits side-by-side (this
> should come before the lazy refcount bit).
>
>> + .bit = QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
>> + .name = "compression type",
>
> This change breaks iotests; at least 31, 36, and 61 need updates (I've
> got a similar patch pending which fixes the fact that we forgot the
> autoclear bit [1]). You'll need to squash in fixes for those at the
> same time.
ok
>
> [1] https://lists.gnu.org/archive/html/qemu-devel/2020-01/msg08069.html
>
>> + },
>> };
>> ret = header_ext_add(buf, QCOW2_EXT_MAGIC_FEATURE_TABLE,
>> @@ -3401,6 +3477,7 @@ qcow2_co_create(BlockdevCreateOptions
>> *create_options, Error **errp)
>> .refcount_table_offset = cpu_to_be64(cluster_size),
>> .refcount_table_clusters = cpu_to_be32(1),
>> .refcount_order = cpu_to_be32(refcount_order),
>> + .compression_type = (uint8_t)
>> QCOW2_COMPRESSION_TYPE_ZLIB,
>
> Is the cast necessary?
>
>> .header_length = cpu_to_be32(sizeof(*header)),
>> };
>> @@ -3420,6 +3497,26 @@ qcow2_co_create(BlockdevCreateOptions
>> *create_options, Error **errp)
>> cpu_to_be64(QCOW2_AUTOCLEAR_DATA_FILE_RAW);
>> }
>> + if (qcow2_opts->has_compression_type &&
>> + qcow2_opts->compression_type != QCOW2_COMPRESSION_TYPE_ZLIB) {
>> +
>> + if (qcow2_opts->compression_type > 0xff) {
>> + error_setg_errno(errp, -EINVAL, "Too big compression
>> type value");
>> + goto out;
>> + }
>> +
>> + switch (qcow2_opts->compression_type) {
>> + default:
>> + error_setg_errno(errp, -EINVAL, "Unknown compression
>> type");
>> + goto out;
>> + }
>
> This should probably be an assert that qcow2_opts->compression_type is
> in range, rather than a switch statement and error_setg. Callers of
> qcow2_co_create should not be handing us unknown values.
The intention was to express what has actually happened in the case of a
"bad" image to easy the problem investigation.
>
>> +
>> + header->compression_type = (uint8_t)
>> qcow2_opts->compression_type;
>
> Why the cast?
>
>> +
>> + header->incompatible_features |=
>> + cpu_to_be64(QCOW2_INCOMPAT_COMPRESSION_TYPE);
>> + }
>> +
>> ret = blk_pwrite(blk, 0, header, cluster_size, 0);
>> g_free(header);
>> if (ret < 0) {
>> @@ -3602,6 +3699,7 @@ static int coroutine_fn
>> qcow2_co_create_opts(const char *filename, QemuOpts *opt
>
>> +++ b/block/qcow2.h
>> @@ -146,6 +146,12 @@ typedef struct QCowHeader {
>> uint32_t refcount_order;
>> uint32_t header_length;
>> +
>> + /* Additional fields */
>> + uint8_t compression_type;
>> +
>> + /* header must be a multiple of 8 */
>> + uint8_t padding[7];
>> } QEMU_PACKED QCowHeader;
>
> You're changing the size of this struct, which WILL break iotests (and
> even more than just the 3 I pointed out above for the feature name
> table).
>
> /me looks ahead
>
> Aha - you even noticed it: patch 7/8 fixes test 80. That fix needs to
> be squashed in here, where the change is made.
ok, but the patch will be pretty long
>
>> typedef struct QEMU_PACKED QCowSnapshotHeader {
>> @@ -213,16 +219,20 @@ enum {
>> /* Incompatible feature bits */
>> enum {
>> - QCOW2_INCOMPAT_DIRTY_BITNR = 0,
>> - QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
>> - QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
>> - QCOW2_INCOMPAT_DIRTY = 1 << QCOW2_INCOMPAT_DIRTY_BITNR,
>> - QCOW2_INCOMPAT_CORRUPT = 1 <<
>> QCOW2_INCOMPAT_CORRUPT_BITNR,
>> - QCOW2_INCOMPAT_DATA_FILE = 1 <<
>> QCOW2_INCOMPAT_DATA_FILE_BITNR,
>> + QCOW2_INCOMPAT_DIRTY_BITNR = 0,
>> + QCOW2_INCOMPAT_CORRUPT_BITNR = 1,
>> + QCOW2_INCOMPAT_DATA_FILE_BITNR = 2,
>> + QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR = 3,
>> + QCOW2_INCOMPAT_DIRTY = 1 <<
>> QCOW2_INCOMPAT_DIRTY_BITNR,
>> + QCOW2_INCOMPAT_CORRUPT = 1 <<
>> QCOW2_INCOMPAT_CORRUPT_BITNR,
>> + QCOW2_INCOMPAT_DATA_FILE = 1 <<
>> QCOW2_INCOMPAT_DATA_FILE_BITNR,
>> + QCOW2_INCOMPAT_COMPRESSION_TYPE =
>> + 1 << QCOW2_INCOMPAT_COMPRESSION_TYPE_BITNR,
>
> Uggh. I hate realigning = just because we added a longer name,
> especially when you then can't even keep things on one line because of
> length. If it were me, I'd leave the existing lines alone, and/or
> switch everything to just use 'BITNR = ' rather than trying to align =.
>
> Bikeshedding - since the new name is so long, can you get by with the
> shorter QCOW2_INCOMPAT_COMPRESSION_BITNR (drop the _TYPE)?
Good idea, I'll certainly redo that part, because I don't like ether.
Denis
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
2020-02-27 14:01 ` Eric Blake
@ 2020-02-28 11:40 ` Denis Plotnikov
0 siblings, 0 replies; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-28 11:40 UTC (permalink / raw)
To: Eric Blake, qemu-devel; +Cc: kwolf, vsementsov, armbru, qemu-block, mreitz
On 27.02.2020 17:01, Eric Blake wrote:
> On 2/27/20 1:29 AM, Denis Plotnikov wrote:
>> zstd significantly reduces cluster compression time.
>> It provides better compression performance maintaining
>> the same level of the compression ratio in comparison with
>> zlib, which, at the moment, is the only compression
>> method available.
>>
>
>> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
>> + const void *src, size_t src_size)
>> +{
>> + size_t ret;
>> +
>> + /*
>> + * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
>
> beginning
>
>> + * to store compressed chunk size
>> + */
>> + char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
>> +
>> + /*
>> + * sanity check that we can store the compressed data length,
>> + * and there is some space left for the compressor buffer
>> + */
>> + if (dest_size <= ZSTD_LEN_BUF) {
>> + return -ENOMEM;
>> + }
>> +
>> + dest_size -= ZSTD_LEN_BUF;
>> +
>> + ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
>> +
>> + if (ZSTD_isError(ret)) {
>> + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
>> + return -ENOMEM;
>> + } else {
>> + return -EIO;
>> + }
>> + }
>> +
>> + /* paraniod sanity check that we can store the commpressed size */
>
> paranoid, compressed
>
>> + if (ret > UINT_MAX) {
>> + return -ENOMEM;
>> + }
>
> This is pointless. Better is to ensure that we actually compressed
> data (the pigeonhole principle states that there are some inputs that
> MUST result in inflation, in order for most other inputs to result in
> compression). But that check was satisfied by checking for
> ZSTD_error_dstSize_tooSmall, which is what happens for one of those
> uncompressible inputs. Namely, zstd will never return a result larger
> than dest_size, and since dest_size is smaller than UINT_MAX on entry,
> this check is pointless. But if you want something, I'd be okay with:
> assert(ret <= dest_size).
Taking into account that this is "just in case" and I'm trying to
protect the first 4 bytes of the buffer from the overflow and
I can't imagine the situation when we deal with cluster sizes greater
than UINT32_MAX but the input size is size_t which can be > UINT32_MAX
on 64bit archs.
I'd rather stick with
if (ret > UINT32_MAX) {
return -ENOMEM;
}
as Vladimir suggested.
I'm not sure that the assert is good here, since it stops the system
operating and this isn't potential error but a limitation
Does it work for you?
Denis
>
>> +++ b/docs/interop/qcow2.txt
>> @@ -208,6 +208,7 @@ version 2.
>> Available compression type values:
>> 0: zlib <https://www.zlib.net/>
>> + 1: zstd <http://github.com/facebook/zstd>
>> === Header padding ===
>> @@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 -
>> (cluster_bits - 8)):
>> Another compressed cluster may map to the tail
>> of the final
>> sector used by this compressed cluster.
>> + The layout of the compressed data depends on
>> the compression
>> + type used for the image (see compressed cluster
>> layout).
>> +
>> If a cluster is unallocated, read requests shall read the data from
>> the backing
>> file (except if bit 0 in the Standard Cluster Descriptor is set).
>> If there is
>> no backing file or the backing file is smaller than the image, they
>> shall read
>> zeros for all parts that are not covered by the backing file.
>> +=== Compressed Cluster Layout ===
>> +
>> +The compressed cluster data has a layout depending on the compression
>> +type used for the image, as follows:
>> +
>> +Compressed data layout for the available compression types:
>> +(x = data_space_length - 1)
>> +
>> + 0: (default) zlib <http://zlib.net/>:
>> + Byte 0 - x: the compressed data content
>> + all the space provided used for
>> compressed data
>> + 1: zstd <http://github.com/facebook/zstd>:
>> + Byte 0 - 3: the length of compressed data in bytes
>> + 4 - x: the compressed data content
>> == Snapshots ==
>> diff --git a/qapi/block-core.json b/qapi/block-core.json
>> index 873fbef3b5..4b6e576c44 100644
>> --- a/qapi/block-core.json
>> +++ b/qapi/block-core.json
>> @@ -4401,11 +4401,12 @@
>> # Compression type used in qcow2 image file
>> #
>> # @zlib: zlib compression, see <http://zlib.net/>
>> +# @zstd: zstd compression, see <http://github.com/facebook/zstd>
>> #
>> # Since: 5.0
>> ##
>> { 'enum': 'Qcow2CompressionType',
>> - 'data': [ 'zlib' ] }
>> + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' }
>> ] }
>
> The spec and UI changes are okay.
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 4/8] iotests: filter out compression_type
2020-02-27 14:03 ` Eric Blake
@ 2020-02-28 11:53 ` Denis Plotnikov
0 siblings, 0 replies; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-28 11:53 UTC (permalink / raw)
To: Eric Blake, qemu-devel; +Cc: kwolf, vsementsov, armbru, qemu-block, mreitz
On 27.02.2020 17:03, Eric Blake wrote:
> On 2/27/20 1:29 AM, Denis Plotnikov wrote:
>> After adding compression type feature to qcow2 format, qemu framework
>> commands reporting the image settingd, e.g. "qemu-img create", started
>
> settings
>
>> reporting the compression type for the image which breaks the iotests
>> output matching.
>>
>> To fix it, add compression_type=zlib to the list of filtered image
>> parameters.
>>
>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>> ---
>> tests/qemu-iotests/common.filter | 3 ++-
>> 1 file changed, 2 insertions(+), 1 deletion(-)
>
> This should be squashed in to the patch that caused the breakage (3/8,
> if I'm right).
>
>>
>> diff --git a/tests/qemu-iotests/common.filter
>> b/tests/qemu-iotests/common.filter
>> index 3f8ee3e5f7..c6962d199c 100644
>> --- a/tests/qemu-iotests/common.filter
>> +++ b/tests/qemu-iotests/common.filter
>> @@ -152,7 +152,8 @@ _filter_img_create()
>> -e "s# refcount_bits=[0-9]\\+##g" \
>> -e "s# key-secret=[a-zA-Z0-9]\\+##g" \
>> -e "s# iter-time=[0-9]\\+##g" \
>> - -e "s# force_size=\\(on\\|off\\)##g"
>> + -e "s# force_size=\\(on\\|off\\)##g" \
>> + -e "s# compression_type=zlib##g"
>
> Do you really want to hard-code just zlib, or should this be more
> generic as compression_type=[a-zA-Z0-9]\\+ as done on other lines like
> key-secret?
When I did this I meant additional implicit check that the default
compression type is zlib. But non of the other items in the filter don't
do it. So I'll change it to be consistent. Thanks!
Denis
>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
* Re: [PATCH v1 3/8] qcow2: add zstd cluster compression
2020-02-27 14:18 ` Vladimir Sementsov-Ogievskiy
@ 2020-02-28 11:59 ` Denis Plotnikov
0 siblings, 0 replies; 35+ messages in thread
From: Denis Plotnikov @ 2020-02-28 11:59 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-devel
Cc: kwolf, qemu-block, den, armbru, mreitz
On 27.02.2020 17:18, Vladimir Sementsov-Ogievskiy wrote:
> 27.02.2020 17:11, Denis Plotnikov wrote:
>>
>>
>> On 27.02.2020 12:55, Vladimir Sementsov-Ogievskiy wrote:
>>> 27.02.2020 10:29, Denis Plotnikov wrote:
>>>> zstd significantly reduces cluster compression time.
>>>> It provides better compression performance maintaining
>>>> the same level of the compression ratio in comparison with
>>>> zlib, which, at the moment, is the only compression
>>>> method available.
>>>>
>>>> The performance test results:
>>>> Test compresses and decompresses qemu qcow2 image with just
>>>> installed rhel-7.6 guest.
>>>> Image cluster size: 64K. Image on disk size: 2.2G
>>>>
>>>> The test was conducted with brd disk to reduce the influence
>>>> of disk subsystem to the test results.
>>>> The results is given in seconds.
>>>>
>>>> compress cmd:
>>>> time ./qemu-img convert -O qcow2 -c -o compression_type=[zlib|zstd]
>>>> src.img [zlib|zstd]_compressed.img
>>>> decompress cmd
>>>> time ./qemu-img convert -O qcow2
>>>> [zlib|zstd]_compressed.img uncompressed.img
>>>>
>>>> compression decompression
>>>> zlib zstd zlib zstd
>>>> ------------------------------------------------------------
>>>> real 65.5 16.3 (-75 %) 1.9 1.6 (-16 %)
>>>> user 65.0 15.8 5.3 2.5
>>>> sys 3.3 0.2 2.0 2.0
>>>>
>>>> Both ZLIB and ZSTD gave the same compression ratio: 1.57
>>>> compressed image size in both cases: 1.4G
>>>>
>>>> Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
>>>> ---
>>>> block/qcow2-threads.c | 122
>>>> +++++++++++++++++++++++++++++++++++++++++
>>>> block/qcow2.c | 7 +++
>>>> configure | 29 ++++++++++
>>>> docs/interop/qcow2.txt | 18 ++++++
>>>> qapi/block-core.json | 3 +-
>>>> 5 files changed, 178 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/block/qcow2-threads.c b/block/qcow2-threads.c
>>>> index 1c128e9840..e942c4d7e5 100644
>>>> --- a/block/qcow2-threads.c
>>>> +++ b/block/qcow2-threads.c
>>>> @@ -28,6 +28,11 @@
>>>> #define ZLIB_CONST
>>>> #include <zlib.h>
>>>> +#ifdef CONFIG_ZSTD
>>>> +#include <zstd.h>
>>>> +#include <zstd_errors.h>
>>>> +#endif
>>>> +
>>>> #include "qcow2.h"
>>>> #include "block/thread-pool.h"
>>>> #include "crypto.h"
>>>> @@ -164,6 +169,113 @@ static ssize_t qcow2_zlib_decompress(void
>>>> *dest, size_t dest_size,
>>>> return ret;
>>>> }
>>>> +#ifdef CONFIG_ZSTD
>>>> +
>>>> +#define ZSTD_LEN_BUF 4
>>>> +
>>>> +/*
>>>> + * qcow2_zstd_compress()
>>>> + *
>>>> + * Compress @src_size bytes of data using zstd compression method
>>>> + *
>>>> + * @dest - destination buffer, @dest_size bytes
>>>> + * @src - source buffer, @src_size bytes
>>>> + *
>>>> + * Returns: compressed size on success
>>>
>>> This doesn't match qcow2_co_compress definition. You should return 0
>>> on success.
>> does it? I'd rather say it doesn't match to qcow2_co_compress
>> description in the function comment, which we can change actually,
>> because qcow2_co_compress is used like:
>
> Oh, yes, you are right. Then we should change the comment.
>
>>
>> block/qcow2.c:
>>
>> static coroutine_fn int
>> qcow2_co_pwritev_compressed_task(BlockDriverState *bs,
>> uint64_t offset, uint64_t bytes,
>> QEMUIOVector *qiov, size_t
>> qiov_offset)
>> {
>> ...
>> out_buf = g_malloc(s->cluster_size);
>>
>> out_len = qcow2_co_compress(bs, out_buf, s->cluster_size - 1,
>> buf, s->cluster_size);
>> if (out_len == -ENOMEM) {
>> /* could not compress: write normal cluster */
>> ret = qcow2_co_pwritev_part(bs, offset, bytes, qiov,
>> qiov_offset, 0);
>> if (ret < 0) {
>> goto fail;
>> }
>> goto success;
>> } else if (out_len < 0) {
>> ret = -EINVAL;
>> goto fail;
>> }
>>
>> qemu_co_mutex_lock(&s->lock);
>> ret = qcow2_alloc_compressed_cluster_offset(bs, offset, out_len,
>> <<<<<<<<<<<<
>> &cluster_offset);
>> ...
>> }
>>
>>>
>>>> + * -ENOMEM destination buffer is not enough to store
>>>> compressed data
>>>> + * -EIO on any other error
>>>> + */
>>>> +
>>>> +static ssize_t qcow2_zstd_compress(void *dest, size_t dest_size,
>>>> + const void *src, size_t src_size)
>>>> +{
>>>> + size_t ret;
>>>> +
>>>> + /*
>>>> + * steal ZSTD_LEN_BUF bytes in the very beginng of the buffer
>>>
>>> beginning
>>>
>>>> + * to store compressed chunk size
>>>> + */
>>>> + char *d_buf = ((char *) dest) + ZSTD_LEN_BUF;
>>>> +
>>>> + /*
>>>> + * sanity check that we can store the compressed data length,
>>>> + * and there is some space left for the compressor buffer
>>>> + */
>>>> + if (dest_size <= ZSTD_LEN_BUF) {
>>>> + return -ENOMEM;
>>>> + }
>>>> +
>>>> + dest_size -= ZSTD_LEN_BUF;
>>>> +
>>>> + ret = ZSTD_compress(d_buf, dest_size, src, src_size, 5);
>>>> +
>>>> + if (ZSTD_isError(ret)) {
>>>> + if (ZSTD_getErrorCode(ret) == ZSTD_error_dstSize_tooSmall) {
>>>> + return -ENOMEM;
>>>> + } else {
>>>> + return -EIO;
>>>> + }
>>>> + }
>>>> +
>>>> + /* paraniod sanity check that we can store the commpressed
>>>> size */
>>>> + if (ret > UINT_MAX) {
>>>> + return -ENOMEM;
>>>> + }
>>>
>>> I'd use UINT32_MAX, possibly even more paranoid)
>> ok
>>>
>>>
>>>> +
>>>> + /* store the compressed chunk size in the very beginning of
>>>> the buffer */
>>>> + stl_be_p(dest, ret);
>>>> +
>>>> + return ret + ZSTD_LEN_BUF;
>>>
>>> return 0;
>>>
>>>> +}
>>>> +
>>>> +/*
>>>> + * qcow2_zstd_decompress()
>>>> + *
>>>> + * Decompress some data (not more than @src_size bytes) to produce
>>>> exactly
>>>> + * @dest_size bytes using zstd compression method
>>>> + *
>>>> + * @dest - destination buffer, @dest_size bytes
>>>> + * @src - source buffer, @src_size bytes
>>>> + *
>>>> + * Returns: 0 on success
>>>> + * -EIO on any error
>>>> + */
>>>> +
>>>
>>> extra empty line.
>> will be removed
>>>
>>>
>>>> +static ssize_t qcow2_zstd_decompress(void *dest, size_t dest_size,
>>>> + const void *src, size_t
>>>> src_size)
>>>> +{
>>>> + /*
>>>> + * zstd decompress wants to know the exact length of the data.
>>>> + * For that purpose, on compression, the length is stored in
>>>> + * the very beginning of the compressed buffer
>>>> + */
>>>> + size_t s_size;
>>>> + const char *s_buf = ((const char *) src) + ZSTD_LEN_BUF;
>>>> +
>>>> + /*
>>>> + * sanity check that we can read 4 byte the content length and
>>>> + * and there is some content to decompress
>>>> + */
>>>> + if (src_size <= ZSTD_LEN_BUF) {
>>>> + return -EIO;
>>>> + }
>>>> +
>>>> + s_size = ldl_be_p(src);
>>>> +
>>>> + /* sanity check that the buffer is big enough to read the
>>>> content from */
>>>> + if (src_size - ZSTD_LEN_BUF < s_size) {
>>>> + return -EIO;
>>>> + }
>>>> +
>>>> + if (ZSTD_isError(
>>>> + ZSTD_decompress(dest, dest_size, s_buf, s_size))) {
>>>
>>> hmm, it fit into one line actually
>> will be fixed if so
If there is no other objections I'd leave it as is since
if (ZSTD_isError(ZSTD_decompress(dest, dest_size, s_buf, s_size))) {
looks too messy to me although it fits into 80 chars
>>>
>>>
>>>> + return -EIO;
>>>> + }
>>>> +
>>>> + return 0;
>>>> +}
>>>> +#endif
>>>> +
>>>> static int qcow2_compress_pool_func(void *opaque)
>>>> {
>>>> Qcow2CompressData *data = opaque;
>>>> @@ -215,6 +327,11 @@ qcow2_co_compress(BlockDriverState *bs, void
>>>> *dest, size_t dest_size,
>>>> fn = qcow2_zlib_compress;
>>>> break;
>>>> +#ifdef CONFIG_ZSTD
>>>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>>>> + fn = qcow2_zstd_compress;
>>>> + break;
>>>> +#endif
>>>> default:
>>>> return -ENOTSUP;
>>>> }
>>>> @@ -247,6 +364,11 @@ qcow2_co_decompress(BlockDriverState *bs, void
>>>> *dest, size_t dest_size,
>>>> fn = qcow2_zlib_decompress;
>>>> break;
>>>> +#ifdef CONFIG_ZSTD
>>>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>>>> + fn = qcow2_zstd_decompress;
>>>> + break;
>>>> +#endif
>>>> default:
>>>> return -ENOTSUP;
>>>> }
>>>> diff --git a/block/qcow2.c b/block/qcow2.c
>>>> index 2ccb2cabd1..9c8ad9d580 100644
>>>> --- a/block/qcow2.c
>>>> +++ b/block/qcow2.c
>>>> @@ -1257,6 +1257,9 @@ static int
>>>> validate_compression_type(BDRVQcow2State *s, Error **errp)
>>>> switch (s->compression_type) {
>>>> case QCOW2_COMPRESSION_TYPE_ZLIB:
>>>> +#ifdef CONFIG_ZSTD
>>>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>>>> +#endif
>>>> break;
>>>> default:
>>>> @@ -3506,6 +3509,10 @@ qcow2_co_create(BlockdevCreateOptions
>>>> *create_options, Error **errp)
>>>> }
>>>> switch (qcow2_opts->compression_type) {
>>>> +#ifdef CONFIG_ZSTD
>>>> + case QCOW2_COMPRESSION_TYPE_ZSTD:
>>>> + break;
>>>> +#endif
>>>> default:
>>>> error_setg_errno(errp, -EINVAL, "Unknown compression
>>>> type");
>>>> goto out;
>>>> diff --git a/configure b/configure
>>>> index 48d6f89d57..4690a7ea9f 100755
>>>> --- a/configure
>>>> +++ b/configure
>>>> @@ -444,6 +444,7 @@ opengl_dmabuf="no"
>>>> cpuid_h="no"
>>>> avx2_opt=""
>>>> zlib="yes"
>>>> +zstd=""
>>>> capstone=""
>>>> lzo=""
>>>> snappy=""
>>>> @@ -1371,6 +1372,10 @@ for opt do
>>>> ;;
>>>> --disable-lzfse) lzfse="no"
>>>> ;;
>>>> + --enable-zstd) zstd="yes"
>>>> + ;;
>>>> + --disable-zstd) zstd="no"
>>>> + ;;
>>>> --enable-guest-agent) guest_agent="yes"
>>>> ;;
>>>> --disable-guest-agent) guest_agent="no"
>>>> @@ -1829,6 +1834,7 @@ disabled with --disable-FEATURE, default is
>>>> enabled if available:
>>>> (for reading bzip2-compressed dmg images)
>>>> lzfse support of lzfse compression library
>>>> (for reading lzfse-compressed dmg images)
>>>> + zstd support of zstd compression library
>>>> seccomp seccomp support
>>>> coroutine-pool coroutine freelist (better performance)
>>>> glusterfs GlusterFS backend
>>>> @@ -2453,6 +2459,25 @@ EOF
>>>> fi
>>>> fi
>>>> +#########################################
>>>> +# zstd check
>>>> +
>>>> +if test "$zstd" != "no" ; then
>>>> + cat > $TMPC << EOF
>>>> +#include <zstd.h>
>>>> +int main(void) { ZSTD_versionNumber(); return 0; }
>>>> +EOF
>>>> + if compile_prog "" "-lzstd" ; then
>>>> + LIBS="$LIBS -lzstd"
>>>> + zstd="yes"
>>>> + else
>>>> + if test "$zstd" = "yes"; then
>>>> + feature_not_found "zstd" "Install libzstd-devel"
>>>
>>> to correspond to style used around: s/libzstd-devel/libzstd devel/
>> My intention was to give a package name for a user encountered this
>> message to copy-paste it to "yum install".
>
> What about apt users? :)
>
>> If this doesn't make things easier, because the package name can be
>> changed in the future (although I doubt) I'll fix it.
>> If the style isn't that important I'd rather leave it as is.
>
> I don't care, keep it as is.
>
>>>
>>>> + fi
>>>> + zstd="no"
>>>> + fi
>>>> +fi
>>>> +
>>>> ##########################################
>>>> # libseccomp check
>>>> @@ -6668,6 +6693,7 @@ echo "lzo support $lzo"
>>>> echo "snappy support $snappy"
>>>> echo "bzip2 support $bzip2"
>>>> echo "lzfse support $lzfse"
>>>> +echo "zstd support $zstd"
>>>> echo "NUMA host support $numa"
>>>> echo "libxml2 $libxml2"
>>>> echo "tcmalloc support $tcmalloc"
>>>> @@ -7559,6 +7585,9 @@ if test "$plugins" = "yes" ; then
>>>> "\$ld_exported_symbols_list should have been set to 'yes'."
>>>> fi
>>>> fi
>>>> +if test "$zstd" = "yes" ; then
>>>> + echo "CONFIG_ZSTD=y" >> $config_host_mak
>>>> +fi
>>>> if test "$tcg_interpreter" = "yes"; then
>>>> QEMU_INCLUDES="-iquote \$(SRC_PATH)/tcg/tci $QEMU_INCLUDES"
>>>> diff --git a/docs/interop/qcow2.txt b/docs/interop/qcow2.txt
>>>> index 5597e24474..aeca2ddebd 100644
>>>> --- a/docs/interop/qcow2.txt
>>>> +++ b/docs/interop/qcow2.txt
>>>> @@ -208,6 +208,7 @@ version 2.
>>>> Available compression type values:
>>>> 0: zlib <https://www.zlib.net/>
>>>> + 1: zstd <http://github.com/facebook/zstd>
>>>> === Header padding ===
>>>> @@ -575,11 +576,28 @@ Compressed Clusters Descriptor (x = 62 -
>>>> (cluster_bits - 8)):
>>>> Another compressed cluster may map to the
>>>> tail of the final
>>>> sector used by this compressed cluster.
>>>> + The layout of the compressed data depends on
>>>> the compression
>>>> + type used for the image (see compressed
>>>> cluster layout).
>>>> +
>>>> If a cluster is unallocated, read requests shall read the data
>>>> from the backing
>>>> file (except if bit 0 in the Standard Cluster Descriptor is set).
>>>> If there is
>>>> no backing file or the backing file is smaller than the image,
>>>> they shall read
>>>> zeros for all parts that are not covered by the backing file.
>>>> +=== Compressed Cluster Layout ===
>>>> +
>>>> +The compressed cluster data has a layout depending on the compression
>>>> +type used for the image, as follows:
>>>> +
>>>> +Compressed data layout for the available compression types:
>>>> +(x = data_space_length - 1)
>>>
>>> Note, that term 'data_space_lenght' is not defined in the spec..
>>>
>>> Hmm. And it's not trivial to define it correctly, all we have is
>>> offset and
>>> "number of additional sectors", and may be not all bytes of final
>>> sector are
>>> occupied... What about something like this:
>>>
>>> In the following layout description byte 0 corresponds to byte at
>>> host cluster offset,
>>> as defined by "Compressed Clusters Descriptor" paragraph above. The
>>> whole layout occupy
>>> space starting from this offset, using additional 512-byte sectors
>>> defined by
>>> "Compressed Clusters Descriptor" paragraph, not necessarily
>>> occupying all of the bytes
>>> in the final sector. Let x be number of last byte of the layout.
>> Sounds too complicated for me, may be
>> x - data chunk length available to store a compressed cluster. (for
>> more details see "Compressed Clusters Descriptor")
>
> Ok
>
>>>
>>>
>>>> +
>>>> + 0: (default) zlib <http://zlib.net/>:
>>>> + Byte 0 - x: the compressed data content
>>>> + all the space provided used for
>>>> compressed data
>>>> + 1: zstd <http://github.com/facebook/zstd>:
>>>> + Byte 0 - 3: the length of compressed data in bytes
>>>> + 4 - x: the compressed data content
>>>
>>> Maybe, note that x+! == 4 + length, where length is value of first
>>> field of the layout.
>>>
>>>> == Snapshots ==
>>>> diff --git a/qapi/block-core.json b/qapi/block-core.json
>>>> index 873fbef3b5..4b6e576c44 100644
>>>> --- a/qapi/block-core.json
>>>> +++ b/qapi/block-core.json
>>>> @@ -4401,11 +4401,12 @@
>>>> # Compression type used in qcow2 image file
>>>> #
>>>> # @zlib: zlib compression, see <http://zlib.net/>
>>>> +# @zstd: zstd compression, see <http://github.com/facebook/zstd>
>>>> #
>>>> # Since: 5.0
>>>> ##
>>>> { 'enum': 'Qcow2CompressionType',
>>>> - 'data': [ 'zlib' ] }
>>>> + 'data': [ 'zlib', { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)'
>>>> } ] }
>>>> ##
>>>> # @BlockdevCreateOptionsQcow2:
>>>>
>>>
>>>
>>
>
>
^ permalink raw reply [flat|nested] 35+ messages in thread
end of thread, other threads:[~2020-02-28 12:03 UTC | newest]
Thread overview: 35+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-02-27 7:29 [PATCH v1 0/8] qcow2: Implement zstd cluster compression method Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 1/8] qcow2: introduce compression type feature Denis Plotnikov
2020-02-27 8:21 ` Vladimir Sementsov-Ogievskiy
2020-02-27 13:24 ` Eric Blake
2020-02-27 13:48 ` Eric Blake
2020-02-27 13:59 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:13 ` Eric Blake
2020-02-27 14:30 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:39 ` Eric Blake
2020-02-28 8:34 ` Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 2/8] qcow2: rework the cluster compression routine Denis Plotnikov
2020-02-27 8:54 ` Vladimir Sementsov-Ogievskiy
2020-02-27 7:29 ` [PATCH v1 3/8] qcow2: add zstd cluster compression Denis Plotnikov
2020-02-27 9:55 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:11 ` Denis Plotnikov
2020-02-27 14:18 ` Vladimir Sementsov-Ogievskiy
2020-02-28 11:59 ` Denis Plotnikov
2020-02-27 14:01 ` Eric Blake
2020-02-28 11:40 ` Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 4/8] iotests: filter out compression_type Denis Plotnikov
2020-02-27 9:57 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:03 ` Eric Blake
2020-02-28 11:53 ` Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 5/8] iotests: fix header size, feature table size and backing file offset Denis Plotnikov
2020-02-27 9:59 ` Vladimir Sementsov-Ogievskiy
2020-02-27 7:29 ` [PATCH v1 6/8] iotests: add "compression type" for test output matching Denis Plotnikov
2020-02-27 10:04 ` Vladimir Sementsov-Ogievskiy
2020-02-27 10:09 ` Vladimir Sementsov-Ogievskiy
2020-02-27 14:06 ` Eric Blake
2020-02-28 8:13 ` Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 7/8] iotests: 080: update header size value because of adding compression type Denis Plotnikov
2020-02-27 7:29 ` [PATCH v1 8/8] iotests: 287: add qcow2 compression type test Denis Plotnikov
2020-02-27 10:29 ` Vladimir Sementsov-Ogievskiy
2020-02-28 8:23 ` Denis Plotnikov
2020-02-28 8:32 ` Vladimir Sementsov-Ogievskiy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.