* [Qemu-devel] [PATCH 0/3] migration: add sztd compression
@ 2019-02-26 13:15 Denis Plotnikov
2019-02-26 13:15 ` [Qemu-devel] [PATCH 1/3] migration: rework compression code for adding more data compressors Denis Plotnikov
` (5 more replies)
0 siblings, 6 replies; 13+ messages in thread
From: Denis Plotnikov @ 2019-02-26 13:15 UTC (permalink / raw)
To: dgilbert, quintela, eblake, armbru; +Cc: den, qemu-devel
zstd date compression algorithm shows better performance on data compression.
It might be useful to employ the algorithm in VM migration to reduce CPU usage.
A user will be able to choose between those algorithms, therefor compress-type
migration parameter is added.
Here are some results of performance comparison zstd vs gzip:
host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
migration to the same host
VM: 2xVCPU, 8G RAM total
5G RAM used, memory populated with postgreqsl data
produced by pgbench performance benchmark
Threads: 1 compress – 1 decompress
zstd provides slightly less compression ratio with almost the same
CPU usage but copes with RAM compression roghly 2 times faster
compression type zlib | zstd
---------------------------------------------------------
compression level 1 5 | 1 5
compression ratio 6.92 7.05 | 6.69 6.89
cpu idle, % 82 83 | 86 80
time, sec 49 71 | 26 31
time diff to zlib, sec -25 -41
Threads: 8 compress – 2 decompress
zstd provides the same migration time with less cpu consumption
compression type none | gzip(zlib) | zstd
------------------------------------------------------------------------------
compression level - | 1 5 9 | 1 5 15
compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93
time, sec 154 | 22 23 27 | 23 23 25
cpu idle, % 99 | 45 30 12 | 70 52 23
cpu idle diff to zlib | | -25% -22% -11%
Denis Plotnikov (3):
migration: rework compression code for adding more data compressors
hmp: add compress-type parameter to migration parameters
migration: add zstd compression
configure | 26 ++++
hmp.c | 8 ++
migration/migration.c | 45 ++++++-
migration/migration.h | 1 +
migration/qemu-file.c | 39 ++----
migration/qemu-file.h | 18 ++-
migration/ram.c | 291 ++++++++++++++++++++++++++++++++++--------
qapi/migration.json | 26 +++-
8 files changed, 369 insertions(+), 85 deletions(-)
--
2.17.0
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [PATCH 1/3] migration: rework compression code for adding more data compressors
2019-02-26 13:15 [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
@ 2019-02-26 13:15 ` Denis Plotnikov
2019-02-26 13:15 ` [Qemu-devel] [PATCH 2/3] hmp: add compress-type parameter to migration parameters Denis Plotnikov
` (4 subsequent siblings)
5 siblings, 0 replies; 13+ messages in thread
From: Denis Plotnikov @ 2019-02-26 13:15 UTC (permalink / raw)
To: dgilbert, quintela, eblake, armbru; +Cc: den, qemu-devel
Also, the patch adds new migration parameter parameter: compress-type
to be able choose between data compressors available.
By the moment, the only available data compressor is gzip (zlib)
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
migration/migration.c | 42 ++++++++-
migration/migration.h | 1 +
migration/qemu-file.c | 39 +++------
migration/qemu-file.h | 17 +++-
migration/ram.c | 196 +++++++++++++++++++++++++++++++-----------
qapi/migration.json | 26 ++++--
6 files changed, 236 insertions(+), 85 deletions(-)
diff --git a/migration/migration.c b/migration/migration.c
index 37e06b76dc..10cecb0eeb 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -739,6 +739,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
params->max_postcopy_bandwidth = s->parameters.max_postcopy_bandwidth;
params->has_max_cpu_throttle = true;
params->max_cpu_throttle = s->parameters.max_cpu_throttle;
+ params->has_compress_type = true;
+ params->compress_type = s->parameters.compress_type;
return params;
}
@@ -1027,10 +1029,27 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
*/
static bool migrate_params_check(MigrationParameters *params, Error **errp)
{
+ int max_compress_level = -1;
+
+ if (params->has_compress_type) {
+ switch (params->compress_type) {
+ case COMPRESSION_TYPE_ZLIB:
+ max_compress_level = 9;
+ break;
+ default:
+ error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "compress_type",
+ "values: 0 - gzip");
+ return false;
+ }
+ }
+
if (params->has_compress_level &&
- (params->compress_level > 9)) {
+ (params->compress_level > max_compress_level)) {
+ char level_range_msg[30];
+ snprintf(level_range_msg, 30, "values from 0 to %d",
+ max_compress_level);
error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "compress_level",
- "is invalid, it should be in the range of 0 to 9");
+ level_range_msg);
return false;
}
@@ -1125,6 +1144,9 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
*dest = migrate_get_current()->parameters;
/* TODO use QAPI_CLONE() instead of duplicating it inline */
+ if (params->has_compress_type) {
+ dest->compress_type = params->compress_type;
+ }
if (params->has_compress_level) {
dest->compress_level = params->compress_level;
@@ -1272,6 +1294,9 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
if (params->has_max_cpu_throttle) {
s->parameters.max_cpu_throttle = params->max_cpu_throttle;
}
+ if (params->has_compress_type) {
+ s->parameters.compress_type = params->compress_type;
+ }
}
void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
@@ -1938,6 +1963,15 @@ bool migrate_use_compression(void)
return s->enabled_capabilities[MIGRATION_CAPABILITY_COMPRESS];
}
+int migrate_compress_type(void)
+{
+ MigrationState *s;
+
+ s = migrate_get_current();
+
+ return s->parameters.compress_type;
+}
+
int migrate_compress_level(void)
{
MigrationState *s;
@@ -3234,6 +3268,9 @@ static Property migration_properties[] = {
decompress_error_check, true),
/* Migration parameters */
+ DEFINE_PROP_UINT8("x-compress-type", MigrationState,
+ parameters.compress_type,
+ COMPRESSION_TYPE_ZLIB),
DEFINE_PROP_UINT8("x-compress-level", MigrationState,
parameters.compress_level,
DEFAULT_MIGRATE_COMPRESS_LEVEL),
@@ -3346,6 +3383,7 @@ static void migration_instance_init(Object *obj)
params->has_xbzrle_cache_size = true;
params->has_max_postcopy_bandwidth = true;
params->has_max_cpu_throttle = true;
+ params->has_compress_type = true;
qemu_sem_init(&ms->postcopy_pause_sem, 0);
qemu_sem_init(&ms->postcopy_pause_rp_sem, 0);
diff --git a/migration/migration.h b/migration/migration.h
index dcd05d9f87..ddb9efec86 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -280,6 +280,7 @@ bool migrate_use_return_path(void);
uint64_t ram_get_total_transferred_pages(void);
bool migrate_use_compression(void);
+int migrate_compress_type(void);
int migrate_compress_level(void);
int migrate_compress_threads(void);
int migrate_compress_wait_thread(void);
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 977b9ae07c..cd95749aa6 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -662,28 +662,10 @@ uint64_t qemu_get_be64(QEMUFile *f)
return v;
}
-/* return the size after compression, or negative value on error */
-static int qemu_compress_data(z_stream *stream, uint8_t *dest, size_t dest_len,
+static int qemu_compress_data(Compression *comp, uint8_t *dest, size_t dest_len,
const uint8_t *source, size_t source_len)
{
- int err;
-
- err = deflateReset(stream);
- if (err != Z_OK) {
- return -1;
- }
-
- stream->avail_in = source_len;
- stream->next_in = (uint8_t *)source;
- stream->avail_out = dest_len;
- stream->next_out = dest;
-
- err = deflate(stream, Z_FINISH);
- if (err != Z_STREAM_END) {
- return -1;
- }
-
- return stream->next_out - dest;
+ return comp->process(comp, dest, dest_len, source, source_len);
}
/* Compress size bytes of data start at p and store the compressed
@@ -695,23 +677,30 @@ static int qemu_compress_data(z_stream *stream, uint8_t *dest, size_t dest_len,
* do fflush first, if f still has no space to save the compressed
* data, return -1.
*/
-ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
+ssize_t qemu_put_compression_data(QEMUFile *f, Compression *comp,
const uint8_t *p, size_t size)
{
- ssize_t blen = IO_BUF_SIZE - f->buf_index - sizeof(int32_t);
+ int blen = IO_BUF_SIZE - f->buf_index - sizeof(int32_t);
+ unsigned long bound;
- if (blen < compressBound(size)) {
+ bound = comp->get_bound(size);
+
+ if (blen < bound) {
if (!qemu_file_is_writable(f)) {
+ error_report("compression: qemu file is not writable");
return -1;
}
+
qemu_fflush(f);
blen = IO_BUF_SIZE - sizeof(int32_t);
- if (blen < compressBound(size)) {
+ if (blen < bound) {
+ error_report("compression: io buffer is too small:%d needed: %lu",
+ IO_BUF_SIZE, bound);
return -1;
}
}
- blen = qemu_compress_data(stream, f->buf + f->buf_index + sizeof(int32_t),
+ blen = qemu_compress_data(comp, f->buf + f->buf_index + sizeof(int32_t),
blen, p, size);
if (blen < 0) {
return -1;
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index 2ccfcfb2a8..24cf0d7e25 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -115,6 +115,21 @@ typedef struct QEMUFileHooks {
QEMURamSaveFunc *save_page;
} QEMUFileHooks;
+typedef enum CompressionType {
+ COMPRESSION_TYPE_ZLIB = 0,
+} CompressionType;
+
+struct Compression {
+ CompressionType type;
+ bool is_decompression;
+ void *stream;
+ int (*process)(struct Compression *comp, uint8_t *dest, size_t dest_len,
+ const uint8_t *source, size_t source_len);
+ unsigned long (*get_bound)(unsigned long);
+};
+
+typedef struct Compression Compression;
+
QEMUFile *qemu_fopen_ops(void *opaque, const QEMUFileOps *ops);
void qemu_file_set_hooks(QEMUFile *f, const QEMUFileHooks *hooks);
int qemu_get_fd(QEMUFile *f);
@@ -134,7 +149,7 @@ bool qemu_file_is_writable(QEMUFile *f);
size_t qemu_peek_buffer(QEMUFile *f, uint8_t **buf, size_t size, size_t offset);
size_t qemu_get_buffer_in_place(QEMUFile *f, uint8_t **buf, size_t size);
-ssize_t qemu_put_compression_data(QEMUFile *f, z_stream *stream,
+ssize_t qemu_put_compression_data(QEMUFile *f, Compression *comp,
const uint8_t *p, size_t size);
int qemu_put_qemu_file(QEMUFile *f_des, QEMUFile *f_src);
diff --git a/migration/ram.c b/migration/ram.c
index 59191c1ed2..9ff154ed7b 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -360,8 +360,8 @@ struct CompressParam {
ram_addr_t offset;
/* internally used fields */
- z_stream stream;
uint8_t *originbuf;
+ Compression comp;
};
typedef struct CompressParam CompressParam;
@@ -373,7 +373,7 @@ struct DecompressParam {
void *des;
uint8_t *compbuf;
int len;
- z_stream stream;
+ Compression comp;
};
typedef struct DecompressParam DecompressParam;
@@ -394,8 +394,114 @@ static QemuThread *decompress_threads;
static QemuMutex decomp_done_lock;
static QemuCond decomp_done_cond;
-static bool do_compress_ram_page(QEMUFile *f, z_stream *stream, RAMBlock *block,
- ram_addr_t offset, uint8_t *source_buf);
+static bool do_compress_ram_page(QEMUFile *f, Compression *comp,
+ RAMBlock *block, ram_addr_t offset,
+ uint8_t *source_buf);
+
+static int zlib_compress(Compression *comp, uint8_t *dest, size_t dest_len,
+ const uint8_t *source, size_t source_len)
+{
+ int err;
+ z_stream *stream = comp->stream;
+
+ err = deflateReset(comp->stream);
+ if (err != Z_OK) {
+ return -1;
+ }
+
+ stream->avail_in = source_len;
+ stream->next_in = (uint8_t *)source;
+ stream->avail_out = dest_len;
+ stream->next_out = dest;
+
+ err = deflate(stream, Z_FINISH);
+ if (err != Z_STREAM_END) {
+ return -1;
+ }
+
+ return stream->next_out - dest;
+}
+
+static int zlib_decompress(Compression *comp, uint8_t *dest, size_t dest_len,
+ const uint8_t *source, size_t source_len)
+{
+ int err;
+ z_stream *stream = comp->stream;
+
+ err = inflateReset(stream);
+ if (err != Z_OK) {
+ return -1;
+ }
+
+ stream->avail_in = source_len;
+ stream->next_in = (uint8_t *)source;
+ stream->avail_out = dest_len;
+ stream->next_out = dest;
+
+ err = inflate(stream, Z_NO_FLUSH);
+ if (err != Z_STREAM_END) {
+ return -1;
+ }
+
+ return stream->total_out;
+}
+
+static int init_compression(Compression *comp, CompressionType type,
+ bool is_decompression)
+{
+ int res;
+
+ switch (type) {
+ case COMPRESSION_TYPE_ZLIB:
+ comp->stream = g_new0(z_stream, 1);
+
+ if (is_decompression) {
+ res = inflateInit(comp->stream);
+ } else {
+ res = deflateInit(comp->stream, migrate_compress_level());
+ }
+
+ if (res != Z_OK) {
+ g_free(comp->stream);
+ return 1;
+ }
+
+ if (is_decompression) {
+ comp->process = zlib_decompress;
+ } else {
+ comp->process = zlib_compress;
+ }
+
+ comp->get_bound = compressBound;
+ break;
+ default:
+ return 1;
+ }
+
+ comp->type = type;
+ comp->is_decompression = is_decompression;
+ return 0;
+}
+
+static void destroy_compression(Compression *comp)
+{
+ assert(comp);
+
+ switch (comp->type) {
+ case COMPRESSION_TYPE_ZLIB:
+ if (comp->is_decompression) {
+ inflateEnd(comp->stream);
+ } else {
+ deflateEnd(comp->stream);
+ }
+ g_free(comp->stream);
+ break;
+ default:
+ assert(false);
+ }
+
+ memset(comp, 0, sizeof(Compression));
+}
static void *do_data_compress(void *opaque)
{
@@ -412,7 +518,7 @@ static void *do_data_compress(void *opaque)
param->block = NULL;
qemu_mutex_unlock(¶m->mutex);
- zero_page = do_compress_ram_page(param->file, ¶m->stream,
+ zero_page = do_compress_ram_page(param->file, ¶m->comp,
block, offset, param->originbuf);
qemu_mutex_lock(&comp_done_lock);
@@ -457,7 +563,7 @@ static void compress_threads_save_cleanup(void)
qemu_thread_join(compress_threads + i);
qemu_mutex_destroy(&comp_param[i].mutex);
qemu_cond_destroy(&comp_param[i].cond);
- deflateEnd(&comp_param[i].stream);
+ destroy_compression(&comp_param->comp);
g_free(comp_param[i].originbuf);
qemu_fclose(comp_param[i].file);
comp_param[i].file = NULL;
@@ -480,31 +586,32 @@ static int compress_threads_save_setup(void)
thread_count = migrate_compress_threads();
compress_threads = g_new0(QemuThread, thread_count);
comp_param = g_new0(CompressParam, thread_count);
+
qemu_cond_init(&comp_done_cond);
qemu_mutex_init(&comp_done_lock);
for (i = 0; i < thread_count; i++) {
- comp_param[i].originbuf = g_try_malloc(TARGET_PAGE_SIZE);
- if (!comp_param[i].originbuf) {
+ CompressParam *comp = &comp_param[i];
+
+ comp->originbuf = g_try_malloc(TARGET_PAGE_SIZE);
+ if (!comp->originbuf) {
goto exit;
}
- if (deflateInit(&comp_param[i].stream,
- migrate_compress_level()) != Z_OK) {
- g_free(comp_param[i].originbuf);
+ if (init_compression(&comp->comp, migrate_compress_type(), false)) {
+ g_free(comp->originbuf);
goto exit;
}
/* comp_param[i].file is just used as a dummy buffer to save data,
* set its ops to empty.
*/
- comp_param[i].file = qemu_fopen_ops(NULL, &empty_ops);
- comp_param[i].done = true;
- comp_param[i].quit = false;
- qemu_mutex_init(&comp_param[i].mutex);
- qemu_cond_init(&comp_param[i].cond);
- qemu_thread_create(compress_threads + i, "compress",
- do_data_compress, comp_param + i,
- QEMU_THREAD_JOINABLE);
+ comp->file = qemu_fopen_ops(NULL, &empty_ops);
+ comp->done = true;
+ comp->quit = false;
+ qemu_mutex_init(&comp->mutex);
+ qemu_cond_init(&comp->cond);
+ qemu_thread_create(compress_threads + i, "compress", do_data_compress,
+ comp, QEMU_THREAD_JOINABLE);
}
return 0;
@@ -1890,8 +1997,9 @@ static int ram_save_multifd_page(RAMState *rs, RAMBlock *block,
return 1;
}
-static bool do_compress_ram_page(QEMUFile *f, z_stream *stream, RAMBlock *block,
- ram_addr_t offset, uint8_t *source_buf)
+static bool do_compress_ram_page(QEMUFile *f, Compression *comp,
+ RAMBlock *block, ram_addr_t offset,
+ uint8_t *source_buf)
{
RAMState *rs = ram_state;
uint8_t *p = block->host + (offset & TARGET_PAGE_MASK);
@@ -1911,7 +2019,7 @@ static bool do_compress_ram_page(QEMUFile *f, z_stream *stream, RAMBlock *block,
* decompression
*/
memcpy(source_buf, p, TARGET_PAGE_SIZE);
- ret = qemu_put_compression_data(f, stream, source_buf, TARGET_PAGE_SIZE);
+ ret = qemu_put_compression_data(f, comp, source_buf, TARGET_PAGE_SIZE);
if (ret < 0) {
qemu_file_set_error(migrate_get_current()->to_dst_file, ret);
error_report("compressed data failed!");
@@ -3502,28 +3610,14 @@ void ram_handle_compressed(void *host, uint8_t ch, uint64_t size)
}
/* return the size after decompression, or negative value on error */
-static int
-qemu_uncompress_data(z_stream *stream, uint8_t *dest, size_t dest_len,
- const uint8_t *source, size_t source_len)
+static int qemu_uncompress_data(Compression *comp, uint8_t *dest,
+ size_t dest_len, const uint8_t *source,
+ size_t source_len)
{
- int err;
-
- err = inflateReset(stream);
- if (err != Z_OK) {
+ if (source_len > comp->get_bound(TARGET_PAGE_SIZE)) {
return -1;
}
-
- stream->avail_in = source_len;
- stream->next_in = (uint8_t *)source;
- stream->avail_out = dest_len;
- stream->next_out = dest;
-
- err = inflate(stream, Z_NO_FLUSH);
- if (err != Z_STREAM_END) {
- return -1;
- }
-
- return stream->total_out;
+ return comp->process(comp, dest, dest_len, source, source_len);
}
static void *do_data_decompress(void *opaque)
@@ -3543,7 +3637,7 @@ static void *do_data_decompress(void *opaque)
pagesize = TARGET_PAGE_SIZE;
- ret = qemu_uncompress_data(¶m->stream, des, pagesize,
+ ret = qemu_uncompress_data(¶m->comp, des, pagesize,
param->compbuf, len);
if (ret < 0 && migrate_get_current()->decompress_error_check) {
error_report("decompress data failed");
@@ -3614,7 +3708,7 @@ static void compress_threads_load_cleanup(void)
qemu_thread_join(decompress_threads + i);
qemu_mutex_destroy(&decomp_param[i].mutex);
qemu_cond_destroy(&decomp_param[i].cond);
- inflateEnd(&decomp_param[i].stream);
+ destroy_compression(&decomp_param[i].comp);
g_free(decomp_param[i].compbuf);
decomp_param[i].compbuf = NULL;
}
@@ -3640,15 +3734,17 @@ static int compress_threads_load_setup(QEMUFile *f)
qemu_cond_init(&decomp_done_cond);
decomp_file = f;
for (i = 0; i < thread_count; i++) {
- if (inflateInit(&decomp_param[i].stream) != Z_OK) {
+ DecompressParam *decomp = &decomp_param[i];
+
+ if (init_compression(&decomp->comp, migrate_compress_type(), true)) {
goto exit;
}
- decomp_param[i].compbuf = g_malloc0(compressBound(TARGET_PAGE_SIZE));
- qemu_mutex_init(&decomp_param[i].mutex);
- qemu_cond_init(&decomp_param[i].cond);
- decomp_param[i].done = true;
- decomp_param[i].quit = false;
+ decomp->compbuf = g_malloc0(decomp->comp.get_bound(TARGET_PAGE_SIZE));
+ qemu_mutex_init(&decomp->mutex);
+ qemu_cond_init(&decomp->cond);
+ decomp->done = true;
+ decomp->quit = false;
qemu_thread_create(decompress_threads + i, "decompress",
do_data_decompress, decomp_param + i,
QEMU_THREAD_JOINABLE);
@@ -4169,7 +4265,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
case RAM_SAVE_FLAG_COMPRESS_PAGE:
len = qemu_get_be32(f);
- if (len < 0 || len > compressBound(TARGET_PAGE_SIZE)) {
+ if (len < 0) {
error_report("Invalid compressed data length: %d", len);
ret = -EINVAL;
break;
diff --git a/qapi/migration.json b/qapi/migration.json
index 7a795ecc16..9a3110e383 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -480,10 +480,15 @@
#
# Migration parameters enumeration
#
+# @compress-type: Set the compression type to be used in live migration,
+# the compression type is an integer from the list:
+# 0 - gzip
+#
# @compress-level: Set the compression level to be used in live migration,
-# the compression level is an integer between 0 and 9, where 0 means
-# no compression, 1 means the best compression speed, and 9 means best
-# compression ratio which will consume more CPU.
+# the compression level is an integer between 0 and 9,
+# where 0 means no compression, 1 means the best compression speed,
+# and the highest value depending on the compression type means
+# the best compression ratio which will consume more CPU.
#
# @compress-threads: Set compression thread count to be used in live migration,
# the compression thread count is an integer between 1 and 255.
@@ -560,8 +565,8 @@
# Since: 2.4
##
{ 'enum': 'MigrationParameter',
- 'data': ['compress-level', 'compress-threads', 'decompress-threads',
- 'compress-wait-thread',
+ 'data': ['compress-type', 'compress-level', 'compress-threads',
+ 'decompress-threads', 'compress-wait-thread',
'cpu-throttle-initial', 'cpu-throttle-increment',
'tls-creds', 'tls-hostname', 'max-bandwidth',
'downtime-limit', 'x-checkpoint-delay', 'block-incremental',
@@ -572,6 +577,9 @@
##
# @MigrateSetParameters:
#
+# @compress-type: Compression type is used for migration.
+# Available types: 0 - gzip
+#
# @compress-level: compression level
#
# @compress-threads: compression thread count
@@ -653,7 +661,8 @@
# TODO either fuse back into MigrationParameters, or make
# MigrationParameters members mandatory
{ 'struct': 'MigrateSetParameters',
- 'data': { '*compress-level': 'int',
+ 'data': { '*compress-type': 'int',
+ '*compress-level': 'int',
'*compress-threads': 'int',
'*compress-wait-thread': 'bool',
'*decompress-threads': 'int',
@@ -692,6 +701,8 @@
#
# The optional members aren't actually optional.
#
+# @compress-type: compression type
+#
# @compress-level: compression level
#
# @compress-threads: compression thread count
@@ -769,7 +780,8 @@
# Since: 2.4
##
{ 'struct': 'MigrationParameters',
- 'data': { '*compress-level': 'uint8',
+ 'data': { '*compress-type': 'uint8',
+ '*compress-level': 'uint8',
'*compress-threads': 'uint8',
'*compress-wait-thread': 'bool',
'*decompress-threads': 'uint8',
--
2.17.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [Qemu-devel] [PATCH 2/3] hmp: add compress-type parameter to migration parameters
2019-02-26 13:15 [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
2019-02-26 13:15 ` [Qemu-devel] [PATCH 1/3] migration: rework compression code for adding more data compressors Denis Plotnikov
@ 2019-02-26 13:15 ` Denis Plotnikov
2019-02-26 13:15 ` [Qemu-devel] [PATCH 3/3] migration: add zstd compression Denis Plotnikov
` (3 subsequent siblings)
5 siblings, 0 replies; 13+ messages in thread
From: Denis Plotnikov @ 2019-02-26 13:15 UTC (permalink / raw)
To: dgilbert, quintela, eblake, armbru; +Cc: den, qemu-devel
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
hmp.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/hmp.c b/hmp.c
index b2a2b1f84e..5f105b816f 100644
--- a/hmp.c
+++ b/hmp.c
@@ -334,6 +334,10 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
params = qmp_query_migrate_parameters(NULL);
if (params) {
+ assert(params->has_compress_type);
+ monitor_printf(mon, "%s: %u\n",
+ MigrationParameter_str(MIGRATION_PARAMETER_COMPRESS_TYPE),
+ params->compress_type);
assert(params->has_compress_level);
monitor_printf(mon, "%s: %u\n",
MigrationParameter_str(MIGRATION_PARAMETER_COMPRESS_LEVEL),
@@ -1757,6 +1761,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
p->has_max_postcopy_bandwidth = true;
visit_type_size(v, param, &p->max_postcopy_bandwidth, &err);
break;
+ case MIGRATION_PARAMETER_COMPRESS_TYPE:
+ p->has_compress_type = true;
+ visit_type_int(v, param, &p->compress_type, &err);
+ break;
default:
assert(0);
}
--
2.17.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* [Qemu-devel] [PATCH 3/3] migration: add zstd compression
2019-02-26 13:15 [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
2019-02-26 13:15 ` [Qemu-devel] [PATCH 1/3] migration: rework compression code for adding more data compressors Denis Plotnikov
2019-02-26 13:15 ` [Qemu-devel] [PATCH 2/3] hmp: add compress-type parameter to migration parameters Denis Plotnikov
@ 2019-02-26 13:15 ` Denis Plotnikov
2019-03-04 15:10 ` [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
` (2 subsequent siblings)
5 siblings, 0 replies; 13+ messages in thread
From: Denis Plotnikov @ 2019-02-26 13:15 UTC (permalink / raw)
To: dgilbert, quintela, eblake, armbru; +Cc: den, qemu-devel
zstd allows to migrate with less cpu consumption maintaining the
the same level of data compression as qzip (zlib).
Compression level for zstd is set with migration parameter "compress-level"
in the range 1 - 22. 1 - the best speed, 22 - the best compression.
Levels in the range of 20-22 should be used with care because they lead
to significant growth of CPU and memory usage.
Signed-off-by: Denis Plotnikov <dplotnikov@virtuozzo.com>
---
configure | 26 ++++++++++++
migration/migration.c | 5 ++-
migration/qemu-file.h | 1 +
migration/ram.c | 95 +++++++++++++++++++++++++++++++++++++++++++
qapi/migration.json | 6 +--
5 files changed, 129 insertions(+), 4 deletions(-)
diff --git a/configure b/configure
index f8176b3c40..9dd1c18650 100755
--- a/configure
+++ b/configure
@@ -432,6 +432,7 @@ opengl_dmabuf="no"
cpuid_h="no"
avx2_opt=""
zlib="yes"
+zstd="yes"
capstone=""
lzo=""
snappy=""
@@ -1301,6 +1302,8 @@ for opt do
;;
--disable-zlib-test) zlib="no"
;;
+ --disable-zstd-test) zstd="no"
+ ;;
--disable-lzo) lzo="no"
;;
--enable-lzo) lzo="yes"
@@ -3586,6 +3589,29 @@ EOF
fi
fi
+#########################################
+# zstd check
+
+if test "$zstd" != "no" ; then
+ if $pkg_config --exists libzstd; then
+ zstd_cflags=$($pkg_config --cflags libzstd)
+ zstd_libs=$($pkg_config --libs libzstd)
+ QEMU_CFLAGS="$zstd_cflags $QEMU_CFLAGS"
+ LIBS="$zstd_libs $LIBS"
+ else
+ cat > $TMPC << EOF
+#include <zstd.h>
+int main(void) { ZSTD_versionNumber(); return 0; }
+EOF
+ if compile_prog "" "-lzstd" ; then
+ LIBS="$LIBS -lzstd"
+ else
+ error_exit "zstd check failed" \
+ "Make sure to have the zstd libs and headers installed."
+ fi
+ fi
+fi
+
##########################################
# SHA command probe for modules
if test "$modules" = yes; then
diff --git a/migration/migration.c b/migration/migration.c
index 10cecb0eeb..a7875bbb47 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1036,9 +1036,12 @@ static bool migrate_params_check(MigrationParameters *params, Error **errp)
case COMPRESSION_TYPE_ZLIB:
max_compress_level = 9;
break;
+ case COMPRESSION_TYPE_ZSTD:
+ max_compress_level = 22;
+ break;
default:
error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "compress_type",
- "values: 0 - gzip");
+ "values: 0 - gzip, 1 - zstd");
return false;
}
}
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index 24cf0d7e25..7cd054f73e 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -117,6 +117,7 @@ typedef struct QEMUFileHooks {
typedef enum CompressionType {
COMPRESSION_TYPE_ZLIB = 0,
+ COMPRESSION_TYPE_ZSTD = 1,
} CompressionType;
struct Compression {
diff --git a/migration/ram.c b/migration/ram.c
index 9ff154ed7b..4be5d100df 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -57,6 +57,7 @@
#include "qemu/uuid.h"
#include "savevm.h"
#include "qemu/iov.h"
+#include <zstd.h>
/***********************************************************/
/* ram save/restore */
@@ -446,6 +447,59 @@ static int zlib_decompress(Compression *comp, uint8_t *dest, size_t dest_len,
return stream->total_out;
}
+static int zstd_compress(Compression *comp, uint8_t *dest, size_t dest_len,
+ const uint8_t *source, size_t source_len)
+{
+ int res;
+ ZSTD_inBuffer input = {source, source_len, 0};
+ ZSTD_outBuffer output = {dest, dest_len, 0};
+
+ res = ZSTD_initCStream(comp->stream, migrate_compress_level());
+
+ if (ZSTD_isError(res)) {
+ error_report("zstd: compression stream initialization error: %s",
+ ZSTD_getErrorName(res));
+ return -1;
+ }
+
+ res = ZSTD_compressStream(comp->stream, &output, &input);
+
+ if (ZSTD_isError(res)) {
+ error_report("zstd: compression error: %s",
+ ZSTD_getErrorName(res));
+ return -1;
+ }
+
+ res = ZSTD_endStream(comp->stream, &output);
+
+ if (ZSTD_isError(res)) {
+ error_report("zstd: end stream error: %s",
+ ZSTD_getErrorName(res));
+ return -1;
+ }
+
+ return output.pos;
+}
+
+static int zstd_decompress(Compression *comp, uint8_t *dest, size_t dest_len,
+ const uint8_t *source, size_t source_len)
+{
+ int res;
+ ZSTD_inBuffer input = {source, source_len, 0};
+ ZSTD_outBuffer output = {dest, dest_len, 0};
+
+ res = ZSTD_decompressStream(comp->stream, &output, &input);
+
+ if (ZSTD_isError(res)) {
+ error_report("zstd: decompression error: %s",
+ ZSTD_getErrorName(res));
+ return -1;
+ }
+
+ return output.pos;
+}
+
+
static int init_compression(Compression *comp, CompressionType type,
bool is_decompression)
{
@@ -474,6 +528,40 @@ static int init_compression(Compression *comp, CompressionType type,
comp->get_bound = compressBound;
break;
+ case COMPRESSION_TYPE_ZSTD:
+ if (is_decompression) {
+ int res;
+
+ comp->stream = ZSTD_createDStream();
+
+ if (comp->stream == NULL) {
+ error_report("zstd: can't create decompression stream");
+ return 1;
+ }
+
+ res = ZSTD_initDStream(comp->stream);
+
+ if (ZSTD_isError(res)) {
+ error_report("zstd: can't initialzie decompression: %s",
+ ZSTD_getErrorName(res));
+ ZSTD_freeDStream(comp->stream);
+ return 1;
+ }
+
+ comp->process = zstd_decompress;
+ } else {
+ comp->stream = ZSTD_createCStream();
+
+ if (comp->stream == NULL) {
+ error_report("zstd: can't create compression stream");
+ return 1;
+ }
+
+ comp->process = zstd_compress;
+ }
+
+ comp->get_bound = ZSTD_compressBound;
+ break;
default:
return 1;
}
@@ -496,6 +584,13 @@ static void destroy_compression(Compression *comp)
}
g_free(comp->stream);
break;
+ case COMPRESSION_TYPE_ZSTD:
+ if (comp->is_decompression) {
+ ZSTD_freeDStream(comp->stream);
+ } else {
+ ZSTD_freeCStream(comp->stream);
+ }
+ break;
default:
assert(false);
}
diff --git a/qapi/migration.json b/qapi/migration.json
index 9a3110e383..c0d48d21d4 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -482,10 +482,10 @@
#
# @compress-type: Set the compression type to be used in live migration,
# the compression type is an integer from the list:
-# 0 - gzip
+# 0 - gzip, 1 -zstd
#
# @compress-level: Set the compression level to be used in live migration,
-# the compression level is an integer between 0 and 9,
+# the compression level is an integer between 0 and gzip:9, zstd:22
# where 0 means no compression, 1 means the best compression speed,
# and the highest value depending on the compression type means
# the best compression ratio which will consume more CPU.
@@ -578,7 +578,7 @@
# @MigrateSetParameters:
#
# @compress-type: Compression type is used for migration.
-# Available types: 0 - gzip
+# Available types: 0 - gzip, 1 - zstd
#
# @compress-level: compression level
#
--
2.17.0
^ permalink raw reply related [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression
2019-02-26 13:15 [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
` (2 preceding siblings ...)
2019-02-26 13:15 ` [Qemu-devel] [PATCH 3/3] migration: add zstd compression Denis Plotnikov
@ 2019-03-04 15:10 ` Denis Plotnikov
2019-03-07 8:39 ` Denis Plotnikov
2019-03-11 8:20 ` Denis Plotnikov
2019-03-06 11:32 ` [Qemu-devel] " no-reply
2020-01-24 12:43 ` Juan Quintela
5 siblings, 2 replies; 13+ messages in thread
From: Denis Plotnikov @ 2019-03-04 15:10 UTC (permalink / raw)
To: dgilbert, quintela, eblake, armbru; +Cc: Denis Lunev, qemu-devel
ping!
On 26.02.2019 16:15, Denis Plotnikov wrote:
> zstd date compression algorithm shows better performance on data compression.
> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
> A user will be able to choose between those algorithms, therefor compress-type
> migration parameter is added.
>
> Here are some results of performance comparison zstd vs gzip:
>
> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
> migration to the same host
> VM: 2xVCPU, 8G RAM total
> 5G RAM used, memory populated with postgreqsl data
> produced by pgbench performance benchmark
>
>
> Threads: 1 compress – 1 decompress
>
> zstd provides slightly less compression ratio with almost the same
> CPU usage but copes with RAM compression roghly 2 times faster
>
> compression type zlib | zstd
> ---------------------------------------------------------
> compression level 1 5 | 1 5
> compression ratio 6.92 7.05 | 6.69 6.89
> cpu idle, % 82 83 | 86 80
> time, sec 49 71 | 26 31
> time diff to zlib, sec -25 -41
>
>
> Threads: 8 compress – 2 decompress
>
> zstd provides the same migration time with less cpu consumption
>
> compression type none | gzip(zlib) | zstd
> ------------------------------------------------------------------------------
> compression level - | 1 5 9 | 1 5 15
> compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93
> time, sec 154 | 22 23 27 | 23 23 25
> cpu idle, % 99 | 45 30 12 | 70 52 23
> cpu idle diff to zlib | | -25% -22% -11%
>
>
> Denis Plotnikov (3):
> migration: rework compression code for adding more data compressors
> hmp: add compress-type parameter to migration parameters
> migration: add zstd compression
>
> configure | 26 ++++
> hmp.c | 8 ++
> migration/migration.c | 45 ++++++-
> migration/migration.h | 1 +
> migration/qemu-file.c | 39 ++----
> migration/qemu-file.h | 18 ++-
> migration/ram.c | 291 ++++++++++++++++++++++++++++++++++--------
> qapi/migration.json | 26 +++-
> 8 files changed, 369 insertions(+), 85 deletions(-)
>
--
Best,
Denis
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression
2019-02-26 13:15 [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
` (3 preceding siblings ...)
2019-03-04 15:10 ` [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
@ 2019-03-06 11:32 ` no-reply
2020-01-24 12:43 ` Juan Quintela
5 siblings, 0 replies; 13+ messages in thread
From: no-reply @ 2019-03-06 11:32 UTC (permalink / raw)
To: dplotnikov; +Cc: fam, dgilbert, quintela, eblake, armbru, den, qemu-devel
Patchew URL: https://patchew.org/QEMU/20190226131535.30361-1-dplotnikov@virtuozzo.com/
Hi,
This series failed the docker-mingw@fedora build test. Please find the testing commands and
their output below. If you have Docker installed, you can probably reproduce it
locally.
=== TEST SCRIPT BEGIN ===
#!/bin/bash
time make docker-test-mingw@fedora SHOW_ENV=1 J=14 NETWORK=1
=== TEST SCRIPT END ===
Configure options:
--enable-werror --target-list=x86_64-softmmu,aarch64-softmmu --prefix=/tmp/qemu-test/install --python=/usr/bin/python3 --cross-prefix=x86_64-w64-mingw32- --enable-trace-backends=simple --enable-gnutls --enable-nettle --enable-curl --enable-vnc --enable-bzip2 --enable-guest-agent
ERROR: zstd check failed
Make sure to have the zstd libs and headers installed.
# QEMU configure log Wed Mar 6 11:31:58 UTC 2019
---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 636 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __linux__ not defined
#error __linux__ not defined
^~~~~
---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 688 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __i386__ not defined
#error __i386__ not defined
^~~~~
---
funcs: do_compiler do_cc compile_object check_define main
lines: 92 122 619 691 0
x86_64-w64-mingw32-gcc -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:2:2: error: #error __ILP32__ not defined
#error __ILP32__ not defined
^~~~~
---
lines: 92 128 923 0
x86_64-w64-mingw32-gcc -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -g -liberty
/usr/lib/gcc/x86_64-w64-mingw32/8.2.0/../../../../x86_64-w64-mingw32/bin/ld: cannot find -liberty
collect2: error: ld returned 1 exit status
funcs: do_compiler do_cc compile_object main
lines: 92 122 1825 0
---
funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Werror -Wstring-plus-int -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
x86_64-w64-mingw32-gcc: error: unrecognized command line option '-Wstring-plus-int'; did you mean '-Wstrict-aliasing'?
funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Werror -Winitializer-overrides -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
x86_64-w64-mingw32-gcc: error: unrecognized command line option '-Winitializer-overrides'; did you mean '-Wno-suggest-override'?
funcs: do_compiler do_cc compile_prog cc_has_warning_flag main
lines: 92 128 1906 1910 0
---
lines: 92 122 2165 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -Werror -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c: In function 'main':
config-temp/qemu-conf.c:4:21: error: unknown conversion type character 'z' in format [-Werror=format=]
return printf("%zu", SIZE_MAX);
^
config-temp/qemu-conf.c:4:19: error: too many arguments for format [-Werror=format-extra-args]
return printf("%zu", SIZE_MAX);
^~~~~
config-temp/qemu-conf.c:4:21: error: unknown conversion type character 'z' in format [-Werror=format=]
return printf("%zu", SIZE_MAX);
^
config-temp/qemu-conf.c:4:19: error: too many arguments for format [-Werror=format-extra-args]
return printf("%zu", SIZE_MAX);
^~~~~
cc1: all warnings being treated as errors
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2177 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
config-temp/qemu-conf.c:1:10: fatal error: sys/socket.h: No such file or directory
#include <sys/socket.h>
^~~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2248 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -llzo2
config-temp/qemu-conf.c:1:10: fatal error: lzo/lzo1x.h: No such file or directory
#include <lzo/lzo1x.h>
^~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2267 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lsnappy
config-temp/qemu-conf.c:1:10: fatal error: snappy-c.h: No such file or directory
#include <snappy-c.h>
^~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2304 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -llzfse
config-temp/qemu-conf.c:1:10: fatal error: lzfse.h: No such file or directory
#include <lzfse.h>
^~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2387 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lxenstore -lxenctrl -lxenguest
config-temp/qemu-conf.c:1:10: fatal error: xenctrl.h: No such file or directory
#include <xenctrl.h>
^~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 2892 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lpam
config-temp/qemu-conf.c:1:10: fatal error: security/pam_appl.h: No such file or directory
#include <security/pam_appl.h>
^~~~~~~~~~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_object check_include main
lines: 92 122 627 2907 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -c -o config-temp/qemu-conf.o config-temp/qemu-conf.c
config-temp/qemu-conf.c:1:10: fatal error: ifaddrs.h: No such file or directory
#include <ifaddrs.h>
^~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3045 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -Dmain=SDL_main -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/SDL2 -Wno-undef -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -L/usr/x86_64-w64-mingw32/sys-root/mingw/lib -lmingw32 -lSDL2main -lSDL2 -mwindows
config-temp/qemu-conf.c:5:2: error: #error No x11 support
#error No x11 support
^~~~~
In file included from /usr/x86_64-w64-mingw32/sys-root/mingw/include/SDL2/SDL.h:32,
from config-temp/qemu-conf.c:1:
/usr/x86_64-w64-mingw32/sys-root/mingw/include/SDL2/SDL_main.h:111:17: error: conflicting types for 'SDL_main'
#define main SDL_main
^~~~~~~~
config-temp/qemu-conf.c:7:5: note: in expansion of macro 'main'
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3060 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lrdmacm -libverbs -libumad
config-temp/qemu-conf.c:1:10: fatal error: rdma/rdma_cma.h: No such file or directory
#include <rdma/rdma_cma.h>
^~~~~~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3130 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lsasl2
config-temp/qemu-conf.c:1:10: fatal error: sasl/sasl.h: No such file or directory
#include <sasl/sasl.h>
^~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3224 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
config-temp/qemu-conf.c:2:10: fatal error: xfs/xfs.h: No such file or directory
#include <xfs/xfs.h>
^~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3248 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lvdeplug
config-temp/qemu-conf.c:1:10: fatal error: libvdeplug.h: No such file or directory
#include <libvdeplug.h>
^~~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3298 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lcap-ng
config-temp/qemu-conf.c:1:10: fatal error: cap-ng.h: No such file or directory
#include <cap-ng.h>
^~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3394 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lbrlapi
config-temp/qemu-conf.c:1:10: fatal error: brlapi.h: No such file or directory
#include <brlapi.h>
^~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3436 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -DNCURSES_WIDECHAR -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
config-temp/qemu-conf.c:2:10: fatal error: curses.h: No such file or directory
#include <curses.h>
^~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3436 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -DNCURSES_WIDECHAR -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lpdcurses
config-temp/qemu-conf.c:2:10: fatal error: curses.h: No such file or directory
#include <curses.h>
^~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3489 0
x86_64-w64-mingw32-gcc -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g
config-temp/qemu-conf.c:1:10: fatal error: bluetooth/bluetooth.h: No such file or directory
#include <bluetooth/bluetooth.h>
^~~~~~~~~~~~~~~~~~~~~~~
compilation terminated.
---
funcs: do_compiler do_cc compile_prog main
lines: 92 128 3609 0
x86_64-w64-mingw32-gcc -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -mms-bitfields -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/glib-2.0 -I/usr/x86_64-w64-mingw32/sys-root/mingw/lib/glib-2.0/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -m64 -mcx16 -mthreads -D_GNU_SOURCE -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -Wstrict-prototypes -Wredundant-decls -Wall -Wundef -Wwrite-strings -Wmissing-prototypes -fno-strict-aliasing -fno-common -fwrapv -std=gnu99 -Wexpansion-to-defined -Wendif-labels -Wno-shift-negative-value -Wno-missing-include-dirs -Wempty-body -Wnested-externs -Wformat-security -Wformat-y2k -Winit-self -Wignored-qualifiers -Wold-style-declaration -Wold-style-definition -Wtype-limits -fstack-protector-strong -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/p11-kit-1 -I/usr/x86_64-w64-mingw32/sys-root/mingw/include -I/usr/x86_64-w64-mingw32/sys-root/mingw/include/libpng16 -o config-temp/qemu-conf.exe config-temp/qemu-conf.c -m64 -g -lzstd
config-temp/qemu-conf.c:1:10: fatal error: zstd.h: No such file or directory
#include <zstd.h>
^~~~~~~~
compilation terminated.
The full log is available at
http://patchew.org/logs/20190226131535.30361-1-dplotnikov@virtuozzo.com/testing.docker-mingw@fedora/?type=message.
---
Email generated automatically by Patchew [http://patchew.org/].
Please send your feedback to patchew-devel@redhat.com
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression
2019-03-04 15:10 ` [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
@ 2019-03-07 8:39 ` Denis Plotnikov
2019-03-11 8:20 ` Denis Plotnikov
1 sibling, 0 replies; 13+ messages in thread
From: Denis Plotnikov @ 2019-03-07 8:39 UTC (permalink / raw)
To: qemu-devel
ping ping
On 04.03.2019 18:10, Denis Plotnikov wrote:
> ping!
>
> On 26.02.2019 16:15, Denis Plotnikov wrote:
>> zstd date compression algorithm shows better performance on data compression.
>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>> A user will be able to choose between those algorithms, therefor compress-type
>> migration parameter is added.
>>
>> Here are some results of performance comparison zstd vs gzip:
>>
>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>> migration to the same host
>> VM: 2xVCPU, 8G RAM total
>> 5G RAM used, memory populated with postgreqsl data
>> produced by pgbench performance benchmark
>>
>>
>> Threads: 1 compress – 1 decompress
>>
>> zstd provides slightly less compression ratio with almost the same
>> CPU usage but copes with RAM compression roghly 2 times faster
>>
>> compression type zlib | zstd
>> ---------------------------------------------------------
>> compression level 1 5 | 1 5
>> compression ratio 6.92 7.05 | 6.69 6.89
>> cpu idle, % 82 83 | 86 80
>> time, sec 49 71 | 26 31
>> time diff to zlib, sec -25 -41
>>
>>
>> Threads: 8 compress – 2 decompress
>>
>> zstd provides the same migration time with less cpu consumption
>>
>> compression type none | gzip(zlib) | zstd
>> ------------------------------------------------------------------------------
>> compression level - | 1 5 9 | 1 5 15
>> compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93
>> time, sec 154 | 22 23 27 | 23 23 25
>> cpu idle, % 99 | 45 30 12 | 70 52 23
>> cpu idle diff to zlib | | -25% -22% -11%
>>
>>
>> Denis Plotnikov (3):
>> migration: rework compression code for adding more data compressors
>> hmp: add compress-type parameter to migration parameters
>> migration: add zstd compression
>>
>> configure | 26 ++++
>> hmp.c | 8 ++
>> migration/migration.c | 45 ++++++-
>> migration/migration.h | 1 +
>> migration/qemu-file.c | 39 ++----
>> migration/qemu-file.h | 18 ++-
>> migration/ram.c | 291 ++++++++++++++++++++++++++++++++++--------
>> qapi/migration.json | 26 +++-
>> 8 files changed, 369 insertions(+), 85 deletions(-)
>>
>
--
Best,
Denis
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PATCH 0/3] migration: add sztd compression
2019-03-04 15:10 ` [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
2019-03-07 8:39 ` Denis Plotnikov
@ 2019-03-11 8:20 ` Denis Plotnikov
[not found] ` <5c342066-30b6-e492-e23a-cb5accbfb11a@virtuozzo.com>
1 sibling, 1 reply; 13+ messages in thread
From: Denis Plotnikov @ 2019-03-11 8:20 UTC (permalink / raw)
To: dgilbert, quintela, eblake, armbru; +Cc: Denis Lunev, qemu-devel
ping ping ping!
On 04.03.2019 18:10, Denis Plotnikov wrote:
> ping!
>
> On 26.02.2019 16:15, Denis Plotnikov wrote:
>> zstd date compression algorithm shows better performance on data compression.
>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>> A user will be able to choose between those algorithms, therefor compress-type
>> migration parameter is added.
>>
>> Here are some results of performance comparison zstd vs gzip:
>>
>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>> migration to the same host
>> VM: 2xVCPU, 8G RAM total
>> 5G RAM used, memory populated with postgreqsl data
>> produced by pgbench performance benchmark
>>
>>
>> Threads: 1 compress – 1 decompress
>>
>> zstd provides slightly less compression ratio with almost the same
>> CPU usage but copes with RAM compression roghly 2 times faster
>>
>> compression type zlib | zstd
>> ---------------------------------------------------------
>> compression level 1 5 | 1 5
>> compression ratio 6.92 7.05 | 6.69 6.89
>> cpu idle, % 82 83 | 86 80
>> time, sec 49 71 | 26 31
>> time diff to zlib, sec -25 -41
>>
>>
>> Threads: 8 compress – 2 decompress
>>
>> zstd provides the same migration time with less cpu consumption
>>
>> compression type none | gzip(zlib) | zstd
>> ------------------------------------------------------------------------------
>> compression level - | 1 5 9 | 1 5 15
>> compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93
>> time, sec 154 | 22 23 27 | 23 23 25
>> cpu idle, % 99 | 45 30 12 | 70 52 23
>> cpu idle diff to zlib | | -25% -22% -11%
>>
>>
>> Denis Plotnikov (3):
>> migration: rework compression code for adding more data compressors
>> hmp: add compress-type parameter to migration parameters
>> migration: add zstd compression
>>
>> configure | 26 ++++
>> hmp.c | 8 ++
>> migration/migration.c | 45 ++++++-
>> migration/migration.h | 1 +
>> migration/qemu-file.c | 39 ++----
>> migration/qemu-file.h | 18 ++-
>> migration/ram.c | 291 ++++++++++++++++++++++++++++++++++--------
>> qapi/migration.json | 26 +++-
>> 8 files changed, 369 insertions(+), 85 deletions(-)
>>
>
--
Best,
Denis
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [PINGl] [PATCH 0/3] migration: add sztd compression
[not found] ` <53b5c677-870e-56a0-402d-7fdc985ec3f5@virtuozzo.com>
@ 2019-04-05 11:40 ` Denis Plotnikov
2019-04-05 11:40 ` [Qemu-devel] [PING] " Denis Plotnikov
1 sibling, 0 replies; 13+ messages in thread
From: Denis Plotnikov @ 2019-04-05 11:40 UTC (permalink / raw)
To: dgilbert, quintela, eblake, armbru; +Cc: Denis Lunev, qemu-devel
ping!
On 26.03.2019 18:51, Denis Plotnikov wrote:
> ping ping ping ping ping!
>
> On 18.03.2019 10:53, Denis Plotnikov wrote:
>> ping ping ping ping!
>>
>> On 11.03.2019 11:20, Denis Plotnikov wrote:
>>> ping ping ping!
>>>
>>> On 04.03.2019 18:10, Denis Plotnikov wrote:
>>>> ping!
>>>>
>>>> On 26.02.2019 16:15, Denis Plotnikov wrote:
>>>>> zstd date compression algorithm shows better performance on data compression.
>>>>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>>>>> A user will be able to choose between those algorithms, therefor compress-type
>>>>> migration parameter is added.
>>>>>
>>>>> Here are some results of performance comparison zstd vs gzip:
>>>>>
>>>>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>>>>> migration to the same host
>>>>> VM: 2xVCPU, 8G RAM total
>>>>> 5G RAM used, memory populated with postgreqsl data
>>>>> produced by pgbench performance benchmark
>>>>>
>>>>>
>>>>> Threads: 1 compress – 1 decompress
>>>>>
>>>>> zstd provides slightly less compression ratio with almost the same
>>>>> CPU usage but copes with RAM compression roghly 2 times faster
>>>>>
>>>>> compression type zlib | zstd
>>>>> ---------------------------------------------------------
>>>>> compression level 1 5 | 1 5
>>>>> compression ratio 6.92 7.05 | 6.69 6.89
>>>>> cpu idle, % 82 83 | 86 80
>>>>> time, sec 49 71 | 26 31
>>>>> time diff to zlib, sec -25 -41
>>>>>
>>>>>
>>>>> Threads: 8 compress – 2 decompress
>>>>>
>>>>> zstd provides the same migration time with less cpu consumption
>>>>>
>>>>> compression type none | gzip(zlib) | zstd
>>>>> ------------------------------------------------------------------------------
>>>>> compression level - | 1 5 9 | 1 5 15
>>>>> compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93
>>>>> time, sec 154 | 22 23 27 | 23 23 25
>>>>> cpu idle, % 99 | 45 30 12 | 70 52 23
>>>>> cpu idle diff to zlib | | -25% -22% -11%
>>>>>
>>>>>
>>>>> Denis Plotnikov (3):
>>>>> migration: rework compression code for adding more data compressors
>>>>> hmp: add compress-type parameter to migration parameters
>>>>> migration: add zstd compression
>>>>>
>>>>> configure | 26 ++++
>>>>> hmp.c | 8 ++
>>>>> migration/migration.c | 45 ++++++-
>>>>> migration/migration.h | 1 +
>>>>> migration/qemu-file.c | 39 ++----
>>>>> migration/qemu-file.h | 18 ++-
>>>>> migration/ram.c | 291 ++++++++++++++++++++++++++++++++++--------
>>>>> qapi/migration.json | 26 +++-
>>>>> 8 files changed, 369 insertions(+), 85 deletions(-)
>>>>>
>>>>
>>>
>>
>
--
Best,
Denis
^ permalink raw reply [flat|nested] 13+ messages in thread
* [Qemu-devel] [PING] [PATCH 0/3] migration: add sztd compression
[not found] ` <53b5c677-870e-56a0-402d-7fdc985ec3f5@virtuozzo.com>
2019-04-05 11:40 ` [Qemu-devel] [PINGl] " Denis Plotnikov
@ 2019-04-05 11:40 ` Denis Plotnikov
2019-04-25 16:17 ` [Qemu-devel] [PING PING PING] " Denis Plotnikov
1 sibling, 1 reply; 13+ messages in thread
From: Denis Plotnikov @ 2019-04-05 11:40 UTC (permalink / raw)
To: dgilbert, quintela, eblake, armbru; +Cc: Denis Lunev, qemu-devel
ping!
On 26.03.2019 18:51, Denis Plotnikov wrote:
> ping ping ping ping ping!
>
> On 18.03.2019 10:53, Denis Plotnikov wrote:
>> ping ping ping ping!
>>
>> On 11.03.2019 11:20, Denis Plotnikov wrote:
>>> ping ping ping!
>>>
>>> On 04.03.2019 18:10, Denis Plotnikov wrote:
>>>> ping!
>>>>
>>>> On 26.02.2019 16:15, Denis Plotnikov wrote:
>>>>> zstd date compression algorithm shows better performance on data compression.
>>>>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>>>>> A user will be able to choose between those algorithms, therefor compress-type
>>>>> migration parameter is added.
>>>>>
>>>>> Here are some results of performance comparison zstd vs gzip:
>>>>>
>>>>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>>>>> migration to the same host
>>>>> VM: 2xVCPU, 8G RAM total
>>>>> 5G RAM used, memory populated with postgreqsl data
>>>>> produced by pgbench performance benchmark
>>>>>
>>>>>
>>>>> Threads: 1 compress – 1 decompress
>>>>>
>>>>> zstd provides slightly less compression ratio with almost the same
>>>>> CPU usage but copes with RAM compression roghly 2 times faster
>>>>>
>>>>> compression type zlib | zstd
>>>>> ---------------------------------------------------------
>>>>> compression level 1 5 | 1 5
>>>>> compression ratio 6.92 7.05 | 6.69 6.89
>>>>> cpu idle, % 82 83 | 86 80
>>>>> time, sec 49 71 | 26 31
>>>>> time diff to zlib, sec -25 -41
>>>>>
>>>>>
>>>>> Threads: 8 compress – 2 decompress
>>>>>
>>>>> zstd provides the same migration time with less cpu consumption
>>>>>
>>>>> compression type none | gzip(zlib) | zstd
>>>>> ------------------------------------------------------------------------------
>>>>> compression level - | 1 5 9 | 1 5 15
>>>>> compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93
>>>>> time, sec 154 | 22 23 27 | 23 23 25
>>>>> cpu idle, % 99 | 45 30 12 | 70 52 23
>>>>> cpu idle diff to zlib | | -25% -22% -11%
>>>>>
>>>>>
>>>>> Denis Plotnikov (3):
>>>>> migration: rework compression code for adding more data compressors
>>>>> hmp: add compress-type parameter to migration parameters
>>>>> migration: add zstd compression
>>>>>
>>>>> configure | 26 ++++
>>>>> hmp.c | 8 ++
>>>>> migration/migration.c | 45 ++++++-
>>>>> migration/migration.h | 1 +
>>>>> migration/qemu-file.c | 39 ++----
>>>>> migration/qemu-file.h | 18 ++-
>>>>> migration/ram.c | 291 ++++++++++++++++++++++++++++++++++--------
>>>>> qapi/migration.json | 26 +++-
>>>>> 8 files changed, 369 insertions(+), 85 deletions(-)
>>>>>
>>>>
>>>
>>
>
--
Best,
Denis
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [Qemu-devel] [PING PING PING] [PATCH 0/3] migration: add sztd compression
2019-04-05 11:40 ` [Qemu-devel] [PING] " Denis Plotnikov
@ 2019-04-25 16:17 ` Denis Plotnikov
0 siblings, 0 replies; 13+ messages in thread
From: Denis Plotnikov @ 2019-04-25 16:17 UTC (permalink / raw)
To: dgilbert, quintela, eblake, armbru; +Cc: Denis Lunev, qemu-devel
On 05.04.2019 14:40, Denis Plotnikov wrote:
> ping!
>
> On 26.03.2019 18:51, Denis Plotnikov wrote:
>> ping ping ping ping ping!
>>
>> On 18.03.2019 10:53, Denis Plotnikov wrote:
>>> ping ping ping ping!
>>>
>>> On 11.03.2019 11:20, Denis Plotnikov wrote:
>>>> ping ping ping!
>>>>
>>>> On 04.03.2019 18:10, Denis Plotnikov wrote:
>>>>> ping!
>>>>>
>>>>> On 26.02.2019 16:15, Denis Plotnikov wrote:
>>>>>> zstd date compression algorithm shows better performance on data
>>>>>> compression.
>>>>>> It might be useful to employ the algorithm in VM migration to
>>>>>> reduce CPU usage.
>>>>>> A user will be able to choose between those algorithms, therefor
>>>>>> compress-type
>>>>>> migration parameter is added.
>>>>>>
>>>>>> Here are some results of performance comparison zstd vs gzip:
>>>>>>
>>>>>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>>>>>> migration to the same host
>>>>>> VM: 2xVCPU, 8G RAM total
>>>>>> 5G RAM used, memory populated with postgreqsl data
>>>>>> produced by pgbench performance benchmark
>>>>>>
>>>>>>
>>>>>> Threads: 1 compress – 1 decompress
>>>>>>
>>>>>> zstd provides slightly less compression ratio with almost the same
>>>>>> CPU usage but copes with RAM compression roghly 2 times faster
>>>>>>
>>>>>> compression type zlib | zstd
>>>>>> ---------------------------------------------------------
>>>>>> compression level 1 5 | 1 5
>>>>>> compression ratio 6.92 7.05 | 6.69 6.89
>>>>>> cpu idle, % 82 83 | 86 80
>>>>>> time, sec 49 71 | 26 31
>>>>>> time diff to zlib, sec -25 -41
>>>>>>
>>>>>>
>>>>>> Threads: 8 compress – 2 decompress
>>>>>>
>>>>>> zstd provides the same migration time with less cpu consumption
>>>>>>
>>>>>> compression type none | gzip(zlib) | zstd
>>>>>> ------------------------------------------------------------------------------
>>>>>>
>>>>>> compression level - | 1 5 9 | 1
>>>>>> 5 15
>>>>>> compression ratio - | 6.94 6.99 7.14 | 6.64
>>>>>> 6.89 6.93
>>>>>> time, sec 154 | 22 23 27 | 23
>>>>>> 23 25
>>>>>> cpu idle, % 99 | 45 30 12 | 70
>>>>>> 52 23
>>>>>> cpu idle diff to zlib | | -25%
>>>>>> -22% -11%
>>>>>>
>>>>>>
>>>>>> Denis Plotnikov (3):
>>>>>> migration: rework compression code for adding more data
>>>>>> compressors
>>>>>> hmp: add compress-type parameter to migration parameters
>>>>>> migration: add zstd compression
>>>>>>
>>>>>> configure | 26 ++++
>>>>>> hmp.c | 8 ++
>>>>>> migration/migration.c | 45 ++++++-
>>>>>> migration/migration.h | 1 +
>>>>>> migration/qemu-file.c | 39 ++----
>>>>>> migration/qemu-file.h | 18 ++-
>>>>>> migration/ram.c | 291
>>>>>> ++++++++++++++++++++++++++++++++++--------
>>>>>> qapi/migration.json | 26 +++-
>>>>>> 8 files changed, 369 insertions(+), 85 deletions(-)
>>>>>>
>>>>>
>>>>
>>>
>>
>
--
Best,
Denis
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 0/3] migration: add sztd compression
2019-02-26 13:15 [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
` (4 preceding siblings ...)
2019-03-06 11:32 ` [Qemu-devel] " no-reply
@ 2020-01-24 12:43 ` Juan Quintela
2020-01-27 7:19 ` Denis Plotnikov
5 siblings, 1 reply; 13+ messages in thread
From: Juan Quintela @ 2020-01-24 12:43 UTC (permalink / raw)
To: Denis Plotnikov; +Cc: den, qemu-devel, dgilbert, armbru
Denis Plotnikov <dplotnikov@virtuozzo.com> wrote:
> zstd date compression algorithm shows better performance on data compression.
> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
> A user will be able to choose between those algorithms, therefor compress-type
> migration parameter is added.
>
> Here are some results of performance comparison zstd vs gzip:
Please, could you comment on the series:
[PATCH v3 00/21] Multifd Migration Compression
That series integrated zstd/zlib compression on top of multifd,
advantages over "old" compression code are:
- We don't have to copy data back and forth
- The unit of compression is 512KB instead of 4kb
- We "conserve" the compression state between packets (this is specially
interesting for zstd, that uses dictionaries)
> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
> migration to the same host
> VM: 2xVCPU, 8G RAM total
> 5G RAM used, memory populated with postgreqsl data
> produced by pgbench performance benchmark
>
>
> Threads: 1 compress – 1 decompress
>
> zstd provides slightly less compression ratio with almost the same
> CPU usage but copes with RAM compression roghly 2 times faster
>
> compression type zlib | zstd
> ---------------------------------------------------------
> compression level 1 5 | 1 5
> compression ratio 6.92 7.05 | 6.69 6.89
> cpu idle, % 82 83 | 86 80
> time, sec 49 71 | 26 31
> time diff to zlib, sec -25 -41
>
>
> Threads: 8 compress – 2 decompress
>
> zstd provides the same migration time with less cpu consumption
>
> compression type none | gzip(zlib) | zstd
> ------------------------------------------------------------------------------
> compression level - | 1 5 9 | 1 5 15
> compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93
> time, sec 154 | 22 23 27 | 23 23 25
> cpu idle, % 99 | 45 30 12 | 70 52 23
> cpu idle diff to zlib | | -25% -22% -11%
I don't have handy results, but it looked for me like you:
- zstd has a way better latency than zlib (i.e. the packet cames sooner)
- And it compress much better
On the migration test (best possible case for a compressor, as we are
writting just one byte of each page, and we write the same value in all
pages):
- zlib: compress 512KB -> 2500 bytes
- zstd: compess 512KB -> 52 bytes (yeap, I tested several times, it
looked too small)
See that I posted another patch to "delete" the old compression code.
Why?
- I have been unable to modify migration-test to test it and work
reliablely (only way was to allow a really huge downtime)
- Even with slow networking (1Gigabit) I got really mixed results,
because as it is so slow, the guest continue dirtying memory, and in
my tests it was never a winner
Thanks, Juan.
^ permalink raw reply [flat|nested] 13+ messages in thread
* Re: [PATCH 0/3] migration: add sztd compression
2020-01-24 12:43 ` Juan Quintela
@ 2020-01-27 7:19 ` Denis Plotnikov
0 siblings, 0 replies; 13+ messages in thread
From: Denis Plotnikov @ 2020-01-27 7:19 UTC (permalink / raw)
To: quintela; +Cc: Denis Lunev, qemu-devel, dgilbert, armbru
Hi, Juan
I'll read the series soon. Thanks for sending that to me!
Denis
On 24.01.2020 15:43, Juan Quintela wrote:
> Denis Plotnikov <dplotnikov@virtuozzo.com> wrote:
>> zstd date compression algorithm shows better performance on data compression.
>> It might be useful to employ the algorithm in VM migration to reduce CPU usage.
>> A user will be able to choose between those algorithms, therefor compress-type
>> migration parameter is added.
>>
>> Here are some results of performance comparison zstd vs gzip:
> Please, could you comment on the series:
>
> [PATCH v3 00/21] Multifd Migration Compression
>
> That series integrated zstd/zlib compression on top of multifd,
> advantages over "old" compression code are:
> - We don't have to copy data back and forth
> - The unit of compression is 512KB instead of 4kb
> - We "conserve" the compression state between packets (this is specially
> interesting for zstd, that uses dictionaries)
>
>> host: i7-4790 8xCPU @ 3.60GHz, 16G RAM
>> migration to the same host
>> VM: 2xVCPU, 8G RAM total
>> 5G RAM used, memory populated with postgreqsl data
>> produced by pgbench performance benchmark
>>
>>
>> Threads: 1 compress – 1 decompress
>>
>> zstd provides slightly less compression ratio with almost the same
>> CPU usage but copes with RAM compression roghly 2 times faster
>>
>> compression type zlib | zstd
>> ---------------------------------------------------------
>> compression level 1 5 | 1 5
>> compression ratio 6.92 7.05 | 6.69 6.89
>> cpu idle, % 82 83 | 86 80
>> time, sec 49 71 | 26 31
>> time diff to zlib, sec -25 -41
>>
>>
>> Threads: 8 compress – 2 decompress
>>
>> zstd provides the same migration time with less cpu consumption
>>
>> compression type none | gzip(zlib) | zstd
>> ------------------------------------------------------------------------------
>> compression level - | 1 5 9 | 1 5 15
>> compression ratio - | 6.94 6.99 7.14 | 6.64 6.89 6.93
>> time, sec 154 | 22 23 27 | 23 23 25
>> cpu idle, % 99 | 45 30 12 | 70 52 23
>> cpu idle diff to zlib | | -25% -22% -11%
> I don't have handy results, but it looked for me like you:
> - zstd has a way better latency than zlib (i.e. the packet cames sooner)
> - And it compress much better
>
> On the migration test (best possible case for a compressor, as we are
> writting just one byte of each page, and we write the same value in all
> pages):
>
> - zlib: compress 512KB -> 2500 bytes
> - zstd: compess 512KB -> 52 bytes (yeap, I tested several times, it
> looked too small)
>
> See that I posted another patch to "delete" the old compression code.
> Why?
> - I have been unable to modify migration-test to test it and work
> reliablely (only way was to allow a really huge downtime)
> - Even with slow networking (1Gigabit) I got really mixed results,
> because as it is so slow, the guest continue dirtying memory, and in
> my tests it was never a winner
>
> Thanks, Juan.
>
^ permalink raw reply [flat|nested] 13+ messages in thread
end of thread, other threads:[~2020-01-27 7:19 UTC | newest]
Thread overview: 13+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-26 13:15 [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
2019-02-26 13:15 ` [Qemu-devel] [PATCH 1/3] migration: rework compression code for adding more data compressors Denis Plotnikov
2019-02-26 13:15 ` [Qemu-devel] [PATCH 2/3] hmp: add compress-type parameter to migration parameters Denis Plotnikov
2019-02-26 13:15 ` [Qemu-devel] [PATCH 3/3] migration: add zstd compression Denis Plotnikov
2019-03-04 15:10 ` [Qemu-devel] [PATCH 0/3] migration: add sztd compression Denis Plotnikov
2019-03-07 8:39 ` Denis Plotnikov
2019-03-11 8:20 ` Denis Plotnikov
[not found] ` <5c342066-30b6-e492-e23a-cb5accbfb11a@virtuozzo.com>
[not found] ` <53b5c677-870e-56a0-402d-7fdc985ec3f5@virtuozzo.com>
2019-04-05 11:40 ` [Qemu-devel] [PINGl] " Denis Plotnikov
2019-04-05 11:40 ` [Qemu-devel] [PING] " Denis Plotnikov
2019-04-25 16:17 ` [Qemu-devel] [PING PING PING] " Denis Plotnikov
2019-03-06 11:32 ` [Qemu-devel] " no-reply
2020-01-24 12:43 ` Juan Quintela
2020-01-27 7:19 ` Denis Plotnikov
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.