From: Juan Quintela <quintela@redhat.com>
To: qemu-devel@nongnu.org
Cc: "Laurent Vivier" <lvivier@redhat.com>,
"Thomas Huth" <thuth@redhat.com>,
"Daniel P. Berrangé" <berrange@redhat.com>,
"Eduardo Habkost" <ehabkost@redhat.com>,
"Juan Quintela" <quintela@redhat.com>,
"Dr. David Alan Gilbert" <dgilbert@redhat.com>,
"Markus Armbruster" <armbru@redhat.com>,
"Paolo Bonzini" <pbonzini@redhat.com>
Subject: [PATCH v5 0/8] Multifd Migration Compression
Date: Wed, 29 Jan 2020 12:56:47 +0100 [thread overview]
Message-ID: <20200129115655.10414-1-quintela@redhat.com> (raw)
Based-on: 20200129111536.9497-1-quintela@redhat.com
[v5]
- rebased on top of latest pull request
- check zlib/zstd return codes in loops (denis suggestions)
- check that we do the right thing in error cases (I corrupted the
stream, make buffers too small, etc by hand)
[v4]
- create new parameters: multifd-zlib-level & multifd-zstd-level
- use proper "conditionals" for qapi (thanks markus)
- more than half of the patches moved to the migration PULL request
that this series are based on
- method type has moved to one int from a set of flags
- fixed all reviews comments
Please review.
[v3]
- rebased on top of upstream + previous multifd cancel series
- split multifd code into its own file (multifd.[ch])
- split zstd/zlib compression methods (multifd-zstd/zlib.c)
- use qemu module feauture to avoid ifdefs
(my understanding is that zlib needs to be present, but
we setup zstd only if it is not there or is disabled)
- multifd-method: none|zlib|zstd
As far as I can see, there is no easy way to convince qapi that zstd
option could/couldn't be there depending on compliation flags. I
ended just checking in migrate_parameters_check() if it is enabled
and giving an error message otherwise.
Questions:
- I am "reusing" the compress-level parameter for both zstd and zlib,
but it poses a problem:
* zlib values: 1-9 (default: 6?)
* zstd values: 1-19 (default: 3)
So, what should I do:
* create multifd-zstd-level and multifd-zlib-level (easier)
* reuse compress-level, and change its maximum values depending on
multifd-method
* any other good option?
Please, review.
[v2] - rebase on top of previous arguments posted to the list -
introduces zlib compression - introduces zstd compression
Please help if you know anything about zstd/zlib compression.
This puts compression on top of multifd. Advantages about current
compression:
- We copy all pages in a single packet and then compress the whole
thing.
- We reuse the compression stream for all the packets sent through the
same channel.
- We can select nocomp/zlib/zstd levels of compression.
Please, review.
Juan Quintela (8):
multifd: Add multifd-method parameter
migration: Add support for modules
multifd: Make no compression operations into its own structure
multifd: Add multifd-zlib-level parameter
multifd: Add zlib compression multifd support
configure: Enable test and libs for zstd
multifd: Add multifd-zstd-level parameter
multifd: Add zstd compression multifd support
configure | 30 ++++
hw/core/qdev-properties.c | 13 ++
include/hw/qdev-properties.h | 3 +
include/qemu/module.h | 2 +
migration/Makefile.objs | 2 +
migration/migration.c | 70 ++++++++
migration/migration.h | 3 +
migration/multifd-zlib.c | 325 +++++++++++++++++++++++++++++++++
migration/multifd-zstd.c | 337 +++++++++++++++++++++++++++++++++++
migration/multifd.c | 191 +++++++++++++++++++-
migration/multifd.h | 29 +++
migration/ram.c | 2 +-
monitor/hmp-cmds.c | 21 +++
qapi/migration.json | 80 ++++++++-
tests/qtest/migration-test.c | 30 +++-
vl.c | 1 +
16 files changed, 1123 insertions(+), 16 deletions(-)
create mode 100644 migration/multifd-zlib.c
create mode 100644 migration/multifd-zstd.c
--
2.24.1
next reply other threads:[~2020-01-29 11:57 UTC|newest]
Thread overview: 39+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-01-29 11:56 Juan Quintela [this message]
2020-01-29 11:56 ` [PATCH v5 1/8] multifd: Add multifd-method parameter Juan Quintela
2020-01-30 7:54 ` Markus Armbruster
2020-01-30 9:11 ` Juan Quintela
2020-01-30 12:17 ` Markus Armbruster
2020-02-11 18:50 ` Daniel P. Berrangé
2020-02-13 19:29 ` Juan Quintela
2020-01-29 11:56 ` [PATCH v5 2/8] migration: Add support for modules Juan Quintela
2020-02-11 10:54 ` Dr. David Alan Gilbert
2020-02-13 19:38 ` Juan Quintela
2020-01-29 11:56 ` [PATCH v5 3/8] multifd: Make no compression operations into its own structure Juan Quintela
2020-02-07 18:45 ` Dr. David Alan Gilbert
2020-02-11 11:23 ` Juan Quintela
2020-01-29 11:56 ` [PATCH v5 4/8] multifd: Add multifd-zlib-level parameter Juan Quintela
2020-01-30 8:03 ` Markus Armbruster
2020-01-30 8:56 ` Juan Quintela
2020-02-11 18:57 ` Daniel P. Berrangé
2020-02-13 13:27 ` Markus Armbruster
2020-02-13 16:33 ` Juan Quintela
2020-01-29 11:56 ` [PATCH v5 5/8] multifd: Add zlib compression multifd support Juan Quintela
2020-01-30 8:04 ` Markus Armbruster
2020-02-11 18:43 ` Dr. David Alan Gilbert
2020-02-13 20:24 ` Juan Quintela
2020-01-29 11:56 ` [PATCH v5 6/8] configure: Enable test and libs for zstd Juan Quintela
2020-02-11 20:11 ` Daniel P. Berrangé
2020-02-13 21:08 ` Juan Quintela
2020-02-14 10:26 ` Daniel P. Berrangé
2020-01-29 11:56 ` [PATCH v5 7/8] multifd: Add multifd-zstd-level parameter Juan Quintela
2020-01-30 8:08 ` Markus Armbruster
2020-02-11 18:47 ` Dr. David Alan Gilbert
2020-02-13 14:04 ` Markus Armbruster
2020-02-13 14:28 ` Dr. David Alan Gilbert
2020-02-13 15:33 ` Juan Quintela
2020-02-14 8:49 ` Markus Armbruster
2020-02-14 18:50 ` Dr. David Alan Gilbert
2020-01-29 11:56 ` [PATCH v5 8/8] multifd: Add zstd compression multifd support Juan Quintela
2020-01-30 8:08 ` Markus Armbruster
2020-02-11 20:01 ` Dr. David Alan Gilbert
2020-02-13 20:39 ` Juan Quintela
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20200129115655.10414-1-quintela@redhat.com \
--to=quintela@redhat.com \
--cc=armbru@redhat.com \
--cc=berrange@redhat.com \
--cc=dgilbert@redhat.com \
--cc=ehabkost@redhat.com \
--cc=lvivier@redhat.com \
--cc=pbonzini@redhat.com \
--cc=qemu-devel@nongnu.org \
--cc=thuth@redhat.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).