qemu-devel.nongnu.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v5 0/8] Multifd Migration Compression
@ 2020-01-29 11:56 Juan Quintela
  2020-01-29 11:56 ` [PATCH v5 1/8] multifd: Add multifd-method parameter Juan Quintela
                   ` (7 more replies)
  0 siblings, 8 replies; 39+ messages in thread
From: Juan Quintela @ 2020-01-29 11:56 UTC (permalink / raw)
  To: qemu-devel
  Cc: Laurent Vivier, Thomas Huth, Daniel P. Berrangé,
	Eduardo Habkost, Juan Quintela, Dr. David Alan Gilbert,
	Markus Armbruster, Paolo Bonzini

Based-on: 20200129111536.9497-1-quintela@redhat.com

[v5]
- rebased on top of latest pull request
- check zlib/zstd return codes in loops (denis suggestions)
- check that we do the right thing in error cases (I corrupted the
  stream, make buffers too small, etc by hand)

[v4]
- create new parameters: multifd-zlib-level & multifd-zstd-level
- use proper "conditionals" for qapi (thanks markus)
- more than half of the patches moved to the migration PULL request
  that this series are based on
- method type has moved to one int from a set of flags
- fixed all reviews comments

Please review.

[v3]
- rebased on top of upstream + previous multifd cancel series
- split multifd code into its own file (multifd.[ch])
- split zstd/zlib compression methods (multifd-zstd/zlib.c)
- use qemu module feauture to avoid ifdefs
  (my understanding is that zlib needs to be present, but
  we setup zstd only if it is not there or is disabled)
- multifd-method: none|zlib|zstd

  As far as I can see, there is no easy way to convince qapi that zstd
  option could/couldn't be there depending on compliation flags. I
  ended just checking in migrate_parameters_check() if it is enabled
  and giving an error message otherwise.

Questions:
- I am "reusing" the compress-level parameter for both zstd and zlib,
  but it poses a problem:
  * zlib values: 1-9 (default: 6?)
  * zstd values: 1-19 (default: 3)
So, what should I do:
  * create multifd-zstd-level and multifd-zlib-level (easier)
  * reuse compress-level, and change its maximum values depending on
    multifd-method
  * any other good option?

Please, review.

[v2] - rebase on top of previous arguments posted to the list -
introduces zlib compression - introduces zstd compression

Please help if you know anything about zstd/zlib compression.

This puts compression on top of multifd. Advantages about current
compression:

- We copy all pages in a single packet and then compress the whole
  thing.

- We reuse the compression stream for all the packets sent through the
  same channel.

- We can select nocomp/zlib/zstd levels of compression.

Please, review.

Juan Quintela (8):
  multifd: Add multifd-method parameter
  migration: Add support for modules
  multifd: Make no compression operations into its own structure
  multifd: Add multifd-zlib-level parameter
  multifd: Add zlib compression multifd support
  configure: Enable test and libs for zstd
  multifd: Add multifd-zstd-level parameter
  multifd: Add zstd compression multifd support

 configure                    |  30 ++++
 hw/core/qdev-properties.c    |  13 ++
 include/hw/qdev-properties.h |   3 +
 include/qemu/module.h        |   2 +
 migration/Makefile.objs      |   2 +
 migration/migration.c        |  70 ++++++++
 migration/migration.h        |   3 +
 migration/multifd-zlib.c     | 325 +++++++++++++++++++++++++++++++++
 migration/multifd-zstd.c     | 337 +++++++++++++++++++++++++++++++++++
 migration/multifd.c          | 191 +++++++++++++++++++-
 migration/multifd.h          |  29 +++
 migration/ram.c              |   2 +-
 monitor/hmp-cmds.c           |  21 +++
 qapi/migration.json          |  80 ++++++++-
 tests/qtest/migration-test.c |  30 +++-
 vl.c                         |   1 +
 16 files changed, 1123 insertions(+), 16 deletions(-)
 create mode 100644 migration/multifd-zlib.c
 create mode 100644 migration/multifd-zstd.c

-- 
2.24.1



^ permalink raw reply	[flat|nested] 39+ messages in thread

end of thread, other threads:[~2020-02-14 18:52 UTC | newest]

Thread overview: 39+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-01-29 11:56 [PATCH v5 0/8] Multifd Migration Compression Juan Quintela
2020-01-29 11:56 ` [PATCH v5 1/8] multifd: Add multifd-method parameter Juan Quintela
2020-01-30  7:54   ` Markus Armbruster
2020-01-30  9:11     ` Juan Quintela
2020-01-30 12:17       ` Markus Armbruster
2020-02-11 18:50   ` Daniel P. Berrangé
2020-02-13 19:29     ` Juan Quintela
2020-01-29 11:56 ` [PATCH v5 2/8] migration: Add support for modules Juan Quintela
2020-02-11 10:54   ` Dr. David Alan Gilbert
2020-02-13 19:38     ` Juan Quintela
2020-01-29 11:56 ` [PATCH v5 3/8] multifd: Make no compression operations into its own structure Juan Quintela
2020-02-07 18:45   ` Dr. David Alan Gilbert
2020-02-11 11:23     ` Juan Quintela
2020-01-29 11:56 ` [PATCH v5 4/8] multifd: Add multifd-zlib-level parameter Juan Quintela
2020-01-30  8:03   ` Markus Armbruster
2020-01-30  8:56     ` Juan Quintela
2020-02-11 18:57     ` Daniel P. Berrangé
2020-02-13 13:27       ` Markus Armbruster
2020-02-13 16:33       ` Juan Quintela
2020-01-29 11:56 ` [PATCH v5 5/8] multifd: Add zlib compression multifd support Juan Quintela
2020-01-30  8:04   ` Markus Armbruster
2020-02-11 18:43   ` Dr. David Alan Gilbert
2020-02-13 20:24     ` Juan Quintela
2020-01-29 11:56 ` [PATCH v5 6/8] configure: Enable test and libs for zstd Juan Quintela
2020-02-11 20:11   ` Daniel P. Berrangé
2020-02-13 21:08     ` Juan Quintela
2020-02-14 10:26       ` Daniel P. Berrangé
2020-01-29 11:56 ` [PATCH v5 7/8] multifd: Add multifd-zstd-level parameter Juan Quintela
2020-01-30  8:08   ` Markus Armbruster
2020-02-11 18:47   ` Dr. David Alan Gilbert
2020-02-13 14:04     ` Markus Armbruster
2020-02-13 14:28       ` Dr. David Alan Gilbert
2020-02-13 15:33       ` Juan Quintela
2020-02-14  8:49         ` Markus Armbruster
2020-02-14 18:50           ` Dr. David Alan Gilbert
2020-01-29 11:56 ` [PATCH v5 8/8] multifd: Add zstd compression multifd support Juan Quintela
2020-01-30  8:08   ` Markus Armbruster
2020-02-11 20:01   ` Dr. David Alan Gilbert
2020-02-13 20:39     ` Juan Quintela

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).