* [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration
@ 2018-02-07 15:58 Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 01/12] block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor() Vladimir Sementsov-Ogievskiy
` (14 more replies)
0 siblings, 15 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Hi all!
There is a new version of dirty bitmap postcopy migration series.
Now it is based on Max's block tree: https://github.com/XanClic/qemu/commits/block,
where it needs only one patch: "block: maintain persistent disabled bitmaps",
but I hope it is near to be merged.
v10
clone: tag postcopy-v10 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v10
01,02: r-b Fam
03: adjust comments about locking
04: fixed 124 iotest (was broken because of small mistake in block/dirty-bitmap.c)
05: rebased on master, staff from migration_thread is moved to migration_iteration_run, so
drop r-b by John and Juan
06: 2.11->2.12, r-b Fam
07,08,09,: r-b Fam
10: move to device names instead of node names, looks like libvirt don't care about
same node-names.
flag AUTOLOAD is ignored for now
use QEMU_ALIGN_UP and DIV_ROUND_UP
skip automatically inserted nodes, when search for dirty bitmaps
allow migration of no bitmaps (see in dirty_bitmap_load_header new logic
with nothing variable, which avoids extra errors)
handle return code of dirty_bitmap_load_header
avoid iteration if there are no bitmaps (see new .no_bitmaps field of
dirty_bitmap_mig_state)
call dirty_bitmap_mig_before_vm_start from process_incoming_migration_bh too,
to enable bitmaps in case of postcopy not actually started.
11: not add r-b Fam
tiny reorganisation of do_test_migration parameters: remove useless default
values and make shared_storage to be the last
disable shared storage test for now, until it will be fixed (it will be separate
series, more related to qcow2 than to migration)
12: r-b Fam
also, "iotests: add default node-name" is dropped, as not more needed.
v9
clone: tag postcopy-v9 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v9
01: r-b John
02: was incomplete, now add here bdrv_reclaim_dirty_bitmap fix
03: new
04: new
05: r-b John
07: fix type in commit message, r-b John
09: add comment about is_active_iterate, r-b Snow and keep Juan's r-b, hope comment is ok
10: change copyright to Virtuozzo
reword comment at the top of the file
rewrite init_dirty_bitmap_migration, to not do same things twice (John)
and skip _only_ unnamed bitmaps, error out for unnamed nodes (John)
use new "locked" state of bitmaps instead of frozen on source vm
do not support migrating bitmap to existent one with the same name,
keep only create-new-bitmap way
break loop in dirty_bitmap_load_complete when bitmap is found
use bitmap locking instead of context acquire
12: rewrite, to add more cases. (note, that 169 iotest is also in my
"[PATCH v2 0/3] fix bitmaps migration through shared storage", which probably should
go to qemu-stable. So this patch should rewrite it, but here I make it like new patch,
to simplify review. When "[PATCH v2..." merged I'll rebase this on it), drop r-b
13: move to separate test, drop r-b
v8.1
clone: tag postcopy-v8.1 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v8.1
05: fix compilation, add new version for cmma_save_pending too.
v8
clone: tag postcopy-v8 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v8
- rebased on master
- patches 01-03 from v7 are already merged to master
- patch order is changed to make it possible to merge block/dirty-bitmap patches
in separate if is needed
01: new patch
03: fixed to use _locked version of bdrv_release_dirty_bitmap
06: qapi-schema.json -> qapi/migration.json
2.9 -> 2.11
10: protocol changed a bit:
instead of 1 byte "bitmap enabled flag" this byte becomes just "flags"
and have "enabled", "persistent" and "autoloading" flags inside.
also, make all migrated bitmaps to be not persistent (to prevent their
storing on source vm)
14: new patch
patches status:
01-04 - are only about block/dirty-bitmap and have no r-b. Fam, John, Paolo (about bitmap lock),
please look at. These patches are ok to be merged in separate (but before 05-14)
other patches are about migration
05-09 has Juan's r-b (and some of them has John's and Eric's r-bs)
10 - the main patch (dirty bitmaps migration), has no r-b.
11 - preparation for tests, not related to migration directly, has Max's r-b, ok to be merged
separately (but before 12-14)
12-14 - tests, 12 and 13 have Max's r-b, 14 is new
v7
clone: tag postcopy-v7 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v7
- rebased on dirty-bitmap byte-based interfaces
(based on git://repo.or.cz/qemu/ericb.git branch nbd-byte-dirty-v4)
- migration of persistent bitmaps should fail for shared storage migration for now,
as persistent dirty bitmaps are stored/load on inactivate/invalidate-cache.
also, even for non-shared storage migration there would be useless saving of dirty
bitmaps on source. This all will be optimized later.
01: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
03: some structural changes due to rebase - drop r-b
04: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
staff from include/sysemu/sysemu.h moved to migration/savevm.h (rebase)
05: fix patch header: block -> block/dirty-bitmap
add locking, drop r-b
06: staff from include/migration/migration.h moved to migration/migration.h (rebase)
07: add locking, drop r-b
09: staff from include/migration/qemu-file.h moved to migration/qemu-file.h (rebase)
10: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
11: new patch
12: a lot of changes/fixes (mostly by Fam's comments) + rebase
header-definition movement
remove include <assert.h>
add some includes
fix/refactor bitmap flags send
byte-based interface for dirty bitmaps (rebase)
froze bitmaps on source
init_dirty_bitmap_migration can return error, if some of bitmaps are already
frozen
bdrv_ref drives with bitmaps
fprintf -> error_report
check version_id in _load function
v6:
clone: tag postcopy-v6 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v6
rebase on master.
03 - tiny contextual change
12 - little change, but it should be reviewed. Call of init_dirty_bitmap_incoming_migration()
(which only initialize mutex) moved from start of process_incoming_migration_co (it was
immediately after "mis = migration_incoming_state_new(f)") to migration_incoming_get_current()
to stay with initialization code.
I remove r-b's, but hope that this will not be a problem. The only change in this patch - is moved
call of init_dirty_bitmap_incoming_migration.
I do so because of recent
commit b4b076daf324894dd288cbdb67ff1e3c7434df7b
Author: Juan Quintela <quintela@redhat.com>
Date: Mon Jan 23 22:32:06 2017 +0100
migration: create Migration Incoming State at init time
15 - add Max's r-b
v5:
clone: tag postcopy-v5 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v5
- move 'signed-off' over 'reviewed-by' in patches.
03,04 - add comments. Hope they will be ok for you, so add Juan's r-b.
If not ok - let me know and I'll resend.
06,08,12 - add Max's r-b
07,09,10,11,12 - add Juan's r-b
14 - used last version of this patch from qcow2-bitmap series with
Max's r-b. It has contextual changes due to different base.
15 - fix 041 iotest, add default node-name only if path is specified and
node-name is not specified
16 - handle whitespaces
s/"exec: cat " + fifo/"exec: cat '" + fifo + "'"/
fix indentation
add Max's r-b
17 - fix typos, wrong size in comment, s/md5/sha256/
add Max's r-b
v4:
clone: tag postcopy-v4 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v4
reroll, to fix "failed automatic build test"
rebase on master!
07: since 2.8 -> since 2.9
14: fix build of test-hbitmap
since 2.8 -> since 2.9 and small rewording of comment (this change was not done for
same patch in may parallels series about qcow2 bitmap extension)
v3:
rebased on Max's block branch: https://github.com/XanClic/qemu/commits/block
clone: tag postcopy-v3 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v3
01 - r-b by Juan
02 - r-b by Juan and David
04 - fix indent
07 - s/since 2.6/since 2.8/
10 - change variable name s/buf/str/
12 - improve copyright message and move it up
fix memory loss (thanks to Juan)
switch from DPRINTF to trace events
14* - switch to sha256 and qcrypto_hash_*
separate qmp command x-debug-block-dirty-bitmap-sha256
16 - use path_suffix for multi-vm test
fix indent
fix copyright
use x-debug-block-dirty-bitmap-sha256 instead of md5
17 - use x-debug-block-dirty-bitmap-sha256 instead of md5
remove not existing 170 test from qemu-iotests/group
*Note: patch 14 is also used in my second series 'qcow2 persistent dirty bitmaps'
v2:
some bugs fixed, iotests a bit changed and merged into one test.
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2
v1:
These series are derived from my 'Dirty bitmaps migration' series. The
core idea is switch to postcopy migration and drop usage of meta
bitmaps.
These patches provide dirty bitmap postcopy migration feature. Only
named dirty bitmaps are to be migrated. Migration may be enabled using
migration capabilities.
The overall method (thanks to John Snow):
1. migrate bitmaps meta data in .save_live_setup
- create/find related bitmaps on target
- disable them
- create successors (anonimous children) only for enabled migrated
bitmaps
2. do nothing in precopy stage
3. just before target vm start: enable successors, created in (1)
4. migrate bitmap data
5. reclaime bitmaps (merge successors to their parents)
6. enable bitmaps (only bitmaps, which was enabled in source)
Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
(DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
and I'll drop them in the following version.
So, relatively to last DBMv7:
01-04: new patches, splitting common postcopy migration out of ram
postcopy migration
05: equal to DBMv7.05
06: new
07: equal to DBMv7.06
08: new
09: equal to DBMv7.07
10: new
11: derived from DBMv7.08, see below
12-15: equal to DBMv7.09-12
16: derived from DBMv7.13
- switch from fifo to socket, as postcopy don't work with fifo
for now
- change parameters: size, granularity, regions
- add time.sleep, to wait for postcopy migration phase (bad
temporary solution.
- drop Reviewed-by
17: new
11: the core patch of the series, it is derived from
[DBMv7.08: migration: add migration_block-dirty-bitmap.c]
There are a lot of changes related to switching from precopy to
postcopy, but functions related to migration stream itself
(structs, send/load sequences) are mostly unchnaged.
So, changes, to switch from precopy to postcopy:
- removed all staff related to meta bitmaps and dirty phase!!!
- add dirty_bitmap_mig_enable_successors, and call it before
target vm start in loadvm_postcopy_handle_run
- add enabled_bitmaps list of bitmaps for
dirty_bitmap_mig_enable_successors
- enabled flag is send with start bitmap chunk instead of
completion chunk
- sectors_per_chunk is calculated directly from CHUNK_SIZE, not
using meta bitmap granularity
- dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
to postcopy stage
- dirty_bitmap_save_pending: remove dirty phase related pending,
switch pending to non-postcopyable
- dirty_bitmap_load_start: get enabled flag and prepare
successors for enabled bitmaps, also add them to
enabled_bitmaps list
- dirty_bitmap_load_complete: for enabled bitmaps: merge them
with successors and enable
- savevm handlers:
* remove separate savevm_dirty_bitmap_live_iterate_handlers state
(it was bad idea, any way), and move its save_live_iterate to
savevm_dirty_bitmap_handlers
* add is_active_iterate savevm handler, which allows iterations
only in postcopy stage (after stopping source vm)
* add has_postcopy savevm handler. (ofcourse, just returning true)
* use save_live_complete_postcopy instead of
save_live_complete_precopy
Other changes:
- some debug output changed
- remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
was needed to omit iterations if bitmap data is small, possibly
this should be reimplemented)
Vladimir Sementsov-Ogievskiy (12):
block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor()
block/dirty-bitmap: fix locking in bdrv_reclaim_dirty_bitmap
block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap
dirty-bitmap: add locked state
migration: introduce postcopy-only pending
qapi: add dirty-bitmaps migration capability
migration: include migrate_dirty_bitmaps in migrate_postcopy
migration/qemu-file: add qemu_put_counted_string()
migration: add is_active_iterate handler
migration: add postcopy migration of dirty bitmaps
iotests: add dirty bitmap migration test
iotests: add dirty bitmap postcopy test
qapi/block-core.json | 5 +-
qapi/migration.json | 6 +-
include/block/dirty-bitmap.h | 7 +
include/migration/misc.h | 3 +
include/migration/register.h | 26 +-
migration/migration.h | 4 +
migration/qemu-file.h | 2 +
migration/savevm.h | 5 +-
block/dirty-bitmap.c | 123 +++++--
blockdev.c | 19 ++
hw/s390x/s390-stattrib.c | 7 +-
migration/block-dirty-bitmap.c | 737 +++++++++++++++++++++++++++++++++++++++++
migration/block.c | 7 +-
migration/migration.c | 32 +-
migration/qemu-file.c | 13 +
migration/ram.c | 9 +-
migration/savevm.c | 20 +-
vl.c | 1 +
migration/Makefile.objs | 1 +
migration/trace-events | 16 +-
tests/qemu-iotests/169 | 141 ++++++++
tests/qemu-iotests/169.out | 5 +
tests/qemu-iotests/199 | 105 ++++++
tests/qemu-iotests/199.out | 5 +
tests/qemu-iotests/group | 2 +
tests/qemu-iotests/iotests.py | 7 +-
26 files changed, 1242 insertions(+), 66 deletions(-)
create mode 100644 migration/block-dirty-bitmap.c
create mode 100755 tests/qemu-iotests/169
create mode 100644 tests/qemu-iotests/169.out
create mode 100755 tests/qemu-iotests/199
create mode 100644 tests/qemu-iotests/199.out
--
2.11.1
^ permalink raw reply [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 01/12] block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor()
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 02/12] block/dirty-bitmap: fix locking in bdrv_reclaim_dirty_bitmap Vladimir Sementsov-Ogievskiy
` (13 subsequent siblings)
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Enabling bitmap successor is necessary to enable successors of bitmaps
being migrated before target vm start.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
---
include/block/dirty-bitmap.h | 1 +
block/dirty-bitmap.c | 8 ++++++++
2 files changed, 9 insertions(+)
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 89dc50946b..93e128f81b 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -20,6 +20,7 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap,
Error **errp);
+void bdrv_dirty_bitmap_enable_successor(BdrvDirtyBitmap *bitmap);
BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
const char *name);
void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap);
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 909f0517f8..0d0e807216 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -234,6 +234,14 @@ int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
return 0;
}
+/* Called with BQL taken. */
+void bdrv_dirty_bitmap_enable_successor(BdrvDirtyBitmap *bitmap)
+{
+ qemu_mutex_lock(bitmap->mutex);
+ bdrv_enable_dirty_bitmap(bitmap->successor);
+ qemu_mutex_unlock(bitmap->mutex);
+}
+
/**
* For a bitmap with a successor, yield our name to the successor,
* delete the old bitmap, and return a handle to the new bitmap.
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 02/12] block/dirty-bitmap: fix locking in bdrv_reclaim_dirty_bitmap
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 01/12] block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor() Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 03/12] block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap Vladimir Sementsov-Ogievskiy
` (12 subsequent siblings)
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Like other setters here these functions should take a lock.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
---
block/dirty-bitmap.c | 85 ++++++++++++++++++++++++++++++++--------------------
1 file changed, 53 insertions(+), 32 deletions(-)
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 0d0e807216..75435f6c2f 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -242,6 +242,51 @@ void bdrv_dirty_bitmap_enable_successor(BdrvDirtyBitmap *bitmap)
qemu_mutex_unlock(bitmap->mutex);
}
+/* Called within bdrv_dirty_bitmap_lock..unlock */
+static void bdrv_do_release_matching_dirty_bitmap_locked(
+ BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+ bool (*cond)(BdrvDirtyBitmap *bitmap))
+{
+ BdrvDirtyBitmap *bm, *next;
+
+ QLIST_FOREACH_SAFE(bm, &bs->dirty_bitmaps, list, next) {
+ if ((!bitmap || bm == bitmap) && (!cond || cond(bm))) {
+ assert(!bm->active_iterators);
+ assert(!bdrv_dirty_bitmap_frozen(bm));
+ assert(!bm->meta);
+ QLIST_REMOVE(bm, list);
+ hbitmap_free(bm->bitmap);
+ g_free(bm->name);
+ g_free(bm);
+
+ if (bitmap) {
+ return;
+ }
+ }
+ }
+
+ if (bitmap) {
+ abort();
+ }
+}
+
+/* Called with BQL taken. */
+static void bdrv_do_release_matching_dirty_bitmap(
+ BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
+ bool (*cond)(BdrvDirtyBitmap *bitmap))
+{
+ bdrv_dirty_bitmaps_lock(bs);
+ bdrv_do_release_matching_dirty_bitmap_locked(bs, bitmap, cond);
+ bdrv_dirty_bitmaps_unlock(bs);
+}
+
+/* Called within bdrv_dirty_bitmap_lock..unlock */
+static void bdrv_release_dirty_bitmap_locked(BlockDriverState *bs,
+ BdrvDirtyBitmap *bitmap)
+{
+ bdrv_do_release_matching_dirty_bitmap_locked(bs, bitmap, NULL);
+}
+
/**
* For a bitmap with a successor, yield our name to the successor,
* delete the old bitmap, and return a handle to the new bitmap.
@@ -281,7 +326,11 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
BdrvDirtyBitmap *parent,
Error **errp)
{
- BdrvDirtyBitmap *successor = parent->successor;
+ BdrvDirtyBitmap *successor;
+
+ qemu_mutex_lock(parent->mutex);
+
+ successor = parent->successor;
if (!successor) {
error_setg(errp, "Cannot reclaim a successor when none is present");
@@ -292,9 +341,11 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
error_setg(errp, "Merging of parent and successor bitmap failed");
return NULL;
}
- bdrv_release_dirty_bitmap(bs, successor);
+ bdrv_release_dirty_bitmap_locked(bs, successor);
parent->successor = NULL;
+ qemu_mutex_unlock(parent->mutex);
+
return parent;
}
@@ -322,36 +373,6 @@ static bool bdrv_dirty_bitmap_has_name(BdrvDirtyBitmap *bitmap)
}
/* Called with BQL taken. */
-static void bdrv_do_release_matching_dirty_bitmap(
- BlockDriverState *bs, BdrvDirtyBitmap *bitmap,
- bool (*cond)(BdrvDirtyBitmap *bitmap))
-{
- BdrvDirtyBitmap *bm, *next;
- bdrv_dirty_bitmaps_lock(bs);
- QLIST_FOREACH_SAFE(bm, &bs->dirty_bitmaps, list, next) {
- if ((!bitmap || bm == bitmap) && (!cond || cond(bm))) {
- assert(!bm->active_iterators);
- assert(!bdrv_dirty_bitmap_frozen(bm));
- assert(!bm->meta);
- QLIST_REMOVE(bm, list);
- hbitmap_free(bm->bitmap);
- g_free(bm->name);
- g_free(bm);
-
- if (bitmap) {
- goto out;
- }
- }
- }
- if (bitmap) {
- abort();
- }
-
-out:
- bdrv_dirty_bitmaps_unlock(bs);
-}
-
-/* Called with BQL taken. */
void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
{
bdrv_do_release_matching_dirty_bitmap(bs, bitmap, NULL);
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 03/12] block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 01/12] block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor() Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 02/12] block/dirty-bitmap: fix locking in bdrv_reclaim_dirty_bitmap Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 04/12] dirty-bitmap: add locked state Vladimir Sementsov-Ogievskiy
` (11 subsequent siblings)
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
include/block/dirty-bitmap.h | 3 +++
block/dirty-bitmap.c | 28 ++++++++++++++++++----------
2 files changed, 21 insertions(+), 10 deletions(-)
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 93e128f81b..b2e1fd913a 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -92,5 +92,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap);
char *bdrv_dirty_bitmap_sha256(const BdrvDirtyBitmap *bitmap, Error **errp);
int64_t bdrv_dirty_bitmap_next_zero(BdrvDirtyBitmap *bitmap, uint64_t start);
+BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap_locked(BlockDriverState *bs,
+ BdrvDirtyBitmap *bitmap,
+ Error **errp);
#endif
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 75435f6c2f..ce00ff3474 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -320,17 +320,13 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
* In cases of failure where we can no longer safely delete the parent,
* we may wish to re-join the parent and child/successor.
* The merged parent will be un-frozen, but not explicitly re-enabled.
- * Called with BQL taken.
+ * Called within bdrv_dirty_bitmap_lock..unlock and with BQL taken.
*/
-BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
- BdrvDirtyBitmap *parent,
- Error **errp)
+BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap_locked(BlockDriverState *bs,
+ BdrvDirtyBitmap *parent,
+ Error **errp)
{
- BdrvDirtyBitmap *successor;
-
- qemu_mutex_lock(parent->mutex);
-
- successor = parent->successor;
+ BdrvDirtyBitmap *successor = parent->successor;
if (!successor) {
error_setg(errp, "Cannot reclaim a successor when none is present");
@@ -344,9 +340,21 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
bdrv_release_dirty_bitmap_locked(bs, successor);
parent->successor = NULL;
+ return parent;
+}
+
+/* Called with BQL taken. */
+BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
+ BdrvDirtyBitmap *parent,
+ Error **errp)
+{
+ BdrvDirtyBitmap *ret;
+
+ qemu_mutex_lock(parent->mutex);
+ ret = bdrv_reclaim_dirty_bitmap_locked(bs, parent, errp);
qemu_mutex_unlock(parent->mutex);
- return parent;
+ return ret;
}
/**
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 04/12] dirty-bitmap: add locked state
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (2 preceding siblings ...)
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 03/12] block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending Vladimir Sementsov-Ogievskiy
` (10 subsequent siblings)
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Add special state, when qmp operations on the bitmap are disabled.
It is needed during bitmap migration. "Frozen" state is not
appropriate here, because it looks like bitmap is unchanged.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
qapi/block-core.json | 5 ++++-
include/block/dirty-bitmap.h | 3 +++
block/dirty-bitmap.c | 16 ++++++++++++++++
blockdev.c | 19 +++++++++++++++++++
4 files changed, 42 insertions(+), 1 deletion(-)
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 1d44c439e9..236586a5e1 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -426,10 +426,13 @@
# @active: The bitmap is actively monitoring for new writes, and can be cleared,
# deleted, or used for backup operations.
#
+# @locked: The bitmap is currently in-use by some operation and can not be
+# cleared, deleted, or used for backup operations. (Since 2.12)
+#
# Since: 2.4
##
{ 'enum': 'DirtyBitmapStatus',
- 'data': ['active', 'disabled', 'frozen'] }
+ 'data': ['active', 'disabled', 'frozen', 'locked'] }
##
# @BlockDirtyInfo:
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index b2e1fd913a..9daaf38897 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -68,6 +68,8 @@ void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
void bdrv_dirty_bitmap_set_readonly(BdrvDirtyBitmap *bitmap, bool value);
void bdrv_dirty_bitmap_set_persistance(BdrvDirtyBitmap *bitmap,
bool persistent);
+void bdrv_dirty_bitmap_set_qmp_locked(BdrvDirtyBitmap *bitmap, bool qmp_locked);
+
/* Functions that require manual locking. */
void bdrv_dirty_bitmap_lock(BdrvDirtyBitmap *bitmap);
@@ -87,6 +89,7 @@ bool bdrv_dirty_bitmap_readonly(const BdrvDirtyBitmap *bitmap);
bool bdrv_has_readonly_bitmaps(BlockDriverState *bs);
bool bdrv_dirty_bitmap_get_autoload(const BdrvDirtyBitmap *bitmap);
bool bdrv_dirty_bitmap_get_persistance(BdrvDirtyBitmap *bitmap);
+bool bdrv_dirty_bitmap_qmp_locked(BdrvDirtyBitmap *bitmap);
bool bdrv_has_changed_persistent_bitmaps(BlockDriverState *bs);
BdrvDirtyBitmap *bdrv_dirty_bitmap_next(BlockDriverState *bs,
BdrvDirtyBitmap *bitmap);
diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index ce00ff3474..221bcb9996 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -40,6 +40,8 @@ struct BdrvDirtyBitmap {
QemuMutex *mutex;
HBitmap *bitmap; /* Dirty bitmap implementation */
HBitmap *meta; /* Meta dirty bitmap */
+ bool qmp_locked; /* Bitmap is frozen, it can't be modified
+ through QMP */
BdrvDirtyBitmap *successor; /* Anonymous child; implies frozen status */
char *name; /* Optional non-empty unique ID */
int64_t size; /* Size of the bitmap, in bytes */
@@ -183,6 +185,18 @@ bool bdrv_dirty_bitmap_frozen(BdrvDirtyBitmap *bitmap)
return bitmap->successor;
}
+void bdrv_dirty_bitmap_set_qmp_locked(BdrvDirtyBitmap *bitmap, bool qmp_locked)
+{
+ qemu_mutex_lock(bitmap->mutex);
+ bitmap->qmp_locked = qmp_locked;
+ qemu_mutex_unlock(bitmap->mutex);
+}
+
+bool bdrv_dirty_bitmap_qmp_locked(BdrvDirtyBitmap *bitmap)
+{
+ return bitmap->qmp_locked;
+}
+
/* Called with BQL taken. */
bool bdrv_dirty_bitmap_enabled(BdrvDirtyBitmap *bitmap)
{
@@ -194,6 +208,8 @@ DirtyBitmapStatus bdrv_dirty_bitmap_status(BdrvDirtyBitmap *bitmap)
{
if (bdrv_dirty_bitmap_frozen(bitmap)) {
return DIRTY_BITMAP_STATUS_FROZEN;
+ } else if (bdrv_dirty_bitmap_qmp_locked(bitmap)) {
+ return DIRTY_BITMAP_STATUS_LOCKED;
} else if (!bdrv_dirty_bitmap_enabled(bitmap)) {
return DIRTY_BITMAP_STATUS_DISABLED;
} else {
diff --git a/blockdev.c b/blockdev.c
index a1dc39345a..2cbe9ff8ea 100644
--- a/blockdev.c
+++ b/blockdev.c
@@ -2113,6 +2113,9 @@ static void block_dirty_bitmap_clear_prepare(BlkActionState *common,
if (bdrv_dirty_bitmap_frozen(state->bitmap)) {
error_setg(errp, "Cannot modify a frozen bitmap");
return;
+ } else if (bdrv_dirty_bitmap_qmp_locked(state->bitmap)) {
+ error_setg(errp, "Cannot modify a locked bitmap");
+ return;
} else if (!bdrv_dirty_bitmap_enabled(state->bitmap)) {
error_setg(errp, "Cannot clear a disabled bitmap");
return;
@@ -2857,6 +2860,11 @@ void qmp_block_dirty_bitmap_remove(const char *node, const char *name,
"Bitmap '%s' is currently frozen and cannot be removed",
name);
return;
+ } else if (bdrv_dirty_bitmap_qmp_locked(bitmap)) {
+ error_setg(errp,
+ "Bitmap '%s' is currently locked and cannot be removed",
+ name);
+ return;
}
if (bdrv_dirty_bitmap_get_persistance(bitmap)) {
@@ -2891,6 +2899,11 @@ void qmp_block_dirty_bitmap_clear(const char *node, const char *name,
"Bitmap '%s' is currently frozen and cannot be modified",
name);
return;
+ } else if (bdrv_dirty_bitmap_qmp_locked(bitmap)) {
+ error_setg(errp,
+ "Bitmap '%s' is currently locked and cannot be modified",
+ name);
+ return;
} else if (!bdrv_dirty_bitmap_enabled(bitmap)) {
error_setg(errp,
"Bitmap '%s' is currently disabled and cannot be cleared",
@@ -3365,6 +3378,12 @@ static BlockJob *do_drive_backup(DriveBackup *backup, BlockJobTxn *txn,
bdrv_unref(target_bs);
goto out;
}
+ if (bdrv_dirty_bitmap_qmp_locked(bmap)) {
+ error_setg(errp,
+ "Bitmap '%s' is currently locked and cannot be used for "
+ "backup", backup->bitmap);
+ goto out;
+ }
}
job = backup_job_create(backup->job_id, bs, target_bs, backup->speed,
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (3 preceding siblings ...)
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 04/12] dirty-bitmap: add locked state Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-03-12 15:30 ` Dr. David Alan Gilbert
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 06/12] qapi: add dirty-bitmaps migration capability Vladimir Sementsov-Ogievskiy
` (9 subsequent siblings)
14 siblings, 1 reply; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
There would be savevm states (dirty-bitmap) which can migrate only in
postcopy stage. The corresponding pending is introduced here.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
include/migration/register.h | 17 +++++++++++++++--
migration/savevm.h | 5 +++--
hw/s390x/s390-stattrib.c | 7 ++++---
migration/block.c | 7 ++++---
migration/migration.c | 16 +++++++++-------
migration/ram.c | 9 +++++----
migration/savevm.c | 13 ++++++++-----
migration/trace-events | 2 +-
8 files changed, 49 insertions(+), 27 deletions(-)
diff --git a/include/migration/register.h b/include/migration/register.h
index f4f7bdc177..9436a87678 100644
--- a/include/migration/register.h
+++ b/include/migration/register.h
@@ -37,8 +37,21 @@ typedef struct SaveVMHandlers {
int (*save_setup)(QEMUFile *f, void *opaque);
void (*save_live_pending)(QEMUFile *f, void *opaque,
uint64_t threshold_size,
- uint64_t *non_postcopiable_pending,
- uint64_t *postcopiable_pending);
+ uint64_t *res_precopy_only,
+ uint64_t *res_compatible,
+ uint64_t *res_postcopy_only);
+ /* Note for save_live_pending:
+ * - res_precopy_only is for data which must be migrated in precopy phase
+ * or in stopped state, in other words - before target vm start
+ * - res_compatible is for data which may be migrated in any phase
+ * - res_postcopy_only is for data which must be migrated in postcopy phase
+ * or in stopped state, in other words - after source vm stop
+ *
+ * Sum of res_postcopy_only, res_compatible and res_postcopy_only is the
+ * whole amount of pending data.
+ */
+
+
LoadStateHandler *load_state;
int (*load_setup)(QEMUFile *f, void *opaque);
int (*load_cleanup)(void *opaque);
diff --git a/migration/savevm.h b/migration/savevm.h
index 295c4a1f2c..cf4f0d37ca 100644
--- a/migration/savevm.h
+++ b/migration/savevm.h
@@ -38,8 +38,9 @@ void qemu_savevm_state_complete_postcopy(QEMUFile *f);
int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only,
bool inactivate_disks);
void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
- uint64_t *res_non_postcopiable,
- uint64_t *res_postcopiable);
+ uint64_t *res_precopy_only,
+ uint64_t *res_compatible,
+ uint64_t *res_postcopy_only);
void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
void qemu_savevm_send_open_return_path(QEMUFile *f);
int qemu_savevm_send_packaged(QEMUFile *f, const uint8_t *buf, size_t len);
diff --git a/hw/s390x/s390-stattrib.c b/hw/s390x/s390-stattrib.c
index 2902f54f11..dd3fbfd1eb 100644
--- a/hw/s390x/s390-stattrib.c
+++ b/hw/s390x/s390-stattrib.c
@@ -183,15 +183,16 @@ static int cmma_save_setup(QEMUFile *f, void *opaque)
}
static void cmma_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
- uint64_t *non_postcopiable_pending,
- uint64_t *postcopiable_pending)
+ uint64_t *res_precopy_only,
+ uint64_t *res_compatible,
+ uint64_t *res_postcopy_only)
{
S390StAttribState *sas = S390_STATTRIB(opaque);
S390StAttribClass *sac = S390_STATTRIB_GET_CLASS(sas);
long long res = sac->get_dirtycount(sas);
if (res >= 0) {
- *non_postcopiable_pending += res;
+ *res_precopy_only += res;
}
}
diff --git a/migration/block.c b/migration/block.c
index 1f03946797..5652ca3337 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -866,8 +866,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)
}
static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
- uint64_t *non_postcopiable_pending,
- uint64_t *postcopiable_pending)
+ uint64_t *res_precopy_only,
+ uint64_t *res_compatible,
+ uint64_t *res_postcopy_only)
{
/* Estimate pending number of bytes to send */
uint64_t pending;
@@ -888,7 +889,7 @@ static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
DPRINTF("Enter save live pending %" PRIu64 "\n", pending);
/* We don't do postcopy */
- *non_postcopiable_pending += pending;
+ *res_precopy_only += pending;
}
static int block_load(QEMUFile *f, void *opaque, int version_id)
diff --git a/migration/migration.c b/migration/migration.c
index c99a4e62d7..3beedd676e 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -2215,21 +2215,23 @@ typedef enum {
*/
static MigIterateState migration_iteration_run(MigrationState *s)
{
- uint64_t pending_size, pend_post, pend_nonpost;
+ uint64_t pending_size, pend_pre, pend_compat, pend_post;
bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE;
- qemu_savevm_state_pending(s->to_dst_file, s->threshold_size,
- &pend_nonpost, &pend_post);
- pending_size = pend_nonpost + pend_post;
+ qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre,
+ &pend_compat, &pend_post);
+ pending_size = pend_pre + pend_compat + pend_post;
trace_migrate_pending(pending_size, s->threshold_size,
- pend_post, pend_nonpost);
+ pend_pre, pend_compat, pend_post);
if (pending_size && pending_size >= s->threshold_size) {
/* Still a significant amount to transfer */
if (migrate_postcopy() && !in_postcopy &&
- pend_nonpost <= s->threshold_size &&
- atomic_read(&s->start_postcopy)) {
+ pend_pre <= s->threshold_size &&
+ (atomic_read(&s->start_postcopy) ||
+ (pend_pre + pend_compat <= s->threshold_size)))
+ {
if (postcopy_start(s)) {
error_report("%s: postcopy failed to start", __func__);
}
diff --git a/migration/ram.c b/migration/ram.c
index cb1950f3eb..38b1b2486a 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2359,8 +2359,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
}
static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
- uint64_t *non_postcopiable_pending,
- uint64_t *postcopiable_pending)
+ uint64_t *res_precopy_only,
+ uint64_t *res_compatible,
+ uint64_t *res_postcopy_only)
{
RAMState **temp = opaque;
RAMState *rs = *temp;
@@ -2380,9 +2381,9 @@ static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
if (migrate_postcopy_ram()) {
/* We can do postcopy, and all the data is postcopiable */
- *postcopiable_pending += remaining_size;
+ *res_compatible += remaining_size;
} else {
- *non_postcopiable_pending += remaining_size;
+ *res_precopy_only += remaining_size;
}
}
diff --git a/migration/savevm.c b/migration/savevm.c
index b7908f62be..c3c60a15e3 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1218,13 +1218,15 @@ int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only,
* for units that can't do postcopy.
*/
void qemu_savevm_state_pending(QEMUFile *f, uint64_t threshold_size,
- uint64_t *res_non_postcopiable,
- uint64_t *res_postcopiable)
+ uint64_t *res_precopy_only,
+ uint64_t *res_compatible,
+ uint64_t *res_postcopy_only)
{
SaveStateEntry *se;
- *res_non_postcopiable = 0;
- *res_postcopiable = 0;
+ *res_precopy_only = 0;
+ *res_compatible = 0;
+ *res_postcopy_only = 0;
QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
@@ -1237,7 +1239,8 @@ void qemu_savevm_state_pending(QEMUFile *f, uint64_t threshold_size,
}
}
se->ops->save_live_pending(f, se->opaque, threshold_size,
- res_non_postcopiable, res_postcopiable);
+ res_precopy_only, res_compatible,
+ res_postcopy_only);
}
}
diff --git a/migration/trace-events b/migration/trace-events
index 6f29fcc686..a04fffb877 100644
--- a/migration/trace-events
+++ b/migration/trace-events
@@ -86,7 +86,7 @@ migrate_fd_cleanup(void) ""
migrate_fd_error(const char *error_desc) "error=%s"
migrate_fd_cancel(void) ""
migrate_handle_rp_req_pages(const char *rbname, size_t start, size_t len) "in %s at 0x%zx len 0x%zx"
-migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")"
+migrate_pending(uint64_t size, uint64_t max, uint64_t pre, uint64_t compat, uint64_t post) "pending size %" PRIu64 " max %" PRIu64 " (pre = %" PRIu64 " compat=%" PRIu64 " post=%" PRIu64 ")"
migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
migration_completion_file_err(void) ""
migration_completion_postcopy_end(void) ""
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 06/12] qapi: add dirty-bitmaps migration capability
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (4 preceding siblings ...)
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 07/12] migration: include migrate_dirty_bitmaps in migrate_postcopy Vladimir Sementsov-Ogievskiy
` (8 subsequent siblings)
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
---
qapi/migration.json | 6 +++++-
migration/migration.h | 1 +
migration/migration.c | 9 +++++++++
3 files changed, 15 insertions(+), 1 deletion(-)
diff --git a/qapi/migration.json b/qapi/migration.json
index 4cd3d13158..6d8181c45f 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -352,12 +352,16 @@
#
# @x-multifd: Use more than one fd for migration (since 2.11)
#
+# @dirty-bitmaps: If enabled, QEMU will migrate named dirty bitmaps.
+# (since 2.12)
+#
# Since: 1.2
##
{ 'enum': 'MigrationCapability',
'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
'compress', 'events', 'postcopy-ram', 'x-colo', 'release-ram',
- 'block', 'return-path', 'pause-before-switchover', 'x-multifd' ] }
+ 'block', 'return-path', 'pause-before-switchover', 'x-multifd',
+ 'dirty-bitmaps' ] }
##
# @MigrationCapabilityStatus:
diff --git a/migration/migration.h b/migration/migration.h
index 786d971ce2..861cdfaa96 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -203,6 +203,7 @@ bool migrate_postcopy(void);
bool migrate_release_ram(void);
bool migrate_postcopy_ram(void);
bool migrate_zero_blocks(void);
+bool migrate_dirty_bitmaps(void);
bool migrate_auto_converge(void);
bool migrate_use_multifd(void);
diff --git a/migration/migration.c b/migration/migration.c
index 3beedd676e..d683c3d693 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1543,6 +1543,15 @@ int migrate_decompress_threads(void)
return s->parameters.decompress_threads;
}
+bool migrate_dirty_bitmaps(void)
+{
+ MigrationState *s;
+
+ s = migrate_get_current();
+
+ return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS];
+}
+
bool migrate_use_events(void)
{
MigrationState *s;
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 07/12] migration: include migrate_dirty_bitmaps in migrate_postcopy
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (5 preceding siblings ...)
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 06/12] qapi: add dirty-bitmaps migration capability Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 08/12] migration/qemu-file: add qemu_put_counted_string() Vladimir Sementsov-Ogievskiy
` (7 subsequent siblings)
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Enable postcopy if dirty bitmap migration is enabled.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
---
migration/migration.c | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/migration/migration.c b/migration/migration.c
index d683c3d693..d09f53d6c3 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1486,7 +1486,7 @@ bool migrate_postcopy_ram(void)
bool migrate_postcopy(void)
{
- return migrate_postcopy_ram();
+ return migrate_postcopy_ram() || migrate_dirty_bitmaps();
}
bool migrate_auto_converge(void)
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 08/12] migration/qemu-file: add qemu_put_counted_string()
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (6 preceding siblings ...)
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 07/12] migration: include migrate_dirty_bitmaps in migrate_postcopy Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 09/12] migration: add is_active_iterate handler Vladimir Sementsov-Ogievskiy
` (6 subsequent siblings)
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Add function opposite to qemu_get_counted_string.
qemu_put_counted_string puts one-byte length of the string (string
should not be longer than 255 characters), and then it puts the string,
without last zero byte.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
---
migration/qemu-file.h | 2 ++
migration/qemu-file.c | 13 +++++++++++++
2 files changed, 15 insertions(+)
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index aae4e5ed36..f4f356ab12 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -174,4 +174,6 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
ram_addr_t offset, size_t size,
uint64_t *bytes_sent);
+void qemu_put_counted_string(QEMUFile *f, const char *name);
+
#endif
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 2ab2bf362d..e85f501f86 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -734,6 +734,19 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])
}
/*
+ * Put a string with one preceding byte containing its length. The length of
+ * the string should be less than 256.
+ */
+void qemu_put_counted_string(QEMUFile *f, const char *str)
+{
+ size_t len = strlen(str);
+
+ assert(len < 256);
+ qemu_put_byte(f, len);
+ qemu_put_buffer(f, (const uint8_t *)str, len);
+}
+
+/*
* Set the blocking state of the QEMUFile.
* Note: On some transports the OS only keeps a single blocking state for
* both directions, and thus changing the blocking on the main
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 09/12] migration: add is_active_iterate handler
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (7 preceding siblings ...)
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 08/12] migration/qemu-file: add qemu_put_counted_string() Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 10/12] migration: add postcopy migration of dirty bitmaps Vladimir Sementsov-Ogievskiy
` (5 subsequent siblings)
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Only-postcopy savevm states (dirty-bitmap) don't need live iteration, so
to disable them and stop transporting empty sections there is a new
savevm handler.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
---
include/migration/register.h | 9 +++++++++
migration/savevm.c | 5 +++++
2 files changed, 14 insertions(+)
diff --git a/include/migration/register.h b/include/migration/register.h
index 9436a87678..f6f12f9b1a 100644
--- a/include/migration/register.h
+++ b/include/migration/register.h
@@ -26,6 +26,15 @@ typedef struct SaveVMHandlers {
bool (*is_active)(void *opaque);
bool (*has_postcopy)(void *opaque);
+ /* is_active_iterate
+ * If it is not NULL then qemu_savevm_state_iterate will skip iteration if
+ * it returns false. For example, it is needed for only-postcopy-states,
+ * which needs to be handled by qemu_savevm_state_setup and
+ * qemu_savevm_state_pending, but do not need iterations until not in
+ * postcopy stage.
+ */
+ bool (*is_active_iterate)(void *opaque);
+
/* This runs outside the iothread lock in the migration case, and
* within the lock in the savevm case. The callback had better only
* use data that is local to the migration thread or protected
diff --git a/migration/savevm.c b/migration/savevm.c
index c3c60a15e3..e5d557458e 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1026,6 +1026,11 @@ int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy)
continue;
}
}
+ if (se->ops && se->ops->is_active_iterate) {
+ if (!se->ops->is_active_iterate(se->opaque)) {
+ continue;
+ }
+ }
/*
* In the postcopy phase, any device that doesn't know how to
* do postcopy should have saved it's state in the _complete
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 10/12] migration: add postcopy migration of dirty bitmaps
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (8 preceding siblings ...)
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 09/12] migration: add is_active_iterate handler Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-03-12 16:09 ` Dr. David Alan Gilbert
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 11/12] iotests: add dirty bitmap migration test Vladimir Sementsov-Ogievskiy
` (4 subsequent siblings)
14 siblings, 1 reply; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Postcopy migration of dirty bitmaps. Only named dirty bitmaps are migrated.
If destination qemu is already containing a dirty bitmap with the same name
as a migrated bitmap (for the same node), then, if their granularities are
the same the migration will be done, otherwise the error will be generated.
If destination qemu doesn't contain such bitmap it will be created.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
include/migration/misc.h | 3 +
migration/migration.h | 3 +
migration/block-dirty-bitmap.c | 737 +++++++++++++++++++++++++++++++++++++++++
migration/migration.c | 5 +
migration/savevm.c | 2 +
vl.c | 1 +
migration/Makefile.objs | 1 +
migration/trace-events | 14 +
8 files changed, 766 insertions(+)
create mode 100644 migration/block-dirty-bitmap.c
diff --git a/include/migration/misc.h b/include/migration/misc.h
index 77fd4f587c..4ebf24c6c2 100644
--- a/include/migration/misc.h
+++ b/include/migration/misc.h
@@ -56,4 +56,7 @@ bool migration_has_failed(MigrationState *);
bool migration_in_postcopy_after_devices(MigrationState *);
void migration_global_dump(Monitor *mon);
+/* migration/block-dirty-bitmap.c */
+void dirty_bitmap_mig_init(void);
+
#endif
diff --git a/migration/migration.h b/migration/migration.h
index 861cdfaa96..79f72b7e50 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -233,4 +233,7 @@ void migrate_send_rp_pong(MigrationIncomingState *mis,
void migrate_send_rp_req_pages(MigrationIncomingState *mis, const char* rbname,
ram_addr_t start, size_t len);
+void dirty_bitmap_mig_before_vm_start(void);
+void init_dirty_bitmap_incoming_migration(void);
+
#endif
diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
new file mode 100644
index 0000000000..5b41f7140d
--- /dev/null
+++ b/migration/block-dirty-bitmap.c
@@ -0,0 +1,737 @@
+/*
+ * Block dirty bitmap postcopy migration
+ *
+ * Copyright IBM, Corp. 2009
+ * Copyright (c) 2016-2017 Virtuozzo International GmbH. All rights reserved.
+ *
+ * Authors:
+ * Liran Schour <lirans@il.ibm.com>
+ * Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2. See
+ * the COPYING file in the top-level directory.
+ * This file is derived from migration/block.c, so it's author and IBM copyright
+ * are here, although content is quite different.
+ *
+ * Contributions after 2012-01-13 are licensed under the terms of the
+ * GNU GPL, version 2 or (at your option) any later version.
+ *
+ * ***
+ *
+ * Here postcopy migration of dirty bitmaps is realized. Only QMP-addressable
+ * bitmaps are migrated.
+ *
+ * Bitmap migration implies creating bitmap with the same name and granularity
+ * in destination QEMU. If the bitmap with the same name (for the same node)
+ * already exists on destination an error will be generated.
+ *
+ * format of migration:
+ *
+ * # Header (shared for different chunk types)
+ * 1, 2 or 4 bytes: flags (see qemu_{put,put}_flags)
+ * [ 1 byte: node name size ] \ flags & DEVICE_NAME
+ * [ n bytes: node name ] /
+ * [ 1 byte: bitmap name size ] \ flags & BITMAP_NAME
+ * [ n bytes: bitmap name ] /
+ *
+ * # Start of bitmap migration (flags & START)
+ * header
+ * be64: granularity
+ * 1 byte: bitmap flags (corresponds to BdrvDirtyBitmap)
+ * bit 0 - bitmap is enabled
+ * bit 1 - bitmap is persistent
+ * bit 2 - bitmap is autoloading
+ * bits 3-7 - reserved, must be zero
+ *
+ * # Complete of bitmap migration (flags & COMPLETE)
+ * header
+ *
+ * # Data chunk of bitmap migration
+ * header
+ * be64: start sector
+ * be32: number of sectors
+ * [ be64: buffer size ] \ ! (flags & ZEROES)
+ * [ n bytes: buffer ] /
+ *
+ * The last chunk in stream should contain flags & EOS. The chunk may skip
+ * device and/or bitmap names, assuming them to be the same with the previous
+ * chunk.
+ */
+
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "block/block_int.h"
+#include "sysemu/block-backend.h"
+#include "qemu/main-loop.h"
+#include "qemu/error-report.h"
+#include "migration/misc.h"
+#include "migration/migration.h"
+#include "migration/qemu-file.h"
+#include "migration/vmstate.h"
+#include "migration/register.h"
+#include "qemu/hbitmap.h"
+#include "sysemu/sysemu.h"
+#include "qemu/cutils.h"
+#include "qapi/error.h"
+#include "trace.h"
+
+#define CHUNK_SIZE (1 << 10)
+
+/* Flags occupy one, two or four bytes (Big Endian). The size is determined as
+ * follows:
+ * in first (most significant) byte bit 8 is clear --> one byte
+ * in first byte bit 8 is set --> two or four bytes, depending on second
+ * byte:
+ * | in second byte bit 8 is clear --> two bytes
+ * | in second byte bit 8 is set --> four bytes
+ */
+#define DIRTY_BITMAP_MIG_FLAG_EOS 0x01
+#define DIRTY_BITMAP_MIG_FLAG_ZEROES 0x02
+#define DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME 0x04
+#define DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME 0x08
+#define DIRTY_BITMAP_MIG_FLAG_START 0x10
+#define DIRTY_BITMAP_MIG_FLAG_COMPLETE 0x20
+#define DIRTY_BITMAP_MIG_FLAG_BITS 0x40
+
+#define DIRTY_BITMAP_MIG_EXTRA_FLAGS 0x80
+
+#define DIRTY_BITMAP_MIG_START_FLAG_ENABLED 0x01
+#define DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT 0x02
+/* 0x04 was "AUTOLOAD" flags on elder versions, no it is ignored */
+#define DIRTY_BITMAP_MIG_START_FLAG_RESERVED_MASK 0xf8
+
+typedef struct DirtyBitmapMigBitmapState {
+ /* Written during setup phase. */
+ BlockDriverState *bs;
+ const char *node_name;
+ BdrvDirtyBitmap *bitmap;
+ uint64_t total_sectors;
+ uint64_t sectors_per_chunk;
+ QSIMPLEQ_ENTRY(DirtyBitmapMigBitmapState) entry;
+ uint8_t flags;
+
+ /* For bulk phase. */
+ bool bulk_completed;
+ uint64_t cur_sector;
+} DirtyBitmapMigBitmapState;
+
+typedef struct DirtyBitmapMigState {
+ QSIMPLEQ_HEAD(dbms_list, DirtyBitmapMigBitmapState) dbms_list;
+
+ bool bulk_completed;
+ bool no_bitmaps;
+
+ /* for send_bitmap_bits() */
+ BlockDriverState *prev_bs;
+ BdrvDirtyBitmap *prev_bitmap;
+} DirtyBitmapMigState;
+
+typedef struct DirtyBitmapLoadState {
+ uint32_t flags;
+ char node_name[256];
+ char bitmap_name[256];
+ BlockDriverState *bs;
+ BdrvDirtyBitmap *bitmap;
+} DirtyBitmapLoadState;
+
+static DirtyBitmapMigState dirty_bitmap_mig_state;
+
+typedef struct DirtyBitmapLoadBitmapState {
+ BlockDriverState *bs;
+ BdrvDirtyBitmap *bitmap;
+ bool migrated;
+} DirtyBitmapLoadBitmapState;
+static GSList *enabled_bitmaps;
+QemuMutex finish_lock;
+
+void init_dirty_bitmap_incoming_migration(void)
+{
+ qemu_mutex_init(&finish_lock);
+}
+
+static uint32_t qemu_get_bitmap_flags(QEMUFile *f)
+{
+ uint8_t flags = qemu_get_byte(f);
+ if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
+ flags = flags << 8 | qemu_get_byte(f);
+ if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
+ flags = flags << 16 | qemu_get_be16(f);
+ }
+ }
+
+ return flags;
+}
+
+static void qemu_put_bitmap_flags(QEMUFile *f, uint32_t flags)
+{
+ /* The code currently do not send flags more than one byte */
+ assert(!(flags & (0xffffff00 | DIRTY_BITMAP_MIG_EXTRA_FLAGS)));
+
+ qemu_put_byte(f, flags);
+}
+
+static void send_bitmap_header(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
+ uint32_t additional_flags)
+{
+ BlockDriverState *bs = dbms->bs;
+ BdrvDirtyBitmap *bitmap = dbms->bitmap;
+ uint32_t flags = additional_flags;
+ trace_send_bitmap_header_enter();
+
+ if (bs != dirty_bitmap_mig_state.prev_bs) {
+ dirty_bitmap_mig_state.prev_bs = bs;
+ flags |= DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME;
+ }
+
+ if (bitmap != dirty_bitmap_mig_state.prev_bitmap) {
+ dirty_bitmap_mig_state.prev_bitmap = bitmap;
+ flags |= DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME;
+ }
+
+ qemu_put_bitmap_flags(f, flags);
+
+ if (flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
+ qemu_put_counted_string(f, dbms->node_name);
+ }
+
+ if (flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
+ qemu_put_counted_string(f, bdrv_dirty_bitmap_name(bitmap));
+ }
+}
+
+static void send_bitmap_start(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
+{
+ send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_START);
+ qemu_put_be32(f, bdrv_dirty_bitmap_granularity(dbms->bitmap));
+ qemu_put_byte(f, dbms->flags);
+}
+
+static void send_bitmap_complete(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
+{
+ send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_COMPLETE);
+}
+
+static void send_bitmap_bits(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
+ uint64_t start_sector, uint32_t nr_sectors)
+{
+ /* align for buffer_is_zero() */
+ uint64_t align = 4 * sizeof(long);
+ uint64_t unaligned_size =
+ bdrv_dirty_bitmap_serialization_size(
+ dbms->bitmap, start_sector << BDRV_SECTOR_BITS,
+ (uint64_t)nr_sectors << BDRV_SECTOR_BITS);
+ uint64_t buf_size = QEMU_ALIGN_UP(unaligned_size, align);
+ uint8_t *buf = g_malloc0(buf_size);
+ uint32_t flags = DIRTY_BITMAP_MIG_FLAG_BITS;
+
+ bdrv_dirty_bitmap_serialize_part(
+ dbms->bitmap, buf, start_sector << BDRV_SECTOR_BITS,
+ (uint64_t)nr_sectors << BDRV_SECTOR_BITS);
+
+ if (buffer_is_zero(buf, buf_size)) {
+ g_free(buf);
+ buf = NULL;
+ flags |= DIRTY_BITMAP_MIG_FLAG_ZEROES;
+ }
+
+ trace_send_bitmap_bits(flags, start_sector, nr_sectors, buf_size);
+
+ send_bitmap_header(f, dbms, flags);
+
+ qemu_put_be64(f, start_sector);
+ qemu_put_be32(f, nr_sectors);
+
+ /* if a block is zero we need to flush here since the network
+ * bandwidth is now a lot higher than the storage device bandwidth.
+ * thus if we queue zero blocks we slow down the migration. */
+ if (flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
+ qemu_fflush(f);
+ } else {
+ qemu_put_be64(f, buf_size);
+ qemu_put_buffer(f, buf, buf_size);
+ }
+
+ g_free(buf);
+}
+
+/* Called with iothread lock taken. */
+static void dirty_bitmap_mig_cleanup(void)
+{
+ DirtyBitmapMigBitmapState *dbms;
+
+ while ((dbms = QSIMPLEQ_FIRST(&dirty_bitmap_mig_state.dbms_list)) != NULL) {
+ QSIMPLEQ_REMOVE_HEAD(&dirty_bitmap_mig_state.dbms_list, entry);
+ bdrv_dirty_bitmap_set_qmp_locked(dbms->bitmap, false);
+ bdrv_unref(dbms->bs);
+ g_free(dbms);
+ }
+}
+
+/* Called with iothread lock taken. */
+static int init_dirty_bitmap_migration(void)
+{
+ BlockDriverState *bs;
+ BdrvDirtyBitmap *bitmap;
+ DirtyBitmapMigBitmapState *dbms;
+ BdrvNextIterator it;
+
+ dirty_bitmap_mig_state.bulk_completed = false;
+ dirty_bitmap_mig_state.prev_bs = NULL;
+ dirty_bitmap_mig_state.prev_bitmap = NULL;
+ dirty_bitmap_mig_state.no_bitmaps = false;
+
+ for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
+ const char *drive_name = bdrv_get_device_or_node_name(bs);
+
+ /* skip automatically inserted nodes */
+ while (bs && bs->drv && bs->implicit) {
+ bs = backing_bs(bs);
+ }
+
+ for (bitmap = bdrv_dirty_bitmap_next(bs, NULL); bitmap;
+ bitmap = bdrv_dirty_bitmap_next(bs, bitmap))
+ {
+ if (!bdrv_dirty_bitmap_name(bitmap)) {
+ continue;
+ }
+
+ if (drive_name == NULL) {
+ error_report("Found bitmap '%s' in unnamed node %p. It can't "
+ "be migrated", bdrv_dirty_bitmap_name(bitmap), bs);
+ goto fail;
+ }
+
+ if (bdrv_dirty_bitmap_frozen(bitmap)) {
+ error_report("Can't migrate frozen dirty bitmap: '%s",
+ bdrv_dirty_bitmap_name(bitmap));
+ goto fail;
+ }
+
+ if (bdrv_dirty_bitmap_qmp_locked(bitmap)) {
+ error_report("Can't migrate locked dirty bitmap: '%s",
+ bdrv_dirty_bitmap_name(bitmap));
+ goto fail;
+ }
+
+ bdrv_ref(bs);
+ bdrv_dirty_bitmap_set_qmp_locked(bitmap, true);
+
+ dbms = g_new0(DirtyBitmapMigBitmapState, 1);
+ dbms->bs = bs;
+ dbms->node_name = drive_name;
+ dbms->bitmap = bitmap;
+ dbms->total_sectors = bdrv_nb_sectors(bs);
+ dbms->sectors_per_chunk = CHUNK_SIZE * 8 *
+ bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS;
+ if (bdrv_dirty_bitmap_enabled(bitmap)) {
+ dbms->flags |= DIRTY_BITMAP_MIG_START_FLAG_ENABLED;
+ }
+ if (bdrv_dirty_bitmap_get_persistance(bitmap)) {
+ dbms->flags |= DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT;
+ }
+
+ QSIMPLEQ_INSERT_TAIL(&dirty_bitmap_mig_state.dbms_list,
+ dbms, entry);
+ }
+ }
+
+ /* unset persistance here, to not roll back it */
+ QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+ bdrv_dirty_bitmap_set_persistance(dbms->bitmap, false);
+ }
+
+ if (QSIMPLEQ_EMPTY(&dirty_bitmap_mig_state.dbms_list)) {
+ dirty_bitmap_mig_state.no_bitmaps = true;
+ }
+
+ return 0;
+
+fail:
+ dirty_bitmap_mig_cleanup();
+
+ return -1;
+}
+
+/* Called with no lock taken. */
+static void bulk_phase_send_chunk(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
+{
+ uint32_t nr_sectors = MIN(dbms->total_sectors - dbms->cur_sector,
+ dbms->sectors_per_chunk);
+
+ send_bitmap_bits(f, dbms, dbms->cur_sector, nr_sectors);
+
+ dbms->cur_sector += nr_sectors;
+ if (dbms->cur_sector >= dbms->total_sectors) {
+ dbms->bulk_completed = true;
+ }
+}
+
+/* Called with no lock taken. */
+static void bulk_phase(QEMUFile *f, bool limit)
+{
+ DirtyBitmapMigBitmapState *dbms;
+
+ QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+ while (!dbms->bulk_completed) {
+ bulk_phase_send_chunk(f, dbms);
+ if (limit && qemu_file_rate_limit(f)) {
+ return;
+ }
+ }
+ }
+
+ dirty_bitmap_mig_state.bulk_completed = true;
+}
+
+/* for SaveVMHandlers */
+static void dirty_bitmap_save_cleanup(void *opaque)
+{
+ dirty_bitmap_mig_cleanup();
+}
+
+static int dirty_bitmap_save_iterate(QEMUFile *f, void *opaque)
+{
+ trace_dirty_bitmap_save_iterate(migration_in_postcopy());
+
+ if (migration_in_postcopy() && !dirty_bitmap_mig_state.bulk_completed) {
+ bulk_phase(f, true);
+ }
+
+ qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
+
+ return dirty_bitmap_mig_state.bulk_completed;
+}
+
+/* Called with iothread lock taken. */
+
+static int dirty_bitmap_save_complete(QEMUFile *f, void *opaque)
+{
+ DirtyBitmapMigBitmapState *dbms;
+ trace_dirty_bitmap_save_complete_enter();
+
+ if (!dirty_bitmap_mig_state.bulk_completed) {
+ bulk_phase(f, false);
+ }
+
+ QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+ send_bitmap_complete(f, dbms);
+ }
+
+ qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
+
+ trace_dirty_bitmap_save_complete_finish();
+
+ dirty_bitmap_mig_cleanup();
+ return 0;
+}
+
+static void dirty_bitmap_save_pending(QEMUFile *f, void *opaque,
+ uint64_t max_size,
+ uint64_t *res_precopy_only,
+ uint64_t *res_compatible,
+ uint64_t *res_postcopy_only)
+{
+ DirtyBitmapMigBitmapState *dbms;
+ uint64_t pending = 0;
+
+ qemu_mutex_lock_iothread();
+
+ QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+ uint64_t gran = bdrv_dirty_bitmap_granularity(dbms->bitmap);
+ uint64_t sectors = dbms->bulk_completed ? 0 :
+ dbms->total_sectors - dbms->cur_sector;
+
+ pending += DIV_ROUND_UP(sectors * BDRV_SECTOR_SIZE, gran);
+ }
+
+ qemu_mutex_unlock_iothread();
+
+ trace_dirty_bitmap_save_pending(pending, max_size);
+
+ *res_postcopy_only += pending;
+}
+
+/* First occurrence of this bitmap. It should be created if doesn't exist */
+static int dirty_bitmap_load_start(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+ Error *local_err = NULL;
+ uint32_t granularity = qemu_get_be32(f);
+ uint8_t flags = qemu_get_byte(f);
+
+ if (s->bitmap) {
+ error_report("Bitmap with the same name ('%s') already exists on "
+ "destination", bdrv_dirty_bitmap_name(s->bitmap));
+ return -EINVAL;
+ } else {
+ s->bitmap = bdrv_create_dirty_bitmap(s->bs, granularity,
+ s->bitmap_name, &local_err);
+ if (!s->bitmap) {
+ error_report_err(local_err);
+ return -EINVAL;
+ }
+ }
+
+ if (flags & DIRTY_BITMAP_MIG_START_FLAG_RESERVED_MASK) {
+ error_report("Unknown flags in migrated dirty bitmap header: %x",
+ flags);
+ return -EINVAL;
+ }
+
+ if (flags & DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT) {
+ bdrv_dirty_bitmap_set_persistance(s->bitmap, true);
+ }
+
+ bdrv_disable_dirty_bitmap(s->bitmap);
+ if (flags & DIRTY_BITMAP_MIG_START_FLAG_ENABLED) {
+ DirtyBitmapLoadBitmapState *b;
+
+ bdrv_dirty_bitmap_create_successor(s->bs, s->bitmap, &local_err);
+ if (local_err) {
+ error_report_err(local_err);
+ return -EINVAL;
+ }
+
+ b = g_new(DirtyBitmapLoadBitmapState, 1);
+ b->bs = s->bs;
+ b->bitmap = s->bitmap;
+ b->migrated = false;
+ enabled_bitmaps = g_slist_prepend(enabled_bitmaps, b);
+ }
+
+ return 0;
+}
+
+void dirty_bitmap_mig_before_vm_start(void)
+{
+ GSList *item;
+
+ qemu_mutex_lock(&finish_lock);
+
+ for (item = enabled_bitmaps; item; item = g_slist_next(item)) {
+ DirtyBitmapLoadBitmapState *b = item->data;
+
+ if (b->migrated) {
+ bdrv_enable_dirty_bitmap(b->bitmap);
+ } else {
+ bdrv_dirty_bitmap_enable_successor(b->bitmap);
+ }
+
+ g_free(b);
+ }
+
+ g_slist_free(enabled_bitmaps);
+ enabled_bitmaps = NULL;
+
+ qemu_mutex_unlock(&finish_lock);
+}
+
+static void dirty_bitmap_load_complete(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+ GSList *item;
+ trace_dirty_bitmap_load_complete();
+ bdrv_dirty_bitmap_deserialize_finish(s->bitmap);
+
+ qemu_mutex_lock(&finish_lock);
+
+ for (item = enabled_bitmaps; item; item = g_slist_next(item)) {
+ DirtyBitmapLoadBitmapState *b = item->data;
+
+ if (b->bitmap == s->bitmap) {
+ b->migrated = true;
+ break;
+ }
+ }
+
+ if (bdrv_dirty_bitmap_frozen(s->bitmap)) {
+ bdrv_dirty_bitmap_lock(s->bitmap);
+ if (enabled_bitmaps == NULL) {
+ /* in postcopy */
+ bdrv_reclaim_dirty_bitmap_locked(s->bs, s->bitmap, &error_abort);
+ bdrv_enable_dirty_bitmap(s->bitmap);
+ } else {
+ /* target not started, successor must be empty */
+ int64_t count = bdrv_get_dirty_count(s->bitmap);
+ BdrvDirtyBitmap *ret = bdrv_reclaim_dirty_bitmap_locked(s->bs,
+ s->bitmap,
+ NULL);
+ /* bdrv_reclaim_dirty_bitmap can fail only on no successor (it
+ * must be) or on merge fail, but merge can't fail when second
+ * bitmap is empty
+ */
+ assert(ret == s->bitmap &&
+ count == bdrv_get_dirty_count(s->bitmap));
+ }
+ bdrv_dirty_bitmap_unlock(s->bitmap);
+ }
+
+ qemu_mutex_unlock(&finish_lock);
+}
+
+static int dirty_bitmap_load_bits(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+ uint64_t first_byte = qemu_get_be64(f) << BDRV_SECTOR_BITS;
+ uint64_t nr_bytes = (uint64_t)qemu_get_be32(f) << BDRV_SECTOR_BITS;
+ trace_dirty_bitmap_load_bits_enter(first_byte >> BDRV_SECTOR_BITS,
+ nr_bytes >> BDRV_SECTOR_BITS);
+
+ if (s->flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
+ trace_dirty_bitmap_load_bits_zeroes();
+ bdrv_dirty_bitmap_deserialize_zeroes(s->bitmap, first_byte, nr_bytes,
+ false);
+ } else {
+ uint8_t *buf;
+ uint64_t buf_size = qemu_get_be64(f);
+ uint64_t needed_size =
+ bdrv_dirty_bitmap_serialization_size(s->bitmap,
+ first_byte, nr_bytes);
+
+ if (needed_size > buf_size) {
+ error_report("Error: Migrated bitmap granularity doesn't "
+ "match the destination bitmap '%s' granularity",
+ bdrv_dirty_bitmap_name(s->bitmap));
+ return -EINVAL;
+ }
+
+ buf = g_malloc(buf_size);
+ qemu_get_buffer(f, buf, buf_size);
+ bdrv_dirty_bitmap_deserialize_part(s->bitmap, buf, first_byte, nr_bytes,
+ false);
+ g_free(buf);
+ }
+
+ return 0;
+}
+
+static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+ Error *local_err = NULL;
+ bool nothing;
+ s->flags = qemu_get_bitmap_flags(f);
+ trace_dirty_bitmap_load_header(s->flags);
+
+ nothing = s->flags == (s->flags & DIRTY_BITMAP_MIG_FLAG_EOS);
+
+ if (s->flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
+ if (!qemu_get_counted_string(f, s->node_name)) {
+ error_report("Unable to read node name string");
+ return -EINVAL;
+ }
+ s->bs = bdrv_lookup_bs(s->node_name, s->node_name, &local_err);
+ if (!s->bs) {
+ error_report_err(local_err);
+ return -EINVAL;
+ }
+ } else if (!s->bs && !nothing) {
+ error_report("Error: block device name is not set");
+ return -EINVAL;
+ }
+
+ if (s->flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
+ if (!qemu_get_counted_string(f, s->bitmap_name)) {
+ error_report("Unable to read bitmap name string");
+ return -EINVAL;
+ }
+ s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);
+
+ /* bitmap may be NULL here, it wouldn't be an error if it is the
+ * first occurrence of the bitmap */
+ if (!s->bitmap && !(s->flags & DIRTY_BITMAP_MIG_FLAG_START)) {
+ error_report("Error: unknown dirty bitmap "
+ "'%s' for block device '%s'",
+ s->bitmap_name, s->node_name);
+ return -EINVAL;
+ }
+ } else if (!s->bitmap && !nothing) {
+ error_report("Error: block device name is not set");
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int dirty_bitmap_load(QEMUFile *f, void *opaque, int version_id)
+{
+ static DirtyBitmapLoadState s;
+ int ret = 0;
+
+ trace_dirty_bitmap_load_enter();
+
+ if (version_id != 1) {
+ return -EINVAL;
+ }
+
+ do {
+ ret = dirty_bitmap_load_header(f, &s);
+
+ if (s.flags & DIRTY_BITMAP_MIG_FLAG_START) {
+ ret = dirty_bitmap_load_start(f, &s);
+ } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_COMPLETE) {
+ dirty_bitmap_load_complete(f, &s);
+ } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_BITS) {
+ ret = dirty_bitmap_load_bits(f, &s);
+ }
+
+ if (!ret) {
+ ret = qemu_file_get_error(f);
+ }
+
+ if (ret) {
+ return ret;
+ }
+ } while (!(s.flags & DIRTY_BITMAP_MIG_FLAG_EOS));
+
+ trace_dirty_bitmap_load_success();
+ return 0;
+}
+
+static int dirty_bitmap_save_setup(QEMUFile *f, void *opaque)
+{
+ DirtyBitmapMigBitmapState *dbms = NULL;
+ if (init_dirty_bitmap_migration() < 0) {
+ return -1;
+ }
+
+ QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+ send_bitmap_start(f, dbms);
+ }
+ qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
+
+ return 0;
+}
+
+static bool dirty_bitmap_is_active(void *opaque)
+{
+ return migrate_dirty_bitmaps() && !dirty_bitmap_mig_state.no_bitmaps;
+}
+
+static bool dirty_bitmap_is_active_iterate(void *opaque)
+{
+ return dirty_bitmap_is_active(opaque) && !runstate_is_running();
+}
+
+static bool dirty_bitmap_has_postcopy(void *opaque)
+{
+ return true;
+}
+
+static SaveVMHandlers savevm_dirty_bitmap_handlers = {
+ .save_setup = dirty_bitmap_save_setup,
+ .save_live_complete_postcopy = dirty_bitmap_save_complete,
+ .save_live_complete_precopy = dirty_bitmap_save_complete,
+ .has_postcopy = dirty_bitmap_has_postcopy,
+ .save_live_pending = dirty_bitmap_save_pending,
+ .save_live_iterate = dirty_bitmap_save_iterate,
+ .is_active_iterate = dirty_bitmap_is_active_iterate,
+ .load_state = dirty_bitmap_load,
+ .save_cleanup = dirty_bitmap_save_cleanup,
+ .is_active = dirty_bitmap_is_active,
+};
+
+void dirty_bitmap_mig_init(void)
+{
+ QSIMPLEQ_INIT(&dirty_bitmap_mig_state.dbms_list);
+
+ register_savevm_live(NULL, "dirty-bitmap", 0, 1,
+ &savevm_dirty_bitmap_handlers,
+ &dirty_bitmap_mig_state);
+}
diff --git a/migration/migration.c b/migration/migration.c
index d09f53d6c3..4128e05d7c 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -155,6 +155,9 @@ MigrationIncomingState *migration_incoming_get_current(void)
memset(&mis_current, 0, sizeof(MigrationIncomingState));
qemu_mutex_init(&mis_current.rp_mutex);
qemu_event_init(&mis_current.main_thread_load_event, false);
+
+ init_dirty_bitmap_incoming_migration();
+
once = true;
}
return &mis_current;
@@ -297,6 +300,8 @@ static void process_incoming_migration_bh(void *opaque)
state, we need to obey autostart. Any other state is set with
runstate_set. */
+ dirty_bitmap_mig_before_vm_start();
+
if (!global_state_received() ||
global_state_get_runstate() == RUN_STATE_RUNNING) {
if (autostart) {
diff --git a/migration/savevm.c b/migration/savevm.c
index e5d557458e..93b339646b 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1673,6 +1673,8 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)
trace_loadvm_postcopy_handle_run_vmstart();
+ dirty_bitmap_mig_before_vm_start();
+
if (autostart) {
/* Hold onto your hats, starting the CPU */
vm_start();
diff --git a/vl.c b/vl.c
index e517a8d995..0ef3f2b5a2 100644
--- a/vl.c
+++ b/vl.c
@@ -4514,6 +4514,7 @@ int main(int argc, char **argv, char **envp)
blk_mig_init();
ram_mig_init();
+ dirty_bitmap_mig_init();
/* If the currently selected machine wishes to override the units-per-bus
* property of its default HBA interface type, do so now. */
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index 99e038024d..c83ec47ba8 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -6,6 +6,7 @@ common-obj-y += qemu-file.o global_state.o
common-obj-y += qemu-file-channel.o
common-obj-y += xbzrle.o postcopy-ram.o
common-obj-y += qjson.o
+common-obj-y += block-dirty-bitmap.o
common-obj-$(CONFIG_RDMA) += rdma.o
diff --git a/migration/trace-events b/migration/trace-events
index a04fffb877..e9eb8078d4 100644
--- a/migration/trace-events
+++ b/migration/trace-events
@@ -227,3 +227,17 @@ colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
colo_send_message(const char *msg) "Send '%s' message"
colo_receive_message(const char *msg) "Receive '%s' message"
colo_failover_set_state(const char *new_state) "new state %s"
+
+# migration/block-dirty-bitmap.c
+send_bitmap_header_enter(void) ""
+send_bitmap_bits(uint32_t flags, uint64_t start_sector, uint32_t nr_sectors, uint64_t data_size) "\n flags: 0x%x\n start_sector: %" PRIu64 "\n nr_sectors: %" PRIu32 "\n data_size: %" PRIu64 "\n"
+dirty_bitmap_save_iterate(int in_postcopy) "in postcopy: %d"
+dirty_bitmap_save_complete_enter(void) ""
+dirty_bitmap_save_complete_finish(void) ""
+dirty_bitmap_save_pending(uint64_t pending, uint64_t max_size) "pending %" PRIu64 " max: %" PRIu64
+dirty_bitmap_load_complete(void) ""
+dirty_bitmap_load_bits_enter(uint64_t first_sector, uint32_t nr_sectors) "chunk: %" PRIu64 " %" PRIu32
+dirty_bitmap_load_bits_zeroes(void) ""
+dirty_bitmap_load_header(uint32_t flags) "flags 0x%x"
+dirty_bitmap_load_enter(void) ""
+dirty_bitmap_load_success(void) ""
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 11/12] iotests: add dirty bitmap migration test
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (9 preceding siblings ...)
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 10/12] migration: add postcopy migration of dirty bitmaps Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-03-12 16:19 ` Dr. David Alan Gilbert
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 12/12] iotests: add dirty bitmap postcopy test Vladimir Sementsov-Ogievskiy
` (3 subsequent siblings)
14 siblings, 1 reply; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
The test starts two vms (vm_a, vm_b), create dirty bitmap in
the first one, do several writes to corresponding device and
then migrate vm_a to vm_b.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
tests/qemu-iotests/169 | 141 +++++++++++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/169.out | 5 ++
tests/qemu-iotests/group | 1 +
3 files changed, 147 insertions(+)
create mode 100755 tests/qemu-iotests/169
create mode 100644 tests/qemu-iotests/169.out
diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
new file mode 100755
index 0000000000..8801f73b10
--- /dev/null
+++ b/tests/qemu-iotests/169
@@ -0,0 +1,141 @@
+#!/usr/bin/env python
+#
+# Tests for dirty bitmaps migration.
+#
+# Copyright (c) 2016-2017 Virtuozzo International GmbH. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import iotests
+import time
+import itertools
+import operator
+import new
+from iotests import qemu_img
+
+
+disk_a = os.path.join(iotests.test_dir, 'disk_a')
+disk_b = os.path.join(iotests.test_dir, 'disk_b')
+size = '1M'
+mig_file = os.path.join(iotests.test_dir, 'mig_file')
+
+
+class TestDirtyBitmapMigration(iotests.QMPTestCase):
+ def tearDown(self):
+ self.vm_a.shutdown()
+ self.vm_b.shutdown()
+ os.remove(disk_a)
+ os.remove(disk_b)
+ os.remove(mig_file)
+
+ def setUp(self):
+ qemu_img('create', '-f', iotests.imgfmt, disk_a, size)
+ qemu_img('create', '-f', iotests.imgfmt, disk_b, size)
+
+ self.vm_a = iotests.VM(path_suffix='a').add_drive(disk_a)
+ self.vm_a.launch()
+
+ self.vm_b = iotests.VM(path_suffix='b')
+ self.vm_b.add_incoming("exec: cat '" + mig_file + "'")
+
+ def add_bitmap(self, vm, granularity, persistent):
+ params = {'node': 'drive0',
+ 'name': 'bitmap0',
+ 'granularity': granularity}
+ if persistent:
+ params['persistent'] = True
+ params['autoload'] = True
+
+ result = vm.qmp('block-dirty-bitmap-add', **params)
+ self.assert_qmp(result, 'return', {});
+
+ def get_bitmap_hash(self, vm):
+ result = vm.qmp('x-debug-block-dirty-bitmap-sha256',
+ node='drive0', name='bitmap0')
+ return result['return']['sha256']
+
+ def check_bitmap(self, vm, sha256):
+ result = vm.qmp('x-debug-block-dirty-bitmap-sha256',
+ node='drive0', name='bitmap0')
+ if sha256:
+ self.assert_qmp(result, 'return/sha256', sha256);
+ else:
+ self.assert_qmp(result, 'error/desc',
+ "Dirty bitmap 'bitmap0' not found");
+
+ def do_test_migration(self, persistent, migrate_bitmaps, online,
+ shared_storage):
+ granularity = 512
+
+ # regions = ((start, count), ...)
+ regions = ((0, 0x10000),
+ (0xf0000, 0x10000),
+ (0xa0201, 0x1000))
+
+ should_migrate = migrate_bitmaps or persistent and shared_storage
+
+ self.vm_b.add_drive(disk_a if shared_storage else disk_b)
+
+ if online:
+ os.mkfifo(mig_file)
+ self.vm_b.launch()
+
+ self.add_bitmap(self.vm_a, granularity, persistent)
+ for r in regions:
+ self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % r)
+ sha256 = self.get_bitmap_hash(self.vm_a)
+
+ if migrate_bitmaps:
+ result = self.vm_a.qmp('migrate-set-capabilities',
+ capabilities=[{'capability': 'dirty-bitmaps',
+ 'state': True}])
+ self.assert_qmp(result, 'return', {})
+
+ result = self.vm_a.qmp('migrate', uri='exec:cat>' + mig_file)
+ self.vm_a.event_wait("STOP")
+
+ if not online:
+ self.vm_a.shutdown()
+ self.vm_b.launch()
+
+ self.vm_b.event_wait("RESUME", timeout=10.0)
+
+ self.check_bitmap(self.vm_b, sha256 if should_migrate else False)
+
+ if should_migrate:
+ self.vm_b.shutdown()
+ self.vm_b.launch()
+ self.check_bitmap(self.vm_b, sha256 if persistent else False)
+
+
+def inject_test_case(klass, name, method, *args, **kwargs):
+ mc = operator.methodcaller(method, *args, **kwargs)
+ setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
+
+for cmb in list(itertools.product((True, False), repeat=3)):
+ name = ('_' if cmb[0] else '_not_') + 'persistent_'
+ name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
+ name += '_online' if cmb[2] else '_offline'
+
+ # TODO fix shared-storage bitmap migration and enable cases for it
+ args = list(cmb) + [False]
+
+ inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
+ *args)
+
+
+if __name__ == '__main__':
+ iotests.main(supported_fmts=['qcow2'])
diff --git a/tests/qemu-iotests/169.out b/tests/qemu-iotests/169.out
new file mode 100644
index 0000000000..594c16f49f
--- /dev/null
+++ b/tests/qemu-iotests/169.out
@@ -0,0 +1,5 @@
+........
+----------------------------------------------------------------------
+Ran 8 tests
+
+OK
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index a2dfe79d86..498c52c4c1 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -169,6 +169,7 @@
162 auto quick
163 rw auto quick
165 rw auto quick
+169 rw auto quick
170 rw auto quick
171 rw auto quick
172 auto
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] [PATCH v10 12/12] iotests: add dirty bitmap postcopy test
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (10 preceding siblings ...)
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 11/12] iotests: add dirty bitmap migration test Vladimir Sementsov-Ogievskiy
@ 2018-02-07 15:58 ` Vladimir Sementsov-Ogievskiy
2018-03-12 16:24 ` Dr. David Alan Gilbert
2018-02-15 9:38 ` [Qemu-devel] ping Re: [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (2 subsequent siblings)
14 siblings, 1 reply; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-07 15:58 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
lirans
Test
- start two vms (vm_a, vm_b)
- in a
- do writes from set A
- do writes from set B
- fix bitmap sha256
- clear bitmap
- do writes from set A
- start migration
- than, in b
- wait vm start (postcopy should start)
- do writes from set B
- check bitmap sha256
The test should verify postcopy migration and then merging with delta
(changes in target, during postcopy process).
Reduce supported cache modes to only 'none', because with cache on time
from source.STOP to target.RESUME is unpredictable and we can fail with
timout while waiting for target.RESUME.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Fam Zheng <famz@redhat.com>
---
tests/qemu-iotests/199 | 105 ++++++++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/199.out | 5 ++
tests/qemu-iotests/group | 1 +
tests/qemu-iotests/iotests.py | 7 ++-
4 files changed, 117 insertions(+), 1 deletion(-)
create mode 100755 tests/qemu-iotests/199
create mode 100644 tests/qemu-iotests/199.out
diff --git a/tests/qemu-iotests/199 b/tests/qemu-iotests/199
new file mode 100755
index 0000000000..f872040a81
--- /dev/null
+++ b/tests/qemu-iotests/199
@@ -0,0 +1,105 @@
+#!/usr/bin/env python
+#
+# Tests for dirty bitmaps postcopy migration.
+#
+# Copyright (c) 2016-2017 Virtuozzo International GmbH. All rights reserved.
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import iotests
+import time
+from iotests import qemu_img
+
+disk_a = os.path.join(iotests.test_dir, 'disk_a')
+disk_b = os.path.join(iotests.test_dir, 'disk_b')
+size = '256G'
+fifo = os.path.join(iotests.test_dir, 'mig_fifo')
+
+class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
+
+ def tearDown(self):
+ self.vm_a.shutdown()
+ self.vm_b.shutdown()
+ os.remove(disk_a)
+ os.remove(disk_b)
+ os.remove(fifo)
+
+ def setUp(self):
+ os.mkfifo(fifo)
+ qemu_img('create', '-f', iotests.imgfmt, disk_a, size)
+ qemu_img('create', '-f', iotests.imgfmt, disk_b, size)
+ self.vm_a = iotests.VM(path_suffix='a').add_drive(disk_a)
+ self.vm_b = iotests.VM(path_suffix='b').add_drive(disk_b)
+ self.vm_b.add_incoming("exec: cat '" + fifo + "'")
+ self.vm_a.launch()
+ self.vm_b.launch()
+
+ def test_postcopy(self):
+ write_size = 0x40000000
+ granularity = 512
+ chunk = 4096
+
+ result = self.vm_a.qmp('block-dirty-bitmap-add', node='drive0',
+ name='bitmap', granularity=granularity)
+ self.assert_qmp(result, 'return', {});
+
+ s = 0
+ while s < write_size:
+ self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
+ s += 0x10000
+ s = 0x8000
+ while s < write_size:
+ self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
+ s += 0x10000
+
+ result = self.vm_a.qmp('x-debug-block-dirty-bitmap-sha256',
+ node='drive0', name='bitmap')
+ sha256 = result['return']['sha256']
+
+ result = self.vm_a.qmp('block-dirty-bitmap-clear', node='drive0',
+ name='bitmap')
+ self.assert_qmp(result, 'return', {});
+ s = 0
+ while s < write_size:
+ self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
+ s += 0x10000
+
+ result = self.vm_a.qmp('migrate-set-capabilities',
+ capabilities=[{'capability': 'dirty-bitmaps',
+ 'state': True}])
+ self.assert_qmp(result, 'return', {})
+
+ result = self.vm_a.qmp('migrate', uri='exec:cat>' + fifo)
+ self.assertNotEqual(self.vm_a.event_wait("STOP"), None)
+ self.assertNotEqual(self.vm_b.event_wait("RESUME"), None)
+
+ s = 0x8000
+ while s < write_size:
+ self.vm_b.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
+ s += 0x10000
+
+ result = self.vm_b.qmp('query-block');
+ while len(result['return'][0]['dirty-bitmaps']) > 1:
+ time.sleep(2)
+ result = self.vm_b.qmp('query-block');
+
+ result = self.vm_b.qmp('x-debug-block-dirty-bitmap-sha256',
+ node='drive0', name='bitmap')
+
+ self.assert_qmp(result, 'return/sha256', sha256);
+
+if __name__ == '__main__':
+ iotests.main(supported_fmts=['qcow2'], supported_cache_modes=['none'])
diff --git a/tests/qemu-iotests/199.out b/tests/qemu-iotests/199.out
new file mode 100644
index 0000000000..ae1213e6f8
--- /dev/null
+++ b/tests/qemu-iotests/199.out
@@ -0,0 +1,5 @@
+.
+----------------------------------------------------------------------
+Ran 1 tests
+
+OK
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 498c52c4c1..282c5ebd5c 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -197,6 +197,7 @@
196 rw auto quick
197 rw auto quick
198 rw auto
+199 rw auto
200 rw auto
201 rw auto migration
202 rw auto quick
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 5a10b2d534..a9bded77e4 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -473,6 +473,10 @@ def verify_platform(supported_oses=['linux']):
if True not in [sys.platform.startswith(x) for x in supported_oses]:
notrun('not suitable for this OS: %s' % sys.platform)
+def verify_cache_mode(supported_cache_modes=[]):
+ if supported_cache_modes and (cachemode not in supported_cache_modes):
+ notrun('not suitable for this cache mode: %s' % cachemode)
+
def supports_quorum():
return 'quorum' in qemu_img_pipe('--help')
@@ -481,7 +485,7 @@ def verify_quorum():
if not supports_quorum():
notrun('quorum support missing')
-def main(supported_fmts=[], supported_oses=['linux']):
+def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[]):
'''Run tests'''
global debug
@@ -498,6 +502,7 @@ def main(supported_fmts=[], supported_oses=['linux']):
verbosity = 1
verify_image_format(supported_fmts)
verify_platform(supported_oses)
+ verify_cache_mode(supported_cache_modes)
# We need to filter out the time taken from the output so that qemu-iotest
# can reliably diff the results against master output.
--
2.11.1
^ permalink raw reply related [flat|nested] 33+ messages in thread
* [Qemu-devel] ping Re: [PATCH v10 00/12] Dirty bitmaps postcopy migration
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (11 preceding siblings ...)
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 12/12] iotests: add dirty bitmap postcopy test Vladimir Sementsov-Ogievskiy
@ 2018-02-15 9:38 ` Vladimir Sementsov-Ogievskiy
2018-02-16 15:03 ` [Qemu-devel] " Vladimir Sementsov-Ogievskiy
2018-03-02 13:05 ` [Qemu-devel] ping " Vladimir Sementsov-Ogievskiy
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-15 9:38 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, lirans
ping
07.02.2018 18:58, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
>
> There is a new version of dirty bitmap postcopy migration series.
>
> Now it is based on Max's block tree: https://github.com/XanClic/qemu/commits/block,
> where it needs only one patch: "block: maintain persistent disabled bitmaps",
> but I hope it is near to be merged.
>
> v10
>
> clone: tag postcopy-v10 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v10
>
> 01,02: r-b Fam
> 03: adjust comments about locking
> 04: fixed 124 iotest (was broken because of small mistake in block/dirty-bitmap.c)
> 05: rebased on master, staff from migration_thread is moved to migration_iteration_run, so
> drop r-b by John and Juan
> 06: 2.11->2.12, r-b Fam
> 07,08,09,: r-b Fam
>
> 10: move to device names instead of node names, looks like libvirt don't care about
> same node-names.
> flag AUTOLOAD is ignored for now
> use QEMU_ALIGN_UP and DIV_ROUND_UP
> skip automatically inserted nodes, when search for dirty bitmaps
> allow migration of no bitmaps (see in dirty_bitmap_load_header new logic
> with nothing variable, which avoids extra errors)
> handle return code of dirty_bitmap_load_header
> avoid iteration if there are no bitmaps (see new .no_bitmaps field of
> dirty_bitmap_mig_state)
> call dirty_bitmap_mig_before_vm_start from process_incoming_migration_bh too,
> to enable bitmaps in case of postcopy not actually started.
> 11: not add r-b Fam
> tiny reorganisation of do_test_migration parameters: remove useless default
> values and make shared_storage to be the last
> disable shared storage test for now, until it will be fixed (it will be separate
> series, more related to qcow2 than to migration)
> 12: r-b Fam
>
> also, "iotests: add default node-name" is dropped, as not more needed.
>
>
> v9
>
> clone: tag postcopy-v9 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v9
>
> 01: r-b John
> 02: was incomplete, now add here bdrv_reclaim_dirty_bitmap fix
> 03: new
> 04: new
> 05: r-b John
> 07: fix type in commit message, r-b John
> 09: add comment about is_active_iterate, r-b Snow and keep Juan's r-b, hope comment is ok
> 10: change copyright to Virtuozzo
> reword comment at the top of the file
> rewrite init_dirty_bitmap_migration, to not do same things twice (John)
> and skip _only_ unnamed bitmaps, error out for unnamed nodes (John)
> use new "locked" state of bitmaps instead of frozen on source vm
> do not support migrating bitmap to existent one with the same name,
> keep only create-new-bitmap way
> break loop in dirty_bitmap_load_complete when bitmap is found
> use bitmap locking instead of context acquire
> 12: rewrite, to add more cases. (note, that 169 iotest is also in my
> "[PATCH v2 0/3] fix bitmaps migration through shared storage", which probably should
> go to qemu-stable. So this patch should rewrite it, but here I make it like new patch,
> to simplify review. When "[PATCH v2..." merged I'll rebase this on it), drop r-b
> 13: move to separate test, drop r-b
>
>
> v8.1
>
> clone: tag postcopy-v8.1 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v8.1
>
> 05: fix compilation, add new version for cmma_save_pending too.
>
>
> v8
>
> clone: tag postcopy-v8 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v8
>
> - rebased on master
> - patches 01-03 from v7 are already merged to master
> - patch order is changed to make it possible to merge block/dirty-bitmap patches
> in separate if is needed
> 01: new patch
> 03: fixed to use _locked version of bdrv_release_dirty_bitmap
> 06: qapi-schema.json -> qapi/migration.json
> 2.9 -> 2.11
> 10: protocol changed a bit:
> instead of 1 byte "bitmap enabled flag" this byte becomes just "flags"
> and have "enabled", "persistent" and "autoloading" flags inside.
> also, make all migrated bitmaps to be not persistent (to prevent their
> storing on source vm)
> 14: new patch
>
>
> patches status:
> 01-04 - are only about block/dirty-bitmap and have no r-b. Fam, John, Paolo (about bitmap lock),
> please look at. These patches are ok to be merged in separate (but before 05-14)
> other patches are about migration
> 05-09 has Juan's r-b (and some of them has John's and Eric's r-bs)
> 10 - the main patch (dirty bitmaps migration), has no r-b.
> 11 - preparation for tests, not related to migration directly, has Max's r-b, ok to be merged
> separately (but before 12-14)
> 12-14 - tests, 12 and 13 have Max's r-b, 14 is new
>
>
> v7
>
> clone: tag postcopy-v7 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v7
>
> - rebased on dirty-bitmap byte-based interfaces
> (based on git://repo.or.cz/qemu/ericb.git branch nbd-byte-dirty-v4)
> - migration of persistent bitmaps should fail for shared storage migration for now,
> as persistent dirty bitmaps are stored/load on inactivate/invalidate-cache.
> also, even for non-shared storage migration there would be useless saving of dirty
> bitmaps on source. This all will be optimized later.
>
> 01: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
> 03: some structural changes due to rebase - drop r-b
> 04: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
> staff from include/sysemu/sysemu.h moved to migration/savevm.h (rebase)
> 05: fix patch header: block -> block/dirty-bitmap
> add locking, drop r-b
> 06: staff from include/migration/migration.h moved to migration/migration.h (rebase)
> 07: add locking, drop r-b
> 09: staff from include/migration/qemu-file.h moved to migration/qemu-file.h (rebase)
> 10: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
> 11: new patch
> 12: a lot of changes/fixes (mostly by Fam's comments) + rebase
> header-definition movement
> remove include <assert.h>
> add some includes
> fix/refactor bitmap flags send
> byte-based interface for dirty bitmaps (rebase)
> froze bitmaps on source
> init_dirty_bitmap_migration can return error, if some of bitmaps are already
> frozen
> bdrv_ref drives with bitmaps
> fprintf -> error_report
> check version_id in _load function
>
> v6:
>
> clone: tag postcopy-v6 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v6
>
> rebase on master.
>
> 03 - tiny contextual change
>
> 12 - little change, but it should be reviewed. Call of init_dirty_bitmap_incoming_migration()
> (which only initialize mutex) moved from start of process_incoming_migration_co (it was
> immediately after "mis = migration_incoming_state_new(f)") to migration_incoming_get_current()
> to stay with initialization code.
> I remove r-b's, but hope that this will not be a problem. The only change in this patch - is moved
> call of init_dirty_bitmap_incoming_migration.
> I do so because of recent
>
> commit b4b076daf324894dd288cbdb67ff1e3c7434df7b
> Author: Juan Quintela <quintela@redhat.com>
> Date: Mon Jan 23 22:32:06 2017 +0100
>
> migration: create Migration Incoming State at init time
>
> 15 - add Max's r-b
>
> v5:
>
> clone: tag postcopy-v5 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v5
>
> - move 'signed-off' over 'reviewed-by' in patches.
>
> 03,04 - add comments. Hope they will be ok for you, so add Juan's r-b.
> If not ok - let me know and I'll resend.
>
> 06,08,12 - add Max's r-b
> 07,09,10,11,12 - add Juan's r-b
>
> 14 - used last version of this patch from qcow2-bitmap series with
> Max's r-b. It has contextual changes due to different base.
>
> 15 - fix 041 iotest, add default node-name only if path is specified and
> node-name is not specified
> 16 - handle whitespaces
> s/"exec: cat " + fifo/"exec: cat '" + fifo + "'"/
> fix indentation
> add Max's r-b
> 17 - fix typos, wrong size in comment, s/md5/sha256/
> add Max's r-b
>
> v4:
>
> clone: tag postcopy-v4 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v4
>
> reroll, to fix "failed automatic build test"
>
> rebase on master!
>
> 07: since 2.8 -> since 2.9
> 14: fix build of test-hbitmap
> since 2.8 -> since 2.9 and small rewording of comment (this change was not done for
> same patch in may parallels series about qcow2 bitmap extension)
>
> v3:
>
> rebased on Max's block branch: https://github.com/XanClic/qemu/commits/block
> clone: tag postcopy-v3 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v3
>
> 01 - r-b by Juan
> 02 - r-b by Juan and David
> 04 - fix indent
> 07 - s/since 2.6/since 2.8/
> 10 - change variable name s/buf/str/
> 12 - improve copyright message and move it up
> fix memory loss (thanks to Juan)
> switch from DPRINTF to trace events
> 14* - switch to sha256 and qcrypto_hash_*
> separate qmp command x-debug-block-dirty-bitmap-sha256
> 16 - use path_suffix for multi-vm test
> fix indent
> fix copyright
> use x-debug-block-dirty-bitmap-sha256 instead of md5
> 17 - use x-debug-block-dirty-bitmap-sha256 instead of md5
> remove not existing 170 test from qemu-iotests/group
>
> *Note: patch 14 is also used in my second series 'qcow2 persistent dirty bitmaps'
>
>
> v2:
> some bugs fixed, iotests a bit changed and merged into one test.
> based on block-next (https://github.com/XanClic/qemu/commits/block-next)
> clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2
>
> v1:
>
> These series are derived from my 'Dirty bitmaps migration' series. The
> core idea is switch to postcopy migration and drop usage of meta
> bitmaps.
>
> These patches provide dirty bitmap postcopy migration feature. Only
> named dirty bitmaps are to be migrated. Migration may be enabled using
> migration capabilities.
>
> The overall method (thanks to John Snow):
>
> 1. migrate bitmaps meta data in .save_live_setup
> - create/find related bitmaps on target
> - disable them
> - create successors (anonimous children) only for enabled migrated
> bitmaps
> 2. do nothing in precopy stage
> 3. just before target vm start: enable successors, created in (1)
> 4. migrate bitmap data
> 5. reclaime bitmaps (merge successors to their parents)
> 6. enable bitmaps (only bitmaps, which was enabled in source)
>
>
> Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
> (DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
> and I'll drop them in the following version.
>
> So, relatively to last DBMv7:
>
> 01-04: new patches, splitting common postcopy migration out of ram
> postcopy migration
> 05: equal to DBMv7.05
> 06: new
> 07: equal to DBMv7.06
> 08: new
> 09: equal to DBMv7.07
> 10: new
> 11: derived from DBMv7.08, see below
> 12-15: equal to DBMv7.09-12
> 16: derived from DBMv7.13
> - switch from fifo to socket, as postcopy don't work with fifo
> for now
> - change parameters: size, granularity, regions
> - add time.sleep, to wait for postcopy migration phase (bad
> temporary solution.
> - drop Reviewed-by
> 17: new
>
> 11: the core patch of the series, it is derived from
> [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
> There are a lot of changes related to switching from precopy to
> postcopy, but functions related to migration stream itself
> (structs, send/load sequences) are mostly unchnaged.
>
> So, changes, to switch from precopy to postcopy:
> - removed all staff related to meta bitmaps and dirty phase!!!
> - add dirty_bitmap_mig_enable_successors, and call it before
> target vm start in loadvm_postcopy_handle_run
> - add enabled_bitmaps list of bitmaps for
> dirty_bitmap_mig_enable_successors
>
> - enabled flag is send with start bitmap chunk instead of
> completion chunk
> - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
> using meta bitmap granularity
>
> - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
> to postcopy stage
> - dirty_bitmap_save_pending: remove dirty phase related pending,
> switch pending to non-postcopyable
> - dirty_bitmap_load_start: get enabled flag and prepare
> successors for enabled bitmaps, also add them to
> enabled_bitmaps list
> - dirty_bitmap_load_complete: for enabled bitmaps: merge them
> with successors and enable
>
> - savevm handlers:
> * remove separate savevm_dirty_bitmap_live_iterate_handlers state
> (it was bad idea, any way), and move its save_live_iterate to
> savevm_dirty_bitmap_handlers
> * add is_active_iterate savevm handler, which allows iterations
> only in postcopy stage (after stopping source vm)
> * add has_postcopy savevm handler. (ofcourse, just returning true)
> * use save_live_complete_postcopy instead of
> save_live_complete_precopy
>
> Other changes:
> - some debug output changed
> - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
> was needed to omit iterations if bitmap data is small, possibly
> this should be reimplemented)
>
> Vladimir Sementsov-Ogievskiy (12):
> block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor()
> block/dirty-bitmap: fix locking in bdrv_reclaim_dirty_bitmap
> block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap
> dirty-bitmap: add locked state
> migration: introduce postcopy-only pending
> qapi: add dirty-bitmaps migration capability
> migration: include migrate_dirty_bitmaps in migrate_postcopy
> migration/qemu-file: add qemu_put_counted_string()
> migration: add is_active_iterate handler
> migration: add postcopy migration of dirty bitmaps
> iotests: add dirty bitmap migration test
> iotests: add dirty bitmap postcopy test
>
> qapi/block-core.json | 5 +-
> qapi/migration.json | 6 +-
> include/block/dirty-bitmap.h | 7 +
> include/migration/misc.h | 3 +
> include/migration/register.h | 26 +-
> migration/migration.h | 4 +
> migration/qemu-file.h | 2 +
> migration/savevm.h | 5 +-
> block/dirty-bitmap.c | 123 +++++--
> blockdev.c | 19 ++
> hw/s390x/s390-stattrib.c | 7 +-
> migration/block-dirty-bitmap.c | 737 +++++++++++++++++++++++++++++++++++++++++
> migration/block.c | 7 +-
> migration/migration.c | 32 +-
> migration/qemu-file.c | 13 +
> migration/ram.c | 9 +-
> migration/savevm.c | 20 +-
> vl.c | 1 +
> migration/Makefile.objs | 1 +
> migration/trace-events | 16 +-
> tests/qemu-iotests/169 | 141 ++++++++
> tests/qemu-iotests/169.out | 5 +
> tests/qemu-iotests/199 | 105 ++++++
> tests/qemu-iotests/199.out | 5 +
> tests/qemu-iotests/group | 2 +
> tests/qemu-iotests/iotests.py | 7 +-
> 26 files changed, 1242 insertions(+), 66 deletions(-)
> create mode 100644 migration/block-dirty-bitmap.c
> create mode 100755 tests/qemu-iotests/169
> create mode 100644 tests/qemu-iotests/169.out
> create mode 100755 tests/qemu-iotests/199
> create mode 100644 tests/qemu-iotests/199.out
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (12 preceding siblings ...)
2018-02-15 9:38 ` [Qemu-devel] ping Re: [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
@ 2018-02-16 15:03 ` Vladimir Sementsov-Ogievskiy
2018-03-02 13:05 ` [Qemu-devel] ping " Vladimir Sementsov-Ogievskiy
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-02-16 15:03 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, lirans
07.02.2018 18:58, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
>
> There is a new version of dirty bitmap postcopy migration series.
>
> Now it is based on Max's block tree: https://github.com/XanClic/qemu/commits/block,
> where it needs only one patch: "block: maintain persistent disabled bitmaps",
> but I hope it is near to be merged.
Yahoo, it's merged, thank you!
Patches don't apply (by git am) as is to master, but branch may be rebased
onto master with automatic conflicts resolution, so I think it's not needed
to post v11 without actual changes.
>
> v10
>
> clone: tag postcopy-v10 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v10
>
> 01,02: r-b Fam
> 03: adjust comments about locking
> 04: fixed 124 iotest (was broken because of small mistake in block/dirty-bitmap.c)
> 05: rebased on master, staff from migration_thread is moved to migration_iteration_run, so
> drop r-b by John and Juan
> 06: 2.11->2.12, r-b Fam
> 07,08,09,: r-b Fam
>
> 10: move to device names instead of node names, looks like libvirt don't care about
> same node-names.
> flag AUTOLOAD is ignored for now
> use QEMU_ALIGN_UP and DIV_ROUND_UP
> skip automatically inserted nodes, when search for dirty bitmaps
> allow migration of no bitmaps (see in dirty_bitmap_load_header new logic
> with nothing variable, which avoids extra errors)
> handle return code of dirty_bitmap_load_header
> avoid iteration if there are no bitmaps (see new .no_bitmaps field of
> dirty_bitmap_mig_state)
> call dirty_bitmap_mig_before_vm_start from process_incoming_migration_bh too,
> to enable bitmaps in case of postcopy not actually started.
> 11: not add r-b Fam
> tiny reorganisation of do_test_migration parameters: remove useless default
> values and make shared_storage to be the last
> disable shared storage test for now, until it will be fixed (it will be separate
> series, more related to qcow2 than to migration)
> 12: r-b Fam
>
> also, "iotests: add default node-name" is dropped, as not more needed.
>
>
> v9
>
> clone: tag postcopy-v9 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v9
>
> 01: r-b John
> 02: was incomplete, now add here bdrv_reclaim_dirty_bitmap fix
> 03: new
> 04: new
> 05: r-b John
> 07: fix type in commit message, r-b John
> 09: add comment about is_active_iterate, r-b Snow and keep Juan's r-b, hope comment is ok
> 10: change copyright to Virtuozzo
> reword comment at the top of the file
> rewrite init_dirty_bitmap_migration, to not do same things twice (John)
> and skip _only_ unnamed bitmaps, error out for unnamed nodes (John)
> use new "locked" state of bitmaps instead of frozen on source vm
> do not support migrating bitmap to existent one with the same name,
> keep only create-new-bitmap way
> break loop in dirty_bitmap_load_complete when bitmap is found
> use bitmap locking instead of context acquire
> 12: rewrite, to add more cases. (note, that 169 iotest is also in my
> "[PATCH v2 0/3] fix bitmaps migration through shared storage", which probably should
> go to qemu-stable. So this patch should rewrite it, but here I make it like new patch,
> to simplify review. When "[PATCH v2..." merged I'll rebase this on it), drop r-b
> 13: move to separate test, drop r-b
>
>
> v8.1
>
> clone: tag postcopy-v8.1 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v8.1
>
> 05: fix compilation, add new version for cmma_save_pending too.
>
>
> v8
>
> clone: tag postcopy-v8 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v8
>
> - rebased on master
> - patches 01-03 from v7 are already merged to master
> - patch order is changed to make it possible to merge block/dirty-bitmap patches
> in separate if is needed
> 01: new patch
> 03: fixed to use _locked version of bdrv_release_dirty_bitmap
> 06: qapi-schema.json -> qapi/migration.json
> 2.9 -> 2.11
> 10: protocol changed a bit:
> instead of 1 byte "bitmap enabled flag" this byte becomes just "flags"
> and have "enabled", "persistent" and "autoloading" flags inside.
> also, make all migrated bitmaps to be not persistent (to prevent their
> storing on source vm)
> 14: new patch
>
>
> patches status:
> 01-04 - are only about block/dirty-bitmap and have no r-b. Fam, John, Paolo (about bitmap lock),
> please look at. These patches are ok to be merged in separate (but before 05-14)
> other patches are about migration
> 05-09 has Juan's r-b (and some of them has John's and Eric's r-bs)
> 10 - the main patch (dirty bitmaps migration), has no r-b.
> 11 - preparation for tests, not related to migration directly, has Max's r-b, ok to be merged
> separately (but before 12-14)
> 12-14 - tests, 12 and 13 have Max's r-b, 14 is new
>
>
> v7
>
> clone: tag postcopy-v7 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v7
>
> - rebased on dirty-bitmap byte-based interfaces
> (based on git://repo.or.cz/qemu/ericb.git branch nbd-byte-dirty-v4)
> - migration of persistent bitmaps should fail for shared storage migration for now,
> as persistent dirty bitmaps are stored/load on inactivate/invalidate-cache.
> also, even for non-shared storage migration there would be useless saving of dirty
> bitmaps on source. This all will be optimized later.
>
> 01: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
> 03: some structural changes due to rebase - drop r-b
> 04: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
> staff from include/sysemu/sysemu.h moved to migration/savevm.h (rebase)
> 05: fix patch header: block -> block/dirty-bitmap
> add locking, drop r-b
> 06: staff from include/migration/migration.h moved to migration/migration.h (rebase)
> 07: add locking, drop r-b
> 09: staff from include/migration/qemu-file.h moved to migration/qemu-file.h (rebase)
> 10: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
> 11: new patch
> 12: a lot of changes/fixes (mostly by Fam's comments) + rebase
> header-definition movement
> remove include <assert.h>
> add some includes
> fix/refactor bitmap flags send
> byte-based interface for dirty bitmaps (rebase)
> froze bitmaps on source
> init_dirty_bitmap_migration can return error, if some of bitmaps are already
> frozen
> bdrv_ref drives with bitmaps
> fprintf -> error_report
> check version_id in _load function
>
> v6:
>
> clone: tag postcopy-v6 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v6
>
> rebase on master.
>
> 03 - tiny contextual change
>
> 12 - little change, but it should be reviewed. Call of init_dirty_bitmap_incoming_migration()
> (which only initialize mutex) moved from start of process_incoming_migration_co (it was
> immediately after "mis = migration_incoming_state_new(f)") to migration_incoming_get_current()
> to stay with initialization code.
> I remove r-b's, but hope that this will not be a problem. The only change in this patch - is moved
> call of init_dirty_bitmap_incoming_migration.
> I do so because of recent
>
> commit b4b076daf324894dd288cbdb67ff1e3c7434df7b
> Author: Juan Quintela <quintela@redhat.com>
> Date: Mon Jan 23 22:32:06 2017 +0100
>
> migration: create Migration Incoming State at init time
>
> 15 - add Max's r-b
>
> v5:
>
> clone: tag postcopy-v5 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v5
>
> - move 'signed-off' over 'reviewed-by' in patches.
>
> 03,04 - add comments. Hope they will be ok for you, so add Juan's r-b.
> If not ok - let me know and I'll resend.
>
> 06,08,12 - add Max's r-b
> 07,09,10,11,12 - add Juan's r-b
>
> 14 - used last version of this patch from qcow2-bitmap series with
> Max's r-b. It has contextual changes due to different base.
>
> 15 - fix 041 iotest, add default node-name only if path is specified and
> node-name is not specified
> 16 - handle whitespaces
> s/"exec: cat " + fifo/"exec: cat '" + fifo + "'"/
> fix indentation
> add Max's r-b
> 17 - fix typos, wrong size in comment, s/md5/sha256/
> add Max's r-b
>
> v4:
>
> clone: tag postcopy-v4 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v4
>
> reroll, to fix "failed automatic build test"
>
> rebase on master!
>
> 07: since 2.8 -> since 2.9
> 14: fix build of test-hbitmap
> since 2.8 -> since 2.9 and small rewording of comment (this change was not done for
> same patch in may parallels series about qcow2 bitmap extension)
>
> v3:
>
> rebased on Max's block branch: https://github.com/XanClic/qemu/commits/block
> clone: tag postcopy-v3 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v3
>
> 01 - r-b by Juan
> 02 - r-b by Juan and David
> 04 - fix indent
> 07 - s/since 2.6/since 2.8/
> 10 - change variable name s/buf/str/
> 12 - improve copyright message and move it up
> fix memory loss (thanks to Juan)
> switch from DPRINTF to trace events
> 14* - switch to sha256 and qcrypto_hash_*
> separate qmp command x-debug-block-dirty-bitmap-sha256
> 16 - use path_suffix for multi-vm test
> fix indent
> fix copyright
> use x-debug-block-dirty-bitmap-sha256 instead of md5
> 17 - use x-debug-block-dirty-bitmap-sha256 instead of md5
> remove not existing 170 test from qemu-iotests/group
>
> *Note: patch 14 is also used in my second series 'qcow2 persistent dirty bitmaps'
>
>
> v2:
> some bugs fixed, iotests a bit changed and merged into one test.
> based on block-next (https://github.com/XanClic/qemu/commits/block-next)
> clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2
>
> v1:
>
> These series are derived from my 'Dirty bitmaps migration' series. The
> core idea is switch to postcopy migration and drop usage of meta
> bitmaps.
>
> These patches provide dirty bitmap postcopy migration feature. Only
> named dirty bitmaps are to be migrated. Migration may be enabled using
> migration capabilities.
>
> The overall method (thanks to John Snow):
>
> 1. migrate bitmaps meta data in .save_live_setup
> - create/find related bitmaps on target
> - disable them
> - create successors (anonimous children) only for enabled migrated
> bitmaps
> 2. do nothing in precopy stage
> 3. just before target vm start: enable successors, created in (1)
> 4. migrate bitmap data
> 5. reclaime bitmaps (merge successors to their parents)
> 6. enable bitmaps (only bitmaps, which was enabled in source)
>
>
> Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
> (DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
> and I'll drop them in the following version.
>
> So, relatively to last DBMv7:
>
> 01-04: new patches, splitting common postcopy migration out of ram
> postcopy migration
> 05: equal to DBMv7.05
> 06: new
> 07: equal to DBMv7.06
> 08: new
> 09: equal to DBMv7.07
> 10: new
> 11: derived from DBMv7.08, see below
> 12-15: equal to DBMv7.09-12
> 16: derived from DBMv7.13
> - switch from fifo to socket, as postcopy don't work with fifo
> for now
> - change parameters: size, granularity, regions
> - add time.sleep, to wait for postcopy migration phase (bad
> temporary solution.
> - drop Reviewed-by
> 17: new
>
> 11: the core patch of the series, it is derived from
> [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
> There are a lot of changes related to switching from precopy to
> postcopy, but functions related to migration stream itself
> (structs, send/load sequences) are mostly unchnaged.
>
> So, changes, to switch from precopy to postcopy:
> - removed all staff related to meta bitmaps and dirty phase!!!
> - add dirty_bitmap_mig_enable_successors, and call it before
> target vm start in loadvm_postcopy_handle_run
> - add enabled_bitmaps list of bitmaps for
> dirty_bitmap_mig_enable_successors
>
> - enabled flag is send with start bitmap chunk instead of
> completion chunk
> - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
> using meta bitmap granularity
>
> - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
> to postcopy stage
> - dirty_bitmap_save_pending: remove dirty phase related pending,
> switch pending to non-postcopyable
> - dirty_bitmap_load_start: get enabled flag and prepare
> successors for enabled bitmaps, also add them to
> enabled_bitmaps list
> - dirty_bitmap_load_complete: for enabled bitmaps: merge them
> with successors and enable
>
> - savevm handlers:
> * remove separate savevm_dirty_bitmap_live_iterate_handlers state
> (it was bad idea, any way), and move its save_live_iterate to
> savevm_dirty_bitmap_handlers
> * add is_active_iterate savevm handler, which allows iterations
> only in postcopy stage (after stopping source vm)
> * add has_postcopy savevm handler. (ofcourse, just returning true)
> * use save_live_complete_postcopy instead of
> save_live_complete_precopy
>
> Other changes:
> - some debug output changed
> - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
> was needed to omit iterations if bitmap data is small, possibly
> this should be reimplemented)
>
> Vladimir Sementsov-Ogievskiy (12):
> block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor()
> block/dirty-bitmap: fix locking in bdrv_reclaim_dirty_bitmap
> block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap
> dirty-bitmap: add locked state
> migration: introduce postcopy-only pending
> qapi: add dirty-bitmaps migration capability
> migration: include migrate_dirty_bitmaps in migrate_postcopy
> migration/qemu-file: add qemu_put_counted_string()
> migration: add is_active_iterate handler
> migration: add postcopy migration of dirty bitmaps
> iotests: add dirty bitmap migration test
> iotests: add dirty bitmap postcopy test
>
> qapi/block-core.json | 5 +-
> qapi/migration.json | 6 +-
> include/block/dirty-bitmap.h | 7 +
> include/migration/misc.h | 3 +
> include/migration/register.h | 26 +-
> migration/migration.h | 4 +
> migration/qemu-file.h | 2 +
> migration/savevm.h | 5 +-
> block/dirty-bitmap.c | 123 +++++--
> blockdev.c | 19 ++
> hw/s390x/s390-stattrib.c | 7 +-
> migration/block-dirty-bitmap.c | 737 +++++++++++++++++++++++++++++++++++++++++
> migration/block.c | 7 +-
> migration/migration.c | 32 +-
> migration/qemu-file.c | 13 +
> migration/ram.c | 9 +-
> migration/savevm.c | 20 +-
> vl.c | 1 +
> migration/Makefile.objs | 1 +
> migration/trace-events | 16 +-
> tests/qemu-iotests/169 | 141 ++++++++
> tests/qemu-iotests/169.out | 5 +
> tests/qemu-iotests/199 | 105 ++++++
> tests/qemu-iotests/199.out | 5 +
> tests/qemu-iotests/group | 2 +
> tests/qemu-iotests/iotests.py | 7 +-
> 26 files changed, 1242 insertions(+), 66 deletions(-)
> create mode 100644 migration/block-dirty-bitmap.c
> create mode 100755 tests/qemu-iotests/169
> create mode 100644 tests/qemu-iotests/169.out
> create mode 100755 tests/qemu-iotests/199
> create mode 100644 tests/qemu-iotests/199.out
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 33+ messages in thread
* [Qemu-devel] ping Re: [PATCH v10 00/12] Dirty bitmaps postcopy migration
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
` (13 preceding siblings ...)
2018-02-16 15:03 ` [Qemu-devel] " Vladimir Sementsov-Ogievskiy
@ 2018-03-02 13:05 ` Vladimir Sementsov-Ogievskiy
14 siblings, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-03-02 13:05 UTC (permalink / raw)
To: qemu-block, qemu-devel
Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, lirans
ping
07.02.2018 18:58, Vladimir Sementsov-Ogievskiy wrote:
> Hi all!
>
> There is a new version of dirty bitmap postcopy migration series.
>
> Now it is based on Max's block tree: https://github.com/XanClic/qemu/commits/block,
> where it needs only one patch: "block: maintain persistent disabled bitmaps",
> but I hope it is near to be merged.
>
> v10
>
> clone: tag postcopy-v10 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v10
>
> 01,02: r-b Fam
> 03: adjust comments about locking
> 04: fixed 124 iotest (was broken because of small mistake in block/dirty-bitmap.c)
> 05: rebased on master, staff from migration_thread is moved to migration_iteration_run, so
> drop r-b by John and Juan
> 06: 2.11->2.12, r-b Fam
> 07,08,09,: r-b Fam
>
> 10: move to device names instead of node names, looks like libvirt don't care about
> same node-names.
> flag AUTOLOAD is ignored for now
> use QEMU_ALIGN_UP and DIV_ROUND_UP
> skip automatically inserted nodes, when search for dirty bitmaps
> allow migration of no bitmaps (see in dirty_bitmap_load_header new logic
> with nothing variable, which avoids extra errors)
> handle return code of dirty_bitmap_load_header
> avoid iteration if there are no bitmaps (see new .no_bitmaps field of
> dirty_bitmap_mig_state)
> call dirty_bitmap_mig_before_vm_start from process_incoming_migration_bh too,
> to enable bitmaps in case of postcopy not actually started.
> 11: not add r-b Fam
> tiny reorganisation of do_test_migration parameters: remove useless default
> values and make shared_storage to be the last
> disable shared storage test for now, until it will be fixed (it will be separate
> series, more related to qcow2 than to migration)
> 12: r-b Fam
>
> also, "iotests: add default node-name" is dropped, as not more needed.
>
>
> v9
>
> clone: tag postcopy-v9 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v9
>
> 01: r-b John
> 02: was incomplete, now add here bdrv_reclaim_dirty_bitmap fix
> 03: new
> 04: new
> 05: r-b John
> 07: fix type in commit message, r-b John
> 09: add comment about is_active_iterate, r-b Snow and keep Juan's r-b, hope comment is ok
> 10: change copyright to Virtuozzo
> reword comment at the top of the file
> rewrite init_dirty_bitmap_migration, to not do same things twice (John)
> and skip _only_ unnamed bitmaps, error out for unnamed nodes (John)
> use new "locked" state of bitmaps instead of frozen on source vm
> do not support migrating bitmap to existent one with the same name,
> keep only create-new-bitmap way
> break loop in dirty_bitmap_load_complete when bitmap is found
> use bitmap locking instead of context acquire
> 12: rewrite, to add more cases. (note, that 169 iotest is also in my
> "[PATCH v2 0/3] fix bitmaps migration through shared storage", which probably should
> go to qemu-stable. So this patch should rewrite it, but here I make it like new patch,
> to simplify review. When "[PATCH v2..." merged I'll rebase this on it), drop r-b
> 13: move to separate test, drop r-b
>
>
> v8.1
>
> clone: tag postcopy-v8.1 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v8.1
>
> 05: fix compilation, add new version for cmma_save_pending too.
>
>
> v8
>
> clone: tag postcopy-v8 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v8
>
> - rebased on master
> - patches 01-03 from v7 are already merged to master
> - patch order is changed to make it possible to merge block/dirty-bitmap patches
> in separate if is needed
> 01: new patch
> 03: fixed to use _locked version of bdrv_release_dirty_bitmap
> 06: qapi-schema.json -> qapi/migration.json
> 2.9 -> 2.11
> 10: protocol changed a bit:
> instead of 1 byte "bitmap enabled flag" this byte becomes just "flags"
> and have "enabled", "persistent" and "autoloading" flags inside.
> also, make all migrated bitmaps to be not persistent (to prevent their
> storing on source vm)
> 14: new patch
>
>
> patches status:
> 01-04 - are only about block/dirty-bitmap and have no r-b. Fam, John, Paolo (about bitmap lock),
> please look at. These patches are ok to be merged in separate (but before 05-14)
> other patches are about migration
> 05-09 has Juan's r-b (and some of them has John's and Eric's r-bs)
> 10 - the main patch (dirty bitmaps migration), has no r-b.
> 11 - preparation for tests, not related to migration directly, has Max's r-b, ok to be merged
> separately (but before 12-14)
> 12-14 - tests, 12 and 13 have Max's r-b, 14 is new
>
>
> v7
>
> clone: tag postcopy-v7 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v7
>
> - rebased on dirty-bitmap byte-based interfaces
> (based on git://repo.or.cz/qemu/ericb.git branch nbd-byte-dirty-v4)
> - migration of persistent bitmaps should fail for shared storage migration for now,
> as persistent dirty bitmaps are stored/load on inactivate/invalidate-cache.
> also, even for non-shared storage migration there would be useless saving of dirty
> bitmaps on source. This all will be optimized later.
>
> 01: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
> 03: some structural changes due to rebase - drop r-b
> 04: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
> staff from include/sysemu/sysemu.h moved to migration/savevm.h (rebase)
> 05: fix patch header: block -> block/dirty-bitmap
> add locking, drop r-b
> 06: staff from include/migration/migration.h moved to migration/migration.h (rebase)
> 07: add locking, drop r-b
> 09: staff from include/migration/qemu-file.h moved to migration/qemu-file.h (rebase)
> 10: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
> 11: new patch
> 12: a lot of changes/fixes (mostly by Fam's comments) + rebase
> header-definition movement
> remove include <assert.h>
> add some includes
> fix/refactor bitmap flags send
> byte-based interface for dirty bitmaps (rebase)
> froze bitmaps on source
> init_dirty_bitmap_migration can return error, if some of bitmaps are already
> frozen
> bdrv_ref drives with bitmaps
> fprintf -> error_report
> check version_id in _load function
>
> v6:
>
> clone: tag postcopy-v6 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v6
>
> rebase on master.
>
> 03 - tiny contextual change
>
> 12 - little change, but it should be reviewed. Call of init_dirty_bitmap_incoming_migration()
> (which only initialize mutex) moved from start of process_incoming_migration_co (it was
> immediately after "mis = migration_incoming_state_new(f)") to migration_incoming_get_current()
> to stay with initialization code.
> I remove r-b's, but hope that this will not be a problem. The only change in this patch - is moved
> call of init_dirty_bitmap_incoming_migration.
> I do so because of recent
>
> commit b4b076daf324894dd288cbdb67ff1e3c7434df7b
> Author: Juan Quintela <quintela@redhat.com>
> Date: Mon Jan 23 22:32:06 2017 +0100
>
> migration: create Migration Incoming State at init time
>
> 15 - add Max's r-b
>
> v5:
>
> clone: tag postcopy-v5 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v5
>
> - move 'signed-off' over 'reviewed-by' in patches.
>
> 03,04 - add comments. Hope they will be ok for you, so add Juan's r-b.
> If not ok - let me know and I'll resend.
>
> 06,08,12 - add Max's r-b
> 07,09,10,11,12 - add Juan's r-b
>
> 14 - used last version of this patch from qcow2-bitmap series with
> Max's r-b. It has contextual changes due to different base.
>
> 15 - fix 041 iotest, add default node-name only if path is specified and
> node-name is not specified
> 16 - handle whitespaces
> s/"exec: cat " + fifo/"exec: cat '" + fifo + "'"/
> fix indentation
> add Max's r-b
> 17 - fix typos, wrong size in comment, s/md5/sha256/
> add Max's r-b
>
> v4:
>
> clone: tag postcopy-v4 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v4
>
> reroll, to fix "failed automatic build test"
>
> rebase on master!
>
> 07: since 2.8 -> since 2.9
> 14: fix build of test-hbitmap
> since 2.8 -> since 2.9 and small rewording of comment (this change was not done for
> same patch in may parallels series about qcow2 bitmap extension)
>
> v3:
>
> rebased on Max's block branch: https://github.com/XanClic/qemu/commits/block
> clone: tag postcopy-v3 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v3
>
> 01 - r-b by Juan
> 02 - r-b by Juan and David
> 04 - fix indent
> 07 - s/since 2.6/since 2.8/
> 10 - change variable name s/buf/str/
> 12 - improve copyright message and move it up
> fix memory loss (thanks to Juan)
> switch from DPRINTF to trace events
> 14* - switch to sha256 and qcrypto_hash_*
> separate qmp command x-debug-block-dirty-bitmap-sha256
> 16 - use path_suffix for multi-vm test
> fix indent
> fix copyright
> use x-debug-block-dirty-bitmap-sha256 instead of md5
> 17 - use x-debug-block-dirty-bitmap-sha256 instead of md5
> remove not existing 170 test from qemu-iotests/group
>
> *Note: patch 14 is also used in my second series 'qcow2 persistent dirty bitmaps'
>
>
> v2:
> some bugs fixed, iotests a bit changed and merged into one test.
> based on block-next (https://github.com/XanClic/qemu/commits/block-next)
> clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2
>
> v1:
>
> These series are derived from my 'Dirty bitmaps migration' series. The
> core idea is switch to postcopy migration and drop usage of meta
> bitmaps.
>
> These patches provide dirty bitmap postcopy migration feature. Only
> named dirty bitmaps are to be migrated. Migration may be enabled using
> migration capabilities.
>
> The overall method (thanks to John Snow):
>
> 1. migrate bitmaps meta data in .save_live_setup
> - create/find related bitmaps on target
> - disable them
> - create successors (anonimous children) only for enabled migrated
> bitmaps
> 2. do nothing in precopy stage
> 3. just before target vm start: enable successors, created in (1)
> 4. migrate bitmap data
> 5. reclaime bitmaps (merge successors to their parents)
> 6. enable bitmaps (only bitmaps, which was enabled in source)
>
>
> Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
> (DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
> and I'll drop them in the following version.
>
> So, relatively to last DBMv7:
>
> 01-04: new patches, splitting common postcopy migration out of ram
> postcopy migration
> 05: equal to DBMv7.05
> 06: new
> 07: equal to DBMv7.06
> 08: new
> 09: equal to DBMv7.07
> 10: new
> 11: derived from DBMv7.08, see below
> 12-15: equal to DBMv7.09-12
> 16: derived from DBMv7.13
> - switch from fifo to socket, as postcopy don't work with fifo
> for now
> - change parameters: size, granularity, regions
> - add time.sleep, to wait for postcopy migration phase (bad
> temporary solution.
> - drop Reviewed-by
> 17: new
>
> 11: the core patch of the series, it is derived from
> [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
> There are a lot of changes related to switching from precopy to
> postcopy, but functions related to migration stream itself
> (structs, send/load sequences) are mostly unchnaged.
>
> So, changes, to switch from precopy to postcopy:
> - removed all staff related to meta bitmaps and dirty phase!!!
> - add dirty_bitmap_mig_enable_successors, and call it before
> target vm start in loadvm_postcopy_handle_run
> - add enabled_bitmaps list of bitmaps for
> dirty_bitmap_mig_enable_successors
>
> - enabled flag is send with start bitmap chunk instead of
> completion chunk
> - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
> using meta bitmap granularity
>
> - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
> to postcopy stage
> - dirty_bitmap_save_pending: remove dirty phase related pending,
> switch pending to non-postcopyable
> - dirty_bitmap_load_start: get enabled flag and prepare
> successors for enabled bitmaps, also add them to
> enabled_bitmaps list
> - dirty_bitmap_load_complete: for enabled bitmaps: merge them
> with successors and enable
>
> - savevm handlers:
> * remove separate savevm_dirty_bitmap_live_iterate_handlers state
> (it was bad idea, any way), and move its save_live_iterate to
> savevm_dirty_bitmap_handlers
> * add is_active_iterate savevm handler, which allows iterations
> only in postcopy stage (after stopping source vm)
> * add has_postcopy savevm handler. (ofcourse, just returning true)
> * use save_live_complete_postcopy instead of
> save_live_complete_precopy
>
> Other changes:
> - some debug output changed
> - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
> was needed to omit iterations if bitmap data is small, possibly
> this should be reimplemented)
>
> Vladimir Sementsov-Ogievskiy (12):
> block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor()
> block/dirty-bitmap: fix locking in bdrv_reclaim_dirty_bitmap
> block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap
> dirty-bitmap: add locked state
> migration: introduce postcopy-only pending
> qapi: add dirty-bitmaps migration capability
> migration: include migrate_dirty_bitmaps in migrate_postcopy
> migration/qemu-file: add qemu_put_counted_string()
> migration: add is_active_iterate handler
> migration: add postcopy migration of dirty bitmaps
> iotests: add dirty bitmap migration test
> iotests: add dirty bitmap postcopy test
>
> qapi/block-core.json | 5 +-
> qapi/migration.json | 6 +-
> include/block/dirty-bitmap.h | 7 +
> include/migration/misc.h | 3 +
> include/migration/register.h | 26 +-
> migration/migration.h | 4 +
> migration/qemu-file.h | 2 +
> migration/savevm.h | 5 +-
> block/dirty-bitmap.c | 123 +++++--
> blockdev.c | 19 ++
> hw/s390x/s390-stattrib.c | 7 +-
> migration/block-dirty-bitmap.c | 737 +++++++++++++++++++++++++++++++++++++++++
> migration/block.c | 7 +-
> migration/migration.c | 32 +-
> migration/qemu-file.c | 13 +
> migration/ram.c | 9 +-
> migration/savevm.c | 20 +-
> vl.c | 1 +
> migration/Makefile.objs | 1 +
> migration/trace-events | 16 +-
> tests/qemu-iotests/169 | 141 ++++++++
> tests/qemu-iotests/169.out | 5 +
> tests/qemu-iotests/199 | 105 ++++++
> tests/qemu-iotests/199.out | 5 +
> tests/qemu-iotests/group | 2 +
> tests/qemu-iotests/iotests.py | 7 +-
> 26 files changed, 1242 insertions(+), 66 deletions(-)
> create mode 100644 migration/block-dirty-bitmap.c
> create mode 100755 tests/qemu-iotests/169
> create mode 100644 tests/qemu-iotests/169.out
> create mode 100755 tests/qemu-iotests/199
> create mode 100644 tests/qemu-iotests/199.out
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending Vladimir Sementsov-Ogievskiy
@ 2018-03-12 15:30 ` Dr. David Alan Gilbert
2018-03-13 5:38 ` John Snow
2018-03-13 7:47 ` Vladimir Sementsov-Ogievskiy
0 siblings, 2 replies; 33+ messages in thread
From: Dr. David Alan Gilbert @ 2018-03-12 15:30 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
* Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> There would be savevm states (dirty-bitmap) which can migrate only in
> postcopy stage. The corresponding pending is introduced here.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> include/migration/register.h | 17 +++++++++++++++--
> migration/savevm.h | 5 +++--
> hw/s390x/s390-stattrib.c | 7 ++++---
> migration/block.c | 7 ++++---
> migration/migration.c | 16 +++++++++-------
> migration/ram.c | 9 +++++----
> migration/savevm.c | 13 ++++++++-----
> migration/trace-events | 2 +-
> 8 files changed, 49 insertions(+), 27 deletions(-)
>
> diff --git a/include/migration/register.h b/include/migration/register.h
> index f4f7bdc177..9436a87678 100644
> --- a/include/migration/register.h
> +++ b/include/migration/register.h
> @@ -37,8 +37,21 @@ typedef struct SaveVMHandlers {
> int (*save_setup)(QEMUFile *f, void *opaque);
> void (*save_live_pending)(QEMUFile *f, void *opaque,
> uint64_t threshold_size,
> - uint64_t *non_postcopiable_pending,
> - uint64_t *postcopiable_pending);
> + uint64_t *res_precopy_only,
> + uint64_t *res_compatible,
> + uint64_t *res_postcopy_only);
> + /* Note for save_live_pending:
> + * - res_precopy_only is for data which must be migrated in precopy phase
> + * or in stopped state, in other words - before target vm start
> + * - res_compatible is for data which may be migrated in any phase
> + * - res_postcopy_only is for data which must be migrated in postcopy phase
> + * or in stopped state, in other words - after source vm stop
> + *
> + * Sum of res_postcopy_only, res_compatible and res_postcopy_only is the
> + * whole amount of pending data.
> + */
> +
> +
> LoadStateHandler *load_state;
> int (*load_setup)(QEMUFile *f, void *opaque);
> int (*load_cleanup)(void *opaque);
> diff --git a/migration/savevm.h b/migration/savevm.h
> index 295c4a1f2c..cf4f0d37ca 100644
> --- a/migration/savevm.h
> +++ b/migration/savevm.h
> @@ -38,8 +38,9 @@ void qemu_savevm_state_complete_postcopy(QEMUFile *f);
> int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only,
> bool inactivate_disks);
> void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
> - uint64_t *res_non_postcopiable,
> - uint64_t *res_postcopiable);
> + uint64_t *res_precopy_only,
> + uint64_t *res_compatible,
> + uint64_t *res_postcopy_only);
> void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
> void qemu_savevm_send_open_return_path(QEMUFile *f);
> int qemu_savevm_send_packaged(QEMUFile *f, const uint8_t *buf, size_t len);
> diff --git a/hw/s390x/s390-stattrib.c b/hw/s390x/s390-stattrib.c
> index 2902f54f11..dd3fbfd1eb 100644
> --- a/hw/s390x/s390-stattrib.c
> +++ b/hw/s390x/s390-stattrib.c
> @@ -183,15 +183,16 @@ static int cmma_save_setup(QEMUFile *f, void *opaque)
> }
>
> static void cmma_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
> - uint64_t *non_postcopiable_pending,
> - uint64_t *postcopiable_pending)
> + uint64_t *res_precopy_only,
> + uint64_t *res_compatible,
> + uint64_t *res_postcopy_only)
> {
> S390StAttribState *sas = S390_STATTRIB(opaque);
> S390StAttribClass *sac = S390_STATTRIB_GET_CLASS(sas);
> long long res = sac->get_dirtycount(sas);
>
> if (res >= 0) {
> - *non_postcopiable_pending += res;
> + *res_precopy_only += res;
> }
> }
>
> diff --git a/migration/block.c b/migration/block.c
> index 1f03946797..5652ca3337 100644
> --- a/migration/block.c
> +++ b/migration/block.c
> @@ -866,8 +866,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)
> }
>
> static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
> - uint64_t *non_postcopiable_pending,
> - uint64_t *postcopiable_pending)
> + uint64_t *res_precopy_only,
> + uint64_t *res_compatible,
> + uint64_t *res_postcopy_only)
> {
> /* Estimate pending number of bytes to send */
> uint64_t pending;
> @@ -888,7 +889,7 @@ static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
>
> DPRINTF("Enter save live pending %" PRIu64 "\n", pending);
> /* We don't do postcopy */
> - *non_postcopiable_pending += pending;
> + *res_precopy_only += pending;
> }
>
> static int block_load(QEMUFile *f, void *opaque, int version_id)
> diff --git a/migration/migration.c b/migration/migration.c
> index c99a4e62d7..3beedd676e 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -2215,21 +2215,23 @@ typedef enum {
> */
> static MigIterateState migration_iteration_run(MigrationState *s)
> {
> - uint64_t pending_size, pend_post, pend_nonpost;
> + uint64_t pending_size, pend_pre, pend_compat, pend_post;
> bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE;
>
> - qemu_savevm_state_pending(s->to_dst_file, s->threshold_size,
> - &pend_nonpost, &pend_post);
> - pending_size = pend_nonpost + pend_post;
> + qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre,
> + &pend_compat, &pend_post);
> + pending_size = pend_pre + pend_compat + pend_post;
>
> trace_migrate_pending(pending_size, s->threshold_size,
> - pend_post, pend_nonpost);
> + pend_pre, pend_compat, pend_post);
>
> if (pending_size && pending_size >= s->threshold_size) {
> /* Still a significant amount to transfer */
> if (migrate_postcopy() && !in_postcopy &&
> - pend_nonpost <= s->threshold_size &&
> - atomic_read(&s->start_postcopy)) {
> + pend_pre <= s->threshold_size &&
> + (atomic_read(&s->start_postcopy) ||
> + (pend_pre + pend_compat <= s->threshold_size)))
This change does something different from the description;
it causes a postcopy_start even if the user never ran the postcopy-start
command; so sorry, we can't do that; because postcopy for RAM is
something that users can enable but only switch into when they've given
up on it completing normally.
However, I guess that leaves you with a problem; which is what happens
to the system when you've run out of pend_pre+pend_compat but can't
complete because pend_post is non-0; so I don't know the answer to that.
Dave
> + {
> if (postcopy_start(s)) {
> error_report("%s: postcopy failed to start", __func__);
> }
> diff --git a/migration/ram.c b/migration/ram.c
> index cb1950f3eb..38b1b2486a 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -2359,8 +2359,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
> }
>
> static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
> - uint64_t *non_postcopiable_pending,
> - uint64_t *postcopiable_pending)
> + uint64_t *res_precopy_only,
> + uint64_t *res_compatible,
> + uint64_t *res_postcopy_only)
> {
> RAMState **temp = opaque;
> RAMState *rs = *temp;
> @@ -2380,9 +2381,9 @@ static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
>
> if (migrate_postcopy_ram()) {
> /* We can do postcopy, and all the data is postcopiable */
> - *postcopiable_pending += remaining_size;
> + *res_compatible += remaining_size;
> } else {
> - *non_postcopiable_pending += remaining_size;
> + *res_precopy_only += remaining_size;
> }
> }
>
> diff --git a/migration/savevm.c b/migration/savevm.c
> index b7908f62be..c3c60a15e3 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -1218,13 +1218,15 @@ int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only,
> * for units that can't do postcopy.
> */
> void qemu_savevm_state_pending(QEMUFile *f, uint64_t threshold_size,
> - uint64_t *res_non_postcopiable,
> - uint64_t *res_postcopiable)
> + uint64_t *res_precopy_only,
> + uint64_t *res_compatible,
> + uint64_t *res_postcopy_only)
> {
> SaveStateEntry *se;
>
> - *res_non_postcopiable = 0;
> - *res_postcopiable = 0;
> + *res_precopy_only = 0;
> + *res_compatible = 0;
> + *res_postcopy_only = 0;
>
>
> QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
> @@ -1237,7 +1239,8 @@ void qemu_savevm_state_pending(QEMUFile *f, uint64_t threshold_size,
> }
> }
> se->ops->save_live_pending(f, se->opaque, threshold_size,
> - res_non_postcopiable, res_postcopiable);
> + res_precopy_only, res_compatible,
> + res_postcopy_only);
> }
> }
>
> diff --git a/migration/trace-events b/migration/trace-events
> index 6f29fcc686..a04fffb877 100644
> --- a/migration/trace-events
> +++ b/migration/trace-events
> @@ -86,7 +86,7 @@ migrate_fd_cleanup(void) ""
> migrate_fd_error(const char *error_desc) "error=%s"
> migrate_fd_cancel(void) ""
> migrate_handle_rp_req_pages(const char *rbname, size_t start, size_t len) "in %s at 0x%zx len 0x%zx"
> -migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")"
> +migrate_pending(uint64_t size, uint64_t max, uint64_t pre, uint64_t compat, uint64_t post) "pending size %" PRIu64 " max %" PRIu64 " (pre = %" PRIu64 " compat=%" PRIu64 " post=%" PRIu64 ")"
> migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
> migration_completion_file_err(void) ""
> migration_completion_postcopy_end(void) ""
> --
> 2.11.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 10/12] migration: add postcopy migration of dirty bitmaps
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 10/12] migration: add postcopy migration of dirty bitmaps Vladimir Sementsov-Ogievskiy
@ 2018-03-12 16:09 ` Dr. David Alan Gilbert
2018-03-13 16:59 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 33+ messages in thread
From: Dr. David Alan Gilbert @ 2018-03-12 16:09 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
* Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> Postcopy migration of dirty bitmaps. Only named dirty bitmaps are migrated.
>
> If destination qemu is already containing a dirty bitmap with the same name
> as a migrated bitmap (for the same node), then, if their granularities are
> the same the migration will be done, otherwise the error will be generated.
>
> If destination qemu doesn't contain such bitmap it will be created.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> include/migration/misc.h | 3 +
> migration/migration.h | 3 +
> migration/block-dirty-bitmap.c | 737 +++++++++++++++++++++++++++++++++++++++++
> migration/migration.c | 5 +
> migration/savevm.c | 2 +
> vl.c | 1 +
> migration/Makefile.objs | 1 +
> migration/trace-events | 14 +
> 8 files changed, 766 insertions(+)
> create mode 100644 migration/block-dirty-bitmap.c
>
> diff --git a/include/migration/misc.h b/include/migration/misc.h
> index 77fd4f587c..4ebf24c6c2 100644
> --- a/include/migration/misc.h
> +++ b/include/migration/misc.h
> @@ -56,4 +56,7 @@ bool migration_has_failed(MigrationState *);
> bool migration_in_postcopy_after_devices(MigrationState *);
> void migration_global_dump(Monitor *mon);
>
> +/* migration/block-dirty-bitmap.c */
> +void dirty_bitmap_mig_init(void);
> +
> #endif
> diff --git a/migration/migration.h b/migration/migration.h
> index 861cdfaa96..79f72b7e50 100644
> --- a/migration/migration.h
> +++ b/migration/migration.h
> @@ -233,4 +233,7 @@ void migrate_send_rp_pong(MigrationIncomingState *mis,
> void migrate_send_rp_req_pages(MigrationIncomingState *mis, const char* rbname,
> ram_addr_t start, size_t len);
>
> +void dirty_bitmap_mig_before_vm_start(void);
> +void init_dirty_bitmap_incoming_migration(void);
> +
> #endif
> diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
> new file mode 100644
> index 0000000000..5b41f7140d
> --- /dev/null
> +++ b/migration/block-dirty-bitmap.c
> @@ -0,0 +1,737 @@
> +/*
> + * Block dirty bitmap postcopy migration
> + *
> + * Copyright IBM, Corp. 2009
> + * Copyright (c) 2016-2017 Virtuozzo International GmbH. All rights reserved.
> + *
> + * Authors:
> + * Liran Schour <lirans@il.ibm.com>
> + * Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2. See
> + * the COPYING file in the top-level directory.
> + * This file is derived from migration/block.c, so it's author and IBM copyright
> + * are here, although content is quite different.
> + *
> + * Contributions after 2012-01-13 are licensed under the terms of the
> + * GNU GPL, version 2 or (at your option) any later version.
> + *
> + * ***
> + *
> + * Here postcopy migration of dirty bitmaps is realized. Only QMP-addressable
> + * bitmaps are migrated.
> + *
> + * Bitmap migration implies creating bitmap with the same name and granularity
> + * in destination QEMU. If the bitmap with the same name (for the same node)
> + * already exists on destination an error will be generated.
> + *
> + * format of migration:
> + *
> + * # Header (shared for different chunk types)
> + * 1, 2 or 4 bytes: flags (see qemu_{put,put}_flags)
> + * [ 1 byte: node name size ] \ flags & DEVICE_NAME
> + * [ n bytes: node name ] /
> + * [ 1 byte: bitmap name size ] \ flags & BITMAP_NAME
> + * [ n bytes: bitmap name ] /
> + *
> + * # Start of bitmap migration (flags & START)
> + * header
> + * be64: granularity
> + * 1 byte: bitmap flags (corresponds to BdrvDirtyBitmap)
> + * bit 0 - bitmap is enabled
> + * bit 1 - bitmap is persistent
> + * bit 2 - bitmap is autoloading
> + * bits 3-7 - reserved, must be zero
> + *
> + * # Complete of bitmap migration (flags & COMPLETE)
> + * header
> + *
> + * # Data chunk of bitmap migration
> + * header
> + * be64: start sector
> + * be32: number of sectors
> + * [ be64: buffer size ] \ ! (flags & ZEROES)
> + * [ n bytes: buffer ] /
> + *
> + * The last chunk in stream should contain flags & EOS. The chunk may skip
> + * device and/or bitmap names, assuming them to be the same with the previous
> + * chunk.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "block/block.h"
> +#include "block/block_int.h"
> +#include "sysemu/block-backend.h"
> +#include "qemu/main-loop.h"
> +#include "qemu/error-report.h"
> +#include "migration/misc.h"
> +#include "migration/migration.h"
> +#include "migration/qemu-file.h"
> +#include "migration/vmstate.h"
> +#include "migration/register.h"
> +#include "qemu/hbitmap.h"
> +#include "sysemu/sysemu.h"
> +#include "qemu/cutils.h"
> +#include "qapi/error.h"
> +#include "trace.h"
> +
> +#define CHUNK_SIZE (1 << 10)
> +
> +/* Flags occupy one, two or four bytes (Big Endian). The size is determined as
> + * follows:
> + * in first (most significant) byte bit 8 is clear --> one byte
> + * in first byte bit 8 is set --> two or four bytes, depending on second
> + * byte:
> + * | in second byte bit 8 is clear --> two bytes
> + * | in second byte bit 8 is set --> four bytes
> + */
> +#define DIRTY_BITMAP_MIG_FLAG_EOS 0x01
> +#define DIRTY_BITMAP_MIG_FLAG_ZEROES 0x02
> +#define DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME 0x04
> +#define DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME 0x08
> +#define DIRTY_BITMAP_MIG_FLAG_START 0x10
> +#define DIRTY_BITMAP_MIG_FLAG_COMPLETE 0x20
> +#define DIRTY_BITMAP_MIG_FLAG_BITS 0x40
> +
> +#define DIRTY_BITMAP_MIG_EXTRA_FLAGS 0x80
> +
> +#define DIRTY_BITMAP_MIG_START_FLAG_ENABLED 0x01
> +#define DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT 0x02
> +/* 0x04 was "AUTOLOAD" flags on elder versions, no it is ignored */
> +#define DIRTY_BITMAP_MIG_START_FLAG_RESERVED_MASK 0xf8
> +
> +typedef struct DirtyBitmapMigBitmapState {
> + /* Written during setup phase. */
> + BlockDriverState *bs;
> + const char *node_name;
> + BdrvDirtyBitmap *bitmap;
> + uint64_t total_sectors;
> + uint64_t sectors_per_chunk;
> + QSIMPLEQ_ENTRY(DirtyBitmapMigBitmapState) entry;
> + uint8_t flags;
> +
> + /* For bulk phase. */
> + bool bulk_completed;
> + uint64_t cur_sector;
> +} DirtyBitmapMigBitmapState;
> +
> +typedef struct DirtyBitmapMigState {
> + QSIMPLEQ_HEAD(dbms_list, DirtyBitmapMigBitmapState) dbms_list;
> +
> + bool bulk_completed;
> + bool no_bitmaps;
> +
> + /* for send_bitmap_bits() */
> + BlockDriverState *prev_bs;
> + BdrvDirtyBitmap *prev_bitmap;
> +} DirtyBitmapMigState;
> +
> +typedef struct DirtyBitmapLoadState {
> + uint32_t flags;
> + char node_name[256];
> + char bitmap_name[256];
> + BlockDriverState *bs;
> + BdrvDirtyBitmap *bitmap;
> +} DirtyBitmapLoadState;
> +
> +static DirtyBitmapMigState dirty_bitmap_mig_state;
> +
> +typedef struct DirtyBitmapLoadBitmapState {
> + BlockDriverState *bs;
> + BdrvDirtyBitmap *bitmap;
> + bool migrated;
> +} DirtyBitmapLoadBitmapState;
> +static GSList *enabled_bitmaps;
> +QemuMutex finish_lock;
> +
> +void init_dirty_bitmap_incoming_migration(void)
> +{
> + qemu_mutex_init(&finish_lock);
> +}
> +
> +static uint32_t qemu_get_bitmap_flags(QEMUFile *f)
> +{
> + uint8_t flags = qemu_get_byte(f);
> + if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
> + flags = flags << 8 | qemu_get_byte(f);
> + if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
> + flags = flags << 16 | qemu_get_be16(f);
> + }
> + }
> +
> + return flags;
> +}
> +
> +static void qemu_put_bitmap_flags(QEMUFile *f, uint32_t flags)
> +{
> + /* The code currently do not send flags more than one byte */
> + assert(!(flags & (0xffffff00 | DIRTY_BITMAP_MIG_EXTRA_FLAGS)));
> +
> + qemu_put_byte(f, flags);
> +}
> +
> +static void send_bitmap_header(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
> + uint32_t additional_flags)
> +{
> + BlockDriverState *bs = dbms->bs;
> + BdrvDirtyBitmap *bitmap = dbms->bitmap;
> + uint32_t flags = additional_flags;
> + trace_send_bitmap_header_enter();
> +
> + if (bs != dirty_bitmap_mig_state.prev_bs) {
> + dirty_bitmap_mig_state.prev_bs = bs;
> + flags |= DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME;
> + }
> +
> + if (bitmap != dirty_bitmap_mig_state.prev_bitmap) {
> + dirty_bitmap_mig_state.prev_bitmap = bitmap;
> + flags |= DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME;
> + }
> +
> + qemu_put_bitmap_flags(f, flags);
> +
> + if (flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
> + qemu_put_counted_string(f, dbms->node_name);
> + }
> +
> + if (flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
> + qemu_put_counted_string(f, bdrv_dirty_bitmap_name(bitmap));
> + }
> +}
> +
> +static void send_bitmap_start(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
> +{
> + send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_START);
> + qemu_put_be32(f, bdrv_dirty_bitmap_granularity(dbms->bitmap));
> + qemu_put_byte(f, dbms->flags);
> +}
> +
> +static void send_bitmap_complete(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
> +{
> + send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_COMPLETE);
> +}
> +
> +static void send_bitmap_bits(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
> + uint64_t start_sector, uint32_t nr_sectors)
> +{
> + /* align for buffer_is_zero() */
> + uint64_t align = 4 * sizeof(long);
> + uint64_t unaligned_size =
> + bdrv_dirty_bitmap_serialization_size(
> + dbms->bitmap, start_sector << BDRV_SECTOR_BITS,
> + (uint64_t)nr_sectors << BDRV_SECTOR_BITS);
> + uint64_t buf_size = QEMU_ALIGN_UP(unaligned_size, align);
> + uint8_t *buf = g_malloc0(buf_size);
> + uint32_t flags = DIRTY_BITMAP_MIG_FLAG_BITS;
> +
> + bdrv_dirty_bitmap_serialize_part(
> + dbms->bitmap, buf, start_sector << BDRV_SECTOR_BITS,
> + (uint64_t)nr_sectors << BDRV_SECTOR_BITS);
> +
> + if (buffer_is_zero(buf, buf_size)) {
> + g_free(buf);
> + buf = NULL;
> + flags |= DIRTY_BITMAP_MIG_FLAG_ZEROES;
> + }
> +
> + trace_send_bitmap_bits(flags, start_sector, nr_sectors, buf_size);
> +
> + send_bitmap_header(f, dbms, flags);
> +
> + qemu_put_be64(f, start_sector);
> + qemu_put_be32(f, nr_sectors);
> +
> + /* if a block is zero we need to flush here since the network
> + * bandwidth is now a lot higher than the storage device bandwidth.
> + * thus if we queue zero blocks we slow down the migration. */
> + if (flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
> + qemu_fflush(f);
> + } else {
> + qemu_put_be64(f, buf_size);
> + qemu_put_buffer(f, buf, buf_size);
> + }
> +
> + g_free(buf);
> +}
> +
> +/* Called with iothread lock taken. */
> +static void dirty_bitmap_mig_cleanup(void)
> +{
> + DirtyBitmapMigBitmapState *dbms;
> +
> + while ((dbms = QSIMPLEQ_FIRST(&dirty_bitmap_mig_state.dbms_list)) != NULL) {
> + QSIMPLEQ_REMOVE_HEAD(&dirty_bitmap_mig_state.dbms_list, entry);
> + bdrv_dirty_bitmap_set_qmp_locked(dbms->bitmap, false);
> + bdrv_unref(dbms->bs);
> + g_free(dbms);
> + }
> +}
> +
> +/* Called with iothread lock taken. */
> +static int init_dirty_bitmap_migration(void)
> +{
> + BlockDriverState *bs;
> + BdrvDirtyBitmap *bitmap;
> + DirtyBitmapMigBitmapState *dbms;
> + BdrvNextIterator it;
> +
> + dirty_bitmap_mig_state.bulk_completed = false;
> + dirty_bitmap_mig_state.prev_bs = NULL;
> + dirty_bitmap_mig_state.prev_bitmap = NULL;
> + dirty_bitmap_mig_state.no_bitmaps = false;
> +
> + for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
> + const char *drive_name = bdrv_get_device_or_node_name(bs);
> +
> + /* skip automatically inserted nodes */
> + while (bs && bs->drv && bs->implicit) {
> + bs = backing_bs(bs);
> + }
> +
> + for (bitmap = bdrv_dirty_bitmap_next(bs, NULL); bitmap;
> + bitmap = bdrv_dirty_bitmap_next(bs, bitmap))
> + {
> + if (!bdrv_dirty_bitmap_name(bitmap)) {
> + continue;
> + }
> +
> + if (drive_name == NULL) {
> + error_report("Found bitmap '%s' in unnamed node %p. It can't "
> + "be migrated", bdrv_dirty_bitmap_name(bitmap), bs);
> + goto fail;
> + }
> +
> + if (bdrv_dirty_bitmap_frozen(bitmap)) {
> + error_report("Can't migrate frozen dirty bitmap: '%s",
> + bdrv_dirty_bitmap_name(bitmap));
> + goto fail;
> + }
> +
> + if (bdrv_dirty_bitmap_qmp_locked(bitmap)) {
> + error_report("Can't migrate locked dirty bitmap: '%s",
> + bdrv_dirty_bitmap_name(bitmap));
> + goto fail;
> + }
> +
> + bdrv_ref(bs);
> + bdrv_dirty_bitmap_set_qmp_locked(bitmap, true);
> +
> + dbms = g_new0(DirtyBitmapMigBitmapState, 1);
> + dbms->bs = bs;
> + dbms->node_name = drive_name;
> + dbms->bitmap = bitmap;
> + dbms->total_sectors = bdrv_nb_sectors(bs);
> + dbms->sectors_per_chunk = CHUNK_SIZE * 8 *
> + bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS;
> + if (bdrv_dirty_bitmap_enabled(bitmap)) {
> + dbms->flags |= DIRTY_BITMAP_MIG_START_FLAG_ENABLED;
> + }
> + if (bdrv_dirty_bitmap_get_persistance(bitmap)) {
> + dbms->flags |= DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT;
> + }
> +
> + QSIMPLEQ_INSERT_TAIL(&dirty_bitmap_mig_state.dbms_list,
> + dbms, entry);
> + }
> + }
> +
> + /* unset persistance here, to not roll back it */
> + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> + bdrv_dirty_bitmap_set_persistance(dbms->bitmap, false);
> + }
> +
> + if (QSIMPLEQ_EMPTY(&dirty_bitmap_mig_state.dbms_list)) {
> + dirty_bitmap_mig_state.no_bitmaps = true;
> + }
> +
> + return 0;
> +
> +fail:
> + dirty_bitmap_mig_cleanup();
> +
> + return -1;
> +}
> +
> +/* Called with no lock taken. */
> +static void bulk_phase_send_chunk(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
> +{
> + uint32_t nr_sectors = MIN(dbms->total_sectors - dbms->cur_sector,
> + dbms->sectors_per_chunk);
> +
> + send_bitmap_bits(f, dbms, dbms->cur_sector, nr_sectors);
> +
> + dbms->cur_sector += nr_sectors;
> + if (dbms->cur_sector >= dbms->total_sectors) {
> + dbms->bulk_completed = true;
> + }
> +}
> +
> +/* Called with no lock taken. */
> +static void bulk_phase(QEMUFile *f, bool limit)
> +{
> + DirtyBitmapMigBitmapState *dbms;
> +
> + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> + while (!dbms->bulk_completed) {
> + bulk_phase_send_chunk(f, dbms);
> + if (limit && qemu_file_rate_limit(f)) {
> + return;
> + }
> + }
> + }
> +
> + dirty_bitmap_mig_state.bulk_completed = true;
> +}
> +
> +/* for SaveVMHandlers */
> +static void dirty_bitmap_save_cleanup(void *opaque)
> +{
> + dirty_bitmap_mig_cleanup();
> +}
> +
> +static int dirty_bitmap_save_iterate(QEMUFile *f, void *opaque)
> +{
> + trace_dirty_bitmap_save_iterate(migration_in_postcopy());
> +
> + if (migration_in_postcopy() && !dirty_bitmap_mig_state.bulk_completed) {
> + bulk_phase(f, true);
> + }
> +
> + qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
> +
> + return dirty_bitmap_mig_state.bulk_completed;
> +}
> +
> +/* Called with iothread lock taken. */
> +
> +static int dirty_bitmap_save_complete(QEMUFile *f, void *opaque)
> +{
> + DirtyBitmapMigBitmapState *dbms;
> + trace_dirty_bitmap_save_complete_enter();
> +
> + if (!dirty_bitmap_mig_state.bulk_completed) {
> + bulk_phase(f, false);
> + }
> +
> + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> + send_bitmap_complete(f, dbms);
> + }
> +
> + qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
> +
> + trace_dirty_bitmap_save_complete_finish();
> +
> + dirty_bitmap_mig_cleanup();
> + return 0;
> +}
> +
> +static void dirty_bitmap_save_pending(QEMUFile *f, void *opaque,
> + uint64_t max_size,
> + uint64_t *res_precopy_only,
> + uint64_t *res_compatible,
> + uint64_t *res_postcopy_only)
> +{
> + DirtyBitmapMigBitmapState *dbms;
> + uint64_t pending = 0;
> +
> + qemu_mutex_lock_iothread();
> +
> + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> + uint64_t gran = bdrv_dirty_bitmap_granularity(dbms->bitmap);
> + uint64_t sectors = dbms->bulk_completed ? 0 :
> + dbms->total_sectors - dbms->cur_sector;
> +
> + pending += DIV_ROUND_UP(sectors * BDRV_SECTOR_SIZE, gran);
> + }
> +
> + qemu_mutex_unlock_iothread();
> +
> + trace_dirty_bitmap_save_pending(pending, max_size);
> +
> + *res_postcopy_only += pending;
> +}
> +
> +/* First occurrence of this bitmap. It should be created if doesn't exist */
> +static int dirty_bitmap_load_start(QEMUFile *f, DirtyBitmapLoadState *s)
> +{
> + Error *local_err = NULL;
> + uint32_t granularity = qemu_get_be32(f);
> + uint8_t flags = qemu_get_byte(f);
> +
> + if (s->bitmap) {
> + error_report("Bitmap with the same name ('%s') already exists on "
> + "destination", bdrv_dirty_bitmap_name(s->bitmap));
> + return -EINVAL;
> + } else {
> + s->bitmap = bdrv_create_dirty_bitmap(s->bs, granularity,
> + s->bitmap_name, &local_err);
> + if (!s->bitmap) {
> + error_report_err(local_err);
> + return -EINVAL;
> + }
> + }
> +
> + if (flags & DIRTY_BITMAP_MIG_START_FLAG_RESERVED_MASK) {
> + error_report("Unknown flags in migrated dirty bitmap header: %x",
> + flags);
> + return -EINVAL;
> + }
> +
> + if (flags & DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT) {
> + bdrv_dirty_bitmap_set_persistance(s->bitmap, true);
> + }
> +
> + bdrv_disable_dirty_bitmap(s->bitmap);
> + if (flags & DIRTY_BITMAP_MIG_START_FLAG_ENABLED) {
> + DirtyBitmapLoadBitmapState *b;
> +
> + bdrv_dirty_bitmap_create_successor(s->bs, s->bitmap, &local_err);
> + if (local_err) {
> + error_report_err(local_err);
> + return -EINVAL;
> + }
> +
> + b = g_new(DirtyBitmapLoadBitmapState, 1);
> + b->bs = s->bs;
> + b->bitmap = s->bitmap;
> + b->migrated = false;
> + enabled_bitmaps = g_slist_prepend(enabled_bitmaps, b);
> + }
> +
> + return 0;
> +}
> +
> +void dirty_bitmap_mig_before_vm_start(void)
> +{
> + GSList *item;
> +
> + qemu_mutex_lock(&finish_lock);
> +
> + for (item = enabled_bitmaps; item; item = g_slist_next(item)) {
> + DirtyBitmapLoadBitmapState *b = item->data;
> +
> + if (b->migrated) {
> + bdrv_enable_dirty_bitmap(b->bitmap);
> + } else {
> + bdrv_dirty_bitmap_enable_successor(b->bitmap);
> + }
> +
> + g_free(b);
> + }
> +
> + g_slist_free(enabled_bitmaps);
> + enabled_bitmaps = NULL;
> +
> + qemu_mutex_unlock(&finish_lock);
> +}
> +
> +static void dirty_bitmap_load_complete(QEMUFile *f, DirtyBitmapLoadState *s)
> +{
> + GSList *item;
> + trace_dirty_bitmap_load_complete();
> + bdrv_dirty_bitmap_deserialize_finish(s->bitmap);
> +
> + qemu_mutex_lock(&finish_lock);
> +
> + for (item = enabled_bitmaps; item; item = g_slist_next(item)) {
> + DirtyBitmapLoadBitmapState *b = item->data;
> +
> + if (b->bitmap == s->bitmap) {
> + b->migrated = true;
> + break;
> + }
> + }
> +
> + if (bdrv_dirty_bitmap_frozen(s->bitmap)) {
> + bdrv_dirty_bitmap_lock(s->bitmap);
> + if (enabled_bitmaps == NULL) {
> + /* in postcopy */
> + bdrv_reclaim_dirty_bitmap_locked(s->bs, s->bitmap, &error_abort);
> + bdrv_enable_dirty_bitmap(s->bitmap);
> + } else {
> + /* target not started, successor must be empty */
> + int64_t count = bdrv_get_dirty_count(s->bitmap);
> + BdrvDirtyBitmap *ret = bdrv_reclaim_dirty_bitmap_locked(s->bs,
> + s->bitmap,
> + NULL);
> + /* bdrv_reclaim_dirty_bitmap can fail only on no successor (it
> + * must be) or on merge fail, but merge can't fail when second
> + * bitmap is empty
> + */
> + assert(ret == s->bitmap &&
> + count == bdrv_get_dirty_count(s->bitmap));
> + }
> + bdrv_dirty_bitmap_unlock(s->bitmap);
> + }
> +
> + qemu_mutex_unlock(&finish_lock);
> +}
> +
> +static int dirty_bitmap_load_bits(QEMUFile *f, DirtyBitmapLoadState *s)
> +{
> + uint64_t first_byte = qemu_get_be64(f) << BDRV_SECTOR_BITS;
> + uint64_t nr_bytes = (uint64_t)qemu_get_be32(f) << BDRV_SECTOR_BITS;
> + trace_dirty_bitmap_load_bits_enter(first_byte >> BDRV_SECTOR_BITS,
> + nr_bytes >> BDRV_SECTOR_BITS);
> +
> + if (s->flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
> + trace_dirty_bitmap_load_bits_zeroes();
> + bdrv_dirty_bitmap_deserialize_zeroes(s->bitmap, first_byte, nr_bytes,
> + false);
> + } else {
> + uint8_t *buf;
> + uint64_t buf_size = qemu_get_be64(f);
> + uint64_t needed_size =
> + bdrv_dirty_bitmap_serialization_size(s->bitmap,
> + first_byte, nr_bytes);
> +
> + if (needed_size > buf_size) {
> + error_report("Error: Migrated bitmap granularity doesn't "
> + "match the destination bitmap '%s' granularity",
> + bdrv_dirty_bitmap_name(s->bitmap));
> + return -EINVAL;
> + }
> +
> + buf = g_malloc(buf_size);
> + qemu_get_buffer(f, buf, buf_size);
Two things to be careful of there:
a) buf_size is read from the stream, so be careful about whether you
trust it could be a crazy value
b) You don't check the return value of qemu_get_buffer; which actually
we're pretty bad at in lots of places, but it's probably a good idea
before trying to deserialise it.
> + bdrv_dirty_bitmap_deserialize_part(s->bitmap, buf, first_byte, nr_bytes,
> + false);
> + g_free(buf);
> + }
> +
> + return 0;
> +}
> +
> +static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s)
> +{
> + Error *local_err = NULL;
> + bool nothing;
> + s->flags = qemu_get_bitmap_flags(f);
> + trace_dirty_bitmap_load_header(s->flags);
> +
> + nothing = s->flags == (s->flags & DIRTY_BITMAP_MIG_FLAG_EOS);
> +
> + if (s->flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
> + if (!qemu_get_counted_string(f, s->node_name)) {
> + error_report("Unable to read node name string");
> + return -EINVAL;
> + }
> + s->bs = bdrv_lookup_bs(s->node_name, s->node_name, &local_err);
> + if (!s->bs) {
> + error_report_err(local_err);
> + return -EINVAL;
> + }
> + } else if (!s->bs && !nothing) {
> + error_report("Error: block device name is not set");
> + return -EINVAL;
> + }
> +
> + if (s->flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
> + if (!qemu_get_counted_string(f, s->bitmap_name)) {
> + error_report("Unable to read bitmap name string");
> + return -EINVAL;
> + }
> + s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);
> +
> + /* bitmap may be NULL here, it wouldn't be an error if it is the
> + * first occurrence of the bitmap */
> + if (!s->bitmap && !(s->flags & DIRTY_BITMAP_MIG_FLAG_START)) {
> + error_report("Error: unknown dirty bitmap "
> + "'%s' for block device '%s'",
> + s->bitmap_name, s->node_name);
> + return -EINVAL;
> + }
> + } else if (!s->bitmap && !nothing) {
> + error_report("Error: block device name is not set");
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
> +static int dirty_bitmap_load(QEMUFile *f, void *opaque, int version_id)
> +{
> + static DirtyBitmapLoadState s;
> + int ret = 0;
> +
> + trace_dirty_bitmap_load_enter();
> +
> + if (version_id != 1) {
> + return -EINVAL;
> + }
> +
> + do {
> + ret = dirty_bitmap_load_header(f, &s);
> +
> + if (s.flags & DIRTY_BITMAP_MIG_FLAG_START) {
> + ret = dirty_bitmap_load_start(f, &s);
> + } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_COMPLETE) {
> + dirty_bitmap_load_complete(f, &s);
> + } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_BITS) {
> + ret = dirty_bitmap_load_bits(f, &s);
> + }
> +
> + if (!ret) {
> + ret = qemu_file_get_error(f);
> + }
> +
> + if (ret) {
> + return ret;
> + }
> + } while (!(s.flags & DIRTY_BITMAP_MIG_FLAG_EOS));
> +
> + trace_dirty_bitmap_load_success();
> + return 0;
> +}
> +
> +static int dirty_bitmap_save_setup(QEMUFile *f, void *opaque)
> +{
> + DirtyBitmapMigBitmapState *dbms = NULL;
> + if (init_dirty_bitmap_migration() < 0) {
> + return -1;
> + }
> +
> + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> + send_bitmap_start(f, dbms);
> + }
> + qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
> +
> + return 0;
> +}
> +
> +static bool dirty_bitmap_is_active(void *opaque)
> +{
> + return migrate_dirty_bitmaps() && !dirty_bitmap_mig_state.no_bitmaps;
> +}
> +
> +static bool dirty_bitmap_is_active_iterate(void *opaque)
> +{
> + return dirty_bitmap_is_active(opaque) && !runstate_is_running();
> +}
> +
> +static bool dirty_bitmap_has_postcopy(void *opaque)
> +{
> + return true;
> +}
> +
> +static SaveVMHandlers savevm_dirty_bitmap_handlers = {
> + .save_setup = dirty_bitmap_save_setup,
> + .save_live_complete_postcopy = dirty_bitmap_save_complete,
> + .save_live_complete_precopy = dirty_bitmap_save_complete,
> + .has_postcopy = dirty_bitmap_has_postcopy,
> + .save_live_pending = dirty_bitmap_save_pending,
> + .save_live_iterate = dirty_bitmap_save_iterate,
> + .is_active_iterate = dirty_bitmap_is_active_iterate,
> + .load_state = dirty_bitmap_load,
> + .save_cleanup = dirty_bitmap_save_cleanup,
> + .is_active = dirty_bitmap_is_active,
> +};
> +
> +void dirty_bitmap_mig_init(void)
> +{
> + QSIMPLEQ_INIT(&dirty_bitmap_mig_state.dbms_list);
> +
> + register_savevm_live(NULL, "dirty-bitmap", 0, 1,
> + &savevm_dirty_bitmap_handlers,
> + &dirty_bitmap_mig_state);
> +}
> diff --git a/migration/migration.c b/migration/migration.c
> index d09f53d6c3..4128e05d7c 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -155,6 +155,9 @@ MigrationIncomingState *migration_incoming_get_current(void)
> memset(&mis_current, 0, sizeof(MigrationIncomingState));
> qemu_mutex_init(&mis_current.rp_mutex);
> qemu_event_init(&mis_current.main_thread_load_event, false);
> +
> + init_dirty_bitmap_incoming_migration();
> +
You might want to consider if that's better in vl.c near where
ram_mig_init() is, OR whether there should be a call in
migratation_incoming_state_destroy to clean it up.
(Although I doubt the cases where the destroy happens are interesting
for postcopy bitmaps).
> once = true;
> }
> return &mis_current;
> @@ -297,6 +300,8 @@ static void process_incoming_migration_bh(void *opaque)
> state, we need to obey autostart. Any other state is set with
> runstate_set. */
>
> + dirty_bitmap_mig_before_vm_start();
> +
> if (!global_state_received() ||
> global_state_get_runstate() == RUN_STATE_RUNNING) {
> if (autostart) {
> diff --git a/migration/savevm.c b/migration/savevm.c
> index e5d557458e..93b339646b 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -1673,6 +1673,8 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)
>
> trace_loadvm_postcopy_handle_run_vmstart();
>
> + dirty_bitmap_mig_before_vm_start();
> +
> if (autostart) {
> /* Hold onto your hats, starting the CPU */
> vm_start();
> diff --git a/vl.c b/vl.c
> index e517a8d995..0ef3f2b5a2 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -4514,6 +4514,7 @@ int main(int argc, char **argv, char **envp)
>
> blk_mig_init();
> ram_mig_init();
> + dirty_bitmap_mig_init();
>
> /* If the currently selected machine wishes to override the units-per-bus
> * property of its default HBA interface type, do so now. */
> diff --git a/migration/Makefile.objs b/migration/Makefile.objs
> index 99e038024d..c83ec47ba8 100644
> --- a/migration/Makefile.objs
> +++ b/migration/Makefile.objs
> @@ -6,6 +6,7 @@ common-obj-y += qemu-file.o global_state.o
> common-obj-y += qemu-file-channel.o
> common-obj-y += xbzrle.o postcopy-ram.o
> common-obj-y += qjson.o
> +common-obj-y += block-dirty-bitmap.o
>
> common-obj-$(CONFIG_RDMA) += rdma.o
>
> diff --git a/migration/trace-events b/migration/trace-events
> index a04fffb877..e9eb8078d4 100644
> --- a/migration/trace-events
> +++ b/migration/trace-events
> @@ -227,3 +227,17 @@ colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
> colo_send_message(const char *msg) "Send '%s' message"
> colo_receive_message(const char *msg) "Receive '%s' message"
> colo_failover_set_state(const char *new_state) "new state %s"
> +
> +# migration/block-dirty-bitmap.c
> +send_bitmap_header_enter(void) ""
> +send_bitmap_bits(uint32_t flags, uint64_t start_sector, uint32_t nr_sectors, uint64_t data_size) "\n flags: 0x%x\n start_sector: %" PRIu64 "\n nr_sectors: %" PRIu32 "\n data_size: %" PRIu64 "\n"
Tracing doesn't have \n's in
> +dirty_bitmap_save_iterate(int in_postcopy) "in postcopy: %d"
> +dirty_bitmap_save_complete_enter(void) ""
> +dirty_bitmap_save_complete_finish(void) ""
> +dirty_bitmap_save_pending(uint64_t pending, uint64_t max_size) "pending %" PRIu64 " max: %" PRIu64
> +dirty_bitmap_load_complete(void) ""
> +dirty_bitmap_load_bits_enter(uint64_t first_sector, uint32_t nr_sectors) "chunk: %" PRIu64 " %" PRIu32
> +dirty_bitmap_load_bits_zeroes(void) ""
> +dirty_bitmap_load_header(uint32_t flags) "flags 0x%x"
> +dirty_bitmap_load_enter(void) ""
> +dirty_bitmap_load_success(void) ""
So other than minor bits, this one looks OK from a migration side; I
can't say I've followed the block side of the patch though.
Dave
> --
> 2.11.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 11/12] iotests: add dirty bitmap migration test
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 11/12] iotests: add dirty bitmap migration test Vladimir Sementsov-Ogievskiy
@ 2018-03-12 16:19 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 33+ messages in thread
From: Dr. David Alan Gilbert @ 2018-03-12 16:19 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
* Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> The test starts two vms (vm_a, vm_b), create dirty bitmap in
> the first one, do several writes to corresponding device and
> then migrate vm_a to vm_b.
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
> tests/qemu-iotests/169 | 141 +++++++++++++++++++++++++++++++++++++++++++++
> tests/qemu-iotests/169.out | 5 ++
> tests/qemu-iotests/group | 1 +
> 3 files changed, 147 insertions(+)
> create mode 100755 tests/qemu-iotests/169
> create mode 100644 tests/qemu-iotests/169.out
>
> diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
> new file mode 100755
> index 0000000000..8801f73b10
> --- /dev/null
> +++ b/tests/qemu-iotests/169
> @@ -0,0 +1,141 @@
> +#!/usr/bin/env python
> +#
> +# Tests for dirty bitmaps migration.
> +#
> +# Copyright (c) 2016-2017 Virtuozzo International GmbH. All rights reserved.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import os
> +import iotests
> +import time
> +import itertools
> +import operator
> +import new
> +from iotests import qemu_img
> +
> +
> +disk_a = os.path.join(iotests.test_dir, 'disk_a')
> +disk_b = os.path.join(iotests.test_dir, 'disk_b')
> +size = '1M'
> +mig_file = os.path.join(iotests.test_dir, 'mig_file')
> +
> +
> +class TestDirtyBitmapMigration(iotests.QMPTestCase):
> + def tearDown(self):
> + self.vm_a.shutdown()
> + self.vm_b.shutdown()
> + os.remove(disk_a)
> + os.remove(disk_b)
> + os.remove(mig_file)
> +
> + def setUp(self):
> + qemu_img('create', '-f', iotests.imgfmt, disk_a, size)
> + qemu_img('create', '-f', iotests.imgfmt, disk_b, size)
> +
> + self.vm_a = iotests.VM(path_suffix='a').add_drive(disk_a)
> + self.vm_a.launch()
> +
> + self.vm_b = iotests.VM(path_suffix='b')
> + self.vm_b.add_incoming("exec: cat '" + mig_file + "'")
> +
> + def add_bitmap(self, vm, granularity, persistent):
> + params = {'node': 'drive0',
> + 'name': 'bitmap0',
> + 'granularity': granularity}
> + if persistent:
> + params['persistent'] = True
> + params['autoload'] = True
> +
> + result = vm.qmp('block-dirty-bitmap-add', **params)
> + self.assert_qmp(result, 'return', {});
> +
> + def get_bitmap_hash(self, vm):
> + result = vm.qmp('x-debug-block-dirty-bitmap-sha256',
> + node='drive0', name='bitmap0')
> + return result['return']['sha256']
> +
> + def check_bitmap(self, vm, sha256):
> + result = vm.qmp('x-debug-block-dirty-bitmap-sha256',
> + node='drive0', name='bitmap0')
> + if sha256:
> + self.assert_qmp(result, 'return/sha256', sha256);
> + else:
> + self.assert_qmp(result, 'error/desc',
> + "Dirty bitmap 'bitmap0' not found");
> +
> + def do_test_migration(self, persistent, migrate_bitmaps, online,
> + shared_storage):
> + granularity = 512
> +
> + # regions = ((start, count), ...)
> + regions = ((0, 0x10000),
> + (0xf0000, 0x10000),
> + (0xa0201, 0x1000))
> +
> + should_migrate = migrate_bitmaps or persistent and shared_storage
> +
> + self.vm_b.add_drive(disk_a if shared_storage else disk_b)
> +
> + if online:
> + os.mkfifo(mig_file)
> + self.vm_b.launch()
> +
> + self.add_bitmap(self.vm_a, granularity, persistent)
> + for r in regions:
> + self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % r)
> + sha256 = self.get_bitmap_hash(self.vm_a)
> +
> + if migrate_bitmaps:
> + result = self.vm_a.qmp('migrate-set-capabilities',
> + capabilities=[{'capability': 'dirty-bitmaps',
> + 'state': True}])
> + self.assert_qmp(result, 'return', {})
> +
> + result = self.vm_a.qmp('migrate', uri='exec:cat>' + mig_file)
> + self.vm_a.event_wait("STOP")
Stefan recently fixed a similar case in another test; 'STOP' is not
sufficient to know migration has finished, just that it's in the last
stage. The trick is to use the new migration events and wait for the
finish in there.
Other than that I'm not really a python person, but it looks OK fromt he
migration side of it.
Dave
> + if not online:
> + self.vm_a.shutdown()
> + self.vm_b.launch()
> +
> + self.vm_b.event_wait("RESUME", timeout=10.0)
> +
> + self.check_bitmap(self.vm_b, sha256 if should_migrate else False)
> +
> + if should_migrate:
> + self.vm_b.shutdown()
> + self.vm_b.launch()
> + self.check_bitmap(self.vm_b, sha256 if persistent else False)
> +
> +
> +def inject_test_case(klass, name, method, *args, **kwargs):
> + mc = operator.methodcaller(method, *args, **kwargs)
> + setattr(klass, 'test_' + name, new.instancemethod(mc, None, klass))
> +
> +for cmb in list(itertools.product((True, False), repeat=3)):
> + name = ('_' if cmb[0] else '_not_') + 'persistent_'
> + name += ('_' if cmb[1] else '_not_') + 'migbitmap_'
> + name += '_online' if cmb[2] else '_offline'
> +
> + # TODO fix shared-storage bitmap migration and enable cases for it
> + args = list(cmb) + [False]
> +
> + inject_test_case(TestDirtyBitmapMigration, name, 'do_test_migration',
> + *args)
> +
> +
> +if __name__ == '__main__':
> + iotests.main(supported_fmts=['qcow2'])
> diff --git a/tests/qemu-iotests/169.out b/tests/qemu-iotests/169.out
> new file mode 100644
> index 0000000000..594c16f49f
> --- /dev/null
> +++ b/tests/qemu-iotests/169.out
> @@ -0,0 +1,5 @@
> +........
> +----------------------------------------------------------------------
> +Ran 8 tests
> +
> +OK
> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> index a2dfe79d86..498c52c4c1 100644
> --- a/tests/qemu-iotests/group
> +++ b/tests/qemu-iotests/group
> @@ -169,6 +169,7 @@
> 162 auto quick
> 163 rw auto quick
> 165 rw auto quick
> +169 rw auto quick
> 170 rw auto quick
> 171 rw auto quick
> 172 auto
> --
> 2.11.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 12/12] iotests: add dirty bitmap postcopy test
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 12/12] iotests: add dirty bitmap postcopy test Vladimir Sementsov-Ogievskiy
@ 2018-03-12 16:24 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 33+ messages in thread
From: Dr. David Alan Gilbert @ 2018-03-12 16:24 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
* Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> Test
> - start two vms (vm_a, vm_b)
>
> - in a
> - do writes from set A
> - do writes from set B
> - fix bitmap sha256
> - clear bitmap
> - do writes from set A
> - start migration
> - than, in b
> - wait vm start (postcopy should start)
> - do writes from set B
> - check bitmap sha256
>
> The test should verify postcopy migration and then merging with delta
> (changes in target, during postcopy process).
>
> Reduce supported cache modes to only 'none', because with cache on time
> from source.STOP to target.RESUME is unpredictable and we can fail with
> timout while waiting for target.RESUME.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Reviewed-by: Fam Zheng <famz@redhat.com>
> ---
> tests/qemu-iotests/199 | 105 ++++++++++++++++++++++++++++++++++++++++++
> tests/qemu-iotests/199.out | 5 ++
> tests/qemu-iotests/group | 1 +
> tests/qemu-iotests/iotests.py | 7 ++-
> 4 files changed, 117 insertions(+), 1 deletion(-)
> create mode 100755 tests/qemu-iotests/199
> create mode 100644 tests/qemu-iotests/199.out
>
> diff --git a/tests/qemu-iotests/199 b/tests/qemu-iotests/199
> new file mode 100755
> index 0000000000..f872040a81
> --- /dev/null
> +++ b/tests/qemu-iotests/199
> @@ -0,0 +1,105 @@
> +#!/usr/bin/env python
> +#
> +# Tests for dirty bitmaps postcopy migration.
> +#
> +# Copyright (c) 2016-2017 Virtuozzo International GmbH. All rights reserved.
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program. If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import os
> +import iotests
> +import time
> +from iotests import qemu_img
> +
> +disk_a = os.path.join(iotests.test_dir, 'disk_a')
> +disk_b = os.path.join(iotests.test_dir, 'disk_b')
> +size = '256G'
> +fifo = os.path.join(iotests.test_dir, 'mig_fifo')
> +
> +class TestDirtyBitmapPostcopyMigration(iotests.QMPTestCase):
> +
> + def tearDown(self):
> + self.vm_a.shutdown()
> + self.vm_b.shutdown()
> + os.remove(disk_a)
> + os.remove(disk_b)
> + os.remove(fifo)
> +
> + def setUp(self):
> + os.mkfifo(fifo)
> + qemu_img('create', '-f', iotests.imgfmt, disk_a, size)
> + qemu_img('create', '-f', iotests.imgfmt, disk_b, size)
> + self.vm_a = iotests.VM(path_suffix='a').add_drive(disk_a)
> + self.vm_b = iotests.VM(path_suffix='b').add_drive(disk_b)
> + self.vm_b.add_incoming("exec: cat '" + fifo + "'")
> + self.vm_a.launch()
> + self.vm_b.launch()
> +
> + def test_postcopy(self):
> + write_size = 0x40000000
> + granularity = 512
> + chunk = 4096
> +
> + result = self.vm_a.qmp('block-dirty-bitmap-add', node='drive0',
> + name='bitmap', granularity=granularity)
> + self.assert_qmp(result, 'return', {});
> +
> + s = 0
> + while s < write_size:
> + self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
> + s += 0x10000
> + s = 0x8000
> + while s < write_size:
> + self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
> + s += 0x10000
> +
> + result = self.vm_a.qmp('x-debug-block-dirty-bitmap-sha256',
> + node='drive0', name='bitmap')
> + sha256 = result['return']['sha256']
> +
> + result = self.vm_a.qmp('block-dirty-bitmap-clear', node='drive0',
> + name='bitmap')
> + self.assert_qmp(result, 'return', {});
> + s = 0
> + while s < write_size:
> + self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
> + s += 0x10000
> +
> + result = self.vm_a.qmp('migrate-set-capabilities',
> + capabilities=[{'capability': 'dirty-bitmaps',
> + 'state': True}])
Generally these days we set the capability on both sides of the
migration (which helps if the destination tells you it can't do it for
some reason).
Again from the migration side this looks OK, but it's really mostly the
details of the IO that I'll leave to the block people.
Dave
> + self.assert_qmp(result, 'return', {})
> +
> + result = self.vm_a.qmp('migrate', uri='exec:cat>' + fifo)
> + self.assertNotEqual(self.vm_a.event_wait("STOP"), None)
> + self.assertNotEqual(self.vm_b.event_wait("RESUME"), None)
> +
> + s = 0x8000
> + while s < write_size:
> + self.vm_b.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
> + s += 0x10000
> +
> + result = self.vm_b.qmp('query-block');
> + while len(result['return'][0]['dirty-bitmaps']) > 1:
> + time.sleep(2)
> + result = self.vm_b.qmp('query-block');
> +
> + result = self.vm_b.qmp('x-debug-block-dirty-bitmap-sha256',
> + node='drive0', name='bitmap')
> +
> + self.assert_qmp(result, 'return/sha256', sha256);
> +
> +if __name__ == '__main__':
> + iotests.main(supported_fmts=['qcow2'], supported_cache_modes=['none'])
> diff --git a/tests/qemu-iotests/199.out b/tests/qemu-iotests/199.out
> new file mode 100644
> index 0000000000..ae1213e6f8
> --- /dev/null
> +++ b/tests/qemu-iotests/199.out
> @@ -0,0 +1,5 @@
> +.
> +----------------------------------------------------------------------
> +Ran 1 tests
> +
> +OK
> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> index 498c52c4c1..282c5ebd5c 100644
> --- a/tests/qemu-iotests/group
> +++ b/tests/qemu-iotests/group
> @@ -197,6 +197,7 @@
> 196 rw auto quick
> 197 rw auto quick
> 198 rw auto
> +199 rw auto
> 200 rw auto
> 201 rw auto migration
> 202 rw auto quick
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 5a10b2d534..a9bded77e4 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -473,6 +473,10 @@ def verify_platform(supported_oses=['linux']):
> if True not in [sys.platform.startswith(x) for x in supported_oses]:
> notrun('not suitable for this OS: %s' % sys.platform)
>
> +def verify_cache_mode(supported_cache_modes=[]):
> + if supported_cache_modes and (cachemode not in supported_cache_modes):
> + notrun('not suitable for this cache mode: %s' % cachemode)
> +
> def supports_quorum():
> return 'quorum' in qemu_img_pipe('--help')
>
> @@ -481,7 +485,7 @@ def verify_quorum():
> if not supports_quorum():
> notrun('quorum support missing')
>
> -def main(supported_fmts=[], supported_oses=['linux']):
> +def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[]):
> '''Run tests'''
>
> global debug
> @@ -498,6 +502,7 @@ def main(supported_fmts=[], supported_oses=['linux']):
> verbosity = 1
> verify_image_format(supported_fmts)
> verify_platform(supported_oses)
> + verify_cache_mode(supported_cache_modes)
>
> # We need to filter out the time taken from the output so that qemu-iotest
> # can reliably diff the results against master output.
> --
> 2.11.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-12 15:30 ` Dr. David Alan Gilbert
@ 2018-03-13 5:38 ` John Snow
2018-03-13 12:17 ` Dr. David Alan Gilbert
2018-03-13 7:47 ` Vladimir Sementsov-Ogievskiy
1 sibling, 1 reply; 33+ messages in thread
From: John Snow @ 2018-03-13 5:38 UTC (permalink / raw)
To: Dr. David Alan Gilbert, Vladimir Sementsov-Ogievskiy
Cc: kwolf, peter.maydell, famz, lirans, qemu-block, quintela,
qemu-devel, armbru, stefanha, den, amit.shah, pbonzini, mreitz
On 03/12/2018 11:30 AM, Dr. David Alan Gilbert wrote:
> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>> There would be savevm states (dirty-bitmap) which can migrate only in
>> postcopy stage. The corresponding pending is introduced here.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>> include/migration/register.h | 17 +++++++++++++++--
>> migration/savevm.h | 5 +++--
>> hw/s390x/s390-stattrib.c | 7 ++++---
>> migration/block.c | 7 ++++---
>> migration/migration.c | 16 +++++++++-------
>> migration/ram.c | 9 +++++----
>> migration/savevm.c | 13 ++++++++-----
>> migration/trace-events | 2 +-
>> 8 files changed, 49 insertions(+), 27 deletions(-)
>>
>> diff --git a/include/migration/register.h b/include/migration/register.h
>> index f4f7bdc177..9436a87678 100644
>> --- a/include/migration/register.h
>> +++ b/include/migration/register.h
>> @@ -37,8 +37,21 @@ typedef struct SaveVMHandlers {
>> int (*save_setup)(QEMUFile *f, void *opaque);
>> void (*save_live_pending)(QEMUFile *f, void *opaque,
>> uint64_t threshold_size,
>> - uint64_t *non_postcopiable_pending,
>> - uint64_t *postcopiable_pending);
>> + uint64_t *res_precopy_only,
>> + uint64_t *res_compatible,
>> + uint64_t *res_postcopy_only);
>> + /* Note for save_live_pending:
>> + * - res_precopy_only is for data which must be migrated in precopy phase
>> + * or in stopped state, in other words - before target vm start
>> + * - res_compatible is for data which may be migrated in any phase
>> + * - res_postcopy_only is for data which must be migrated in postcopy phase
>> + * or in stopped state, in other words - after source vm stop
>> + *
>> + * Sum of res_postcopy_only, res_compatible and res_postcopy_only is the
>> + * whole amount of pending data.
>> + */
>> +
>> +
>> LoadStateHandler *load_state;
>> int (*load_setup)(QEMUFile *f, void *opaque);
>> int (*load_cleanup)(void *opaque);
>> diff --git a/migration/savevm.h b/migration/savevm.h
>> index 295c4a1f2c..cf4f0d37ca 100644
>> --- a/migration/savevm.h
>> +++ b/migration/savevm.h
>> @@ -38,8 +38,9 @@ void qemu_savevm_state_complete_postcopy(QEMUFile *f);
>> int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only,
>> bool inactivate_disks);
>> void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
>> - uint64_t *res_non_postcopiable,
>> - uint64_t *res_postcopiable);
>> + uint64_t *res_precopy_only,
>> + uint64_t *res_compatible,
>> + uint64_t *res_postcopy_only);
>> void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
>> void qemu_savevm_send_open_return_path(QEMUFile *f);
>> int qemu_savevm_send_packaged(QEMUFile *f, const uint8_t *buf, size_t len);
>> diff --git a/hw/s390x/s390-stattrib.c b/hw/s390x/s390-stattrib.c
>> index 2902f54f11..dd3fbfd1eb 100644
>> --- a/hw/s390x/s390-stattrib.c
>> +++ b/hw/s390x/s390-stattrib.c
>> @@ -183,15 +183,16 @@ static int cmma_save_setup(QEMUFile *f, void *opaque)
>> }
>>
>> static void cmma_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
>> - uint64_t *non_postcopiable_pending,
>> - uint64_t *postcopiable_pending)
>> + uint64_t *res_precopy_only,
>> + uint64_t *res_compatible,
>> + uint64_t *res_postcopy_only)
>> {
>> S390StAttribState *sas = S390_STATTRIB(opaque);
>> S390StAttribClass *sac = S390_STATTRIB_GET_CLASS(sas);
>> long long res = sac->get_dirtycount(sas);
>>
>> if (res >= 0) {
>> - *non_postcopiable_pending += res;
>> + *res_precopy_only += res;
>> }
>> }
>>
>> diff --git a/migration/block.c b/migration/block.c
>> index 1f03946797..5652ca3337 100644
>> --- a/migration/block.c
>> +++ b/migration/block.c
>> @@ -866,8 +866,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)
>> }
>>
>> static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
>> - uint64_t *non_postcopiable_pending,
>> - uint64_t *postcopiable_pending)
>> + uint64_t *res_precopy_only,
>> + uint64_t *res_compatible,
>> + uint64_t *res_postcopy_only)
>> {
>> /* Estimate pending number of bytes to send */
>> uint64_t pending;
>> @@ -888,7 +889,7 @@ static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
>>
>> DPRINTF("Enter save live pending %" PRIu64 "\n", pending);
>> /* We don't do postcopy */
>> - *non_postcopiable_pending += pending;
>> + *res_precopy_only += pending;
>> }
>>
>> static int block_load(QEMUFile *f, void *opaque, int version_id)
>> diff --git a/migration/migration.c b/migration/migration.c
>> index c99a4e62d7..3beedd676e 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -2215,21 +2215,23 @@ typedef enum {
>> */
>> static MigIterateState migration_iteration_run(MigrationState *s)
>> {
>> - uint64_t pending_size, pend_post, pend_nonpost;
>> + uint64_t pending_size, pend_pre, pend_compat, pend_post;
>> bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE;
>>
>> - qemu_savevm_state_pending(s->to_dst_file, s->threshold_size,
>> - &pend_nonpost, &pend_post);
>> - pending_size = pend_nonpost + pend_post;
>> + qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre,
>> + &pend_compat, &pend_post);
>> + pending_size = pend_pre + pend_compat + pend_post;
>>
>> trace_migrate_pending(pending_size, s->threshold_size,
>> - pend_post, pend_nonpost);
>> + pend_pre, pend_compat, pend_post);
>>
>> if (pending_size && pending_size >= s->threshold_size) {
>> /* Still a significant amount to transfer */
>> if (migrate_postcopy() && !in_postcopy &&
>> - pend_nonpost <= s->threshold_size &&
>> - atomic_read(&s->start_postcopy)) {
>> + pend_pre <= s->threshold_size &&
>> + (atomic_read(&s->start_postcopy) ||
>> + (pend_pre + pend_compat <= s->threshold_size)))
>
> This change does something different from the description;
> it causes a postcopy_start even if the user never ran the postcopy-start
> command; so sorry, we can't do that; because postcopy for RAM is
> something that users can enable but only switch into when they've given
> up on it completing normally.
>
> However, I guess that leaves you with a problem; which is what happens
> to the system when you've run out of pend_pre+pend_compat but can't
> complete because pend_post is non-0; so I don't know the answer to that.
>
> Dave
>
I'm willing to pull the bitmap related changes into my tree for 2.12,
but it would be _really_ nice to have this patchset in 2.12 -- is there
anything we can do to get this into the migration queue?
I'm afraid I don't know what the right answer here is either, so I can't
fix this up myself and I imagine Vladimir will need some feedback from
you to change this design.
Are we -hosed- ?
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-12 15:30 ` Dr. David Alan Gilbert
2018-03-13 5:38 ` John Snow
@ 2018-03-13 7:47 ` Vladimir Sementsov-Ogievskiy
2018-03-13 10:30 ` Dr. David Alan Gilbert
1 sibling, 1 reply; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-03-13 7:47 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
12.03.2018 18:30, Dr. David Alan Gilbert wrote:
> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>> There would be savevm states (dirty-bitmap) which can migrate only in
>> postcopy stage. The corresponding pending is introduced here.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
[...]
>> static MigIterateState migration_iteration_run(MigrationState *s)
>> {
>> - uint64_t pending_size, pend_post, pend_nonpost;
>> + uint64_t pending_size, pend_pre, pend_compat, pend_post;
>> bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE;
>>
>> - qemu_savevm_state_pending(s->to_dst_file, s->threshold_size,
>> - &pend_nonpost, &pend_post);
>> - pending_size = pend_nonpost + pend_post;
>> + qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre,
>> + &pend_compat, &pend_post);
>> + pending_size = pend_pre + pend_compat + pend_post;
>>
>> trace_migrate_pending(pending_size, s->threshold_size,
>> - pend_post, pend_nonpost);
>> + pend_pre, pend_compat, pend_post);
>>
>> if (pending_size && pending_size >= s->threshold_size) {
>> /* Still a significant amount to transfer */
>> if (migrate_postcopy() && !in_postcopy &&
>> - pend_nonpost <= s->threshold_size &&
>> - atomic_read(&s->start_postcopy)) {
>> + pend_pre <= s->threshold_size &&
>> + (atomic_read(&s->start_postcopy) ||
>> + (pend_pre + pend_compat <= s->threshold_size)))
> This change does something different from the description;
> it causes a postcopy_start even if the user never ran the postcopy-start
> command; so sorry, we can't do that; because postcopy for RAM is
> something that users can enable but only switch into when they've given
> up on it completing normally.
>
> However, I guess that leaves you with a problem; which is what happens
> to the system when you've run out of pend_pre+pend_compat but can't
> complete because pend_post is non-0; so I don't know the answer to that.
>
>
Hmm. Here, we go to postcopy only if "pend_pre + pend_compat <=
s->threshold_size". Pre-patch, in this case we will go to
migration_completion(). So, precopy stage is finishing anyway. So, we
want in this case to finish ram migration like it was finished by
migration_completion(), and then, run postcopy, which will handle only
dirty bitmaps, yes?
Hmm2. Looked through migration_completion(), I don't understand, how it
finishes ram migration without postcopy. It calls
qemu_savevm_state_complete_precopy(), which skips states with
has_postcopy=true, which is ram...
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-13 7:47 ` Vladimir Sementsov-Ogievskiy
@ 2018-03-13 10:30 ` Dr. David Alan Gilbert
2018-03-13 13:11 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 33+ messages in thread
From: Dr. David Alan Gilbert @ 2018-03-13 10:30 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
* Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> 12.03.2018 18:30, Dr. David Alan Gilbert wrote:
> > * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> > > There would be savevm states (dirty-bitmap) which can migrate only in
> > > postcopy stage. The corresponding pending is introduced here.
> > >
> > > Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> > > ---
>
> [...]
>
> > > static MigIterateState migration_iteration_run(MigrationState *s)
> > > {
> > > - uint64_t pending_size, pend_post, pend_nonpost;
> > > + uint64_t pending_size, pend_pre, pend_compat, pend_post;
> > > bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE;
> > > - qemu_savevm_state_pending(s->to_dst_file, s->threshold_size,
> > > - &pend_nonpost, &pend_post);
> > > - pending_size = pend_nonpost + pend_post;
> > > + qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre,
> > > + &pend_compat, &pend_post);
> > > + pending_size = pend_pre + pend_compat + pend_post;
> > > trace_migrate_pending(pending_size, s->threshold_size,
> > > - pend_post, pend_nonpost);
> > > + pend_pre, pend_compat, pend_post);
> > > if (pending_size && pending_size >= s->threshold_size) {
> > > /* Still a significant amount to transfer */
> > > if (migrate_postcopy() && !in_postcopy &&
> > > - pend_nonpost <= s->threshold_size &&
> > > - atomic_read(&s->start_postcopy)) {
> > > + pend_pre <= s->threshold_size &&
> > > + (atomic_read(&s->start_postcopy) ||
> > > + (pend_pre + pend_compat <= s->threshold_size)))
> > This change does something different from the description;
> > it causes a postcopy_start even if the user never ran the postcopy-start
> > command; so sorry, we can't do that; because postcopy for RAM is
> > something that users can enable but only switch into when they've given
> > up on it completing normally.
> >
> > However, I guess that leaves you with a problem; which is what happens
> > to the system when you've run out of pend_pre+pend_compat but can't
> > complete because pend_post is non-0; so I don't know the answer to that.
> >
> >
>
> Hmm. Here, we go to postcopy only if "pend_pre + pend_compat <=
> s->threshold_size". Pre-patch, in this case we will go to
> migration_completion(). So, precopy stage is finishing anyway.
Right.
> So, we want
> in this case to finish ram migration like it was finished by
> migration_completion(), and then, run postcopy, which will handle only dirty
> bitmaps, yes?
It's a bit tricky; the first important thing is that we can't change the
semantics of the migration without the 'dirty bitmaps'.
So then there's the question of how a migration with both
postcopy-ram+dirty bitmaps should work; again I don't think we should
enter the postcopy-ram phase until start-postcopy is issued.
Then there's the 3rd case; dirty-bitmaps but no postcopy-ram; in that
case I worry less about the semantics of how you want to do it.
> Hmm2. Looked through migration_completion(), I don't understand, how it
> finishes ram migration without postcopy. It calls
> qemu_savevm_state_complete_precopy(), which skips states with
> has_postcopy=true, which is ram...
Because savevm_state_complete_precopy only skips has_postcopy=true in
the in_postcopy case:
(in_postcopy && se->ops->has_postcopy &&
se->ops->has_postcopy(se->opaque)) ||
so when we call it in migration_completion(), if we've not entered
postcopy yet, then that test doesn't trigger.
(Apologies for not spotting this earlier; but I thought this patch was
a nice easy one just adding the postcopy_only_pending - I didn't realise it changed
existing semantics until I spotted that)
Dave
> --
> Best regards,
> Vladimir
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-13 5:38 ` John Snow
@ 2018-03-13 12:17 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 33+ messages in thread
From: Dr. David Alan Gilbert @ 2018-03-13 12:17 UTC (permalink / raw)
To: John Snow
Cc: Vladimir Sementsov-Ogievskiy, kwolf, peter.maydell, famz, lirans,
qemu-block, quintela, qemu-devel, armbru, stefanha, den,
amit.shah, pbonzini, mreitz
* John Snow (jsnow@redhat.com) wrote:
>
>
> On 03/12/2018 11:30 AM, Dr. David Alan Gilbert wrote:
> > * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> >> There would be savevm states (dirty-bitmap) which can migrate only in
> >> postcopy stage. The corresponding pending is introduced here.
> >>
> >> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> >> ---
> >> include/migration/register.h | 17 +++++++++++++++--
> >> migration/savevm.h | 5 +++--
> >> hw/s390x/s390-stattrib.c | 7 ++++---
> >> migration/block.c | 7 ++++---
> >> migration/migration.c | 16 +++++++++-------
> >> migration/ram.c | 9 +++++----
> >> migration/savevm.c | 13 ++++++++-----
> >> migration/trace-events | 2 +-
> >> 8 files changed, 49 insertions(+), 27 deletions(-)
> >>
> >> diff --git a/include/migration/register.h b/include/migration/register.h
> >> index f4f7bdc177..9436a87678 100644
> >> --- a/include/migration/register.h
> >> +++ b/include/migration/register.h
> >> @@ -37,8 +37,21 @@ typedef struct SaveVMHandlers {
> >> int (*save_setup)(QEMUFile *f, void *opaque);
> >> void (*save_live_pending)(QEMUFile *f, void *opaque,
> >> uint64_t threshold_size,
> >> - uint64_t *non_postcopiable_pending,
> >> - uint64_t *postcopiable_pending);
> >> + uint64_t *res_precopy_only,
> >> + uint64_t *res_compatible,
> >> + uint64_t *res_postcopy_only);
> >> + /* Note for save_live_pending:
> >> + * - res_precopy_only is for data which must be migrated in precopy phase
> >> + * or in stopped state, in other words - before target vm start
> >> + * - res_compatible is for data which may be migrated in any phase
> >> + * - res_postcopy_only is for data which must be migrated in postcopy phase
> >> + * or in stopped state, in other words - after source vm stop
> >> + *
> >> + * Sum of res_postcopy_only, res_compatible and res_postcopy_only is the
> >> + * whole amount of pending data.
> >> + */
> >> +
> >> +
> >> LoadStateHandler *load_state;
> >> int (*load_setup)(QEMUFile *f, void *opaque);
> >> int (*load_cleanup)(void *opaque);
> >> diff --git a/migration/savevm.h b/migration/savevm.h
> >> index 295c4a1f2c..cf4f0d37ca 100644
> >> --- a/migration/savevm.h
> >> +++ b/migration/savevm.h
> >> @@ -38,8 +38,9 @@ void qemu_savevm_state_complete_postcopy(QEMUFile *f);
> >> int qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only,
> >> bool inactivate_disks);
> >> void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
> >> - uint64_t *res_non_postcopiable,
> >> - uint64_t *res_postcopiable);
> >> + uint64_t *res_precopy_only,
> >> + uint64_t *res_compatible,
> >> + uint64_t *res_postcopy_only);
> >> void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
> >> void qemu_savevm_send_open_return_path(QEMUFile *f);
> >> int qemu_savevm_send_packaged(QEMUFile *f, const uint8_t *buf, size_t len);
> >> diff --git a/hw/s390x/s390-stattrib.c b/hw/s390x/s390-stattrib.c
> >> index 2902f54f11..dd3fbfd1eb 100644
> >> --- a/hw/s390x/s390-stattrib.c
> >> +++ b/hw/s390x/s390-stattrib.c
> >> @@ -183,15 +183,16 @@ static int cmma_save_setup(QEMUFile *f, void *opaque)
> >> }
> >>
> >> static void cmma_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
> >> - uint64_t *non_postcopiable_pending,
> >> - uint64_t *postcopiable_pending)
> >> + uint64_t *res_precopy_only,
> >> + uint64_t *res_compatible,
> >> + uint64_t *res_postcopy_only)
> >> {
> >> S390StAttribState *sas = S390_STATTRIB(opaque);
> >> S390StAttribClass *sac = S390_STATTRIB_GET_CLASS(sas);
> >> long long res = sac->get_dirtycount(sas);
> >>
> >> if (res >= 0) {
> >> - *non_postcopiable_pending += res;
> >> + *res_precopy_only += res;
> >> }
> >> }
> >>
> >> diff --git a/migration/block.c b/migration/block.c
> >> index 1f03946797..5652ca3337 100644
> >> --- a/migration/block.c
> >> +++ b/migration/block.c
> >> @@ -866,8 +866,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)
> >> }
> >>
> >> static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
> >> - uint64_t *non_postcopiable_pending,
> >> - uint64_t *postcopiable_pending)
> >> + uint64_t *res_precopy_only,
> >> + uint64_t *res_compatible,
> >> + uint64_t *res_postcopy_only)
> >> {
> >> /* Estimate pending number of bytes to send */
> >> uint64_t pending;
> >> @@ -888,7 +889,7 @@ static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
> >>
> >> DPRINTF("Enter save live pending %" PRIu64 "\n", pending);
> >> /* We don't do postcopy */
> >> - *non_postcopiable_pending += pending;
> >> + *res_precopy_only += pending;
> >> }
> >>
> >> static int block_load(QEMUFile *f, void *opaque, int version_id)
> >> diff --git a/migration/migration.c b/migration/migration.c
> >> index c99a4e62d7..3beedd676e 100644
> >> --- a/migration/migration.c
> >> +++ b/migration/migration.c
> >> @@ -2215,21 +2215,23 @@ typedef enum {
> >> */
> >> static MigIterateState migration_iteration_run(MigrationState *s)
> >> {
> >> - uint64_t pending_size, pend_post, pend_nonpost;
> >> + uint64_t pending_size, pend_pre, pend_compat, pend_post;
> >> bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE;
> >>
> >> - qemu_savevm_state_pending(s->to_dst_file, s->threshold_size,
> >> - &pend_nonpost, &pend_post);
> >> - pending_size = pend_nonpost + pend_post;
> >> + qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre,
> >> + &pend_compat, &pend_post);
> >> + pending_size = pend_pre + pend_compat + pend_post;
> >>
> >> trace_migrate_pending(pending_size, s->threshold_size,
> >> - pend_post, pend_nonpost);
> >> + pend_pre, pend_compat, pend_post);
> >>
> >> if (pending_size && pending_size >= s->threshold_size) {
> >> /* Still a significant amount to transfer */
> >> if (migrate_postcopy() && !in_postcopy &&
> >> - pend_nonpost <= s->threshold_size &&
> >> - atomic_read(&s->start_postcopy)) {
> >> + pend_pre <= s->threshold_size &&
> >> + (atomic_read(&s->start_postcopy) ||
> >> + (pend_pre + pend_compat <= s->threshold_size)))
> >
> > This change does something different from the description;
> > it causes a postcopy_start even if the user never ran the postcopy-start
> > command; so sorry, we can't do that; because postcopy for RAM is
> > something that users can enable but only switch into when they've given
> > up on it completing normally.
> >
> > However, I guess that leaves you with a problem; which is what happens
> > to the system when you've run out of pend_pre+pend_compat but can't
> > complete because pend_post is non-0; so I don't know the answer to that.
> >
> > Dave
> >
>
> I'm willing to pull the bitmap related changes into my tree for 2.12,
> but it would be _really_ nice to have this patchset in 2.12 -- is there
> anything we can do to get this into the migration queue?
>
> I'm afraid I don't know what the right answer here is either, so I can't
> fix this up myself and I imagine Vladimir will need some feedback from
> you to change this design.
Well I'm good with everything except the change in semantics for
existing migration; we can't do that because it can break existing users.
> Are we -hosed- ?
Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-13 10:30 ` Dr. David Alan Gilbert
@ 2018-03-13 13:11 ` Vladimir Sementsov-Ogievskiy
2018-03-13 13:32 ` Vladimir Sementsov-Ogievskiy
2018-03-13 15:35 ` Dr. David Alan Gilbert
0 siblings, 2 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-03-13 13:11 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
13.03.2018 13:30, Dr. David Alan Gilbert wrote:
> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>> 12.03.2018 18:30, Dr. David Alan Gilbert wrote:
>>> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>>>> There would be savevm states (dirty-bitmap) which can migrate only in
>>>> postcopy stage. The corresponding pending is introduced here.
>>>>
>>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>>> ---
>> [...]
>>
>>>> static MigIterateState migration_iteration_run(MigrationState *s)
>>>> {
>>>> - uint64_t pending_size, pend_post, pend_nonpost;
>>>> + uint64_t pending_size, pend_pre, pend_compat, pend_post;
>>>> bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE;
>>>> - qemu_savevm_state_pending(s->to_dst_file, s->threshold_size,
>>>> - &pend_nonpost, &pend_post);
>>>> - pending_size = pend_nonpost + pend_post;
>>>> + qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre,
>>>> + &pend_compat, &pend_post);
>>>> + pending_size = pend_pre + pend_compat + pend_post;
>>>> trace_migrate_pending(pending_size, s->threshold_size,
>>>> - pend_post, pend_nonpost);
>>>> + pend_pre, pend_compat, pend_post);
>>>> if (pending_size && pending_size >= s->threshold_size) {
>>>> /* Still a significant amount to transfer */
>>>> if (migrate_postcopy() && !in_postcopy &&
>>>> - pend_nonpost <= s->threshold_size &&
>>>> - atomic_read(&s->start_postcopy)) {
>>>> + pend_pre <= s->threshold_size &&
>>>> + (atomic_read(&s->start_postcopy) ||
>>>> + (pend_pre + pend_compat <= s->threshold_size)))
>>> This change does something different from the description;
>>> it causes a postcopy_start even if the user never ran the postcopy-start
>>> command; so sorry, we can't do that; because postcopy for RAM is
>>> something that users can enable but only switch into when they've given
>>> up on it completing normally.
>>>
>>> However, I guess that leaves you with a problem; which is what happens
>>> to the system when you've run out of pend_pre+pend_compat but can't
>>> complete because pend_post is non-0; so I don't know the answer to that.
>>>
>>>
>> Hmm. Here, we go to postcopy only if "pend_pre + pend_compat <=
>> s->threshold_size". Pre-patch, in this case we will go to
>> migration_completion(). So, precopy stage is finishing anyway.
> Right.
>
>> So, we want
>> in this case to finish ram migration like it was finished by
>> migration_completion(), and then, run postcopy, which will handle only dirty
>> bitmaps, yes?
> It's a bit tricky; the first important thing is that we can't change the
> semantics of the migration without the 'dirty bitmaps'.
>
> So then there's the question of how a migration with both
> postcopy-ram+dirty bitmaps should work; again I don't think we should
> enter the postcopy-ram phase until start-postcopy is issued.
>
> Then there's the 3rd case; dirty-bitmaps but no postcopy-ram; in that
> case I worry less about the semantics of how you want to do it.
I have an idea:
in postcopy_start(), in ram_has_postcopy() (and may be some other
places?), check atomic_read(&s->start_postcopy) instead of
migrate_postcopy_ram()
then:
1. behavior without dirty-bitmaps is not changed, as currently we cant
go into postcopy_start and ram_has_postcopy without s->start_postcopy
2. dirty-bitmaps+ram: if user don't set s->start_postcopy,
postcopy_start() will operate as if migration capability was not
enabled, so ram should complete its migration
3. only dirty-bitmaps: again, postcopy_start() will operate as if
migration capability was not enabled, so ram should complete its migration
>
>> Hmm2. Looked through migration_completion(), I don't understand, how it
>> finishes ram migration without postcopy. It calls
>> qemu_savevm_state_complete_precopy(), which skips states with
>> has_postcopy=true, which is ram...
> Because savevm_state_complete_precopy only skips has_postcopy=true in
> the in_postcopy case:
>
> (in_postcopy && se->ops->has_postcopy &&
> se->ops->has_postcopy(se->opaque)) ||
>
> so when we call it in migration_completion(), if we've not entered
> postcopy yet, then that test doesn't trigger.
>
> (Apologies for not spotting this earlier; but I thought this patch was
> a nice easy one just adding the postcopy_only_pending - I didn't realise it changed
> existing semantics until I spotted that)
oh, yes, I was inattentive :(
>
> Dave
>
>> --
>> Best regards,
>> Vladimir
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-13 13:11 ` Vladimir Sementsov-Ogievskiy
@ 2018-03-13 13:32 ` Vladimir Sementsov-Ogievskiy
2018-03-13 15:35 ` Dr. David Alan Gilbert
1 sibling, 0 replies; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-03-13 13:32 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
13.03.2018 16:11, Vladimir Sementsov-Ogievskiy wrote:
> 13.03.2018 13:30, Dr. David Alan Gilbert wrote:
>> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>>> 12.03.2018 18:30, Dr. David Alan Gilbert wrote:
>>>> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>>>>> There would be savevm states (dirty-bitmap) which can migrate only in
>>>>> postcopy stage. The corresponding pending is introduced here.
>>>>>
>>>>> Signed-off-by: Vladimir Sementsov-Ogievskiy<vsementsov@virtuozzo.com>
>>>>> ---
>>> [...]
>>>
>>>>> static MigIterateState migration_iteration_run(MigrationState *s)
>>>>> {
>>>>> - uint64_t pending_size, pend_post, pend_nonpost;
>>>>> + uint64_t pending_size, pend_pre, pend_compat, pend_post;
>>>>> bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE;
>>>>> - qemu_savevm_state_pending(s->to_dst_file, s->threshold_size,
>>>>> - &pend_nonpost, &pend_post);
>>>>> - pending_size = pend_nonpost + pend_post;
>>>>> + qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre,
>>>>> + &pend_compat, &pend_post);
>>>>> + pending_size = pend_pre + pend_compat + pend_post;
>>>>> trace_migrate_pending(pending_size, s->threshold_size,
>>>>> - pend_post, pend_nonpost);
>>>>> + pend_pre, pend_compat, pend_post);
>>>>> if (pending_size && pending_size >= s->threshold_size) {
>>>>> /* Still a significant amount to transfer */
>>>>> if (migrate_postcopy() && !in_postcopy &&
>>>>> - pend_nonpost <= s->threshold_size &&
>>>>> - atomic_read(&s->start_postcopy)) {
>>>>> + pend_pre <= s->threshold_size &&
>>>>> + (atomic_read(&s->start_postcopy) ||
>>>>> + (pend_pre + pend_compat <= s->threshold_size)))
>>>> This change does something different from the description;
>>>> it causes a postcopy_start even if the user never ran the postcopy-start
>>>> command; so sorry, we can't do that; because postcopy for RAM is
>>>> something that users can enable but only switch into when they've given
>>>> up on it completing normally.
>>>>
>>>> However, I guess that leaves you with a problem; which is what happens
>>>> to the system when you've run out of pend_pre+pend_compat but can't
>>>> complete because pend_post is non-0; so I don't know the answer to that.
>>>>
>>>>
>>> Hmm. Here, we go to postcopy only if "pend_pre + pend_compat <=
>>> s->threshold_size". Pre-patch, in this case we will go to
>>> migration_completion(). So, precopy stage is finishing anyway.
>> Right.
>>
>>> So, we want
>>> in this case to finish ram migration like it was finished by
>>> migration_completion(), and then, run postcopy, which will handle only dirty
>>> bitmaps, yes?
>> It's a bit tricky; the first important thing is that we can't change the
>> semantics of the migration without the 'dirty bitmaps'.
>>
>> So then there's the question of how a migration with both
>> postcopy-ram+dirty bitmaps should work; again I don't think we should
>> enter the postcopy-ram phase until start-postcopy is issued.
>>
>> Then there's the 3rd case; dirty-bitmaps but no postcopy-ram; in that
>> case I worry less about the semantics of how you want to do it.
>
> I have an idea:
>
> in postcopy_start(), in ram_has_postcopy() (and may be some other
> places?), check atomic_read(&s->start_postcopy) instead of
> migrate_postcopy_ram()
>
> then:
>
> 1. behavior without dirty-bitmaps is not changed, as currently we cant
> go into postcopy_start and ram_has_postcopy without s->start_postcopy
> 2. dirty-bitmaps+ram: if user don't set s->start_postcopy,
> postcopy_start() will operate as if migration capability was not
> enabled, so ram should complete its migration
> 3. only dirty-bitmaps: again, postcopy_start() will operate as if
> migration capability was not enabled, so ram should complete its migration
I mean s/migration capability/migration capability for ram postcopy/.
What do you think, will that work?
>
>
>>> Hmm2. Looked through migration_completion(), I don't understand, how it
>>> finishes ram migration without postcopy. It calls
>>> qemu_savevm_state_complete_precopy(), which skips states with
>>> has_postcopy=true, which is ram...
>> Because savevm_state_complete_precopy only skips has_postcopy=true in
>> the in_postcopy case:
>>
>> (in_postcopy && se->ops->has_postcopy &&
>> se->ops->has_postcopy(se->opaque)) ||
>>
>> so when we call it in migration_completion(), if we've not entered
>> postcopy yet, then that test doesn't trigger.
>>
>> (Apologies for not spotting this earlier; but I thought this patch was
>> a nice easy one just adding the postcopy_only_pending - I didn't realise it changed
>> existing semantics until I spotted that)
>
> oh, yes, I was inattentive :(
>
>> Dave
>>
>>> --
>>> Best regards,
>>> Vladimir
>>>
>> --
>> Dr. David Alan Gilbert /dgilbert@redhat.com / Manchester, UK
>
>
> --
> Best regards,
> Vladimir
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-13 13:11 ` Vladimir Sementsov-Ogievskiy
2018-03-13 13:32 ` Vladimir Sementsov-Ogievskiy
@ 2018-03-13 15:35 ` Dr. David Alan Gilbert
2018-03-13 16:14 ` Vladimir Sementsov-Ogievskiy
1 sibling, 1 reply; 33+ messages in thread
From: Dr. David Alan Gilbert @ 2018-03-13 15:35 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
* Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> 13.03.2018 13:30, Dr. David Alan Gilbert wrote:
> > * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> > > 12.03.2018 18:30, Dr. David Alan Gilbert wrote:
> > > > * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> > > > > There would be savevm states (dirty-bitmap) which can migrate only in
> > > > > postcopy stage. The corresponding pending is introduced here.
> > > > >
> > > > > Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> > > > > ---
> > > [...]
> > >
> > > > > static MigIterateState migration_iteration_run(MigrationState *s)
> > > > > {
> > > > > - uint64_t pending_size, pend_post, pend_nonpost;
> > > > > + uint64_t pending_size, pend_pre, pend_compat, pend_post;
> > > > > bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE;
> > > > > - qemu_savevm_state_pending(s->to_dst_file, s->threshold_size,
> > > > > - &pend_nonpost, &pend_post);
> > > > > - pending_size = pend_nonpost + pend_post;
> > > > > + qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre,
> > > > > + &pend_compat, &pend_post);
> > > > > + pending_size = pend_pre + pend_compat + pend_post;
> > > > > trace_migrate_pending(pending_size, s->threshold_size,
> > > > > - pend_post, pend_nonpost);
> > > > > + pend_pre, pend_compat, pend_post);
> > > > > if (pending_size && pending_size >= s->threshold_size) {
> > > > > /* Still a significant amount to transfer */
> > > > > if (migrate_postcopy() && !in_postcopy &&
> > > > > - pend_nonpost <= s->threshold_size &&
> > > > > - atomic_read(&s->start_postcopy)) {
> > > > > + pend_pre <= s->threshold_size &&
> > > > > + (atomic_read(&s->start_postcopy) ||
> > > > > + (pend_pre + pend_compat <= s->threshold_size)))
> > > > This change does something different from the description;
> > > > it causes a postcopy_start even if the user never ran the postcopy-start
> > > > command; so sorry, we can't do that; because postcopy for RAM is
> > > > something that users can enable but only switch into when they've given
> > > > up on it completing normally.
> > > >
> > > > However, I guess that leaves you with a problem; which is what happens
> > > > to the system when you've run out of pend_pre+pend_compat but can't
> > > > complete because pend_post is non-0; so I don't know the answer to that.
> > > >
> > > >
> > > Hmm. Here, we go to postcopy only if "pend_pre + pend_compat <=
> > > s->threshold_size". Pre-patch, in this case we will go to
> > > migration_completion(). So, precopy stage is finishing anyway.
> > Right.
> >
> > > So, we want
> > > in this case to finish ram migration like it was finished by
> > > migration_completion(), and then, run postcopy, which will handle only dirty
> > > bitmaps, yes?
> > It's a bit tricky; the first important thing is that we can't change the
> > semantics of the migration without the 'dirty bitmaps'.
> >
> > So then there's the question of how a migration with both
> > postcopy-ram+dirty bitmaps should work; again I don't think we should
> > enter the postcopy-ram phase until start-postcopy is issued.
> >
> > Then there's the 3rd case; dirty-bitmaps but no postcopy-ram; in that
> > case I worry less about the semantics of how you want to do it.
>
> I have an idea:
>
> in postcopy_start(), in ram_has_postcopy() (and may be some other places?),
> check atomic_read(&s->start_postcopy) instead of migrate_postcopy_ram()
We've got to use migrate_postcopy_ram() to decide whether we should do
ram specific things, e.g. send the ram discard data.
I'm wanting to make sure that if we have another full postcopy device
(like RAM, maybe storage say) that we'll just add that in with
migrate_postcopy_whatever().
> then:
>
> 1. behavior without dirty-bitmaps is not changed, as currently we cant go
> into postcopy_start and ram_has_postcopy without s->start_postcopy
> 2. dirty-bitmaps+ram: if user don't set s->start_postcopy, postcopy_start()
> will operate as if migration capability was not enabled, so ram should
> complete its migration
> 3. only dirty-bitmaps: again, postcopy_start() will operate as if migration
> capability was not enabled, so ram should complete its migration
Why can't we just remove the change to the trigger condition in this
patch? Then I think everything works as long as the management layer
does eventually call migration-start-postcopy ?
(It might waste some bandwidth at the point where there's otherwise
nothing left to send).
Even with the use of migrate-start-postcopy, you're going to need to be
careful about the higher level story; you need to document when to do it
and what the higher levels should do after a migration failure - at the
moment they know that once postcopy starts migration is irrecoverable if
it fails; I suspect that's not true with your dirty bitmaps.
IMHO this still comes back to my original observation from ~18months ago
that in many ways this isn't very postcopy like; in the sense that all
the semantics are quite different from RAM.
Dave
> >
> > > Hmm2. Looked through migration_completion(), I don't understand, how it
> > > finishes ram migration without postcopy. It calls
> > > qemu_savevm_state_complete_precopy(), which skips states with
> > > has_postcopy=true, which is ram...
> > Because savevm_state_complete_precopy only skips has_postcopy=true in
> > the in_postcopy case:
> >
> > (in_postcopy && se->ops->has_postcopy &&
> > se->ops->has_postcopy(se->opaque)) ||
> >
> > so when we call it in migration_completion(), if we've not entered
> > postcopy yet, then that test doesn't trigger.
> >
> > (Apologies for not spotting this earlier; but I thought this patch was
> > a nice easy one just adding the postcopy_only_pending - I didn't realise it changed
> > existing semantics until I spotted that)
>
> oh, yes, I was inattentive :(
>
> >
> > Dave
> >
> > > --
> > > Best regards,
> > > Vladimir
> > >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>
> --
> Best regards,
> Vladimir
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-13 15:35 ` Dr. David Alan Gilbert
@ 2018-03-13 16:14 ` Vladimir Sementsov-Ogievskiy
2018-03-13 16:16 ` John Snow
0 siblings, 1 reply; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-03-13 16:14 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
13.03.2018 18:35, Dr. David Alan Gilbert wrote:
> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>> 13.03.2018 13:30, Dr. David Alan Gilbert wrote:
>>> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>>>> 12.03.2018 18:30, Dr. David Alan Gilbert wrote:
>>>>> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>>>>>> There would be savevm states (dirty-bitmap) which can migrate only in
>>>>>> postcopy stage. The corresponding pending is introduced here.
>>>>>>
>>>>>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>>>>>> ---
>>>> [...]
>>>>
>>>>>> static MigIterateState migration_iteration_run(MigrationState *s)
>>>>>> {
>>>>>> - uint64_t pending_size, pend_post, pend_nonpost;
>>>>>> + uint64_t pending_size, pend_pre, pend_compat, pend_post;
>>>>>> bool in_postcopy = s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE;
>>>>>> - qemu_savevm_state_pending(s->to_dst_file, s->threshold_size,
>>>>>> - &pend_nonpost, &pend_post);
>>>>>> - pending_size = pend_nonpost + pend_post;
>>>>>> + qemu_savevm_state_pending(s->to_dst_file, s->threshold_size, &pend_pre,
>>>>>> + &pend_compat, &pend_post);
>>>>>> + pending_size = pend_pre + pend_compat + pend_post;
>>>>>> trace_migrate_pending(pending_size, s->threshold_size,
>>>>>> - pend_post, pend_nonpost);
>>>>>> + pend_pre, pend_compat, pend_post);
>>>>>> if (pending_size && pending_size >= s->threshold_size) {
>>>>>> /* Still a significant amount to transfer */
>>>>>> if (migrate_postcopy() && !in_postcopy &&
>>>>>> - pend_nonpost <= s->threshold_size &&
>>>>>> - atomic_read(&s->start_postcopy)) {
>>>>>> + pend_pre <= s->threshold_size &&
>>>>>> + (atomic_read(&s->start_postcopy) ||
>>>>>> + (pend_pre + pend_compat <= s->threshold_size)))
>>>>> This change does something different from the description;
>>>>> it causes a postcopy_start even if the user never ran the postcopy-start
>>>>> command; so sorry, we can't do that; because postcopy for RAM is
>>>>> something that users can enable but only switch into when they've given
>>>>> up on it completing normally.
>>>>>
>>>>> However, I guess that leaves you with a problem; which is what happens
>>>>> to the system when you've run out of pend_pre+pend_compat but can't
>>>>> complete because pend_post is non-0; so I don't know the answer to that.
>>>>>
>>>>>
>>>> Hmm. Here, we go to postcopy only if "pend_pre + pend_compat <=
>>>> s->threshold_size". Pre-patch, in this case we will go to
>>>> migration_completion(). So, precopy stage is finishing anyway.
>>> Right.
>>>
>>>> So, we want
>>>> in this case to finish ram migration like it was finished by
>>>> migration_completion(), and then, run postcopy, which will handle only dirty
>>>> bitmaps, yes?
>>> It's a bit tricky; the first important thing is that we can't change the
>>> semantics of the migration without the 'dirty bitmaps'.
>>>
>>> So then there's the question of how a migration with both
>>> postcopy-ram+dirty bitmaps should work; again I don't think we should
>>> enter the postcopy-ram phase until start-postcopy is issued.
>>>
>>> Then there's the 3rd case; dirty-bitmaps but no postcopy-ram; in that
>>> case I worry less about the semantics of how you want to do it.
>> I have an idea:
>>
>> in postcopy_start(), in ram_has_postcopy() (and may be some other places?),
>> check atomic_read(&s->start_postcopy) instead of migrate_postcopy_ram()
> We've got to use migrate_postcopy_ram() to decide whether we should do
> ram specific things, e.g. send the ram discard data.
> I'm wanting to make sure that if we have another full postcopy device
> (like RAM, maybe storage say) that we'll just add that in with
> migrate_postcopy_whatever().
>
>> then:
>>
>> 1. behavior without dirty-bitmaps is not changed, as currently we cant go
>> into postcopy_start and ram_has_postcopy without s->start_postcopy
>> 2. dirty-bitmaps+ram: if user don't set s->start_postcopy, postcopy_start()
>> will operate as if migration capability was not enabled, so ram should
>> complete its migration
>> 3. only dirty-bitmaps: again, postcopy_start() will operate as if migration
>> capability was not enabled, so ram should complete its migration
> Why can't we just remove the change to the trigger condition in this
> patch? Then I think everything works as long as the management layer
> does eventually call migration-start-postcopy ?
> (It might waste some bandwidth at the point where there's otherwise
> nothing left to send).
Hmm, I agree, it is the simplest thing we can do for now, and I'll
rethink later,
how (and is it worth doing) to go to postcopy automatically in case of
only-dirty-bitmaps.
Should I respin?
>
> Even with the use of migrate-start-postcopy, you're going to need to be
> careful about the higher level story; you need to document when to do it
> and what the higher levels should do after a migration failure - at the
> moment they know that once postcopy starts migration is irrecoverable if
> it fails; I suspect that's not true with your dirty bitmaps.
>
> IMHO this still comes back to my original observation from ~18months ago
> that in many ways this isn't very postcopy like; in the sense that all
> the semantics are quite different from RAM.
>
> Dave
>
>>>> Hmm2. Looked through migration_completion(), I don't understand, how it
>>>> finishes ram migration without postcopy. It calls
>>>> qemu_savevm_state_complete_precopy(), which skips states with
>>>> has_postcopy=true, which is ram...
>>> Because savevm_state_complete_precopy only skips has_postcopy=true in
>>> the in_postcopy case:
>>>
>>> (in_postcopy && se->ops->has_postcopy &&
>>> se->ops->has_postcopy(se->opaque)) ||
>>>
>>> so when we call it in migration_completion(), if we've not entered
>>> postcopy yet, then that test doesn't trigger.
>>>
>>> (Apologies for not spotting this earlier; but I thought this patch was
>>> a nice easy one just adding the postcopy_only_pending - I didn't realise it changed
>>> existing semantics until I spotted that)
>> oh, yes, I was inattentive :(
>>
>>> Dave
>>>
>>>> --
>>>> Best regards,
>>>> Vladimir
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>
>> --
>> Best regards,
>> Vladimir
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-13 16:14 ` Vladimir Sementsov-Ogievskiy
@ 2018-03-13 16:16 ` John Snow
2018-03-13 16:33 ` Vladimir Sementsov-Ogievskiy
0 siblings, 1 reply; 33+ messages in thread
From: John Snow @ 2018-03-13 16:16 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, Dr. David Alan Gilbert
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, lirans
On 03/13/2018 12:14 PM, Vladimir Sementsov-Ogievskiy wrote:
>
> Hmm, I agree, it is the simplest thing we can do for now, and I'll
> rethink later,
> how (and is it worth doing) to go to postcopy automatically in case of
> only-dirty-bitmaps.
> Should I respin?
Please do. I already staged patches 1-4 in my branch, so if you'd like,
you can respin just 5+.
https://github.com/jnsnow/qemu/tree/bitmaps
--js
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-13 16:16 ` John Snow
@ 2018-03-13 16:33 ` Vladimir Sementsov-Ogievskiy
2018-03-13 17:10 ` John Snow
0 siblings, 1 reply; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-03-13 16:33 UTC (permalink / raw)
To: John Snow, Dr. David Alan Gilbert
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, lirans
13.03.2018 19:16, John Snow wrote:
>
> On 03/13/2018 12:14 PM, Vladimir Sementsov-Ogievskiy wrote:
>> Hmm, I agree, it is the simplest thing we can do for now, and I'll
>> rethink later,
>> how (and is it worth doing) to go to postcopy automatically in case of
>> only-dirty-bitmaps.
>> Should I respin?
> Please do. I already staged patches 1-4 in my branch, so if you'd like,
> you can respin just 5+.
>
> https://github.com/jnsnow/qemu/tree/bitmaps
>
> --js
Ok, I'll base on your branch. How should I write Based-on: for patchew
in this case?
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 10/12] migration: add postcopy migration of dirty bitmaps
2018-03-12 16:09 ` Dr. David Alan Gilbert
@ 2018-03-13 16:59 ` Vladimir Sementsov-Ogievskiy
2018-03-13 18:02 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 33+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-03-13 16:59 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
12.03.2018 19:09, Dr. David Alan Gilbert wrote:
> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>> Postcopy migration of dirty bitmaps. Only named dirty bitmaps are migrated.
>>
>> If destination qemu is already containing a dirty bitmap with the same name
>> as a migrated bitmap (for the same node), then, if their granularities are
>> the same the migration will be done, otherwise the error will be generated.
>>
>> If destination qemu doesn't contain such bitmap it will be created.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>> include/migration/misc.h | 3 +
>> migration/migration.h | 3 +
>> migration/block-dirty-bitmap.c | 737 +++++++++++++++++++++++++++++++++++++++++
>> migration/migration.c | 5 +
>> migration/savevm.c | 2 +
>> vl.c | 1 +
>> migration/Makefile.objs | 1 +
>> migration/trace-events | 14 +
>> 8 files changed, 766 insertions(+)
>> create mode 100644 migration/block-dirty-bitmap.c
>>
>> diff --git a/include/migration/misc.h b/include/migration/misc.h
>> index 77fd4f587c..4ebf24c6c2 100644
>> --- a/include/migration/misc.h
>> +++ b/include/migration/misc.h
>> @@ -56,4 +56,7 @@ bool migration_has_failed(MigrationState *);
>> bool migration_in_postcopy_after_devices(MigrationState *);
>> void migration_global_dump(Monitor *mon);
>>
>> +/* migration/block-dirty-bitmap.c */
>> +void dirty_bitmap_mig_init(void);
>> +
>> #endif
>> diff --git a/migration/migration.h b/migration/migration.h
>> index 861cdfaa96..79f72b7e50 100644
>> --- a/migration/migration.h
>> +++ b/migration/migration.h
>> @@ -233,4 +233,7 @@ void migrate_send_rp_pong(MigrationIncomingState *mis,
>> void migrate_send_rp_req_pages(MigrationIncomingState *mis, const char* rbname,
>> ram_addr_t start, size_t len);
>>
>> +void dirty_bitmap_mig_before_vm_start(void);
>> +void init_dirty_bitmap_incoming_migration(void);
>> +
>> #endif
>> diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
>> new file mode 100644
>> index 0000000000..5b41f7140d
>> --- /dev/null
>> +++ b/migration/block-dirty-bitmap.c
>> @@ -0,0 +1,737 @@
>> +/*
>> + * Block dirty bitmap postcopy migration
>> + *
>> + * Copyright IBM, Corp. 2009
>> + * Copyright (c) 2016-2017 Virtuozzo International GmbH. All rights reserved.
>> + *
>> + * Authors:
>> + * Liran Schour <lirans@il.ibm.com>
>> + * Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2. See
>> + * the COPYING file in the top-level directory.
>> + * This file is derived from migration/block.c, so it's author and IBM copyright
>> + * are here, although content is quite different.
>> + *
>> + * Contributions after 2012-01-13 are licensed under the terms of the
>> + * GNU GPL, version 2 or (at your option) any later version.
>> + *
>> + * ***
>> + *
>> + * Here postcopy migration of dirty bitmaps is realized. Only QMP-addressable
>> + * bitmaps are migrated.
>> + *
>> + * Bitmap migration implies creating bitmap with the same name and granularity
>> + * in destination QEMU. If the bitmap with the same name (for the same node)
>> + * already exists on destination an error will be generated.
>> + *
>> + * format of migration:
>> + *
>> + * # Header (shared for different chunk types)
>> + * 1, 2 or 4 bytes: flags (see qemu_{put,put}_flags)
>> + * [ 1 byte: node name size ] \ flags & DEVICE_NAME
>> + * [ n bytes: node name ] /
>> + * [ 1 byte: bitmap name size ] \ flags & BITMAP_NAME
>> + * [ n bytes: bitmap name ] /
>> + *
>> + * # Start of bitmap migration (flags & START)
>> + * header
>> + * be64: granularity
>> + * 1 byte: bitmap flags (corresponds to BdrvDirtyBitmap)
>> + * bit 0 - bitmap is enabled
>> + * bit 1 - bitmap is persistent
>> + * bit 2 - bitmap is autoloading
>> + * bits 3-7 - reserved, must be zero
>> + *
>> + * # Complete of bitmap migration (flags & COMPLETE)
>> + * header
>> + *
>> + * # Data chunk of bitmap migration
>> + * header
>> + * be64: start sector
>> + * be32: number of sectors
>> + * [ be64: buffer size ] \ ! (flags & ZEROES)
>> + * [ n bytes: buffer ] /
>> + *
>> + * The last chunk in stream should contain flags & EOS. The chunk may skip
>> + * device and/or bitmap names, assuming them to be the same with the previous
>> + * chunk.
>> + */
>> +
>> +#include "qemu/osdep.h"
>> +#include "block/block.h"
>> +#include "block/block_int.h"
>> +#include "sysemu/block-backend.h"
>> +#include "qemu/main-loop.h"
>> +#include "qemu/error-report.h"
>> +#include "migration/misc.h"
>> +#include "migration/migration.h"
>> +#include "migration/qemu-file.h"
>> +#include "migration/vmstate.h"
>> +#include "migration/register.h"
>> +#include "qemu/hbitmap.h"
>> +#include "sysemu/sysemu.h"
>> +#include "qemu/cutils.h"
>> +#include "qapi/error.h"
>> +#include "trace.h"
>> +
>> +#define CHUNK_SIZE (1 << 10)
>> +
>> +/* Flags occupy one, two or four bytes (Big Endian). The size is determined as
>> + * follows:
>> + * in first (most significant) byte bit 8 is clear --> one byte
>> + * in first byte bit 8 is set --> two or four bytes, depending on second
>> + * byte:
>> + * | in second byte bit 8 is clear --> two bytes
>> + * | in second byte bit 8 is set --> four bytes
>> + */
>> +#define DIRTY_BITMAP_MIG_FLAG_EOS 0x01
>> +#define DIRTY_BITMAP_MIG_FLAG_ZEROES 0x02
>> +#define DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME 0x04
>> +#define DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME 0x08
>> +#define DIRTY_BITMAP_MIG_FLAG_START 0x10
>> +#define DIRTY_BITMAP_MIG_FLAG_COMPLETE 0x20
>> +#define DIRTY_BITMAP_MIG_FLAG_BITS 0x40
>> +
>> +#define DIRTY_BITMAP_MIG_EXTRA_FLAGS 0x80
>> +
>> +#define DIRTY_BITMAP_MIG_START_FLAG_ENABLED 0x01
>> +#define DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT 0x02
>> +/* 0x04 was "AUTOLOAD" flags on elder versions, no it is ignored */
>> +#define DIRTY_BITMAP_MIG_START_FLAG_RESERVED_MASK 0xf8
>> +
>> +typedef struct DirtyBitmapMigBitmapState {
>> + /* Written during setup phase. */
>> + BlockDriverState *bs;
>> + const char *node_name;
>> + BdrvDirtyBitmap *bitmap;
>> + uint64_t total_sectors;
>> + uint64_t sectors_per_chunk;
>> + QSIMPLEQ_ENTRY(DirtyBitmapMigBitmapState) entry;
>> + uint8_t flags;
>> +
>> + /* For bulk phase. */
>> + bool bulk_completed;
>> + uint64_t cur_sector;
>> +} DirtyBitmapMigBitmapState;
>> +
>> +typedef struct DirtyBitmapMigState {
>> + QSIMPLEQ_HEAD(dbms_list, DirtyBitmapMigBitmapState) dbms_list;
>> +
>> + bool bulk_completed;
>> + bool no_bitmaps;
>> +
>> + /* for send_bitmap_bits() */
>> + BlockDriverState *prev_bs;
>> + BdrvDirtyBitmap *prev_bitmap;
>> +} DirtyBitmapMigState;
>> +
>> +typedef struct DirtyBitmapLoadState {
>> + uint32_t flags;
>> + char node_name[256];
>> + char bitmap_name[256];
>> + BlockDriverState *bs;
>> + BdrvDirtyBitmap *bitmap;
>> +} DirtyBitmapLoadState;
>> +
>> +static DirtyBitmapMigState dirty_bitmap_mig_state;
>> +
>> +typedef struct DirtyBitmapLoadBitmapState {
>> + BlockDriverState *bs;
>> + BdrvDirtyBitmap *bitmap;
>> + bool migrated;
>> +} DirtyBitmapLoadBitmapState;
>> +static GSList *enabled_bitmaps;
>> +QemuMutex finish_lock;
>> +
>> +void init_dirty_bitmap_incoming_migration(void)
>> +{
>> + qemu_mutex_init(&finish_lock);
>> +}
>> +
>> +static uint32_t qemu_get_bitmap_flags(QEMUFile *f)
>> +{
>> + uint8_t flags = qemu_get_byte(f);
>> + if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
>> + flags = flags << 8 | qemu_get_byte(f);
>> + if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
>> + flags = flags << 16 | qemu_get_be16(f);
>> + }
>> + }
>> +
>> + return flags;
>> +}
>> +
>> +static void qemu_put_bitmap_flags(QEMUFile *f, uint32_t flags)
>> +{
>> + /* The code currently do not send flags more than one byte */
>> + assert(!(flags & (0xffffff00 | DIRTY_BITMAP_MIG_EXTRA_FLAGS)));
>> +
>> + qemu_put_byte(f, flags);
>> +}
>> +
>> +static void send_bitmap_header(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
>> + uint32_t additional_flags)
>> +{
>> + BlockDriverState *bs = dbms->bs;
>> + BdrvDirtyBitmap *bitmap = dbms->bitmap;
>> + uint32_t flags = additional_flags;
>> + trace_send_bitmap_header_enter();
>> +
>> + if (bs != dirty_bitmap_mig_state.prev_bs) {
>> + dirty_bitmap_mig_state.prev_bs = bs;
>> + flags |= DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME;
>> + }
>> +
>> + if (bitmap != dirty_bitmap_mig_state.prev_bitmap) {
>> + dirty_bitmap_mig_state.prev_bitmap = bitmap;
>> + flags |= DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME;
>> + }
>> +
>> + qemu_put_bitmap_flags(f, flags);
>> +
>> + if (flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
>> + qemu_put_counted_string(f, dbms->node_name);
>> + }
>> +
>> + if (flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
>> + qemu_put_counted_string(f, bdrv_dirty_bitmap_name(bitmap));
>> + }
>> +}
>> +
>> +static void send_bitmap_start(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
>> +{
>> + send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_START);
>> + qemu_put_be32(f, bdrv_dirty_bitmap_granularity(dbms->bitmap));
>> + qemu_put_byte(f, dbms->flags);
>> +}
>> +
>> +static void send_bitmap_complete(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
>> +{
>> + send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_COMPLETE);
>> +}
>> +
>> +static void send_bitmap_bits(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
>> + uint64_t start_sector, uint32_t nr_sectors)
>> +{
>> + /* align for buffer_is_zero() */
>> + uint64_t align = 4 * sizeof(long);
>> + uint64_t unaligned_size =
>> + bdrv_dirty_bitmap_serialization_size(
>> + dbms->bitmap, start_sector << BDRV_SECTOR_BITS,
>> + (uint64_t)nr_sectors << BDRV_SECTOR_BITS);
>> + uint64_t buf_size = QEMU_ALIGN_UP(unaligned_size, align);
>> + uint8_t *buf = g_malloc0(buf_size);
>> + uint32_t flags = DIRTY_BITMAP_MIG_FLAG_BITS;
>> +
>> + bdrv_dirty_bitmap_serialize_part(
>> + dbms->bitmap, buf, start_sector << BDRV_SECTOR_BITS,
>> + (uint64_t)nr_sectors << BDRV_SECTOR_BITS);
>> +
>> + if (buffer_is_zero(buf, buf_size)) {
>> + g_free(buf);
>> + buf = NULL;
>> + flags |= DIRTY_BITMAP_MIG_FLAG_ZEROES;
>> + }
>> +
>> + trace_send_bitmap_bits(flags, start_sector, nr_sectors, buf_size);
>> +
>> + send_bitmap_header(f, dbms, flags);
>> +
>> + qemu_put_be64(f, start_sector);
>> + qemu_put_be32(f, nr_sectors);
>> +
>> + /* if a block is zero we need to flush here since the network
>> + * bandwidth is now a lot higher than the storage device bandwidth.
>> + * thus if we queue zero blocks we slow down the migration. */
>> + if (flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
>> + qemu_fflush(f);
>> + } else {
>> + qemu_put_be64(f, buf_size);
>> + qemu_put_buffer(f, buf, buf_size);
>> + }
>> +
>> + g_free(buf);
>> +}
>> +
>> +/* Called with iothread lock taken. */
>> +static void dirty_bitmap_mig_cleanup(void)
>> +{
>> + DirtyBitmapMigBitmapState *dbms;
>> +
>> + while ((dbms = QSIMPLEQ_FIRST(&dirty_bitmap_mig_state.dbms_list)) != NULL) {
>> + QSIMPLEQ_REMOVE_HEAD(&dirty_bitmap_mig_state.dbms_list, entry);
>> + bdrv_dirty_bitmap_set_qmp_locked(dbms->bitmap, false);
>> + bdrv_unref(dbms->bs);
>> + g_free(dbms);
>> + }
>> +}
>> +
>> +/* Called with iothread lock taken. */
>> +static int init_dirty_bitmap_migration(void)
>> +{
>> + BlockDriverState *bs;
>> + BdrvDirtyBitmap *bitmap;
>> + DirtyBitmapMigBitmapState *dbms;
>> + BdrvNextIterator it;
>> +
>> + dirty_bitmap_mig_state.bulk_completed = false;
>> + dirty_bitmap_mig_state.prev_bs = NULL;
>> + dirty_bitmap_mig_state.prev_bitmap = NULL;
>> + dirty_bitmap_mig_state.no_bitmaps = false;
>> +
>> + for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
>> + const char *drive_name = bdrv_get_device_or_node_name(bs);
>> +
>> + /* skip automatically inserted nodes */
>> + while (bs && bs->drv && bs->implicit) {
>> + bs = backing_bs(bs);
>> + }
>> +
>> + for (bitmap = bdrv_dirty_bitmap_next(bs, NULL); bitmap;
>> + bitmap = bdrv_dirty_bitmap_next(bs, bitmap))
>> + {
>> + if (!bdrv_dirty_bitmap_name(bitmap)) {
>> + continue;
>> + }
>> +
>> + if (drive_name == NULL) {
>> + error_report("Found bitmap '%s' in unnamed node %p. It can't "
>> + "be migrated", bdrv_dirty_bitmap_name(bitmap), bs);
>> + goto fail;
>> + }
>> +
>> + if (bdrv_dirty_bitmap_frozen(bitmap)) {
>> + error_report("Can't migrate frozen dirty bitmap: '%s",
>> + bdrv_dirty_bitmap_name(bitmap));
>> + goto fail;
>> + }
>> +
>> + if (bdrv_dirty_bitmap_qmp_locked(bitmap)) {
>> + error_report("Can't migrate locked dirty bitmap: '%s",
>> + bdrv_dirty_bitmap_name(bitmap));
>> + goto fail;
>> + }
>> +
>> + bdrv_ref(bs);
>> + bdrv_dirty_bitmap_set_qmp_locked(bitmap, true);
>> +
>> + dbms = g_new0(DirtyBitmapMigBitmapState, 1);
>> + dbms->bs = bs;
>> + dbms->node_name = drive_name;
>> + dbms->bitmap = bitmap;
>> + dbms->total_sectors = bdrv_nb_sectors(bs);
>> + dbms->sectors_per_chunk = CHUNK_SIZE * 8 *
>> + bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS;
>> + if (bdrv_dirty_bitmap_enabled(bitmap)) {
>> + dbms->flags |= DIRTY_BITMAP_MIG_START_FLAG_ENABLED;
>> + }
>> + if (bdrv_dirty_bitmap_get_persistance(bitmap)) {
>> + dbms->flags |= DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT;
>> + }
>> +
>> + QSIMPLEQ_INSERT_TAIL(&dirty_bitmap_mig_state.dbms_list,
>> + dbms, entry);
>> + }
>> + }
>> +
>> + /* unset persistance here, to not roll back it */
>> + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
>> + bdrv_dirty_bitmap_set_persistance(dbms->bitmap, false);
>> + }
>> +
>> + if (QSIMPLEQ_EMPTY(&dirty_bitmap_mig_state.dbms_list)) {
>> + dirty_bitmap_mig_state.no_bitmaps = true;
>> + }
>> +
>> + return 0;
>> +
>> +fail:
>> + dirty_bitmap_mig_cleanup();
>> +
>> + return -1;
>> +}
>> +
>> +/* Called with no lock taken. */
>> +static void bulk_phase_send_chunk(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
>> +{
>> + uint32_t nr_sectors = MIN(dbms->total_sectors - dbms->cur_sector,
>> + dbms->sectors_per_chunk);
>> +
>> + send_bitmap_bits(f, dbms, dbms->cur_sector, nr_sectors);
>> +
>> + dbms->cur_sector += nr_sectors;
>> + if (dbms->cur_sector >= dbms->total_sectors) {
>> + dbms->bulk_completed = true;
>> + }
>> +}
>> +
>> +/* Called with no lock taken. */
>> +static void bulk_phase(QEMUFile *f, bool limit)
>> +{
>> + DirtyBitmapMigBitmapState *dbms;
>> +
>> + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
>> + while (!dbms->bulk_completed) {
>> + bulk_phase_send_chunk(f, dbms);
>> + if (limit && qemu_file_rate_limit(f)) {
>> + return;
>> + }
>> + }
>> + }
>> +
>> + dirty_bitmap_mig_state.bulk_completed = true;
>> +}
>> +
>> +/* for SaveVMHandlers */
>> +static void dirty_bitmap_save_cleanup(void *opaque)
>> +{
>> + dirty_bitmap_mig_cleanup();
>> +}
>> +
>> +static int dirty_bitmap_save_iterate(QEMUFile *f, void *opaque)
>> +{
>> + trace_dirty_bitmap_save_iterate(migration_in_postcopy());
>> +
>> + if (migration_in_postcopy() && !dirty_bitmap_mig_state.bulk_completed) {
>> + bulk_phase(f, true);
>> + }
>> +
>> + qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
>> +
>> + return dirty_bitmap_mig_state.bulk_completed;
>> +}
>> +
>> +/* Called with iothread lock taken. */
>> +
>> +static int dirty_bitmap_save_complete(QEMUFile *f, void *opaque)
>> +{
>> + DirtyBitmapMigBitmapState *dbms;
>> + trace_dirty_bitmap_save_complete_enter();
>> +
>> + if (!dirty_bitmap_mig_state.bulk_completed) {
>> + bulk_phase(f, false);
>> + }
>> +
>> + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
>> + send_bitmap_complete(f, dbms);
>> + }
>> +
>> + qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
>> +
>> + trace_dirty_bitmap_save_complete_finish();
>> +
>> + dirty_bitmap_mig_cleanup();
>> + return 0;
>> +}
>> +
>> +static void dirty_bitmap_save_pending(QEMUFile *f, void *opaque,
>> + uint64_t max_size,
>> + uint64_t *res_precopy_only,
>> + uint64_t *res_compatible,
>> + uint64_t *res_postcopy_only)
>> +{
>> + DirtyBitmapMigBitmapState *dbms;
>> + uint64_t pending = 0;
>> +
>> + qemu_mutex_lock_iothread();
>> +
>> + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
>> + uint64_t gran = bdrv_dirty_bitmap_granularity(dbms->bitmap);
>> + uint64_t sectors = dbms->bulk_completed ? 0 :
>> + dbms->total_sectors - dbms->cur_sector;
>> +
>> + pending += DIV_ROUND_UP(sectors * BDRV_SECTOR_SIZE, gran);
>> + }
>> +
>> + qemu_mutex_unlock_iothread();
>> +
>> + trace_dirty_bitmap_save_pending(pending, max_size);
>> +
>> + *res_postcopy_only += pending;
>> +}
>> +
>> +/* First occurrence of this bitmap. It should be created if doesn't exist */
>> +static int dirty_bitmap_load_start(QEMUFile *f, DirtyBitmapLoadState *s)
>> +{
>> + Error *local_err = NULL;
>> + uint32_t granularity = qemu_get_be32(f);
>> + uint8_t flags = qemu_get_byte(f);
>> +
>> + if (s->bitmap) {
>> + error_report("Bitmap with the same name ('%s') already exists on "
>> + "destination", bdrv_dirty_bitmap_name(s->bitmap));
>> + return -EINVAL;
>> + } else {
>> + s->bitmap = bdrv_create_dirty_bitmap(s->bs, granularity,
>> + s->bitmap_name, &local_err);
>> + if (!s->bitmap) {
>> + error_report_err(local_err);
>> + return -EINVAL;
>> + }
>> + }
>> +
>> + if (flags & DIRTY_BITMAP_MIG_START_FLAG_RESERVED_MASK) {
>> + error_report("Unknown flags in migrated dirty bitmap header: %x",
>> + flags);
>> + return -EINVAL;
>> + }
>> +
>> + if (flags & DIRTY_BITMAP_MIG_START_FLAG_PERSISTENT) {
>> + bdrv_dirty_bitmap_set_persistance(s->bitmap, true);
>> + }
>> +
>> + bdrv_disable_dirty_bitmap(s->bitmap);
>> + if (flags & DIRTY_BITMAP_MIG_START_FLAG_ENABLED) {
>> + DirtyBitmapLoadBitmapState *b;
>> +
>> + bdrv_dirty_bitmap_create_successor(s->bs, s->bitmap, &local_err);
>> + if (local_err) {
>> + error_report_err(local_err);
>> + return -EINVAL;
>> + }
>> +
>> + b = g_new(DirtyBitmapLoadBitmapState, 1);
>> + b->bs = s->bs;
>> + b->bitmap = s->bitmap;
>> + b->migrated = false;
>> + enabled_bitmaps = g_slist_prepend(enabled_bitmaps, b);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +void dirty_bitmap_mig_before_vm_start(void)
>> +{
>> + GSList *item;
>> +
>> + qemu_mutex_lock(&finish_lock);
>> +
>> + for (item = enabled_bitmaps; item; item = g_slist_next(item)) {
>> + DirtyBitmapLoadBitmapState *b = item->data;
>> +
>> + if (b->migrated) {
>> + bdrv_enable_dirty_bitmap(b->bitmap);
>> + } else {
>> + bdrv_dirty_bitmap_enable_successor(b->bitmap);
>> + }
>> +
>> + g_free(b);
>> + }
>> +
>> + g_slist_free(enabled_bitmaps);
>> + enabled_bitmaps = NULL;
>> +
>> + qemu_mutex_unlock(&finish_lock);
>> +}
>> +
>> +static void dirty_bitmap_load_complete(QEMUFile *f, DirtyBitmapLoadState *s)
>> +{
>> + GSList *item;
>> + trace_dirty_bitmap_load_complete();
>> + bdrv_dirty_bitmap_deserialize_finish(s->bitmap);
>> +
>> + qemu_mutex_lock(&finish_lock);
>> +
>> + for (item = enabled_bitmaps; item; item = g_slist_next(item)) {
>> + DirtyBitmapLoadBitmapState *b = item->data;
>> +
>> + if (b->bitmap == s->bitmap) {
>> + b->migrated = true;
>> + break;
>> + }
>> + }
>> +
>> + if (bdrv_dirty_bitmap_frozen(s->bitmap)) {
>> + bdrv_dirty_bitmap_lock(s->bitmap);
>> + if (enabled_bitmaps == NULL) {
>> + /* in postcopy */
>> + bdrv_reclaim_dirty_bitmap_locked(s->bs, s->bitmap, &error_abort);
>> + bdrv_enable_dirty_bitmap(s->bitmap);
>> + } else {
>> + /* target not started, successor must be empty */
>> + int64_t count = bdrv_get_dirty_count(s->bitmap);
>> + BdrvDirtyBitmap *ret = bdrv_reclaim_dirty_bitmap_locked(s->bs,
>> + s->bitmap,
>> + NULL);
>> + /* bdrv_reclaim_dirty_bitmap can fail only on no successor (it
>> + * must be) or on merge fail, but merge can't fail when second
>> + * bitmap is empty
>> + */
>> + assert(ret == s->bitmap &&
>> + count == bdrv_get_dirty_count(s->bitmap));
>> + }
>> + bdrv_dirty_bitmap_unlock(s->bitmap);
>> + }
>> +
>> + qemu_mutex_unlock(&finish_lock);
>> +}
>> +
>> +static int dirty_bitmap_load_bits(QEMUFile *f, DirtyBitmapLoadState *s)
>> +{
>> + uint64_t first_byte = qemu_get_be64(f) << BDRV_SECTOR_BITS;
>> + uint64_t nr_bytes = (uint64_t)qemu_get_be32(f) << BDRV_SECTOR_BITS;
>> + trace_dirty_bitmap_load_bits_enter(first_byte >> BDRV_SECTOR_BITS,
>> + nr_bytes >> BDRV_SECTOR_BITS);
>> +
>> + if (s->flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
>> + trace_dirty_bitmap_load_bits_zeroes();
>> + bdrv_dirty_bitmap_deserialize_zeroes(s->bitmap, first_byte, nr_bytes,
>> + false);
>> + } else {
>> + uint8_t *buf;
>> + uint64_t buf_size = qemu_get_be64(f);
>> + uint64_t needed_size =
>> + bdrv_dirty_bitmap_serialization_size(s->bitmap,
>> + first_byte, nr_bytes);
>> +
>> + if (needed_size > buf_size) {
>> + error_report("Error: Migrated bitmap granularity doesn't "
>> + "match the destination bitmap '%s' granularity",
>> + bdrv_dirty_bitmap_name(s->bitmap));
>> + return -EINVAL;
>> + }
>> +
>> + buf = g_malloc(buf_size);
>> + qemu_get_buffer(f, buf, buf_size);
> Two things to be careful of there:
> a) buf_size is read from the stream, so be careful about whether you
> trust it could be a crazy value
> b) You don't check the return value of qemu_get_buffer; which actually
> we're pretty bad at in lots of places, but it's probably a good idea
> before trying to deserialise it.
will fix in respin soon.
>
>> + bdrv_dirty_bitmap_deserialize_part(s->bitmap, buf, first_byte, nr_bytes,
>> + false);
>> + g_free(buf);
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s)
>> +{
>> + Error *local_err = NULL;
>> + bool nothing;
>> + s->flags = qemu_get_bitmap_flags(f);
>> + trace_dirty_bitmap_load_header(s->flags);
>> +
>> + nothing = s->flags == (s->flags & DIRTY_BITMAP_MIG_FLAG_EOS);
>> +
>> + if (s->flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
>> + if (!qemu_get_counted_string(f, s->node_name)) {
>> + error_report("Unable to read node name string");
>> + return -EINVAL;
>> + }
>> + s->bs = bdrv_lookup_bs(s->node_name, s->node_name, &local_err);
>> + if (!s->bs) {
>> + error_report_err(local_err);
>> + return -EINVAL;
>> + }
>> + } else if (!s->bs && !nothing) {
>> + error_report("Error: block device name is not set");
>> + return -EINVAL;
>> + }
>> +
>> + if (s->flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
>> + if (!qemu_get_counted_string(f, s->bitmap_name)) {
>> + error_report("Unable to read bitmap name string");
>> + return -EINVAL;
>> + }
>> + s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);
>> +
>> + /* bitmap may be NULL here, it wouldn't be an error if it is the
>> + * first occurrence of the bitmap */
>> + if (!s->bitmap && !(s->flags & DIRTY_BITMAP_MIG_FLAG_START)) {
>> + error_report("Error: unknown dirty bitmap "
>> + "'%s' for block device '%s'",
>> + s->bitmap_name, s->node_name);
>> + return -EINVAL;
>> + }
>> + } else if (!s->bitmap && !nothing) {
>> + error_report("Error: block device name is not set");
>> + return -EINVAL;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int dirty_bitmap_load(QEMUFile *f, void *opaque, int version_id)
>> +{
>> + static DirtyBitmapLoadState s;
>> + int ret = 0;
>> +
>> + trace_dirty_bitmap_load_enter();
>> +
>> + if (version_id != 1) {
>> + return -EINVAL;
>> + }
>> +
>> + do {
>> + ret = dirty_bitmap_load_header(f, &s);
>> +
>> + if (s.flags & DIRTY_BITMAP_MIG_FLAG_START) {
>> + ret = dirty_bitmap_load_start(f, &s);
>> + } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_COMPLETE) {
>> + dirty_bitmap_load_complete(f, &s);
>> + } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_BITS) {
>> + ret = dirty_bitmap_load_bits(f, &s);
>> + }
>> +
>> + if (!ret) {
>> + ret = qemu_file_get_error(f);
>> + }
>> +
>> + if (ret) {
>> + return ret;
>> + }
>> + } while (!(s.flags & DIRTY_BITMAP_MIG_FLAG_EOS));
>> +
>> + trace_dirty_bitmap_load_success();
>> + return 0;
>> +}
>> +
>> +static int dirty_bitmap_save_setup(QEMUFile *f, void *opaque)
>> +{
>> + DirtyBitmapMigBitmapState *dbms = NULL;
>> + if (init_dirty_bitmap_migration() < 0) {
>> + return -1;
>> + }
>> +
>> + QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
>> + send_bitmap_start(f, dbms);
>> + }
>> + qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
>> +
>> + return 0;
>> +}
>> +
>> +static bool dirty_bitmap_is_active(void *opaque)
>> +{
>> + return migrate_dirty_bitmaps() && !dirty_bitmap_mig_state.no_bitmaps;
>> +}
>> +
>> +static bool dirty_bitmap_is_active_iterate(void *opaque)
>> +{
>> + return dirty_bitmap_is_active(opaque) && !runstate_is_running();
>> +}
>> +
>> +static bool dirty_bitmap_has_postcopy(void *opaque)
>> +{
>> + return true;
>> +}
>> +
>> +static SaveVMHandlers savevm_dirty_bitmap_handlers = {
>> + .save_setup = dirty_bitmap_save_setup,
>> + .save_live_complete_postcopy = dirty_bitmap_save_complete,
>> + .save_live_complete_precopy = dirty_bitmap_save_complete,
>> + .has_postcopy = dirty_bitmap_has_postcopy,
>> + .save_live_pending = dirty_bitmap_save_pending,
>> + .save_live_iterate = dirty_bitmap_save_iterate,
>> + .is_active_iterate = dirty_bitmap_is_active_iterate,
>> + .load_state = dirty_bitmap_load,
>> + .save_cleanup = dirty_bitmap_save_cleanup,
>> + .is_active = dirty_bitmap_is_active,
>> +};
>> +
>> +void dirty_bitmap_mig_init(void)
>> +{
>> + QSIMPLEQ_INIT(&dirty_bitmap_mig_state.dbms_list);
>> +
>> + register_savevm_live(NULL, "dirty-bitmap", 0, 1,
>> + &savevm_dirty_bitmap_handlers,
>> + &dirty_bitmap_mig_state);
>> +}
>> diff --git a/migration/migration.c b/migration/migration.c
>> index d09f53d6c3..4128e05d7c 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -155,6 +155,9 @@ MigrationIncomingState *migration_incoming_get_current(void)
>> memset(&mis_current, 0, sizeof(MigrationIncomingState));
>> qemu_mutex_init(&mis_current.rp_mutex);
>> qemu_event_init(&mis_current.main_thread_load_event, false);
>> +
>> + init_dirty_bitmap_incoming_migration();
>> +
> You might want to consider if that's better in vl.c near where
> ram_mig_init() is, OR whether there should be a call in
> migratation_incoming_state_destroy to clean it up.
> (Although I doubt the cases where the destroy happens are interesting
> for postcopy bitmaps).
If you don't mind, let's leave it as is for now
>
>> once = true;
>> }
>> return &mis_current;
>> @@ -297,6 +300,8 @@ static void process_incoming_migration_bh(void *opaque)
>> state, we need to obey autostart. Any other state is set with
>> runstate_set. */
>>
>> + dirty_bitmap_mig_before_vm_start();
>> +
>> if (!global_state_received() ||
>> global_state_get_runstate() == RUN_STATE_RUNNING) {
>> if (autostart) {
>> diff --git a/migration/savevm.c b/migration/savevm.c
>> index e5d557458e..93b339646b 100644
>> --- a/migration/savevm.c
>> +++ b/migration/savevm.c
>> @@ -1673,6 +1673,8 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)
>>
>> trace_loadvm_postcopy_handle_run_vmstart();
>>
>> + dirty_bitmap_mig_before_vm_start();
>> +
>> if (autostart) {
>> /* Hold onto your hats, starting the CPU */
>> vm_start();
>> diff --git a/vl.c b/vl.c
>> index e517a8d995..0ef3f2b5a2 100644
>> --- a/vl.c
>> +++ b/vl.c
>> @@ -4514,6 +4514,7 @@ int main(int argc, char **argv, char **envp)
>>
>> blk_mig_init();
>> ram_mig_init();
>> + dirty_bitmap_mig_init();
>>
>> /* If the currently selected machine wishes to override the units-per-bus
>> * property of its default HBA interface type, do so now. */
>> diff --git a/migration/Makefile.objs b/migration/Makefile.objs
>> index 99e038024d..c83ec47ba8 100644
>> --- a/migration/Makefile.objs
>> +++ b/migration/Makefile.objs
>> @@ -6,6 +6,7 @@ common-obj-y += qemu-file.o global_state.o
>> common-obj-y += qemu-file-channel.o
>> common-obj-y += xbzrle.o postcopy-ram.o
>> common-obj-y += qjson.o
>> +common-obj-y += block-dirty-bitmap.o
>>
>> common-obj-$(CONFIG_RDMA) += rdma.o
>>
>> diff --git a/migration/trace-events b/migration/trace-events
>> index a04fffb877..e9eb8078d4 100644
>> --- a/migration/trace-events
>> +++ b/migration/trace-events
>> @@ -227,3 +227,17 @@ colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
>> colo_send_message(const char *msg) "Send '%s' message"
>> colo_receive_message(const char *msg) "Receive '%s' message"
>> colo_failover_set_state(const char *new_state) "new state %s"
>> +
>> +# migration/block-dirty-bitmap.c
>> +send_bitmap_header_enter(void) ""
>> +send_bitmap_bits(uint32_t flags, uint64_t start_sector, uint32_t nr_sectors, uint64_t data_size) "\n flags: 0x%x\n start_sector: %" PRIu64 "\n nr_sectors: %" PRIu32 "\n data_size: %" PRIu64 "\n"
> Tracing doesn't have \n's in
will fix.
>
>> +dirty_bitmap_save_iterate(int in_postcopy) "in postcopy: %d"
>> +dirty_bitmap_save_complete_enter(void) ""
>> +dirty_bitmap_save_complete_finish(void) ""
>> +dirty_bitmap_save_pending(uint64_t pending, uint64_t max_size) "pending %" PRIu64 " max: %" PRIu64
>> +dirty_bitmap_load_complete(void) ""
>> +dirty_bitmap_load_bits_enter(uint64_t first_sector, uint32_t nr_sectors) "chunk: %" PRIu64 " %" PRIu32
>> +dirty_bitmap_load_bits_zeroes(void) ""
>> +dirty_bitmap_load_header(uint32_t flags) "flags 0x%x"
>> +dirty_bitmap_load_enter(void) ""
>> +dirty_bitmap_load_success(void) ""
> So other than minor bits, this one looks OK from a migration side; I
> can't say I've followed the block side of the patch though.
>
> Dave
>
>> --
>> 2.11.1
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending
2018-03-13 16:33 ` Vladimir Sementsov-Ogievskiy
@ 2018-03-13 17:10 ` John Snow
0 siblings, 0 replies; 33+ messages in thread
From: John Snow @ 2018-03-13 17:10 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, Dr. David Alan Gilbert
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, lirans
On 03/13/2018 12:33 PM, Vladimir Sementsov-Ogievskiy wrote:
> 13.03.2018 19:16, John Snow wrote:
>>
>> On 03/13/2018 12:14 PM, Vladimir Sementsov-Ogievskiy wrote:
>>> Hmm, I agree, it is the simplest thing we can do for now, and I'll
>>> rethink later,
>>> how (and is it worth doing) to go to postcopy automatically in case of
>>> only-dirty-bitmaps.
>>> Should I respin?
>> Please do. I already staged patches 1-4 in my branch, so if you'd like,
>> you can respin just 5+.
>>
>> https://github.com/jnsnow/qemu/tree/bitmaps
>>
>> --js
>
> Ok, I'll base on your branch. How should I write Based-on: for patchew
> in this case?
>
You know, that's actually a good question... just send out the whole
series for the sake of patchew. I Added an R-B to patches 1-4 and edited
a single comment in #4.
--js
^ permalink raw reply [flat|nested] 33+ messages in thread
* Re: [Qemu-devel] [PATCH v10 10/12] migration: add postcopy migration of dirty bitmaps
2018-03-13 16:59 ` Vladimir Sementsov-Ogievskiy
@ 2018-03-13 18:02 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 33+ messages in thread
From: Dr. David Alan Gilbert @ 2018-03-13 18:02 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy
Cc: qemu-block, qemu-devel, pbonzini, armbru, eblake, famz, stefanha,
amit.shah, quintela, mreitz, kwolf, peter.maydell, den, jsnow,
lirans
* Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> 12.03.2018 19:09, Dr. David Alan Gilbert wrote:
> > * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> > > Postcopy migration of dirty bitmaps. Only named dirty bitmaps are migrated.
> > >
> > > +
> > > + init_dirty_bitmap_incoming_migration();
> > > +
> > You might want to consider if that's better in vl.c near where
> > ram_mig_init() is, OR whether there should be a call in
> > migratation_incoming_state_destroy to clean it up.
> > (Although I doubt the cases where the destroy happens are interesting
> > for postcopy bitmaps).
>
> If you don't mind, let's leave it as is for now
Yep, that's OK.
Dave
> >
> > > once = true;
> > > }
> > > return &mis_current;
> > > @@ -297,6 +300,8 @@ static void process_incoming_migration_bh(void *opaque)
> > > state, we need to obey autostart. Any other state is set with
> > > runstate_set. */
> > > + dirty_bitmap_mig_before_vm_start();
> > > +
> > > if (!global_state_received() ||
> > > global_state_get_runstate() == RUN_STATE_RUNNING) {
> > > if (autostart) {
> > > diff --git a/migration/savevm.c b/migration/savevm.c
> > > index e5d557458e..93b339646b 100644
> > > --- a/migration/savevm.c
> > > +++ b/migration/savevm.c
> > > @@ -1673,6 +1673,8 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)
> > > trace_loadvm_postcopy_handle_run_vmstart();
> > > + dirty_bitmap_mig_before_vm_start();
> > > +
> > > if (autostart) {
> > > /* Hold onto your hats, starting the CPU */
> > > vm_start();
> > > diff --git a/vl.c b/vl.c
> > > index e517a8d995..0ef3f2b5a2 100644
> > > --- a/vl.c
> > > +++ b/vl.c
> > > @@ -4514,6 +4514,7 @@ int main(int argc, char **argv, char **envp)
> > > blk_mig_init();
> > > ram_mig_init();
> > > + dirty_bitmap_mig_init();
> > > /* If the currently selected machine wishes to override the units-per-bus
> > > * property of its default HBA interface type, do so now. */
> > > diff --git a/migration/Makefile.objs b/migration/Makefile.objs
> > > index 99e038024d..c83ec47ba8 100644
> > > --- a/migration/Makefile.objs
> > > +++ b/migration/Makefile.objs
> > > @@ -6,6 +6,7 @@ common-obj-y += qemu-file.o global_state.o
> > > common-obj-y += qemu-file-channel.o
> > > common-obj-y += xbzrle.o postcopy-ram.o
> > > common-obj-y += qjson.o
> > > +common-obj-y += block-dirty-bitmap.o
> > > common-obj-$(CONFIG_RDMA) += rdma.o
> > > diff --git a/migration/trace-events b/migration/trace-events
> > > index a04fffb877..e9eb8078d4 100644
> > > --- a/migration/trace-events
> > > +++ b/migration/trace-events
> > > @@ -227,3 +227,17 @@ colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
> > > colo_send_message(const char *msg) "Send '%s' message"
> > > colo_receive_message(const char *msg) "Receive '%s' message"
> > > colo_failover_set_state(const char *new_state) "new state %s"
> > > +
> > > +# migration/block-dirty-bitmap.c
> > > +send_bitmap_header_enter(void) ""
> > > +send_bitmap_bits(uint32_t flags, uint64_t start_sector, uint32_t nr_sectors, uint64_t data_size) "\n flags: 0x%x\n start_sector: %" PRIu64 "\n nr_sectors: %" PRIu32 "\n data_size: %" PRIu64 "\n"
> > Tracing doesn't have \n's in
>
> will fix.
>
> >
> > > +dirty_bitmap_save_iterate(int in_postcopy) "in postcopy: %d"
> > > +dirty_bitmap_save_complete_enter(void) ""
> > > +dirty_bitmap_save_complete_finish(void) ""
> > > +dirty_bitmap_save_pending(uint64_t pending, uint64_t max_size) "pending %" PRIu64 " max: %" PRIu64
> > > +dirty_bitmap_load_complete(void) ""
> > > +dirty_bitmap_load_bits_enter(uint64_t first_sector, uint32_t nr_sectors) "chunk: %" PRIu64 " %" PRIu32
> > > +dirty_bitmap_load_bits_zeroes(void) ""
> > > +dirty_bitmap_load_header(uint32_t flags) "flags 0x%x"
> > > +dirty_bitmap_load_enter(void) ""
> > > +dirty_bitmap_load_success(void) ""
> > So other than minor bits, this one looks OK from a migration side; I
> > can't say I've followed the block side of the patch though.
> >
> > Dave
> >
> > > --
> > > 2.11.1
> > >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>
> --
> Best regards,
> Vladimir
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 33+ messages in thread
end of thread, other threads:[~2018-03-13 18:03 UTC | newest]
Thread overview: 33+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-02-07 15:58 [Qemu-devel] [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 01/12] block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor() Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 02/12] block/dirty-bitmap: fix locking in bdrv_reclaim_dirty_bitmap Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 03/12] block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 04/12] dirty-bitmap: add locked state Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 05/12] migration: introduce postcopy-only pending Vladimir Sementsov-Ogievskiy
2018-03-12 15:30 ` Dr. David Alan Gilbert
2018-03-13 5:38 ` John Snow
2018-03-13 12:17 ` Dr. David Alan Gilbert
2018-03-13 7:47 ` Vladimir Sementsov-Ogievskiy
2018-03-13 10:30 ` Dr. David Alan Gilbert
2018-03-13 13:11 ` Vladimir Sementsov-Ogievskiy
2018-03-13 13:32 ` Vladimir Sementsov-Ogievskiy
2018-03-13 15:35 ` Dr. David Alan Gilbert
2018-03-13 16:14 ` Vladimir Sementsov-Ogievskiy
2018-03-13 16:16 ` John Snow
2018-03-13 16:33 ` Vladimir Sementsov-Ogievskiy
2018-03-13 17:10 ` John Snow
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 06/12] qapi: add dirty-bitmaps migration capability Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 07/12] migration: include migrate_dirty_bitmaps in migrate_postcopy Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 08/12] migration/qemu-file: add qemu_put_counted_string() Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 09/12] migration: add is_active_iterate handler Vladimir Sementsov-Ogievskiy
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 10/12] migration: add postcopy migration of dirty bitmaps Vladimir Sementsov-Ogievskiy
2018-03-12 16:09 ` Dr. David Alan Gilbert
2018-03-13 16:59 ` Vladimir Sementsov-Ogievskiy
2018-03-13 18:02 ` Dr. David Alan Gilbert
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 11/12] iotests: add dirty bitmap migration test Vladimir Sementsov-Ogievskiy
2018-03-12 16:19 ` Dr. David Alan Gilbert
2018-02-07 15:58 ` [Qemu-devel] [PATCH v10 12/12] iotests: add dirty bitmap postcopy test Vladimir Sementsov-Ogievskiy
2018-03-12 16:24 ` Dr. David Alan Gilbert
2018-02-15 9:38 ` [Qemu-devel] ping Re: [PATCH v10 00/12] Dirty bitmaps postcopy migration Vladimir Sementsov-Ogievskiy
2018-02-16 15:03 ` [Qemu-devel] " Vladimir Sementsov-Ogievskiy
2018-03-02 13:05 ` [Qemu-devel] ping " Vladimir Sementsov-Ogievskiy
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.