[Qemu-devel] [PATCH v11 00/13] Dirty bitmaps postcopy migration

* [Qemu-devel] [PATCH v11 00/13] Dirty bitmaps postcopy migration
@ 2018-03-13 18:03 Vladimir Sementsov-Ogievskiy
  2018-03-13 18:03 ` [Qemu-devel] [PATCH v11 01/13] block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor() Vladimir Sementsov-Ogievskiy
                   ` (13 more replies)
  0 siblings, 14 replies; 25+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2018-03-13 18:03 UTC (permalink / raw)
  To: qemu-block, qemu-devel
  Cc: pbonzini, armbru, eblake, famz, stefanha, amit.shah, quintela,
	mreitz, kwolf, peter.maydell, dgilbert, den, jsnow, vsementsov,
	lirans

Hi all!

There is a new version of dirty bitmap postcopy migration series.

Patches 01-04 are directly from John's branch
  https://github.com/jnsnow/qemu/tree/bitmaps
, they are included only for patchew.

v11
clone: tag postcopy-v11 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v11

05: drop inconsistent behavior change, keeping necessity of setting s->start_postcopy
10: new patch. it is needed because of 05 change, we should allow
    migrate-start-postcopy for dirty-bitmaps too.
11: in dirty_bitmap_load_bits():
    - check too large buffer size
    - check return value of qemu_get_buffer
    drop "\n" from trace-event
12: - set dirty-bitmap capability for target (only for online case and left a TODO for
    offline).
    - move from STOP to MIGRATION event like in 203
13: - drop Fam's r-b
    - set dirty-bitmap capability for target
    - move from STOP to MIGRATION event like in 203
    - add missed self.assert_qmp after migrate cmd
    - add call of migrate-start-postcopy (see 05 changes and patch 10)

v10

clone: tag postcopy-v10 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v10

01,02: r-b Fam
03: adjust comments about locking
04: fixed 124 iotest (was broken because of small mistake in block/dirty-bitmap.c)
05: rebased on master, staff from migration_thread is moved to migration_iteration_run, so
    drop r-b by John and Juan
06: 2.11->2.12, r-b Fam
07,08,09,: r-b Fam

10: move to device names instead of node names, looks like libvirt don't care about
    same node-names.
    flag AUTOLOAD is ignored for now
    use QEMU_ALIGN_UP and DIV_ROUND_UP
    skip automatically inserted nodes, when search for dirty bitmaps
    allow migration of no bitmaps (see in dirty_bitmap_load_header new logic
                                   with nothing variable, which avoids extra errors)
    handle return code of dirty_bitmap_load_header
    avoid iteration if there are no bitmaps (see new .no_bitmaps field of 
                                             dirty_bitmap_mig_state)
    call dirty_bitmap_mig_before_vm_start from process_incoming_migration_bh too,
    to enable bitmaps in case of postcopy not actually started.
11: not add r-b Fam
    tiny reorganisation of do_test_migration parameters: remove useless default
    values and make shared_storage to be the last
    disable shared storage test for now, until it will be fixed (it will be separate
    series, more related to qcow2 than to migration)
12: r-b Fam

also, "iotests: add default node-name" is dropped, as not more needed.

v9

clone: tag postcopy-v9 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v9

01: r-b John
02: was incomplete, now add here bdrv_reclaim_dirty_bitmap fix
03: new
04: new
05: r-b John
07: fix type in commit message, r-b John
09: add comment about is_active_iterate, r-b Snow and keep Juan's r-b, hope comment is ok
10: change copyright to Virtuozzo
    reword comment at the top of the file
    rewrite init_dirty_bitmap_migration, to not do same things twice (John)
      and skip _only_ unnamed bitmaps, error out for unnamed nodes (John)
    use new "locked" state of bitmaps instead of frozen on source vm
    do not support migrating bitmap to existent one with the same name,
      keep only create-new-bitmap way
    break loop in dirty_bitmap_load_complete when bitmap is found
    use bitmap locking instead of context acquire
12: rewrite, to add more cases. (note, that 169 iotest is also in my
    "[PATCH v2 0/3] fix bitmaps migration through shared storage", which probably should
    go to qemu-stable. So this patch should rewrite it, but here I make it like new patch,
    to simplify review. When "[PATCH v2..." merged I'll rebase this on it), drop r-b
13: move to separate test, drop r-b

v8.1

clone: tag postcopy-v8.1 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v8.1

05: fix compilation, add new version for cmma_save_pending too.

v8

clone: tag postcopy-v8 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v8

- rebased on master
- patches 01-03 from v7 are already merged to master
- patch order is changed to make it possible to merge block/dirty-bitmap patches
  in separate if is needed
01: new patch
03: fixed to use _locked version of bdrv_release_dirty_bitmap
06: qapi-schema.json -> qapi/migration.json
    2.9 -> 2.11
10: protocol changed a bit:
  instead of 1 byte "bitmap enabled flag" this byte becomes just "flags"
  and have "enabled", "persistent" and "autoloading" flags inside.
  also, make all migrated bitmaps to be not persistent (to prevent their
  storing on source vm)
14: new patch

patches status:
01-04 - are only about block/dirty-bitmap and have no r-b. Fam, John, Paolo (about bitmap lock),
    please look at. These patches are ok to be merged in separate (but before 05-14)
other patches are about migration
05-09 has Juan's r-b (and some of them has John's and Eric's r-bs)
10 - the main patch (dirty bitmaps migration), has no r-b.
11 - preparation for tests, not related to migration directly, has Max's r-b, ok to be merged
    separately (but before 12-14)
12-14 - tests, 12 and 13 have Max's r-b, 14 is new

v7

clone: tag postcopy-v7 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v7

- rebased on dirty-bitmap byte-based interfaces
    (based on git://repo.or.cz/qemu/ericb.git branch nbd-byte-dirty-v4)
- migration of persistent bitmaps should fail for shared storage migration for now,
  as persistent dirty bitmaps are stored/load on inactivate/invalidate-cache.
  also, even for non-shared storage migration there would be useless saving of dirty
  bitmaps on source. This all will be optimized later.

01: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
03: some structural changes due to rebase - drop r-b
04: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
    staff from include/sysemu/sysemu.h moved to migration/savevm.h (rebase)
05: fix patch header: block -> block/dirty-bitmap
    add locking, drop r-b
06: staff from include/migration/migration.h moved to migration/migration.h (rebase)
07: add locking, drop r-b
09: staff from include/migration/qemu-file.h moved to migration/qemu-file.h (rebase)
10: staff from include/migration/vmstate.h moved to include/migration/register.h (rebase)
11: new patch
12: a lot of changes/fixes (mostly by Fam's comments) + rebase
    header-definition movement
    remove include <assert.h>
    add some includes
    fix/refactor bitmap flags send
    byte-based interface for dirty bitmaps (rebase)
    froze bitmaps on source
    init_dirty_bitmap_migration can return error, if some of bitmaps are already
      frozen
    bdrv_ref drives with bitmaps
    fprintf -> error_report
    check version_id in _load function

v6:

clone: tag postcopy-v6 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v6

rebase on master.

03 - tiny contextual change

12 - little change, but it should be reviewed. Call of init_dirty_bitmap_incoming_migration()
    (which only initialize mutex) moved from start of process_incoming_migration_co (it was
    immediately after "mis = migration_incoming_state_new(f)") to migration_incoming_get_current()
    to stay with initialization code.
    I remove r-b's, but hope that this will not be a problem. The only change in this patch - is moved
    call of init_dirty_bitmap_incoming_migration.
    I do so because of recent

commit b4b076daf324894dd288cbdb67ff1e3c7434df7b
Author: Juan Quintela <quintela@redhat.com>
Date:   Mon Jan 23 22:32:06 2017 +0100

    migration: create Migration Incoming State at init time

15 - add Max's r-b

v5:

clone: tag postcopy-v5 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v5

- move 'signed-off' over 'reviewed-by' in patches.

03,04 - add comments. Hope they will be ok for you, so add Juan's r-b.
    If not ok - let me know and I'll resend.

06,08,12 - add Max's r-b
07,09,10,11,12 - add Juan's r-b

14 - used last version of this patch from qcow2-bitmap series with 
     Max's r-b. It has contextual changes due to different base.

15 - fix 041 iotest, add default node-name only if path is specified and
     node-name is not specified
16 - handle whitespaces
       s/"exec: cat " + fifo/"exec: cat '" + fifo + "'"/
     fix indentation
     add Max's r-b
17 - fix typos, wrong size in comment, s/md5/sha256/
     add Max's r-b

v4:

clone: tag postcopy-v4 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v4

reroll, to fix "failed automatic build test"

rebase on master!

07: since 2.8 -> since 2.9
14: fix build of test-hbitmap
    since 2.8 -> since 2.9 and small rewording of comment (this change was not done for
    same patch in may parallels series about qcow2 bitmap extension)

v3:

rebased on Max's block branch: https://github.com/XanClic/qemu/commits/block
clone: tag postcopy-v3 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=postcopy-v3

01  - r-b by Juan
02  - r-b by Juan and David
04  - fix indent
07  - s/since 2.6/since 2.8/
10  - change variable name s/buf/str/
12  - improve copyright message and move it up
      fix memory loss (thanks to Juan)
      switch from DPRINTF to trace events
14* - switch to sha256 and qcrypto_hash_*
      separate qmp command x-debug-block-dirty-bitmap-sha256
16  - use path_suffix for multi-vm test
      fix indent
      fix copyright
      use x-debug-block-dirty-bitmap-sha256 instead of md5
17  - use x-debug-block-dirty-bitmap-sha256 instead of md5
      remove not existing 170 test from qemu-iotests/group

*Note: patch 14 is also used in my second series 'qcow2 persistent dirty bitmaps'

v2:
some bugs fixed, iotests a bit changed and merged into one test.
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2

v1:

These series are derived from my 'Dirty bitmaps migration' series. The
core idea is switch to postcopy migration and drop usage of meta
bitmaps.

These patches provide dirty bitmap postcopy migration feature. Only
named dirty bitmaps are to be migrated. Migration may be enabled using
migration capabilities.

The overall method (thanks to John Snow):

1. migrate bitmaps meta data in .save_live_setup
   - create/find related bitmaps on target
   - disable them
   - create successors (anonimous children) only for enabled migrated
     bitmaps
2. do nothing in precopy stage
3. just before target vm start: enable successors, created in (1)
4. migrate bitmap data
5. reclaime bitmaps (merge successors to their parents)
6. enable bitmaps (only bitmaps, which was enabled in source)

Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
(DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
and I'll drop them in the following version.

So, relatively to last DBMv7: 

01-04: new patches, splitting common postcopy migration out of ram
       postcopy migration
   05: equal to DBMv7.05
   06: new
   07: equal to DBMv7.06
   08: new
   09: equal to DBMv7.07
   10: new
   11: derived from DBMv7.08, see below
12-15: equal to DBMv7.09-12
   16: derived from DBMv7.13
       - switch from fifo to socket, as postcopy don't work with fifo
         for now
       - change parameters: size, granularity, regions
       - add time.sleep, to wait for postcopy migration phase (bad
         temporary solution.
       - drop Reviewed-by
   17: new

   11: the core patch of the series, it is derived from
       [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
       There are a lot of changes related to switching from precopy to
       postcopy, but functions related to migration stream itself
       (structs, send/load sequences) are mostly unchnaged.

       So, changes, to switch from precopy to postcopy:
       - removed all staff related to meta bitmaps and dirty phase!!!
       - add dirty_bitmap_mig_enable_successors, and call it before
         target vm start in loadvm_postcopy_handle_run
       - add enabled_bitmaps list of bitmaps for
         dirty_bitmap_mig_enable_successors

       - enabled flag is send with start bitmap chunk instead of
         completion chunk
       - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
         using meta bitmap granularity

       - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
         to postcopy stage
       - dirty_bitmap_save_pending: remove dirty phase related pending,
         switch pending to non-postcopyable
       - dirty_bitmap_load_start: get enabled flag and prepare
         successors for enabled bitmaps, also add them to
         enabled_bitmaps list
       - dirty_bitmap_load_complete: for enabled bitmaps: merge them
         with successors and enable

       - savevm handlers:
         * remove separate savevm_dirty_bitmap_live_iterate_handlers state
           (it was bad idea, any way), and move its save_live_iterate to
           savevm_dirty_bitmap_handlers
         * add is_active_iterate savevm handler, which allows iterations
           only in postcopy stage (after stopping source vm)
         * add has_postcopy savevm handler. (ofcourse, just returning true)
         * use save_live_complete_postcopy instead of
           save_live_complete_precopy

       Other changes:
       - some debug output changed
       - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
         was needed to omit iterations if bitmap data is small, possibly
         this should be reimplemented)

Vladimir Sementsov-Ogievskiy (13):
  block/dirty-bitmap: add bdrv_dirty_bitmap_enable_successor()
  block/dirty-bitmap: fix locking in bdrv_reclaim_dirty_bitmap
  block/dirty-bitmap: add _locked version of bdrv_reclaim_dirty_bitmap
  dirty-bitmap: add locked state
  migration: introduce postcopy-only pending
  qapi: add dirty-bitmaps migration capability
  migration: include migrate_dirty_bitmaps in migrate_postcopy
  migration/qemu-file: add qemu_put_counted_string()
  migration: add is_active_iterate handler
  migration: allow qmp command migrate-start-postcopy for any postcopy
  migration: add postcopy migration of dirty bitmaps
  iotests: add dirty bitmap migration test
  iotests: add dirty bitmap postcopy test

 qapi/block-core.json           |   5 +-
 qapi/migration.json            |   6 +-
 include/block/dirty-bitmap.h   |   7 +
 include/migration/misc.h       |   3 +
 include/migration/register.h   |  26 +-
 migration/migration.h          |   4 +
 migration/qemu-file.h          |   2 +
 migration/savevm.h             |   5 +-
 block/dirty-bitmap.c           | 123 +++++--
 blockdev.c                     |  19 ++
 hw/s390x/s390-stattrib.c       |   7 +-
 migration/block-dirty-bitmap.c | 746 +++++++++++++++++++++++++++++++++++++++++
 migration/block.c              |   7 +-
 migration/migration.c          |  30 +-
 migration/qemu-file.c          |  13 +
 migration/ram.c                |   9 +-
 migration/savevm.c             |  20 +-
 vl.c                           |   1 +
 migration/Makefile.objs        |   1 +
 migration/trace-events         |  16 +-
 tests/qemu-iotests/169         | 156 +++++++++
 tests/qemu-iotests/169.out     |   5 +
 tests/qemu-iotests/199         | 118 +++++++
 tests/qemu-iotests/199.out     |   5 +
 tests/qemu-iotests/group       |   2 +
 tests/qemu-iotests/iotests.py  |   7 +-
 26 files changed, 1277 insertions(+), 66 deletions(-)
 create mode 100644 migration/block-dirty-bitmap.c
 create mode 100755 tests/qemu-iotests/169
 create mode 100644 tests/qemu-iotests/169.out
 create mode 100755 tests/qemu-iotests/199
 create mode 100644 tests/qemu-iotests/199.out

-- 
2.11.1

^ permalink raw reply	[flat|nested] 25+ messages in thread