[Qemu-devel] [PATCH 00/18] Dirty bitmaps postcopy migration

[Qemu-devel] [PATCH 00/18] Dirty bitmaps postcopy migration

[Vladimir Sementsov-Ogievskiy]

v2:
some bugs fixed, iotests a bit changed and merged into one test.
based on block-next (https://github.com/XanClic/qemu/commits/block-next)
clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2

v1:

These series are derived from my 'Dirty bitmaps migration' series. The
core idea is switch to postcopy migration and drop usage of meta
bitmaps.

These patches provide dirty bitmap postcopy migration feature. Only
named dirty bitmaps are to be migrated. Migration may be enabled using
migration capabilities.

The overall method (thanks to John Snow):

1. migrate bitmaps meta data in .save_live_setup
   - create/find related bitmaps on target
   - disable them
   - create successors (anonimous children) only for enabled migrated
     bitmaps
2. do nothing in precopy stage
3. just before target vm start: enable successors, created in (1)
4. migrate bitmap data
5. reclaime bitmaps (merge successors to their parents)
6. enable bitmaps (only bitmaps, which was enabled in source)


Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
(DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
and I'll drop them in the following version.

So, relatively to last DBMv7: 

01-04: new patches, splitting common postcopy migration out of ram
       postcopy migration
   05: equal to DBMv7.05
   06: new
   07: equal to DBMv7.06
   08: new
   09: equal to DBMv7.07
   10: new
   11: derived from DBMv7.08, see below
12-15: equal to DBMv7.09-12
   16: derived from DBMv7.13
       - switch from fifo to socket, as postcopy don't work with fifo
         for now
       - change parameters: size, granularity, regions
       - add time.sleep, to wait for postcopy migration phase (bad
         temporary solution.
       - drop Reviewed-by
   17: new

   11: the core patch of the series, it is derived from
       [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
       There are a lot of changes related to switching from precopy to
       postcopy, but functions related to migration stream itself
       (structs, send/load sequences) are mostly unchnaged.

       So, changes, to switch from precopy to postcopy:
       - removed all staff related to meta bitmaps and dirty phase!!!
       - add dirty_bitmap_mig_enable_successors, and call it before
         target vm start in loadvm_postcopy_handle_run
       - add enabled_bitmaps list of bitmaps for
         dirty_bitmap_mig_enable_successors

       - enabled flag is send with start bitmap chunk instead of
         completion chunk
       - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
         using meta bitmap granularity

       - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
         to postcopy stage
       - dirty_bitmap_save_pending: remove dirty phase related pending,
         switch pending to non-postcopyable
       - dirty_bitmap_load_start: get enabled flag and prepare
         successors for enabled bitmaps, also add them to
         enabled_bitmaps list
       - dirty_bitmap_load_complete: for enabled bitmaps: merge them
         with successors and enable

       - savevm handlers:
         * remove separate savevm_dirty_bitmap_live_iterate_handlers state
           (it was bad idea, any way), and move its save_live_iterate to
           savevm_dirty_bitmap_handlers
         * add is_active_iterate savevm handler, which allows iterations
           only in postcopy stage (after stopping source vm)
         * add has_postcopy savevm handler. (ofcourse, just returning true)
         * use save_live_complete_postcopy instead of
           save_live_complete_precopy

       Other changes:
       - some debug output changed
       - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
         was needed to omit iterations if bitmap data is small, possibly
         this should be reimplemented)

Vladimir Sementsov-Ogievskiy (18):
  migration: add has_postcopy savevm handler
  migration: fix ram_save_pending
  migration: split common postcopy out of ram postcopy
  migration: introduce postcopy-only pending
  block: add bdrv_next_dirty_bitmap()
  block: add bdrv_dirty_bitmap_enable_successor()
  qapi: add dirty-bitmaps migration capability
  block/dirty-bitmap: add bdrv_dirty_bitmap_release_successor
  migration: include migrate_dirty_bitmaps in migrate_postcopy
  migration/qemu-file: add qemu_put_counted_string()
  migration: add is_active_iterate handler
  migration: add postcopy migration of dirty bitmaps
  iotests: maintain several vms in test
  iotests: add add_incoming_migration to VM class
  qapi: add md5 checksum of last dirty bitmap level to query-block
  iotests: add default node-name
  iotests: add dirty bitmap migration test 117
  iotests: add dirty bitmap postcopy test

 block/dirty-bitmap.c           |  26 ++
 include/block/dirty-bitmap.h   |   6 +
 include/migration/block.h      |   1 +
 include/migration/migration.h  |   6 +
 include/migration/qemu-file.h  |   2 +
 include/migration/vmstate.h    |   7 +-
 include/qemu/hbitmap.h         |   8 +
 include/sysemu/sysemu.h        |   5 +-
 migration/Makefile.objs        |   2 +-
 migration/block-dirty-bitmap.c | 699 +++++++++++++++++++++++++++++++++++++++++
 migration/block.c              |   7 +-
 migration/migration.c          |  66 ++--
 migration/postcopy-ram.c       |   4 +-
 migration/qemu-file.c          |  13 +
 migration/ram.c                |  19 +-
 migration/savevm.c             |  56 +++-
 migration/trace-events         |   2 +-
 qapi-schema.json               |   4 +-
 qapi/block-core.json           |   5 +-
 tests/qemu-iotests/169         | 134 ++++++++
 tests/qemu-iotests/169.out     |   5 +
 tests/qemu-iotests/group       |   2 +
 tests/qemu-iotests/iotests.py  |  21 +-
 util/hbitmap.c                 |   8 +
 vl.c                           |   1 +
 25 files changed, 1055 insertions(+), 54 deletions(-)
 create mode 100644 migration/block-dirty-bitmap.c
 create mode 100755 tests/qemu-iotests/169
 create mode 100644 tests/qemu-iotests/169.out

-- 
1.8.3.1

[Qemu-devel] [PATCH 01/18] migration: add has_postcopy savevm handler

[Vladimir Sementsov-Ogievskiy]

Now postcopy-able states are recognized by not NULL
save_live_complete_postcopy handler. But when we have several different
postcopy-able states, it is not convenient. Ram postcopy may be
disabled, while some other postcopy enabled, in this case Ram state
should behave as it is not postcopy-able.

This patch add separate has_postcopy handler to specify behaviour of
savevm state.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 include/migration/vmstate.h | 1 +
 migration/ram.c             | 6 ++++++
 migration/savevm.c          | 6 ++++--
 3 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 1638ee5..5c30ef7 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -46,6 +46,7 @@ typedef struct SaveVMHandlers {
 
     /* This runs both outside and inside the iothread lock.  */
     bool (*is_active)(void *opaque);
+    bool (*has_postcopy)(void *opaque);
 
     /* This runs outside the iothread lock in the migration case, and
      * within the lock in the savevm case.  The callback had better only
diff --git a/migration/ram.c b/migration/ram.c
index 815bc0e..fa5c72b 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2578,10 +2578,16 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
     return ret;
 }
 
+static bool ram_has_postcopy(void *opaque)
+{
+    return migrate_postcopy_ram();
+}
+
 static SaveVMHandlers savevm_ram_handlers = {
     .save_live_setup = ram_save_setup,
     .save_live_iterate = ram_save_iterate,
     .save_live_complete_postcopy = ram_save_complete,
+    .has_postcopy = ram_has_postcopy,
     .save_live_complete_precopy = ram_save_complete,
     .save_live_pending = ram_save_pending,
     .load_state = ram_load,
diff --git a/migration/savevm.c b/migration/savevm.c
index 33a2911..d2efeeb 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -940,7 +940,8 @@ int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy)
          * call that's already run, it might get confused if we call
          * iterate afterwards.
          */
-        if (postcopy && !se->ops->save_live_complete_postcopy) {
+        if (postcopy &&
+            !(se->ops->has_postcopy && se->ops->has_postcopy(se->opaque))) {
             continue;
         }
         if (qemu_file_rate_limit(f)) {
@@ -1028,7 +1029,8 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only)
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
         if (!se->ops ||
-            (in_postcopy && se->ops->save_live_complete_postcopy) ||
+            (in_postcopy && se->ops->has_postcopy &&
+             se->ops->has_postcopy(se->opaque)) ||
             (in_postcopy && !iterable_only) ||
             !se->ops->save_live_complete_precopy) {
             continue;
-- 
1.8.3.1

[Qemu-devel] [PATCH 02/18] migration: fix ram_save_pending

[Vladimir Sementsov-Ogievskiy]

Fill postcopy-able pending only if ram postcopy is enabled.
It is necessary because of there will be other postcopy-able states and
when ram postcopy is disabled, it should not spoil common postcopy
related pending.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 migration/ram.c | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index fa5c72b..424ce0b 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2079,8 +2079,12 @@ static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
         remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
     }
 
-    /* We can do postcopy, and all the data is postcopiable */
-    *postcopiable_pending += remaining_size;
+    if (migrate_postcopy_ram()) {
+        /* We can do postcopy, and all the data is postcopiable */
+        *postcopiable_pending += remaining_size;
+    } else {
+        *non_postcopiable_pending += remaining_size;
+    }
 }
 
 static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
-- 
1.8.3.1

[Qemu-devel] [PATCH 03/18] migration: split common postcopy out of ram postcopy

[Vladimir Sementsov-Ogievskiy]

Split common postcopy staff from ram postcopy staff.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 include/migration/migration.h |  1 +
 migration/migration.c         | 39 +++++++++++++++++++++++++++------------
 migration/postcopy-ram.c      |  4 +++-
 migration/savevm.c            | 31 ++++++++++++++++++++++---------
 4 files changed, 53 insertions(+), 22 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 3c96623..ca93765 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -290,6 +290,7 @@ void migrate_add_blocker(Error *reason);
  */
 void migrate_del_blocker(Error *reason);
 
+bool migrate_postcopy(void);
 bool migrate_postcopy_ram(void);
 bool migrate_zero_blocks(void);
 
diff --git a/migration/migration.c b/migration/migration.c
index 955d5ee..4300de0 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1198,6 +1198,11 @@ bool migrate_postcopy_ram(void)
     return s->enabled_capabilities[MIGRATION_CAPABILITY_POSTCOPY_RAM];
 }
 
+bool migrate_postcopy(void)
+{
+    return migrate_postcopy_ram();
+}
+
 bool migrate_auto_converge(void)
 {
     MigrationState *s;
@@ -1525,9 +1530,11 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
      * need to tell the destination to throw any pages it's already received
      * that are dirty
      */
-    if (ram_postcopy_send_discard_bitmap(ms)) {
-        error_report("postcopy send discard bitmap failed");
-        goto fail;
+    if (migrate_postcopy_ram()) {
+        if (ram_postcopy_send_discard_bitmap(ms)) {
+            error_report("postcopy send discard bitmap failed");
+            goto fail;
+        }
     }
 
     /*
@@ -1536,8 +1543,10 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
      * wrap their state up here
      */
     qemu_file_set_rate_limit(ms->to_dst_file, INT64_MAX);
-    /* Ping just for debugging, helps line traces up */
-    qemu_savevm_send_ping(ms->to_dst_file, 2);
+    if (migrate_postcopy_ram()) {
+        /* Ping just for debugging, helps line traces up */
+        qemu_savevm_send_ping(ms->to_dst_file, 2);
+    }
 
     /*
      * While loading the device state we may trigger page transfer
@@ -1561,7 +1570,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
     qemu_savevm_send_postcopy_listen(fb);
 
     qemu_savevm_state_complete_precopy(fb, false);
-    qemu_savevm_send_ping(fb, 3);
+    if (migrate_postcopy_ram()) {
+        qemu_savevm_send_ping(fb, 3);
+    }
 
     qemu_savevm_send_postcopy_run(fb);
 
@@ -1584,11 +1595,13 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
 
     qemu_mutex_unlock_iothread();
 
-    /*
-     * Although this ping is just for debug, it could potentially be
-     * used for getting a better measurement of downtime at the source.
-     */
-    qemu_savevm_send_ping(ms->to_dst_file, 4);
+    if (migrate_postcopy_ram()) {
+        /*
+         * Although this ping is just for debug, it could potentially be
+         * used for getting a better measurement of downtime at the source.
+         */
+        qemu_savevm_send_ping(ms->to_dst_file, 4);
+    }
 
     ret = qemu_file_get_error(ms->to_dst_file);
     if (ret) {
@@ -1725,7 +1738,9 @@ static void *migration_thread(void *opaque)
 
         /* And do a ping that will make stuff easier to debug */
         qemu_savevm_send_ping(s->to_dst_file, 1);
+    }
 
+    if (migrate_postcopy()) {
         /*
          * Tell the destination that we *might* want to do postcopy later;
          * if the other end can't do postcopy it should fail now, nice and
@@ -1759,7 +1774,7 @@ static void *migration_thread(void *opaque)
             if (pending_size && pending_size >= max_size) {
                 /* Still a significant amount to transfer */
 
-                if (migrate_postcopy_ram() &&
+                if (migrate_postcopy() &&
                     s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE &&
                     pend_nonpost <= max_size &&
                     atomic_read(&s->start_postcopy)) {
diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index abe8c60..305e017 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -315,7 +315,9 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
     }
 
     postcopy_state_set(POSTCOPY_INCOMING_END);
-    migrate_send_rp_shut(mis, qemu_file_get_error(mis->from_src_file) != 0);
+    if (migrate_postcopy_ram()) {
+        migrate_send_rp_shut(mis, qemu_file_get_error(mis->from_src_file) != 0);
+    }
 
     if (mis->postcopy_tmp_page) {
         munmap(mis->postcopy_tmp_page, getpagesize());
diff --git a/migration/savevm.c b/migration/savevm.c
index d2efeeb..cce542f 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -73,7 +73,7 @@ static struct mig_cmd_args {
     [MIG_CMD_INVALID]          = { .len = -1, .name = "INVALID" },
     [MIG_CMD_OPEN_RETURN_PATH] = { .len =  0, .name = "OPEN_RETURN_PATH" },
     [MIG_CMD_PING]             = { .len = sizeof(uint32_t), .name = "PING" },
-    [MIG_CMD_POSTCOPY_ADVISE]  = { .len = 16, .name = "POSTCOPY_ADVISE" },
+    [MIG_CMD_POSTCOPY_ADVISE]  = { .len = -1, .name = "POSTCOPY_ADVISE" },
     [MIG_CMD_POSTCOPY_LISTEN]  = { .len =  0, .name = "POSTCOPY_LISTEN" },
     [MIG_CMD_POSTCOPY_RUN]     = { .len =  0, .name = "POSTCOPY_RUN" },
     [MIG_CMD_POSTCOPY_RAM_DISCARD] = {
@@ -778,12 +778,17 @@ int qemu_savevm_send_packaged(QEMUFile *f, const uint8_t *buf, size_t len)
 /* Send prior to any postcopy transfer */
 void qemu_savevm_send_postcopy_advise(QEMUFile *f)
 {
-    uint64_t tmp[2];
-    tmp[0] = cpu_to_be64(getpagesize());
-    tmp[1] = cpu_to_be64(1ul << qemu_target_page_bits());
+    if (migrate_postcopy_ram()) {
+        uint64_t tmp[2];
+        tmp[0] = cpu_to_be64(getpagesize());
+        tmp[1] = cpu_to_be64(1ul << qemu_target_page_bits());
 
-    trace_qemu_savevm_send_postcopy_advise();
-    qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_ADVISE, 16, (uint8_t *)tmp);
+        trace_qemu_savevm_send_postcopy_advise();
+        qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_ADVISE,
+                                 16, (uint8_t *)tmp);
+    } else {
+        qemu_savevm_command_send(f, MIG_CMD_POSTCOPY_ADVISE, 0, NULL);
+    }
 }
 
 /* Sent prior to starting the destination running in postcopy, discard pages
@@ -1266,6 +1271,10 @@ static int loadvm_postcopy_handle_advise(MigrationIncomingState *mis)
         return -1;
     }
 
+    if (!migrate_postcopy_ram()) {
+        return 0;
+    }
+
     if (!postcopy_ram_supported_by_host()) {
         return -1;
     }
@@ -1466,7 +1475,9 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
          * A rare case, we entered listen without having to do any discards,
          * so do the setup that's normally done at the time of the 1st discard.
          */
-        postcopy_ram_prepare_discard(mis);
+        if (migrate_postcopy_ram()) {
+            postcopy_ram_prepare_discard(mis);
+        }
     }
 
     /*
@@ -1474,8 +1485,10 @@ static int loadvm_postcopy_handle_listen(MigrationIncomingState *mis)
      * However, at this point the CPU shouldn't be running, and the IO
      * shouldn't be doing anything yet so don't actually expect requests
      */
-    if (postcopy_ram_enable_notify(mis)) {
-        return -1;
+    if (migrate_postcopy_ram()) {
+        if (postcopy_ram_enable_notify(mis)) {
+            return -1;
+        }
     }
 
     if (mis->have_listen_thread) {
-- 
1.8.3.1

[Qemu-devel] [PATCH 04/18] migration: introduce postcopy-only pending

[Vladimir Sementsov-Ogievskiy]

There would be savevm states (dirty-bitmap) which can migrate only in
postcopy stage. The corresponding pending is introduced here.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 include/migration/vmstate.h |  5 +++--
 include/sysemu/sysemu.h     |  5 +++--
 migration/block.c           |  7 ++++---
 migration/migration.c       | 15 ++++++++-------
 migration/ram.c             |  9 +++++----
 migration/savevm.c          | 12 +++++++-----
 migration/trace-events      |  2 +-
 7 files changed, 31 insertions(+), 24 deletions(-)

diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index 5c30ef7..dc656be 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -58,8 +58,9 @@ typedef struct SaveVMHandlers {
     /* This runs outside the iothread lock!  */
     int (*save_live_setup)(QEMUFile *f, void *opaque);
     void (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size,
-                              uint64_t *non_postcopiable_pending,
-                              uint64_t *postcopiable_pending);
+                              uint64_t *res_precopy_only,
+                              uint64_t *res_compatible,
+                              uint64_t *res_postcopy_only);
     LoadStateHandler *load_state;
 } SaveVMHandlers;
 
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index ee7c760..d88bcda 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -114,8 +114,9 @@ void qemu_savevm_state_cleanup(void);
 void qemu_savevm_state_complete_postcopy(QEMUFile *f);
 void qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only);
 void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
-                               uint64_t *res_non_postcopiable,
-                               uint64_t *res_postcopiable);
+                               uint64_t *res_precopy_only,
+                               uint64_t *res_compatible,
+                               uint64_t *res_postcopy_only);
 void qemu_savevm_command_send(QEMUFile *f, enum qemu_vm_cmd command,
                               uint16_t len, uint8_t *data);
 void qemu_savevm_send_ping(QEMUFile *f, uint32_t value);
diff --git a/migration/block.c b/migration/block.c
index ebc10e6..83b0785 100644
--- a/migration/block.c
+++ b/migration/block.c
@@ -830,8 +830,9 @@ static int block_save_complete(QEMUFile *f, void *opaque)
 }
 
 static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
-                               uint64_t *non_postcopiable_pending,
-                               uint64_t *postcopiable_pending)
+                               uint64_t *res_precopy_only,
+                               uint64_t *res_compatible,
+                               uint64_t *res_postcopy_only)
 {
     /* Estimate pending number of bytes to send */
     uint64_t pending;
@@ -852,7 +853,7 @@ static void block_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
 
     DPRINTF("Enter save live pending  %" PRIu64 "\n", pending);
     /* We don't do postcopy */
-    *non_postcopiable_pending += pending;
+    *res_precopy_only += pending;
 }
 
 static int block_load(QEMUFile *f, void *opaque, int version_id)
diff --git a/migration/migration.c b/migration/migration.c
index 4300de0..01a7bd5 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1764,20 +1764,21 @@ static void *migration_thread(void *opaque)
         uint64_t pending_size;
 
         if (!qemu_file_rate_limit(s->to_dst_file)) {
-            uint64_t pend_post, pend_nonpost;
+            uint64_t pend_pre, pend_compat, pend_post;
 
-            qemu_savevm_state_pending(s->to_dst_file, max_size, &pend_nonpost,
-                                      &pend_post);
-            pending_size = pend_nonpost + pend_post;
+            qemu_savevm_state_pending(s->to_dst_file, max_size, &pend_pre,
+                                      &pend_compat, &pend_post);
+            pending_size = pend_pre + pend_compat + pend_post;
             trace_migrate_pending(pending_size, max_size,
-                                  pend_post, pend_nonpost);
+                                  pend_pre, pend_compat, pend_post);
             if (pending_size && pending_size >= max_size) {
                 /* Still a significant amount to transfer */
 
                 if (migrate_postcopy() &&
                     s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE &&
-                    pend_nonpost <= max_size &&
-                    atomic_read(&s->start_postcopy)) {
+                    pend_pre <= max_size &&
+                    (atomic_read(&s->start_postcopy) ||
+                     (pend_pre + pend_compat <= max_size))) {
 
                     if (!postcopy_start(s, &old_vm_running)) {
                         current_active_state = MIGRATION_STATUS_POSTCOPY_ACTIVE;
diff --git a/migration/ram.c b/migration/ram.c
index 424ce0b..fd843b7 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2062,8 +2062,9 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
 }
 
 static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
-                             uint64_t *non_postcopiable_pending,
-                             uint64_t *postcopiable_pending)
+                             uint64_t *res_precopy_only,
+                             uint64_t *res_compatible,
+                             uint64_t *res_postcopy_only)
 {
     uint64_t remaining_size;
 
@@ -2081,9 +2082,9 @@ static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
 
     if (migrate_postcopy_ram()) {
         /* We can do postcopy, and all the data is postcopiable */
-        *postcopiable_pending += remaining_size;
+        *res_compatible += remaining_size;
     } else {
-        *non_postcopiable_pending += remaining_size;
+        *res_precopy_only += remaining_size;
     }
 }
 
diff --git a/migration/savevm.c b/migration/savevm.c
index cce542f..7757a29 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1114,13 +1114,15 @@ void qemu_savevm_state_complete_precopy(QEMUFile *f, bool iterable_only)
  * for units that can't do postcopy.
  */
 void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
-                               uint64_t *res_non_postcopiable,
-                               uint64_t *res_postcopiable)
+                               uint64_t *res_precopy_only,
+                               uint64_t *res_compatible,
+                               uint64_t *res_postcopy_only)
 {
     SaveStateEntry *se;
 
-    *res_non_postcopiable = 0;
-    *res_postcopiable = 0;
+    *res_precopy_only = 0;
+    *res_compatible = 0;
+    *res_postcopy_only = 0;
 
 
     QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
@@ -1133,7 +1135,7 @@ void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
             }
         }
         se->ops->save_live_pending(f, se->opaque, max_size,
-                                   res_non_postcopiable, res_postcopiable);
+                res_precopy_only, res_compatible, res_postcopy_only);
     }
 }
 
diff --git a/migration/trace-events b/migration/trace-events
index 8568dab..bcc6124 100644
--- a/migration/trace-events
+++ b/migration/trace-events
@@ -74,7 +74,7 @@ migrate_fd_cleanup(void) ""
 migrate_fd_error(const char *error_desc) "error=%s"
 migrate_fd_cancel(void) ""
 migrate_handle_rp_req_pages(const char *rbname, size_t start, size_t len) "in %s at %zx len %zx"
-migrate_pending(uint64_t size, uint64_t max, uint64_t post, uint64_t nonpost) "pending size %" PRIu64 " max %" PRIu64 " (post=%" PRIu64 " nonpost=%" PRIu64 ")"
+migrate_pending(uint64_t size, uint64_t max, uint64_t pre, uint64_t compat, uint64_t post) "pending size %" PRIu64 " max %" PRIu64 " (pre = %" PRIu64 " compat=%" PRIu64 " post=%" PRIu64 ")"
 migrate_send_rp_message(int msg_type, uint16_t len) "%d: len %d"
 migration_completion_file_err(void) ""
 migration_completion_postcopy_end(void) ""
-- 
1.8.3.1

[Qemu-devel] [PATCH 05/18] block: add bdrv_next_dirty_bitmap()

[Vladimir Sementsov-Ogievskiy]

Like bdrv_next()  - bdrv_next_dirty_bitmap() is a function to provide
access to private dirty bitmaps list.

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 block/dirty-bitmap.c         | 10 ++++++++++
 include/block/dirty-bitmap.h |  3 +++
 2 files changed, 13 insertions(+)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 519737c..7547c78 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -533,3 +533,13 @@ int64_t bdrv_get_meta_dirty_count(BdrvDirtyBitmap *bitmap)
 {
     return hbitmap_count(bitmap->meta);
 }
+
+BdrvDirtyBitmap *bdrv_next_dirty_bitmap(BlockDriverState *bs,
+                                        BdrvDirtyBitmap *bitmap)
+{
+    if (bitmap == NULL) {
+        return QLIST_FIRST(&bs->dirty_bitmaps);
+    }
+
+    return QLIST_NEXT(bitmap, list);
+}
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 9dea14b..1a457d5 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -72,4 +72,7 @@ void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
                                           bool finish);
 void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
 
+BdrvDirtyBitmap *bdrv_next_dirty_bitmap(BlockDriverState *bs,
+                                        BdrvDirtyBitmap *bitmap);
+
 #endif
-- 
1.8.3.1

[Qemu-devel] [PATCH 06/18] block: add bdrv_dirty_bitmap_enable_successor()

[Vladimir Sementsov-Ogievskiy]

Enabling bitmap successor is necessary to enable successors of bitmaps
being migrated before target vm start.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 block/dirty-bitmap.c         | 5 +++++
 include/block/dirty-bitmap.h | 1 +
 2 files changed, 6 insertions(+)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 7547c78..ab2c7f5 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -217,6 +217,11 @@ int bdrv_dirty_bitmap_create_successor(BlockDriverState *bs,
     return 0;
 }
 
+void bdrv_dirty_bitmap_enable_successor(BdrvDirtyBitmap *bitmap)
+{
+    bdrv_enable_dirty_bitmap(bitmap->successor);
+}
+
 /**
  * For a bitmap with a successor, yield our name to the successor,
  * delete the old bitmap, and return a handle to the new bitmap.
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 1a457d5..57b476c 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -20,6 +20,7 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
 BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
                                            BdrvDirtyBitmap *bitmap,
                                            Error **errp);
+void bdrv_dirty_bitmap_enable_successor(BdrvDirtyBitmap *bitmap);
 BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
                                         const char *name);
 void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap);
-- 
1.8.3.1

[Qemu-devel] [PATCH 07/18] qapi: add dirty-bitmaps migration capability

[Vladimir Sementsov-Ogievskiy]

Reviewed-by: John Snow <jsnow@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 include/migration/migration.h | 1 +
 migration/migration.c         | 9 +++++++++
 qapi-schema.json              | 4 +++-
 3 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index ca93765..223e9ec 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -293,6 +293,7 @@ void migrate_del_blocker(Error *reason);
 bool migrate_postcopy(void);
 bool migrate_postcopy_ram(void);
 bool migrate_zero_blocks(void);
+bool migrate_dirty_bitmaps(void);
 
 bool migrate_auto_converge(void);
 
diff --git a/migration/migration.c b/migration/migration.c
index 01a7bd5..7d56a8e 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1257,6 +1257,15 @@ int migrate_decompress_threads(void)
     return s->parameters.decompress_threads;
 }
 
+bool migrate_dirty_bitmaps(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS];
+}
+
 bool migrate_use_events(void)
 {
     MigrationState *s;
diff --git a/qapi-schema.json b/qapi-schema.json
index 5658723..6386fbc 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -553,11 +553,13 @@
 #          been migrated, pulling the remaining pages along as needed. NOTE: If
 #          the migration fails during postcopy the VM will fail.  (since 2.6)
 #
+# @dirty-bitmaps: If enabled, QEMU will migrate named dirty bitmaps. (since 2.6)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
   'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
-           'compress', 'events', 'postcopy-ram'] }
+           'compress', 'events', 'postcopy-ram', 'dirty-bitmaps'] }
 
 ##
 # @MigrationCapabilityStatus
-- 
1.8.3.1

[Qemu-devel] [PATCH 08/18] block/dirty-bitmap: add bdrv_dirty_bitmap_release_successor

[Vladimir Sementsov-Ogievskiy]

To just release successor and unfreeze bitmap without any additional
work.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Denis V. Lunev <den@openvz.org>
---
 block/dirty-bitmap.c         | 10 ++++++++++
 include/block/dirty-bitmap.h |  2 ++
 2 files changed, 12 insertions(+)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index ab2c7f5..32aa6eb 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -274,6 +274,16 @@ BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
     return parent;
 }
 
+void bdrv_dirty_bitmap_release_successor(BlockDriverState *bs,
+                                         BdrvDirtyBitmap *parent)
+{
+    BdrvDirtyBitmap *successor = parent->successor;
+
+    if (successor) {
+        bdrv_release_dirty_bitmap(bs, successor);
+        parent->successor = NULL;
+    }
+}
 /**
  * Truncates _all_ bitmaps attached to a BDS.
  */
diff --git a/include/block/dirty-bitmap.h b/include/block/dirty-bitmap.h
index 57b476c..20b3ec7 100644
--- a/include/block/dirty-bitmap.h
+++ b/include/block/dirty-bitmap.h
@@ -20,6 +20,8 @@ BdrvDirtyBitmap *bdrv_dirty_bitmap_abdicate(BlockDriverState *bs,
 BdrvDirtyBitmap *bdrv_reclaim_dirty_bitmap(BlockDriverState *bs,
                                            BdrvDirtyBitmap *bitmap,
                                            Error **errp);
+void bdrv_dirty_bitmap_release_successor(BlockDriverState *bs,
+                                         BdrvDirtyBitmap *bitmap);
 void bdrv_dirty_bitmap_enable_successor(BdrvDirtyBitmap *bitmap);
 BdrvDirtyBitmap *bdrv_find_dirty_bitmap(BlockDriverState *bs,
                                         const char *name);
-- 
1.8.3.1

[Qemu-devel] [PATCH 09/18] migration: include migrate_dirty_bitmaps in migrate_postcopy

[Vladimir Sementsov-Ogievskiy]

Enable postcopy if dirty bitmap migration is endabled.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 migration/migration.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/migration/migration.c b/migration/migration.c
index 7d56a8e..a3bb518 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1200,7 +1200,7 @@ bool migrate_postcopy_ram(void)
 
 bool migrate_postcopy(void)
 {
-    return migrate_postcopy_ram();
+    return migrate_postcopy_ram() || migrate_dirty_bitmaps();
 }
 
 bool migrate_auto_converge(void)
-- 
1.8.3.1

[Qemu-devel] [PATCH 10/18] migration/qemu-file: add qemu_put_counted_string()

[Vladimir Sementsov-Ogievskiy]

Add function opposite to qemu_get_counted_string.
qemu_put_counted_string puts one-byte length of the string (string
should not be longer than 255 characters), and then it puts the string,
without last zero byte.

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 include/migration/qemu-file.h |  2 ++
 migration/qemu-file.c         | 13 +++++++++++++
 2 files changed, 15 insertions(+)

diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index abedd46..d860c92 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -309,4 +309,6 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv)
 
 size_t qemu_get_counted_string(QEMUFile *f, char buf[256]);
 
+void qemu_put_counted_string(QEMUFile *f, const char *name);
+
 #endif
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index bbc565e..6fcdd68 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -690,6 +690,19 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])
 }
 
 /*
+ * Put a string with one preceding byte containing its length. The length of
+ * the string should be less than 256.
+ */
+void qemu_put_counted_string(QEMUFile *f, const char *name)
+{
+    size_t len = strlen(name);
+
+    assert(len < 256);
+    qemu_put_byte(f, len);
+    qemu_put_buffer(f, (const uint8_t *)name, len);
+}
+
+/*
  * Set the blocking state of the QEMUFile.
  * Note: On some transports the OS only keeps a single blocking state for
  *       both directions, and thus changing the blocking on the main
-- 
1.8.3.1

[Qemu-devel] [PATCH 11/18] migration: add is_active_iterate handler

[Vladimir Sementsov-Ogievskiy]

Only-postcopy savevm states (dirty-bitmap) don't need live iteration, so
to disable them and stop transporting empty sections there is a new
savevm handler.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 include/migration/vmstate.h | 1 +
 migration/savevm.c          | 5 +++++
 2 files changed, 6 insertions(+)

diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
index dc656be..2c53d0b 100644
--- a/include/migration/vmstate.h
+++ b/include/migration/vmstate.h
@@ -47,6 +47,7 @@ typedef struct SaveVMHandlers {
     /* This runs both outside and inside the iothread lock.  */
     bool (*is_active)(void *opaque);
     bool (*has_postcopy)(void *opaque);
+    bool (*is_active_iterate)(void *opaque);
 
     /* This runs outside the iothread lock in the migration case, and
      * within the lock in the savevm case.  The callback had better only
diff --git a/migration/savevm.c b/migration/savevm.c
index 7757a29..4fbed75 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -939,6 +939,11 @@ int qemu_savevm_state_iterate(QEMUFile *f, bool postcopy)
                 continue;
             }
         }
+        if (se->ops && se->ops->is_active_iterate) {
+            if (!se->ops->is_active_iterate(se->opaque)) {
+                continue;
+            }
+        }
         /*
          * In the postcopy phase, any device that doesn't know how to
          * do postcopy should have saved it's state in the _complete
-- 
1.8.3.1

[Qemu-devel] [PATCH 12/18] migration: add postcopy migration of dirty bitmaps

[Vladimir Sementsov-Ogievskiy]

Postcopy migration of dirty bitmaps. Only named dirty bitmaps,
associated with root nodes and non-root named nodes are migrated.

If destination qemu is already containing a dirty bitmap with the same name
as a migrated bitmap (for the same node), than, if their granularities are
the same the migration will be done, otherwise the error will be generated.

If destination qemu doesn't contain such bitmap it will be created.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 include/migration/block.h      |   1 +
 include/migration/migration.h  |   4 +
 migration/Makefile.objs        |   2 +-
 migration/block-dirty-bitmap.c | 699 +++++++++++++++++++++++++++++++++++++++++
 migration/migration.c          |   3 +
 migration/savevm.c             |   2 +
 vl.c                           |   1 +
 7 files changed, 711 insertions(+), 1 deletion(-)
 create mode 100644 migration/block-dirty-bitmap.c

diff --git a/include/migration/block.h b/include/migration/block.h
index 41a1ac8..8333c43 100644
--- a/include/migration/block.h
+++ b/include/migration/block.h
@@ -14,6 +14,7 @@
 #ifndef MIGRATION_BLOCK_H
 #define MIGRATION_BLOCK_H
 
+void dirty_bitmap_mig_init(void);
 void blk_mig_init(void);
 int blk_mig_active(void);
 uint64_t blk_mig_bytes_transferred(void);
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 223e9ec..8eb38f1 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -356,4 +356,8 @@ int ram_save_queue_pages(MigrationState *ms, const char *rbname,
 PostcopyState postcopy_state_get(void);
 /* Set the state and return the old state */
 PostcopyState postcopy_state_set(PostcopyState new_state);
+
+void dirty_bitmap_mig_before_vm_start(void);
+void init_dirty_bitmap_incoming_migration(void);
+
 #endif
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index 30ad945..82900ec 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -8,5 +8,5 @@ common-obj-y += qjson.o
 
 common-obj-$(CONFIG_RDMA) += rdma.o
 
-common-obj-y += block.o
+common-obj-y += block.o block-dirty-bitmap.o
 
diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
new file mode 100644
index 0000000..c668d02
--- /dev/null
+++ b/migration/block-dirty-bitmap.c
@@ -0,0 +1,699 @@
+/*
+ * QEMU dirty bitmap migration
+ *
+ * Postcopy migration of dirty bitmaps. Only named dirty bitmaps, associated
+ * with root nodes and non-root named nodes are migrated.
+ *
+ * If destination qemu is already containing a dirty bitmap with the same name
+ * as a migrated bitmap (for the same node), than, if their granularities are
+ * the same the migration will be done, otherwise the error will be generated.
+ *
+ * If destination qemu doesn't contain such bitmap it will be created.
+ *
+ * format of migration:
+ *
+ * # Header (shared for different chunk types)
+ * 1, 2 or 4 bytes: flags (see qemu_{put,put}_flags)
+ * [ 1 byte: node name size ] \  flags & DEVICE_NAME
+ * [ n bytes: node name     ] /
+ * [ 1 byte: bitmap name size ] \  flags & BITMAP_NAME
+ * [ n bytes: bitmap name     ] /
+ *
+ * # Start of bitmap migration (flags & START)
+ * header
+ * be64: granularity
+ * 1 byte: bitmap enabled flag
+ *
+ * # Complete of bitmap migration (flags & COMPLETE)
+ * header
+ *
+ * # Data chunk of bitmap migration
+ * header
+ * be64: start sector
+ * be32: number of sectors
+ * [ be64: buffer size  ] \ ! (flags & ZEROES)
+ * [ n bytes: buffer    ] /
+ *
+ * The last chunk in stream should contain flags & EOS. The chunk may skip
+ * device and/or bitmap names, assuming them to be the same with the previous
+ * chunk.
+ *
+ *
+ * This file is derived from migration/block.c
+ *
+ * Author:
+ * Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
+ *
+ * original copyright message:
+ * =====================================================================
+ * Copyright IBM, Corp. 2009
+ *
+ * Authors:
+ *  Liran Schour   <lirans@il.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Contributions after 2012-01-13 are licensed under the terms of the
+ * GNU GPL, version 2 or (at your option) any later version.
+ * =====================================================================
+ */
+
+#include "qemu/osdep.h"
+#include "block/block.h"
+#include "block/block_int.h"
+#include "sysemu/block-backend.h"
+#include "qemu/main-loop.h"
+#include "qemu/error-report.h"
+#include "migration/block.h"
+#include "migration/migration.h"
+#include "qemu/hbitmap.h"
+#include "sysemu/sysemu.h"
+#include "qemu/cutils.h"
+#include "qapi/error.h"
+#include <assert.h>
+
+#define CHUNK_SIZE     (1 << 10)
+
+/* Flags occupy from one to four bytes. In all but one the 7-th (EXTRA_FLAGS)
+ * bit should be set. */
+#define DIRTY_BITMAP_MIG_FLAG_EOS           0x01
+#define DIRTY_BITMAP_MIG_FLAG_ZEROES        0x02
+#define DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME   0x04
+#define DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME   0x08
+#define DIRTY_BITMAP_MIG_FLAG_START         0x10
+#define DIRTY_BITMAP_MIG_FLAG_COMPLETE      0x20
+#define DIRTY_BITMAP_MIG_FLAG_BITS          0x40
+
+#define DIRTY_BITMAP_MIG_EXTRA_FLAGS        0x80
+#define DIRTY_BITMAP_MIG_FLAGS_SIZE_16      0x8000
+#define DIRTY_BITMAP_MIG_FLAGS_SIZE_32      0x8080
+
+#define DEBUG_DIRTY_BITMAP_MIGRATION 0
+
+#define DPRINTF(fmt, args...) \
+    do { \
+        if (DEBUG_DIRTY_BITMAP_MIGRATION) { \
+            printf("DMIG %s:%d ", __func__, __LINE__); \
+            printf(fmt, ##args); \
+        } \
+    } while (0)
+
+typedef struct DirtyBitmapMigBitmapState {
+    /* Written during setup phase. */
+    BlockDriverState *bs;
+    const char *node_name;
+    BdrvDirtyBitmap *bitmap;
+    uint64_t total_sectors;
+    uint64_t sectors_per_chunk;
+    QSIMPLEQ_ENTRY(DirtyBitmapMigBitmapState) entry;
+
+    /* For bulk phase. */
+    bool bulk_completed;
+    uint64_t cur_sector;
+} DirtyBitmapMigBitmapState;
+
+typedef struct DirtyBitmapMigState {
+    QSIMPLEQ_HEAD(dbms_list, DirtyBitmapMigBitmapState) dbms_list;
+
+    bool bulk_completed;
+
+    /* for send_bitmap_bits() */
+    BlockDriverState *prev_bs;
+    BdrvDirtyBitmap *prev_bitmap;
+} DirtyBitmapMigState;
+
+typedef struct DirtyBitmapLoadState {
+    uint32_t flags;
+    char node_name[256];
+    char bitmap_name[256];
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *bitmap;
+} DirtyBitmapLoadState;
+
+static DirtyBitmapMigState dirty_bitmap_mig_state;
+
+typedef struct DirtyBitmapLoadBitmapState {
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *bitmap;
+    bool migrated;
+} DirtyBitmapLoadBitmapState;
+static GSList *enabled_bitmaps;
+QemuMutex finish_lock;
+
+void init_dirty_bitmap_incoming_migration(void)
+{
+    qemu_mutex_init(&finish_lock);
+}
+
+static uint32_t qemu_get_bitmap_flags(QEMUFile *f)
+{
+    uint8_t flags = qemu_get_byte(f);
+    if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
+        flags = flags << 8 | qemu_get_byte(f);
+        if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
+            flags = flags << 16 | qemu_get_be16(f);
+        }
+    }
+
+    return flags;
+}
+
+static void qemu_put_bitmap_flags(QEMUFile *f, uint32_t flags)
+{
+    if (!(flags & 0xffffff00)) {
+        qemu_put_byte(f, flags);
+        return;
+    }
+
+    if (!(flags & 0xffff0000)) {
+        qemu_put_be16(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_16);
+        return;
+    }
+
+    qemu_put_be32(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_32);
+}
+
+static void send_bitmap_header(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
+                               uint32_t additional_flags)
+{
+    BlockDriverState *bs = dbms->bs;
+    BdrvDirtyBitmap *bitmap = dbms->bitmap;
+    uint32_t flags = additional_flags;
+    DPRINTF("enter\n");
+
+    if (bs != dirty_bitmap_mig_state.prev_bs) {
+        dirty_bitmap_mig_state.prev_bs = bs;
+        flags |= DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME;
+    }
+
+    if (bitmap != dirty_bitmap_mig_state.prev_bitmap) {
+        dirty_bitmap_mig_state.prev_bitmap = bitmap;
+        flags |= DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME;
+    }
+
+    qemu_put_bitmap_flags(f, flags);
+
+    if (flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
+        qemu_put_counted_string(f, dbms->node_name);
+    }
+
+    if (flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
+        qemu_put_counted_string(f, bdrv_dirty_bitmap_name(bitmap));
+    }
+}
+
+static void send_bitmap_start(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
+{
+    send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_START);
+    qemu_put_be32(f, bdrv_dirty_bitmap_granularity(dbms->bitmap));
+    qemu_put_byte(f, bdrv_dirty_bitmap_enabled(dbms->bitmap));
+}
+
+static void send_bitmap_complete(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
+{
+    send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_COMPLETE);
+}
+
+static void send_bitmap_bits(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
+                             uint64_t start_sector, uint32_t nr_sectors)
+{
+    /* align for buffer_is_zero() */
+    uint64_t align = 4 * sizeof(long);
+    uint64_t unaligned_size =
+        bdrv_dirty_bitmap_serialization_size(dbms->bitmap,
+                                             start_sector, nr_sectors);
+    uint64_t buf_size = (unaligned_size + align - 1) & ~(align - 1);
+    uint8_t *buf = g_malloc0(buf_size);
+    uint32_t flags = DIRTY_BITMAP_MIG_FLAG_BITS;
+
+    bdrv_dirty_bitmap_serialize_part(dbms->bitmap, buf,
+                                     start_sector, nr_sectors);
+
+    if (buffer_is_zero(buf, buf_size)) {
+        g_free(buf);
+        buf = NULL;
+        flags |= DIRTY_BITMAP_MIG_FLAG_ZEROES;
+    }
+
+    DPRINTF("parameters:"
+            "\n   flags:        %x"
+            "\n   start_sector: %" PRIu64
+            "\n   nr_sectors:   %" PRIu32
+            "\n   data_size:    %" PRIu64 "\n",
+            flags, start_sector, nr_sectors, buf_size);
+
+    send_bitmap_header(f, dbms, flags);
+
+    qemu_put_be64(f, start_sector);
+    qemu_put_be32(f, nr_sectors);
+
+    /* if a block is zero we need to flush here since the network
+     * bandwidth is now a lot higher than the storage device bandwidth.
+     * thus if we queue zero blocks we slow down the migration.
+     * also, skip writing block when migrate only dirty bitmaps. */
+    if (flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
+        qemu_fflush(f);
+        return;
+    }
+
+    qemu_put_be64(f, buf_size);
+    qemu_put_buffer(f, buf, buf_size);
+    g_free(buf);
+}
+
+
+/* Called with iothread lock taken.  */
+
+static void init_dirty_bitmap_migration(void)
+{
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *bitmap;
+    DirtyBitmapMigBitmapState *dbms;
+    BdrvNextIterator it;
+    uint64_t total_bytes = 0;
+
+    dirty_bitmap_mig_state.bulk_completed = false;
+    dirty_bitmap_mig_state.prev_bs = NULL;
+    dirty_bitmap_mig_state.prev_bitmap = NULL;
+
+    for (bs = bdrv_first(&it); bs; bs = bdrv_next(&it)) {
+        for (bitmap = bdrv_next_dirty_bitmap(bs, NULL); bitmap;
+             bitmap = bdrv_next_dirty_bitmap(bs, bitmap)) {
+            if (!bdrv_dirty_bitmap_name(bitmap)) {
+                continue;
+            }
+
+            if (!bdrv_get_device_or_node_name(bs)) {
+                /* not named non-root node */
+                continue;
+            }
+
+            dbms = g_new0(DirtyBitmapMigBitmapState, 1);
+            dbms->bs = bs;
+            dbms->node_name = bdrv_get_node_name(bs);
+            if (!dbms->node_name || dbms->node_name[0] == '\0') {
+                dbms->node_name = bdrv_get_device_name(bs);
+            }
+            dbms->bitmap = bitmap;
+            dbms->total_sectors = bdrv_nb_sectors(bs);
+            dbms->sectors_per_chunk = CHUNK_SIZE * 8 *
+                bdrv_dirty_bitmap_granularity(bitmap) >> BDRV_SECTOR_BITS;
+
+            total_bytes +=
+                bdrv_dirty_bitmap_serialization_size(bitmap,
+                                                     0, dbms->total_sectors);
+
+            QSIMPLEQ_INSERT_TAIL(&dirty_bitmap_mig_state.dbms_list,
+                                 dbms, entry);
+        }
+    }
+}
+
+/* Called with no lock taken.  */
+static void bulk_phase_send_chunk(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
+{
+    uint32_t nr_sectors = MIN(dbms->total_sectors - dbms->cur_sector,
+                             dbms->sectors_per_chunk);
+
+    send_bitmap_bits(f, dbms, dbms->cur_sector, nr_sectors);
+
+    dbms->cur_sector += nr_sectors;
+    if (dbms->cur_sector >= dbms->total_sectors) {
+        dbms->bulk_completed = true;
+    }
+}
+
+/* Called with no lock taken.  */
+static void bulk_phase(QEMUFile *f, bool limit)
+{
+    DirtyBitmapMigBitmapState *dbms;
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        while (!dbms->bulk_completed) {
+            bulk_phase_send_chunk(f, dbms);
+            if (limit && qemu_file_rate_limit(f)) {
+                return;
+            }
+        }
+    }
+
+    dirty_bitmap_mig_state.bulk_completed = true;
+}
+
+/* Called with iothread lock taken.  */
+static void dirty_bitmap_mig_cleanup(void)
+{
+    DirtyBitmapMigBitmapState *dbms;
+
+    while ((dbms = QSIMPLEQ_FIRST(&dirty_bitmap_mig_state.dbms_list)) != NULL) {
+        QSIMPLEQ_REMOVE_HEAD(&dirty_bitmap_mig_state.dbms_list, entry);
+        g_free(dbms);
+    }
+}
+
+/* for SaveVMHandlers */
+static void dirty_bitmap_migration_cleanup(void *opaque)
+{
+    dirty_bitmap_mig_cleanup();
+}
+
+static int dirty_bitmap_save_iterate(QEMUFile *f, void *opaque)
+{
+    DPRINTF("in postcopy: %s\n", migration_in_postcopy(migrate_get_current()) ?
+            "yes" : "no");
+
+    if (migration_in_postcopy(migrate_get_current()) &&
+        !dirty_bitmap_mig_state.bulk_completed) {
+        bulk_phase(f, true);
+    }
+
+    qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
+
+    return dirty_bitmap_mig_state.bulk_completed;
+}
+
+/* Called with iothread lock taken.  */
+
+static int dirty_bitmap_save_complete(QEMUFile *f, void *opaque)
+{
+    DirtyBitmapMigBitmapState *dbms;
+    DPRINTF("enter\n");
+
+    if (!dirty_bitmap_mig_state.bulk_completed) {
+        bulk_phase(f, false);
+    }
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        send_bitmap_complete(f, dbms);
+    }
+
+    qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
+
+    DPRINTF("Dirty bitmaps migration completed\n");
+
+    dirty_bitmap_mig_cleanup();
+    return 0;
+}
+
+static void dirty_bitmap_save_pending(QEMUFile *f, void *opaque,
+                                      uint64_t max_size,
+                                      uint64_t *res_precopy_only,
+                                      uint64_t *res_compatible,
+                                      uint64_t *res_postcopy_only)
+{
+    DirtyBitmapMigBitmapState *dbms;
+    uint64_t pending = 0;
+
+    qemu_mutex_lock_iothread();
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        uint64_t gran = bdrv_dirty_bitmap_granularity(dbms->bitmap);
+        uint64_t sectors = dbms->bulk_completed ? 0 :
+                           dbms->total_sectors - dbms->cur_sector;
+
+        pending += (sectors * BDRV_SECTOR_SIZE + gran - 1) / gran;
+    }
+
+    qemu_mutex_unlock_iothread();
+
+    DPRINTF("pending %" PRIu64 ", max: %" PRIu64 "\n",
+            pending, max_size);
+
+    *res_postcopy_only += pending;
+}
+
+/* First occurrence of this bitmap. It should be created if doesn't exist */
+static int dirty_bitmap_load_start(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+    Error *local_err = NULL;
+    uint32_t granularity = qemu_get_be32(f);
+    bool enabled = qemu_get_byte(f);
+
+    if (!s->bitmap) {
+        s->bitmap = bdrv_create_dirty_bitmap(s->bs, granularity,
+                                             s->bitmap_name, &local_err);
+        if (!s->bitmap) {
+            error_report_err(local_err);
+            return -EINVAL;
+        }
+    } else {
+        uint32_t dest_granularity =
+            bdrv_dirty_bitmap_granularity(s->bitmap);
+        if (dest_granularity != granularity) {
+            fprintf(stderr,
+                    "Error: "
+                    "Migrated bitmap granularity (%" PRIu32 ") "
+                    "doesn't match the destination bitmap '%s' "
+                    "granularity (%" PRIu32 ")\n",
+                    granularity,
+                    bdrv_dirty_bitmap_name(s->bitmap),
+                    dest_granularity);
+            return -EINVAL;
+        }
+    }
+
+    bdrv_disable_dirty_bitmap(s->bitmap);
+    if (enabled) {
+        DirtyBitmapLoadBitmapState *b;
+
+        bdrv_dirty_bitmap_create_successor(s->bs, s->bitmap, &local_err);
+        if (local_err) {
+            error_report_err(local_err);
+            return -EINVAL;
+        }
+
+        b = g_new(DirtyBitmapLoadBitmapState, 1);
+        b->bs = s->bs;
+        b->bitmap = s->bitmap;
+        b->migrated = false;
+        enabled_bitmaps = g_slist_prepend(enabled_bitmaps, b);
+    }
+
+    return 0;
+}
+
+void dirty_bitmap_mig_before_vm_start(void)
+{
+    GSList *item;
+
+    qemu_mutex_lock(&finish_lock);
+
+    for (item = enabled_bitmaps; item; item = g_slist_next(item)) {
+        DirtyBitmapLoadBitmapState *b = item->data;
+
+        if (b->migrated) {
+            bdrv_enable_dirty_bitmap(b->bitmap);
+        } else {
+            bdrv_dirty_bitmap_enable_successor(b->bitmap);
+        }
+
+        g_free(b);
+    }
+
+    g_slist_free(enabled_bitmaps);
+    enabled_bitmaps = NULL;
+
+    qemu_mutex_unlock(&finish_lock);
+}
+
+static void dirty_bitmap_load_complete(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+    GSList *item;
+    DPRINTF("dirty bitmap load complete\n");
+    bdrv_dirty_bitmap_deserialize_finish(s->bitmap);
+
+    qemu_mutex_lock(&finish_lock);
+
+    for (item = enabled_bitmaps; item; item = g_slist_next(item)) {
+        DirtyBitmapLoadBitmapState *b = item->data;
+
+        if (b->bitmap == s->bitmap) {
+            b->migrated = true;
+        }
+    }
+
+    if (bdrv_dirty_bitmap_frozen(s->bitmap)) {
+        if (enabled_bitmaps == NULL) {
+            /* in postcopy */
+            AioContext *aio_context = bdrv_get_aio_context(s->bs);
+            aio_context_acquire(aio_context);
+
+            bdrv_reclaim_dirty_bitmap(s->bs, s->bitmap, &error_abort);
+            bdrv_enable_dirty_bitmap(s->bitmap);
+
+            aio_context_release(aio_context);
+        } else {
+            /* target not started, successor is empty */
+            bdrv_dirty_bitmap_release_successor(s->bs, s->bitmap);
+        }
+    }
+
+    qemu_mutex_unlock(&finish_lock);
+}
+
+static int dirty_bitmap_load_bits(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+    uint64_t first_sector = qemu_get_be64(f);
+    uint32_t nr_sectors = qemu_get_be32(f);
+    DPRINTF("chunk: %" PRIu64 " %" PRIu32 "\n", first_sector, nr_sectors);
+
+
+    if (s->flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
+        DPRINTF("   - zeroes\n");
+        bdrv_dirty_bitmap_deserialize_zeroes(s->bitmap, first_sector,
+                                             nr_sectors, false);
+    } else {
+        uint8_t *buf;
+        uint64_t buf_size = qemu_get_be64(f);
+        uint64_t needed_size =
+            bdrv_dirty_bitmap_serialization_size(s->bitmap,
+                                                 first_sector, nr_sectors);
+
+        if (needed_size > buf_size) {
+            fprintf(stderr,
+                    "Error: Migrated bitmap granularity doesn't "
+                    "match the destination bitmap '%s' granularity\n",
+                    bdrv_dirty_bitmap_name(s->bitmap));
+            return -EINVAL;
+        }
+
+        buf = g_malloc(buf_size);
+        qemu_get_buffer(f, buf, buf_size);
+        bdrv_dirty_bitmap_deserialize_part(s->bitmap, buf,
+                                           first_sector,
+                                           nr_sectors, false);
+        g_free(buf);
+    }
+
+    return 0;
+}
+
+static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+    Error *local_err = NULL;
+    s->flags = qemu_get_bitmap_flags(f);
+    DPRINTF("flags: %x\n", s->flags);
+
+    if (s->flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
+        if (!qemu_get_counted_string(f, s->node_name)) {
+            fprintf(stderr, "Unable to read node name string\n");
+            return -EINVAL;
+        }
+        s->bs = bdrv_lookup_bs(s->node_name, s->node_name, &local_err);
+        if (!s->bs) {
+            error_report("%s", error_get_pretty(local_err));
+            error_free(local_err);
+            return -EINVAL;
+        }
+    } else if (!s->bs) {
+        fprintf(stderr, "Error: block device name is not set\n");
+        return -EINVAL;
+    }
+
+    if (s->flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
+        if (!qemu_get_counted_string(f, s->bitmap_name)) {
+            fprintf(stderr, "Unable to read node name string\n");
+            return -EINVAL;
+        }
+        s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);
+
+        /* bitmap may be NULL here, it wouldn't be an error if it is the
+         * first occurrence of the bitmap */
+        if (!s->bitmap && !(s->flags & DIRTY_BITMAP_MIG_FLAG_START)) {
+            fprintf(stderr, "Error: unknown dirty bitmap "
+                    "'%s' for block device '%s'\n",
+                    s->bitmap_name, s->node_name);
+            return -EINVAL;
+        }
+    } else if (!s->bitmap) {
+        fprintf(stderr, "Error: block device name is not set\n");
+        return -EINVAL;
+    }
+
+    return 0;
+}
+
+static int dirty_bitmap_load(QEMUFile *f, void *opaque, int version_id)
+{
+    static DirtyBitmapLoadState s;
+
+    int ret = 0;
+
+    DPRINTF("load start\n");
+
+    do {
+        dirty_bitmap_load_header(f, &s);
+
+        if (s.flags & DIRTY_BITMAP_MIG_FLAG_START) {
+            ret = dirty_bitmap_load_start(f, &s);
+        } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_COMPLETE) {
+            dirty_bitmap_load_complete(f, &s);
+        } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_BITS) {
+            ret = dirty_bitmap_load_bits(f, &s);
+        }
+
+        DPRINTF("ret: %d\n", ret);
+        if (!ret) {
+            ret = qemu_file_get_error(f);
+        }
+
+        DPRINTF("ret: %d\n", ret);
+        if (ret) {
+            return ret;
+        }
+    } while (!(s.flags & DIRTY_BITMAP_MIG_FLAG_EOS));
+
+    DPRINTF("load finish\n");
+    return 0;
+}
+
+static int dirty_bitmap_save_setup(QEMUFile *f, void *opaque)
+{
+    DirtyBitmapMigBitmapState *dbms = NULL;
+    init_dirty_bitmap_migration();
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        send_bitmap_start(f, dbms);
+    }
+    qemu_put_bitmap_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
+
+    return 0;
+}
+
+static bool dirty_bitmap_is_active(void *opaque)
+{
+    return migrate_dirty_bitmaps();
+}
+
+static bool dirty_bitmap_is_active_iterate(void *opaque)
+{
+    return dirty_bitmap_is_active(opaque) && !runstate_is_running();
+}
+
+static bool dirty_bitmap_has_postcopy(void *opaque)
+{
+    return true;
+}
+
+static SaveVMHandlers savevm_dirty_bitmap_handlers = {
+    .save_live_setup = dirty_bitmap_save_setup,
+    .save_live_complete_postcopy = dirty_bitmap_save_complete,
+    .save_live_complete_precopy = dirty_bitmap_save_complete,
+    .has_postcopy = dirty_bitmap_has_postcopy,
+    .save_live_pending = dirty_bitmap_save_pending,
+    .save_live_iterate = dirty_bitmap_save_iterate,
+    .is_active_iterate = dirty_bitmap_is_active_iterate,
+    .load_state = dirty_bitmap_load,
+    .cleanup = dirty_bitmap_migration_cleanup,
+    .is_active = dirty_bitmap_is_active,
+};
+
+void dirty_bitmap_mig_init(void)
+{
+    QSIMPLEQ_INIT(&dirty_bitmap_mig_state.dbms_list);
+
+    register_savevm_live(NULL, "dirty-bitmap", 0, 1,
+                         &savevm_dirty_bitmap_handlers,
+                         &dirty_bitmap_mig_state);
+}
diff --git a/migration/migration.c b/migration/migration.c
index a3bb518..e633a95 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -375,6 +375,9 @@ static void process_incoming_migration_co(void *opaque)
     int ret;
 
     mis = migration_incoming_state_new(f);
+
+    init_dirty_bitmap_incoming_migration();
+
     postcopy_state_set(POSTCOPY_INCOMING_NONE);
     migrate_set_state(&mis->state, MIGRATION_STATUS_NONE,
                       MIGRATION_STATUS_ACTIVE);
diff --git a/migration/savevm.c b/migration/savevm.c
index 4fbed75..ee7588f 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1543,6 +1543,8 @@ static void loadvm_postcopy_handle_run_bh(void *opaque)
 
     trace_loadvm_postcopy_handle_run_vmstart();
 
+    dirty_bitmap_mig_before_vm_start();
+
     if (autostart) {
         /* Hold onto your hats, starting the CPU */
         vm_start();
diff --git a/vl.c b/vl.c
index e7c2c62..6791510 100644
--- a/vl.c
+++ b/vl.c
@@ -4384,6 +4384,7 @@ int main(int argc, char **argv, char **envp)
 
     blk_mig_init();
     ram_mig_init();
+    dirty_bitmap_mig_init();
 
     /* If the currently selected machine wishes to override the units-per-bus
      * property of its default HBA interface type, do so now. */
-- 
1.8.3.1

[Qemu-devel] [PATCH 13/18] iotests: maintain several vms in test

[Vladimir Sementsov-Ogievskiy]

The only problem with it is the same qmp socket name (which is
vm._monitor_path) for all vms. And because of this second vm couldn't be
lauched (vm.launch() fails because of socket is already in use).
This patch adds a number of vm into vm._monitor_path

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 tests/qemu-iotests/iotests.py | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index dbe0ee5..fade81e 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -130,11 +130,14 @@ def log(msg, filters=[]):
 
 class VM(qtest.QEMUQtestMachine):
     '''A QEMU VM'''
+    nb_vms = 0
 
     def __init__(self):
-        super(VM, self).__init__(qemu_prog, qemu_opts, test_dir=test_dir,
+        super(VM, self).__init__(qemu_prog, qemu_opts,
+                                 name = 'vm-%d' % VM.nb_vms, test_dir=test_dir,
                                  socket_scm_helper=socket_scm_helper)
         self._num_drives = 0
+        VM.nb_vms += 1
 
     def add_drive_raw(self, opts):
         self._args.append('-drive')
-- 
1.8.3.1

[Qemu-devel] [PATCH 14/18] iotests: add add_incoming_migration to VM class

[Vladimir Sementsov-Ogievskiy]

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 tests/qemu-iotests/iotests.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index fade81e..37c0f63 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -162,6 +162,12 @@ class VM(qtest.QEMUQtestMachine):
         self._num_drives += 1
         return self
 
+    def add_incoming_migration(self, desc):
+        '''Add an incoming migration to the VM'''
+        self._args.append('-incoming')
+        self._args.append(desc)
+        return self
+
     def pause_drive(self, drive, event=None):
         '''Pause drive r/w operations'''
         if not event:
-- 
1.8.3.1

[Qemu-devel] [PATCH 15/18] qapi: add md5 checksum of last dirty bitmap level to query-block

[Vladimir Sementsov-Ogievskiy]

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 block/dirty-bitmap.c   | 1 +
 include/qemu/hbitmap.h | 8 ++++++++
 qapi/block-core.json   | 5 ++++-
 util/hbitmap.c         | 8 ++++++++
 4 files changed, 21 insertions(+), 1 deletion(-)

diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
index 32aa6eb..f895fce 100644
--- a/block/dirty-bitmap.c
+++ b/block/dirty-bitmap.c
@@ -365,6 +365,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
         info->has_name = !!bm->name;
         info->name = g_strdup(bm->name);
         info->status = bdrv_dirty_bitmap_status(bm);
+        info->md5 = hbitmap_md5(bm->bitmap);
         entry->value = info;
         *plist = entry;
         plist = &entry->next;
diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index eb46475..8d4f836 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -225,6 +225,14 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count,
 void hbitmap_deserialize_finish(HBitmap *hb);
 
 /**
+ * hbitmap_md5:
+ * @bitmap: HBitmap to operate on.
+ *
+ * Returns md5 checksum of the last level.
+ */
+char *hbitmap_md5(const HBitmap *bitmap);
+
+/**
  * hbitmap_free:
  * @hb: HBitmap to operate on.
  *
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 2bbc027..5642a8a 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -419,11 +419,14 @@
 #
 # @status: current status of the dirty bitmap (since 2.4)
 #
+# @md5: md5 checksum (as a hexadecimal string) of the last bitmap level
+#       (since 2.6)
+#
 # Since: 1.3
 ##
 { 'struct': 'BlockDirtyInfo',
   'data': {'*name': 'str', 'count': 'int', 'granularity': 'uint32',
-           'status': 'DirtyBitmapStatus'} }
+           'status': 'DirtyBitmapStatus', 'md5': 'str'} }
 
 ##
 # @BlockInfo:
diff --git a/util/hbitmap.c b/util/hbitmap.c
index 6a13c12..4afb3d5 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -669,3 +669,11 @@ void hbitmap_free_meta(HBitmap *hb)
     hbitmap_free(hb->meta);
     hb->meta = NULL;
 }
+
+char *hbitmap_md5(const HBitmap *bitmap)
+{
+    uint64_t size =
+        MAX((bitmap->size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
+    const guchar *data = (const guchar *)bitmap->levels[HBITMAP_LEVELS - 1];
+    return g_compute_checksum_for_data(G_CHECKSUM_MD5, data, size);
+}
-- 
1.8.3.1

[Qemu-devel] [PATCH 16/18] iotests: add default node-name

[Vladimir Sementsov-Ogievskiy]

When testing migration, auto-generated by qemu node-names differs in
source and destination qemu and migration fails. After this patch,
auto-generated by iotest nodenames will be the same.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 tests/qemu-iotests/iotests.py | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 37c0f63..0332589 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -147,7 +147,8 @@ class VM(qtest.QEMUQtestMachine):
     def add_drive(self, path, opts='', interface='virtio'):
         '''Add a virtio-blk drive to the VM'''
         options = ['if=%s' % interface,
-                   'id=drive%d' % self._num_drives]
+                   'id=drive%d' % self._num_drives,
+                   'node-name=drivenode%d' % self._num_drives]
 
         if path is not None:
             options.append('file=%s' % path)
-- 
1.8.3.1

[Qemu-devel] [PATCH 17/18] iotests: add dirty bitmap migration test 169

[Vladimir Sementsov-Ogievskiy]

The test starts two vms (vm_a, vm_b), create dirty bitmap in
the first one, do several writes to corresponding device and
then migrate vm_a to vm_b with dirty bitmaps.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 tests/qemu-iotests/169     | 84 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/169.out |  5 +++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 90 insertions(+)
 create mode 100755 tests/qemu-iotests/169
 create mode 100644 tests/qemu-iotests/169.out

diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
new file mode 100755
index 0000000..d40be1d
--- /dev/null
+++ b/tests/qemu-iotests/169
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+#
+# Tests for dirty bitmaps migration.
+#
+# (C) Vladimir Sementsov-Ogievskiy 2015
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import iotests
+import time
+from iotests import qemu_img
+
+disk_a = os.path.join(iotests.test_dir, 'disk_a')
+disk_b = os.path.join(iotests.test_dir, 'disk_b')
+fifo = os.path.join(iotests.test_dir, 'mig_fifo')
+
+class TestDirtyBitmapMigration(iotests.QMPTestCase):
+
+    def setUp(self):
+        size = 0x400000000 # 1G
+        os.mkfifo(fifo)
+        qemu_img('create', '-f', iotests.imgfmt, disk_a, str(size))
+        qemu_img('create', '-f', iotests.imgfmt, disk_b, str(size))
+        self.vm_a = iotests.VM().add_drive(disk_a)
+        self.vm_b = iotests.VM().add_drive(disk_b)
+        self.vm_b.add_incoming_migration("exec: cat " + fifo)
+        self.vm_a.launch()
+        self.vm_b.launch()
+
+    def tearDown(self):
+        self.vm_a.shutdown()
+        self.vm_b.shutdown()
+        os.remove(disk_a)
+        os.remove(disk_b)
+        os.remove(fifo)
+
+    def test_migration(self):
+        granularity = 512
+        regions = [
+            { 'start': 0,          'count': 0x100000 },
+            { 'start': 0x100000000, 'count': 0x200000  },
+            { 'start': 0x399900000, 'count': 0x100000  }
+            ]
+
+        result = self.vm_a.qmp('block-dirty-bitmap-add', node='drive0',
+                               name='bitmap', granularity=granularity)
+        self.assert_qmp(result, 'return', {});
+
+        for r in regions:
+          self.vm_a.hmp_qemu_io('drive0',
+                                'write %d %d' % (r['start'], r['count']))
+
+        result = self.vm_a.qmp('query-block');
+        md5 = result['return'][0]['dirty-bitmaps'][0]['md5']
+
+        result = self.vm_a.qmp('migrate-set-capabilities',
+                               capabilities=[{'capability': 'dirty-bitmaps',
+                                              'state': True}])
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm_a.qmp('migrate', uri='exec:cat>' + fifo)
+        self.assertNotEqual(self.vm_a.event_wait("STOP"), None)
+        self.assertNotEqual(self.vm_b.event_wait("RESUME"), None)
+        time.sleep(2)
+
+        result = self.vm_b.qmp('query-block');
+        self.assert_qmp(result, 'return[0]/dirty-bitmaps[0]/md5', md5);
+
+
+if __name__ == '__main__':
+    iotests.main()
diff --git a/tests/qemu-iotests/169.out b/tests/qemu-iotests/169.out
new file mode 100644
index 0000000..ae1213e
--- /dev/null
+++ b/tests/qemu-iotests/169.out
@@ -0,0 +1,5 @@
+.
+----------------------------------------------------------------------
+Ran 1 tests
+
+OK
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 3a3973e..a939718 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -157,3 +157,4 @@
 155 rw auto
 156 rw auto quick
 157 auto
+169 rw auto quick
-- 
1.8.3.1

[Qemu-devel] [PATCH 18/18] iotests: add dirty bitmap postcopy test

[Vladimir Sementsov-Ogievskiy]

Test
- start two vms (vm_a, vm_b)

- in a
    - do writes from set A
    - do writes from set B
    - fix bitmap md5
    - clear bitmap
    - do writes from set A
    - start migration
- than, in b
    - wait vm start (postcopy should start)
    - do writes from set B
    - check btimap md5

The test should verify postcopy migration and than merging with delta
(changes in target, during postcopy process).

Reduce supported cache modes to only 'none', because with cache on time
from source.STOP to target.RESUME is unpredictable and we can fail with
timout while waiting for target.RESUME.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 tests/qemu-iotests/169        | 70 ++++++++++++++++++++++++++++++++++++-------
 tests/qemu-iotests/169.out    |  4 +--
 tests/qemu-iotests/group      |  1 +
 tests/qemu-iotests/iotests.py |  7 ++++-
 4 files changed, 69 insertions(+), 13 deletions(-)

diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
index d40be1d..31fe4b2 100755
--- a/tests/qemu-iotests/169
+++ b/tests/qemu-iotests/169
@@ -29,8 +29,14 @@ fifo = os.path.join(iotests.test_dir, 'mig_fifo')
 
 class TestDirtyBitmapMigration(iotests.QMPTestCase):
 
-    def setUp(self):
-        size = 0x400000000 # 1G
+    def tearDown(self):
+        self.vm_a.shutdown()
+        self.vm_b.shutdown()
+        os.remove(disk_a)
+        os.remove(disk_b)
+        os.remove(fifo)
+
+    def init(self, size):
         os.mkfifo(fifo)
         qemu_img('create', '-f', iotests.imgfmt, disk_a, str(size))
         qemu_img('create', '-f', iotests.imgfmt, disk_b, str(size))
@@ -40,14 +46,8 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
         self.vm_a.launch()
         self.vm_b.launch()
 
-    def tearDown(self):
-        self.vm_a.shutdown()
-        self.vm_b.shutdown()
-        os.remove(disk_a)
-        os.remove(disk_b)
-        os.remove(fifo)
-
     def test_migration(self):
+        self.init(0x400000000) # 1G
         granularity = 512
         regions = [
             { 'start': 0,          'count': 0x100000 },
@@ -79,6 +79,56 @@ class TestDirtyBitmapMigration(iotests.QMPTestCase):
         result = self.vm_b.qmp('query-block');
         self.assert_qmp(result, 'return[0]/dirty-bitmaps[0]/md5', md5);
 
+    def test_postcopy(self):
+        self.init(0x4000000000) # 128G
+        write_size = 0x40000000
+        granularity = 512
+        chunk = 4096
+
+        result = self.vm_a.qmp('block-dirty-bitmap-add', node='drive0',
+                               name='bitmap', granularity=granularity)
+        self.assert_qmp(result, 'return', {});
+
+        s = 0
+        while s < write_size:
+            self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
+            s += 0x10000
+        s = 0x8000
+        while s < write_size:
+            self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
+            s += 0x10000
+
+        result = self.vm_a.qmp('query-block');
+        md5 = result['return'][0]['dirty-bitmaps'][0]['md5']
+
+        result = self.vm_a.qmp('block-dirty-bitmap-clear', node='drive0',
+                               name='bitmap')
+        self.assert_qmp(result, 'return', {});
+        s = 0
+        while s < write_size:
+            self.vm_a.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
+            s += 0x10000
+
+        result = self.vm_a.qmp('migrate-set-capabilities',
+                               capabilities=[{'capability': 'dirty-bitmaps',
+                                              'state': True}])
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm_a.qmp('migrate', uri='exec:cat>' + fifo)
+        self.assertNotEqual(self.vm_a.event_wait("STOP"), None)
+        self.assertNotEqual(self.vm_b.event_wait("RESUME"), None)
+
+        s = 0x8000
+        while s < write_size:
+            self.vm_b.hmp_qemu_io('drive0', 'write %d %d' % (s, chunk))
+            s += 0x10000
+
+        result = self.vm_b.qmp('query-block');
+        while len(result['return'][0]['dirty-bitmaps']) > 1:
+            time.sleep(2)
+            result = self.vm_b.qmp('query-block');
+
+        self.assert_qmp(result, 'return[0]/dirty-bitmaps[0]/md5', md5);
 
 if __name__ == '__main__':
-    iotests.main()
+    iotests.main(supported_fmts=['qcow2'], supported_cache_modes=['none'])
diff --git a/tests/qemu-iotests/169.out b/tests/qemu-iotests/169.out
index ae1213e..fbc63e6 100644
--- a/tests/qemu-iotests/169.out
+++ b/tests/qemu-iotests/169.out
@@ -1,5 +1,5 @@
-.
+..
 ----------------------------------------------------------------------
-Ran 1 tests
+Ran 2 tests
 
 OK
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index a939718..3355ce4 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -158,3 +158,4 @@
 156 rw auto quick
 157 auto
 169 rw auto quick
+170 rw auto
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 0332589..5fa0603 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -320,12 +320,16 @@ def verify_platform(supported_oses=['linux']):
     if True not in [sys.platform.startswith(x) for x in supported_oses]:
         notrun('not suitable for this OS: %s' % sys.platform)
 
+def verify_cache_mode(supported_cache_modes=[]):
+    if supported_cache_modes and (cachemode not in supported_cache_modes):
+        notrun('not suitable for this cache mode: %s' % cachemode)
+
 def verify_quorum():
     '''Skip test suite if quorum support is not available'''
     if 'quorum' not in qemu_img_pipe('--help'):
         notrun('quorum support missing')
 
-def main(supported_fmts=[], supported_oses=['linux']):
+def main(supported_fmts=[], supported_oses=['linux'], supported_cache_modes=[]):
     '''Run tests'''
 
     # We are using TEST_DIR and QEMU_DEFAULT_MACHINE as proxies to
@@ -340,6 +344,7 @@ def main(supported_fmts=[], supported_oses=['linux']):
     verbosity = 1
     verify_image_format(supported_fmts)
     verify_platform(supported_oses)
+    verify_cache_mode(supported_cache_modes)
 
     # We need to filter out the time taken from the output so that qemu-iotest
     # can reliably diff the results against master output.
-- 
1.8.3.1

Re: [Qemu-devel] [PATCH 15/18] qapi: add md5 checksum of last dirty bitmap level to query-block

[Daniel P. Berrange]

On Tue, Aug 16, 2016 at 01:26:12PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> Reviewed-by: John Snow <jsnow@redhat.com>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>  block/dirty-bitmap.c   | 1 +
>  include/qemu/hbitmap.h | 8 ++++++++
>  qapi/block-core.json   | 5 ++++-
>  util/hbitmap.c         | 8 ++++++++
>  4 files changed, 21 insertions(+), 1 deletion(-)
> 
> diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
> index 32aa6eb..f895fce 100644
> --- a/block/dirty-bitmap.c
> +++ b/block/dirty-bitmap.c
> @@ -365,6 +365,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
>          info->has_name = !!bm->name;
>          info->name = g_strdup(bm->name);
>          info->status = bdrv_dirty_bitmap_status(bm);
> +        info->md5 = hbitmap_md5(bm->bitmap);
>          entry->value = info;
>          *plist = entry;
>          plist = &entry->next;
> diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
> index eb46475..8d4f836 100644
> --- a/include/qemu/hbitmap.h
> +++ b/include/qemu/hbitmap.h
> @@ -225,6 +225,14 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count,
>  void hbitmap_deserialize_finish(HBitmap *hb);
>  
>  /**
> + * hbitmap_md5:
> + * @bitmap: HBitmap to operate on.
> + *
> + * Returns md5 checksum of the last level.
> + */
> +char *hbitmap_md5(const HBitmap *bitmap);
> +
> +/**
>   * hbitmap_free:
>   * @hb: HBitmap to operate on.
>   *
> diff --git a/qapi/block-core.json b/qapi/block-core.json
> index 2bbc027..5642a8a 100644
> --- a/qapi/block-core.json
> +++ b/qapi/block-core.json
> @@ -419,11 +419,14 @@
>  #
>  # @status: current status of the dirty bitmap (since 2.4)
>  #
> +# @md5: md5 checksum (as a hexadecimal string) of the last bitmap level
> +#       (since 2.6)
> +#
>  # Since: 1.3
>  ##
>  { 'struct': 'BlockDirtyInfo',
>    'data': {'*name': 'str', 'count': 'int', 'granularity': 'uint32',
> -           'status': 'DirtyBitmapStatus'} }
> +           'status': 'DirtyBitmapStatus', 'md5': 'str'} }
>  
>  ##
>  # @BlockInfo:
> diff --git a/util/hbitmap.c b/util/hbitmap.c
> index 6a13c12..4afb3d5 100644
> --- a/util/hbitmap.c
> +++ b/util/hbitmap.c
> @@ -669,3 +669,11 @@ void hbitmap_free_meta(HBitmap *hb)
>      hbitmap_free(hb->meta);
>      hb->meta = NULL;
>  }
> +
> +char *hbitmap_md5(const HBitmap *bitmap)
> +{
> +    uint64_t size =
> +        MAX((bitmap->size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
> +    const guchar *data = (const guchar *)bitmap->levels[HBITMAP_LEVELS - 1];
> +    return g_compute_checksum_for_data(G_CHECKSUM_MD5, data, size);
> +}

Please use one of the qcrypto_hash_* functions, not GChecksum directly,
as this ensures we use the hash impls that have been through crypto
certification when needed by people shipping QEM.


Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 15/18] qapi: add md5 checksum of last dirty bitmap level to query-block

[Daniel P. Berrange]

On Tue, Aug 16, 2016 at 11:37:28AM +0100, Daniel P. Berrange wrote:
> On Tue, Aug 16, 2016 at 01:26:12PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> > Reviewed-by: John Snow <jsnow@redhat.com>
> > Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> > ---
> >  block/dirty-bitmap.c   | 1 +
> >  include/qemu/hbitmap.h | 8 ++++++++
> >  qapi/block-core.json   | 5 ++++-
> >  util/hbitmap.c         | 8 ++++++++
> >  4 files changed, 21 insertions(+), 1 deletion(-)
> > 
> > diff --git a/block/dirty-bitmap.c b/block/dirty-bitmap.c
> > index 32aa6eb..f895fce 100644
> > --- a/block/dirty-bitmap.c
> > +++ b/block/dirty-bitmap.c
> > @@ -365,6 +365,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
> >          info->has_name = !!bm->name;
> >          info->name = g_strdup(bm->name);
> >          info->status = bdrv_dirty_bitmap_status(bm);
> > +        info->md5 = hbitmap_md5(bm->bitmap);
> >          entry->value = info;
> >          *plist = entry;
> >          plist = &entry->next;
> > diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
> > index eb46475..8d4f836 100644
> > --- a/include/qemu/hbitmap.h
> > +++ b/include/qemu/hbitmap.h
> > @@ -225,6 +225,14 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count,
> >  void hbitmap_deserialize_finish(HBitmap *hb);
> >  
> >  /**
> > + * hbitmap_md5:
> > + * @bitmap: HBitmap to operate on.
> > + *
> > + * Returns md5 checksum of the last level.
> > + */
> > +char *hbitmap_md5(const HBitmap *bitmap);
> > +
> > +/**
> >   * hbitmap_free:
> >   * @hb: HBitmap to operate on.
> >   *
> > diff --git a/qapi/block-core.json b/qapi/block-core.json
> > index 2bbc027..5642a8a 100644
> > --- a/qapi/block-core.json
> > +++ b/qapi/block-core.json
> > @@ -419,11 +419,14 @@
> >  #
> >  # @status: current status of the dirty bitmap (since 2.4)
> >  #
> > +# @md5: md5 checksum (as a hexadecimal string) of the last bitmap level
> > +#       (since 2.6)
> > +#
> >  # Since: 1.3
> >  ##
> >  { 'struct': 'BlockDirtyInfo',
> >    'data': {'*name': 'str', 'count': 'int', 'granularity': 'uint32',
> > -           'status': 'DirtyBitmapStatus'} }
> > +           'status': 'DirtyBitmapStatus', 'md5': 'str'} }
> >  
> >  ##
> >  # @BlockInfo:
> > diff --git a/util/hbitmap.c b/util/hbitmap.c
> > index 6a13c12..4afb3d5 100644
> > --- a/util/hbitmap.c
> > +++ b/util/hbitmap.c
> > @@ -669,3 +669,11 @@ void hbitmap_free_meta(HBitmap *hb)
> >      hbitmap_free(hb->meta);
> >      hb->meta = NULL;
> >  }
> > +
> > +char *hbitmap_md5(const HBitmap *bitmap)
> > +{
> > +    uint64_t size =
> > +        MAX((bitmap->size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
> > +    const guchar *data = (const guchar *)bitmap->levels[HBITMAP_LEVELS - 1];
> > +    return g_compute_checksum_for_data(G_CHECKSUM_MD5, data, size);
> > +}
> 
> Please use one of the qcrypto_hash_* functions, not GChecksum directly,
> as this ensures we use the hash impls that have been through crypto
> certification when needed by people shipping QEM.

Oh and why MD5 here. Can we use a modern strong algorithm SHA256 instead
of MD5 which is known to be susceptible to collisions

Regards,
Daniel
-- 
|: http://berrange.com      -o-    http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org              -o-             http://virt-manager.org :|
|: http://autobuild.org       -o-         http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org       -o-       http://live.gnome.org/gtk-vnc :|

Re: [Qemu-devel] [PATCH 07/18] qapi: add dirty-bitmaps migration capability

[Fam Zheng]

On Tue, 08/16 13:26, Vladimir Sementsov-Ogievskiy wrote:
>  bool migrate_use_events(void)
>  {
>      MigrationState *s;
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 5658723..6386fbc 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -553,11 +553,13 @@
>  #          been migrated, pulling the remaining pages along as needed. NOTE: If
>  #          the migration fails during postcopy the VM will fail.  (since 2.6)
>  #
> +# @dirty-bitmaps: If enabled, QEMU will migrate named dirty bitmaps. (since 2.6)

This need to be 2.8.

Fam

Re: [Qemu-devel] [PATCH 10/18] migration/qemu-file: add qemu_put_counted_string()

[Fam Zheng]

On Tue, 08/16 13:26, Vladimir Sementsov-Ogievskiy wrote:
> Add function opposite to qemu_get_counted_string.
> qemu_put_counted_string puts one-byte length of the string (string
> should not be longer than 255 characters), and then it puts the string,
> without last zero byte.
> 
> Reviewed-by: John Snow <jsnow@redhat.com>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>  include/migration/qemu-file.h |  2 ++
>  migration/qemu-file.c         | 13 +++++++++++++
>  2 files changed, 15 insertions(+)
> 
> diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
> index abedd46..d860c92 100644
> --- a/include/migration/qemu-file.h
> +++ b/include/migration/qemu-file.h
> @@ -309,4 +309,6 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv)
>  
>  size_t qemu_get_counted_string(QEMUFile *f, char buf[256]);
>  
> +void qemu_put_counted_string(QEMUFile *f, const char *name);
> +
>  #endif
> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
> index bbc565e..6fcdd68 100644
> --- a/migration/qemu-file.c
> +++ b/migration/qemu-file.c
> @@ -690,6 +690,19 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])
>  }
>  
>  /*
> + * Put a string with one preceding byte containing its length. The length of
> + * the string should be less than 256.
> + */
> +void qemu_put_counted_string(QEMUFile *f, const char *name)

"Name" is a poor name. Perhaps call it "buf" like qemu_get_counted_string?

Fam

> +{
> +    size_t len = strlen(name);
> +
> +    assert(len < 256);
> +    qemu_put_byte(f, len);
> +    qemu_put_buffer(f, (const uint8_t *)name, len);
> +}
> +
> +/*
>   * Set the blocking state of the QEMUFile.
>   * Note: On some transports the OS only keeps a single blocking state for
>   *       both directions, and thus changing the blocking on the main
> -- 
> 1.8.3.1
> 
> 

Re: [Qemu-devel] [PATCH 16/18] iotests: add default node-name

[Fam Zheng]

On Tue, 08/16 13:26, Vladimir Sementsov-Ogievskiy wrote:
> When testing migration, auto-generated by qemu node-names differs in
> source and destination qemu and migration fails. After this patch,
> auto-generated by iotest nodenames will be the same.

Does this mean if a real user doesn't specify node-name, migration will fail
too?

Fam

> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>  tests/qemu-iotests/iotests.py | 3 ++-
>  1 file changed, 2 insertions(+), 1 deletion(-)
> 
> diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
> index 37c0f63..0332589 100644
> --- a/tests/qemu-iotests/iotests.py
> +++ b/tests/qemu-iotests/iotests.py
> @@ -147,7 +147,8 @@ class VM(qtest.QEMUQtestMachine):
>      def add_drive(self, path, opts='', interface='virtio'):
>          '''Add a virtio-blk drive to the VM'''
>          options = ['if=%s' % interface,
> -                   'id=drive%d' % self._num_drives]
> +                   'id=drive%d' % self._num_drives,
> +                   'node-name=drivenode%d' % self._num_drives]
>  
>          if path is not None:
>              options.append('file=%s' % path)
> -- 
> 1.8.3.1
> 
> 

Re: [Qemu-devel] [PATCH 17/18] iotests: add dirty bitmap migration test 169

[Fam Zheng]

On Tue, 08/16 13:26, Vladimir Sementsov-Ogievskiy wrote:
> The test starts two vms (vm_a, vm_b), create dirty bitmap in
> the first one, do several writes to corresponding device and
> then migrate vm_a to vm_b with dirty bitmaps.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>  tests/qemu-iotests/169     | 84 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/169.out |  5 +++
>  tests/qemu-iotests/group   |  1 +
>  3 files changed, 90 insertions(+)
>  create mode 100755 tests/qemu-iotests/169
>  create mode 100644 tests/qemu-iotests/169.out
> 
> diff --git a/tests/qemu-iotests/169 b/tests/qemu-iotests/169
> new file mode 100755
> index 0000000..d40be1d
> --- /dev/null
> +++ b/tests/qemu-iotests/169
> @@ -0,0 +1,84 @@
> +#!/usr/bin/env python
> +#
> +# Tests for dirty bitmaps migration.
> +#
> +# (C) Vladimir Sementsov-Ogievskiy 2015

2016?

> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import os
> +import iotests
> +import time
> +from iotests import qemu_img
> +
> +disk_a = os.path.join(iotests.test_dir, 'disk_a')
> +disk_b = os.path.join(iotests.test_dir, 'disk_b')
> +fifo = os.path.join(iotests.test_dir, 'mig_fifo')
> +
> +class TestDirtyBitmapMigration(iotests.QMPTestCase):
> +
> +    def setUp(self):
> +        size = 0x400000000 # 1G
> +        os.mkfifo(fifo)
> +        qemu_img('create', '-f', iotests.imgfmt, disk_a, str(size))
> +        qemu_img('create', '-f', iotests.imgfmt, disk_b, str(size))
> +        self.vm_a = iotests.VM().add_drive(disk_a)
> +        self.vm_b = iotests.VM().add_drive(disk_b)
> +        self.vm_b.add_incoming_migration("exec: cat " + fifo)
> +        self.vm_a.launch()
> +        self.vm_b.launch()
> +
> +    def tearDown(self):
> +        self.vm_a.shutdown()
> +        self.vm_b.shutdown()
> +        os.remove(disk_a)
> +        os.remove(disk_b)
> +        os.remove(fifo)
> +
> +    def test_migration(self):
> +        granularity = 512
> +        regions = [
> +            { 'start': 0,          'count': 0x100000 },
                                      ^
                                      Misaligned by one column?

> +            { 'start': 0x100000000, 'count': 0x200000  },
> +            { 'start': 0x399900000, 'count': 0x100000  }
> +            ]
> +
> +        result = self.vm_a.qmp('block-dirty-bitmap-add', node='drive0',
> +                               name='bitmap', granularity=granularity)
> +        self.assert_qmp(result, 'return', {});
> +
> +        for r in regions:
> +          self.vm_a.hmp_qemu_io('drive0',
> +                                'write %d %d' % (r['start'], r['count']))
> +
> +        result = self.vm_a.qmp('query-block');
> +        md5 = result['return'][0]['dirty-bitmaps'][0]['md5']
> +
> +        result = self.vm_a.qmp('migrate-set-capabilities',
> +                               capabilities=[{'capability': 'dirty-bitmaps',
> +                                              'state': True}])
> +        self.assert_qmp(result, 'return', {})
> +
> +        result = self.vm_a.qmp('migrate', uri='exec:cat>' + fifo)
> +        self.assertNotEqual(self.vm_a.event_wait("STOP"), None)
> +        self.assertNotEqual(self.vm_b.event_wait("RESUME"), None)
> +        time.sleep(2)
> +
> +        result = self.vm_b.qmp('query-block');
> +        self.assert_qmp(result, 'return[0]/dirty-bitmaps[0]/md5', md5);
> +
> +
> +if __name__ == '__main__':
> +    iotests.main()
> diff --git a/tests/qemu-iotests/169.out b/tests/qemu-iotests/169.out
> new file mode 100644
> index 0000000..ae1213e
> --- /dev/null
> +++ b/tests/qemu-iotests/169.out
> @@ -0,0 +1,5 @@
> +.
> +----------------------------------------------------------------------
> +Ran 1 tests
> +
> +OK
> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> index 3a3973e..a939718 100644
> --- a/tests/qemu-iotests/group
> +++ b/tests/qemu-iotests/group
> @@ -157,3 +157,4 @@
>  155 rw auto
>  156 rw auto quick
>  157 auto
> +169 rw auto quick
> -- 
> 1.8.3.1
> 
> 

Fam

Re: [Qemu-devel] [PATCH 00/18] Dirty bitmaps postcopy migration

[Fam Zheng]

On Tue, 08/16 13:25, Vladimir Sementsov-Ogievskiy wrote:

Patches 5-8, 13, 14, 16, 17: Reviewed-by: Fam Zheng <famz@redhat.com>

I'll take a closer look at patch 12 and 18 tomorrow.

Fam

Re: [Qemu-devel] [PATCH 02/18] migration: fix ram_save_pending

[Dr. David Alan Gilbert]

* Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> Fill postcopy-able pending only if ram postcopy is enabled.
> It is necessary because of there will be other postcopy-able states and
> when ram postcopy is disabled, it should not spoil common postcopy
> related pending.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>

> ---
>  migration/ram.c | 8 ++++++--
>  1 file changed, 6 insertions(+), 2 deletions(-)
> 
> diff --git a/migration/ram.c b/migration/ram.c
> index fa5c72b..424ce0b 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -2079,8 +2079,12 @@ static void ram_save_pending(QEMUFile *f, void *opaque, uint64_t max_size,
>          remaining_size = ram_save_remaining() * TARGET_PAGE_SIZE;
>      }
>  
> -    /* We can do postcopy, and all the data is postcopiable */
> -    *postcopiable_pending += remaining_size;
> +    if (migrate_postcopy_ram()) {
> +        /* We can do postcopy, and all the data is postcopiable */
> +        *postcopiable_pending += remaining_size;
> +    } else {
> +        *non_postcopiable_pending += remaining_size;
> +    }
>  }
>  
>  static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
> -- 
> 1.8.3.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH 00/18] Dirty bitmaps postcopy migration

[Vladimir Sementsov-Ogievskiy]

On 17.08.2016 14:47, Fam Zheng wrote:
> On Tue, 08/16 13:25, Vladimir Sementsov-Ogievskiy wrote:
>
> Patches 5-8, 13, 14, 16, 17: Reviewed-by: Fam Zheng <famz@redhat.com>
>
> I'll take a closer look at patch 12 and 18 tomorrow.
>
> Fam
Thank you!

-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH 00/18] Dirty bitmaps postcopy migration

[Dr. David Alan Gilbert]

* Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
> v2:
> some bugs fixed, iotests a bit changed and merged into one test.
> based on block-next (https://github.com/XanClic/qemu/commits/block-next)
> clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2
> 
> v1:
> 
> These series are derived from my 'Dirty bitmaps migration' series. The
> core idea is switch to postcopy migration and drop usage of meta
> bitmaps.
> 
> These patches provide dirty bitmap postcopy migration feature. Only
> named dirty bitmaps are to be migrated. Migration may be enabled using
> migration capabilities.
> 
> The overall method (thanks to John Snow):
> 
> 1. migrate bitmaps meta data in .save_live_setup
>    - create/find related bitmaps on target
>    - disable them
>    - create successors (anonimous children) only for enabled migrated
>      bitmaps
> 2. do nothing in precopy stage
> 3. just before target vm start: enable successors, created in (1)
> 4. migrate bitmap data
> 5. reclaime bitmaps (merge successors to their parents)
> 6. enable bitmaps (only bitmaps, which was enabled in source)

I'd like to just understand a little bit more about the structure;
    a)I don't see anything where the destination asks the source for data;
      am I right or just missing that?
    b)If I'm right about (a) is the only reason it's called 'postcopy'
      because it's done when the destination is already running?
    c)How does it play together with ram postcopy; do they get in each
      others way?

Dave

> 
> Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
> (DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
> and I'll drop them in the following version.
> 
> So, relatively to last DBMv7: 
> 
> 01-04: new patches, splitting common postcopy migration out of ram
>        postcopy migration
>    05: equal to DBMv7.05
>    06: new
>    07: equal to DBMv7.06
>    08: new
>    09: equal to DBMv7.07
>    10: new
>    11: derived from DBMv7.08, see below
> 12-15: equal to DBMv7.09-12
>    16: derived from DBMv7.13
>        - switch from fifo to socket, as postcopy don't work with fifo
>          for now
>        - change parameters: size, granularity, regions
>        - add time.sleep, to wait for postcopy migration phase (bad
>          temporary solution.
>        - drop Reviewed-by
>    17: new
> 
>    11: the core patch of the series, it is derived from
>        [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
>        There are a lot of changes related to switching from precopy to
>        postcopy, but functions related to migration stream itself
>        (structs, send/load sequences) are mostly unchnaged.
> 
>        So, changes, to switch from precopy to postcopy:
>        - removed all staff related to meta bitmaps and dirty phase!!!
>        - add dirty_bitmap_mig_enable_successors, and call it before
>          target vm start in loadvm_postcopy_handle_run
>        - add enabled_bitmaps list of bitmaps for
>          dirty_bitmap_mig_enable_successors
> 
>        - enabled flag is send with start bitmap chunk instead of
>          completion chunk
>        - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
>          using meta bitmap granularity
> 
>        - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
>          to postcopy stage
>        - dirty_bitmap_save_pending: remove dirty phase related pending,
>          switch pending to non-postcopyable
>        - dirty_bitmap_load_start: get enabled flag and prepare
>          successors for enabled bitmaps, also add them to
>          enabled_bitmaps list
>        - dirty_bitmap_load_complete: for enabled bitmaps: merge them
>          with successors and enable
> 
>        - savevm handlers:
>          * remove separate savevm_dirty_bitmap_live_iterate_handlers state
>            (it was bad idea, any way), and move its save_live_iterate to
>            savevm_dirty_bitmap_handlers
>          * add is_active_iterate savevm handler, which allows iterations
>            only in postcopy stage (after stopping source vm)
>          * add has_postcopy savevm handler. (ofcourse, just returning true)
>          * use save_live_complete_postcopy instead of
>            save_live_complete_precopy
> 
>        Other changes:
>        - some debug output changed
>        - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
>          was needed to omit iterations if bitmap data is small, possibly
>          this should be reimplemented)
> 
> Vladimir Sementsov-Ogievskiy (18):
>   migration: add has_postcopy savevm handler
>   migration: fix ram_save_pending
>   migration: split common postcopy out of ram postcopy
>   migration: introduce postcopy-only pending
>   block: add bdrv_next_dirty_bitmap()
>   block: add bdrv_dirty_bitmap_enable_successor()
>   qapi: add dirty-bitmaps migration capability
>   block/dirty-bitmap: add bdrv_dirty_bitmap_release_successor
>   migration: include migrate_dirty_bitmaps in migrate_postcopy
>   migration/qemu-file: add qemu_put_counted_string()
>   migration: add is_active_iterate handler
>   migration: add postcopy migration of dirty bitmaps
>   iotests: maintain several vms in test
>   iotests: add add_incoming_migration to VM class
>   qapi: add md5 checksum of last dirty bitmap level to query-block
>   iotests: add default node-name
>   iotests: add dirty bitmap migration test 117
>   iotests: add dirty bitmap postcopy test
> 
>  block/dirty-bitmap.c           |  26 ++
>  include/block/dirty-bitmap.h   |   6 +
>  include/migration/block.h      |   1 +
>  include/migration/migration.h  |   6 +
>  include/migration/qemu-file.h  |   2 +
>  include/migration/vmstate.h    |   7 +-
>  include/qemu/hbitmap.h         |   8 +
>  include/sysemu/sysemu.h        |   5 +-
>  migration/Makefile.objs        |   2 +-
>  migration/block-dirty-bitmap.c | 699 +++++++++++++++++++++++++++++++++++++++++
>  migration/block.c              |   7 +-
>  migration/migration.c          |  66 ++--
>  migration/postcopy-ram.c       |   4 +-
>  migration/qemu-file.c          |  13 +
>  migration/ram.c                |  19 +-
>  migration/savevm.c             |  56 +++-
>  migration/trace-events         |   2 +-
>  qapi-schema.json               |   4 +-
>  qapi/block-core.json           |   5 +-
>  tests/qemu-iotests/169         | 134 ++++++++
>  tests/qemu-iotests/169.out     |   5 +
>  tests/qemu-iotests/group       |   2 +
>  tests/qemu-iotests/iotests.py  |  21 +-
>  util/hbitmap.c                 |   8 +
>  vl.c                           |   1 +
>  25 files changed, 1055 insertions(+), 54 deletions(-)
>  create mode 100644 migration/block-dirty-bitmap.c
>  create mode 100755 tests/qemu-iotests/169
>  create mode 100644 tests/qemu-iotests/169.out
> 
> -- 
> 1.8.3.1
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: [Qemu-devel] [PATCH 00/18] Dirty bitmaps postcopy migration

[Vladimir Sementsov-Ogievskiy]

On 17.08.2016 15:35, Dr. David Alan Gilbert wrote:
> * Vladimir Sementsov-Ogievskiy (vsementsov@virtuozzo.com) wrote:
>> v2:
>> some bugs fixed, iotests a bit changed and merged into one test.
>> based on block-next (https://github.com/XanClic/qemu/commits/block-next)
>> clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
>> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2
>>
>> v1:
>>
>> These series are derived from my 'Dirty bitmaps migration' series. The
>> core idea is switch to postcopy migration and drop usage of meta
>> bitmaps.
>>
>> These patches provide dirty bitmap postcopy migration feature. Only
>> named dirty bitmaps are to be migrated. Migration may be enabled using
>> migration capabilities.
>>
>> The overall method (thanks to John Snow):
>>
>> 1. migrate bitmaps meta data in .save_live_setup
>>     - create/find related bitmaps on target
>>     - disable them
>>     - create successors (anonimous children) only for enabled migrated
>>       bitmaps
>> 2. do nothing in precopy stage
>> 3. just before target vm start: enable successors, created in (1)
>> 4. migrate bitmap data
>> 5. reclaime bitmaps (merge successors to their parents)
>> 6. enable bitmaps (only bitmaps, which was enabled in source)
> I'd like to just understand a little bit more about the structure;
>      a)I don't see anything where the destination asks the source for data;
>        am I right or just missing that?

You are right

>      b)If I'm right about (a) is the only reason it's called 'postcopy'
>        because it's done when the destination is already running?

Yes

>      c)How does it play together with ram postcopy; do they get in each
>        others way?

They should work together, but I have no tests for this case yet.

>
> Dave
>
>> Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
>> (DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
>> and I'll drop them in the following version.
>>
>> So, relatively to last DBMv7:
>>
>> 01-04: new patches, splitting common postcopy migration out of ram
>>         postcopy migration
>>     05: equal to DBMv7.05
>>     06: new
>>     07: equal to DBMv7.06
>>     08: new
>>     09: equal to DBMv7.07
>>     10: new
>>     11: derived from DBMv7.08, see below
>> 12-15: equal to DBMv7.09-12
>>     16: derived from DBMv7.13
>>         - switch from fifo to socket, as postcopy don't work with fifo
>>           for now
>>         - change parameters: size, granularity, regions
>>         - add time.sleep, to wait for postcopy migration phase (bad
>>           temporary solution.
>>         - drop Reviewed-by
>>     17: new
>>
>>     11: the core patch of the series, it is derived from
>>         [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
>>         There are a lot of changes related to switching from precopy to
>>         postcopy, but functions related to migration stream itself
>>         (structs, send/load sequences) are mostly unchnaged.
>>
>>         So, changes, to switch from precopy to postcopy:
>>         - removed all staff related to meta bitmaps and dirty phase!!!
>>         - add dirty_bitmap_mig_enable_successors, and call it before
>>           target vm start in loadvm_postcopy_handle_run
>>         - add enabled_bitmaps list of bitmaps for
>>           dirty_bitmap_mig_enable_successors
>>
>>         - enabled flag is send with start bitmap chunk instead of
>>           completion chunk
>>         - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
>>           using meta bitmap granularity
>>
>>         - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
>>           to postcopy stage
>>         - dirty_bitmap_save_pending: remove dirty phase related pending,
>>           switch pending to non-postcopyable
>>         - dirty_bitmap_load_start: get enabled flag and prepare
>>           successors for enabled bitmaps, also add them to
>>           enabled_bitmaps list
>>         - dirty_bitmap_load_complete: for enabled bitmaps: merge them
>>           with successors and enable
>>
>>         - savevm handlers:
>>           * remove separate savevm_dirty_bitmap_live_iterate_handlers state
>>             (it was bad idea, any way), and move its save_live_iterate to
>>             savevm_dirty_bitmap_handlers
>>           * add is_active_iterate savevm handler, which allows iterations
>>             only in postcopy stage (after stopping source vm)
>>           * add has_postcopy savevm handler. (ofcourse, just returning true)
>>           * use save_live_complete_postcopy instead of
>>             save_live_complete_precopy
>>
>>         Other changes:
>>         - some debug output changed
>>         - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
>>           was needed to omit iterations if bitmap data is small, possibly
>>           this should be reimplemented)
>>
>> Vladimir Sementsov-Ogievskiy (18):
>>    migration: add has_postcopy savevm handler
>>    migration: fix ram_save_pending
>>    migration: split common postcopy out of ram postcopy
>>    migration: introduce postcopy-only pending
>>    block: add bdrv_next_dirty_bitmap()
>>    block: add bdrv_dirty_bitmap_enable_successor()
>>    qapi: add dirty-bitmaps migration capability
>>    block/dirty-bitmap: add bdrv_dirty_bitmap_release_successor
>>    migration: include migrate_dirty_bitmaps in migrate_postcopy
>>    migration/qemu-file: add qemu_put_counted_string()
>>    migration: add is_active_iterate handler
>>    migration: add postcopy migration of dirty bitmaps
>>    iotests: maintain several vms in test
>>    iotests: add add_incoming_migration to VM class
>>    qapi: add md5 checksum of last dirty bitmap level to query-block
>>    iotests: add default node-name
>>    iotests: add dirty bitmap migration test 117
>>    iotests: add dirty bitmap postcopy test
>>
>>   block/dirty-bitmap.c           |  26 ++
>>   include/block/dirty-bitmap.h   |   6 +
>>   include/migration/block.h      |   1 +
>>   include/migration/migration.h  |   6 +
>>   include/migration/qemu-file.h  |   2 +
>>   include/migration/vmstate.h    |   7 +-
>>   include/qemu/hbitmap.h         |   8 +
>>   include/sysemu/sysemu.h        |   5 +-
>>   migration/Makefile.objs        |   2 +-
>>   migration/block-dirty-bitmap.c | 699 +++++++++++++++++++++++++++++++++++++++++
>>   migration/block.c              |   7 +-
>>   migration/migration.c          |  66 ++--
>>   migration/postcopy-ram.c       |   4 +-
>>   migration/qemu-file.c          |  13 +
>>   migration/ram.c                |  19 +-
>>   migration/savevm.c             |  56 +++-
>>   migration/trace-events         |   2 +-
>>   qapi-schema.json               |   4 +-
>>   qapi/block-core.json           |   5 +-
>>   tests/qemu-iotests/169         | 134 ++++++++
>>   tests/qemu-iotests/169.out     |   5 +
>>   tests/qemu-iotests/group       |   2 +
>>   tests/qemu-iotests/iotests.py  |  21 +-
>>   util/hbitmap.c                 |   8 +
>>   vl.c                           |   1 +
>>   25 files changed, 1055 insertions(+), 54 deletions(-)
>>   create mode 100644 migration/block-dirty-bitmap.c
>>   create mode 100755 tests/qemu-iotests/169
>>   create mode 100644 tests/qemu-iotests/169.out
>>
>> -- 
>> 1.8.3.1
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH 00/18] Dirty bitmaps postcopy migration

[Fam Zheng]

On Wed, 08/17 15:15, Vladimir Sementsov-Ogievskiy wrote:
> On 17.08.2016 14:47, Fam Zheng wrote:
> >On Tue, 08/16 13:25, Vladimir Sementsov-Ogievskiy wrote:
> >
> >Patches 5-8, 13, 14, 16, 17: Reviewed-by: Fam Zheng <famz@redhat.com>
> >
> >I'll take a closer look at patch 12 and 18 tomorrow.
> >
> >Fam
> Thank you!

My son was born and this will be heavily delayed, sorry!

Fam

Re: [Qemu-devel] [PATCH 00/18] Dirty bitmaps postcopy migration

[Vladimir Sementsov-Ogievskiy]

ping

For now there are some notes mostly about accessory patches. What about 
migration itself?

On 16.08.2016 13:25, Vladimir Sementsov-Ogievskiy wrote:
> v2:
> some bugs fixed, iotests a bit changed and merged into one test.
> based on block-next (https://github.com/XanClic/qemu/commits/block-next)
> clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2
>
> v1:
>
> These series are derived from my 'Dirty bitmaps migration' series. The
> core idea is switch to postcopy migration and drop usage of meta
> bitmaps.
>
> These patches provide dirty bitmap postcopy migration feature. Only
> named dirty bitmaps are to be migrated. Migration may be enabled using
> migration capabilities.
>
> The overall method (thanks to John Snow):
>
> 1. migrate bitmaps meta data in .save_live_setup
>     - create/find related bitmaps on target
>     - disable them
>     - create successors (anonimous children) only for enabled migrated
>       bitmaps
> 2. do nothing in precopy stage
> 3. just before target vm start: enable successors, created in (1)
> 4. migrate bitmap data
> 5. reclaime bitmaps (merge successors to their parents)
> 6. enable bitmaps (only bitmaps, which was enabled in source)
>
>
> Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
> (DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
> and I'll drop them in the following version.
>
> So, relatively to last DBMv7:
>
> 01-04: new patches, splitting common postcopy migration out of ram
>         postcopy migration
>     05: equal to DBMv7.05
>     06: new
>     07: equal to DBMv7.06
>     08: new
>     09: equal to DBMv7.07
>     10: new
>     11: derived from DBMv7.08, see below
> 12-15: equal to DBMv7.09-12
>     16: derived from DBMv7.13
>         - switch from fifo to socket, as postcopy don't work with fifo
>           for now
>         - change parameters: size, granularity, regions
>         - add time.sleep, to wait for postcopy migration phase (bad
>           temporary solution.
>         - drop Reviewed-by
>     17: new
>
>     11: the core patch of the series, it is derived from
>         [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
>         There are a lot of changes related to switching from precopy to
>         postcopy, but functions related to migration stream itself
>         (structs, send/load sequences) are mostly unchnaged.
>
>         So, changes, to switch from precopy to postcopy:
>         - removed all staff related to meta bitmaps and dirty phase!!!
>         - add dirty_bitmap_mig_enable_successors, and call it before
>           target vm start in loadvm_postcopy_handle_run
>         - add enabled_bitmaps list of bitmaps for
>           dirty_bitmap_mig_enable_successors
>
>         - enabled flag is send with start bitmap chunk instead of
>           completion chunk
>         - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
>           using meta bitmap granularity
>
>         - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
>           to postcopy stage
>         - dirty_bitmap_save_pending: remove dirty phase related pending,
>           switch pending to non-postcopyable
>         - dirty_bitmap_load_start: get enabled flag and prepare
>           successors for enabled bitmaps, also add them to
>           enabled_bitmaps list
>         - dirty_bitmap_load_complete: for enabled bitmaps: merge them
>           with successors and enable
>
>         - savevm handlers:
>           * remove separate savevm_dirty_bitmap_live_iterate_handlers state
>             (it was bad idea, any way), and move its save_live_iterate to
>             savevm_dirty_bitmap_handlers
>           * add is_active_iterate savevm handler, which allows iterations
>             only in postcopy stage (after stopping source vm)
>           * add has_postcopy savevm handler. (ofcourse, just returning true)
>           * use save_live_complete_postcopy instead of
>             save_live_complete_precopy
>
>         Other changes:
>         - some debug output changed
>         - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
>           was needed to omit iterations if bitmap data is small, possibly
>           this should be reimplemented)
>
> Vladimir Sementsov-Ogievskiy (18):
>    migration: add has_postcopy savevm handler
>    migration: fix ram_save_pending
>    migration: split common postcopy out of ram postcopy
>    migration: introduce postcopy-only pending
>    block: add bdrv_next_dirty_bitmap()
>    block: add bdrv_dirty_bitmap_enable_successor()
>    qapi: add dirty-bitmaps migration capability
>    block/dirty-bitmap: add bdrv_dirty_bitmap_release_successor
>    migration: include migrate_dirty_bitmaps in migrate_postcopy
>    migration/qemu-file: add qemu_put_counted_string()
>    migration: add is_active_iterate handler
>    migration: add postcopy migration of dirty bitmaps
>    iotests: maintain several vms in test
>    iotests: add add_incoming_migration to VM class
>    qapi: add md5 checksum of last dirty bitmap level to query-block
>    iotests: add default node-name
>    iotests: add dirty bitmap migration test 117
>    iotests: add dirty bitmap postcopy test
>
>   block/dirty-bitmap.c           |  26 ++
>   include/block/dirty-bitmap.h   |   6 +
>   include/migration/block.h      |   1 +
>   include/migration/migration.h  |   6 +
>   include/migration/qemu-file.h  |   2 +
>   include/migration/vmstate.h    |   7 +-
>   include/qemu/hbitmap.h         |   8 +
>   include/sysemu/sysemu.h        |   5 +-
>   migration/Makefile.objs        |   2 +-
>   migration/block-dirty-bitmap.c | 699 +++++++++++++++++++++++++++++++++++++++++
>   migration/block.c              |   7 +-
>   migration/migration.c          |  66 ++--
>   migration/postcopy-ram.c       |   4 +-
>   migration/qemu-file.c          |  13 +
>   migration/ram.c                |  19 +-
>   migration/savevm.c             |  56 +++-
>   migration/trace-events         |   2 +-
>   qapi-schema.json               |   4 +-
>   qapi/block-core.json           |   5 +-
>   tests/qemu-iotests/169         | 134 ++++++++
>   tests/qemu-iotests/169.out     |   5 +
>   tests/qemu-iotests/group       |   2 +
>   tests/qemu-iotests/iotests.py  |  21 +-
>   util/hbitmap.c                 |   8 +
>   vl.c                           |   1 +
>   25 files changed, 1055 insertions(+), 54 deletions(-)
>   create mode 100644 migration/block-dirty-bitmap.c
>   create mode 100755 tests/qemu-iotests/169
>   create mode 100644 tests/qemu-iotests/169.out
>


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH 00/18] Dirty bitmaps postcopy migration

[Vladimir Sementsov-Ogievskiy]

ping

For now there are some notes mostly about accessory patches. What about 
migration itself? Is it ok?

On 16.08.2016 13:25, Vladimir Sementsov-Ogievskiy wrote:
> v2:
> some bugs fixed, iotests a bit changed and merged into one test.
> based on block-next (https://github.com/XanClic/qemu/commits/block-next)
> clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2
>
> v1:
>
> These series are derived from my 'Dirty bitmaps migration' series. The
> core idea is switch to postcopy migration and drop usage of meta
> bitmaps.
>
> These patches provide dirty bitmap postcopy migration feature. Only
> named dirty bitmaps are to be migrated. Migration may be enabled using
> migration capabilities.
>
> The overall method (thanks to John Snow):
>
> 1. migrate bitmaps meta data in .save_live_setup
>     - create/find related bitmaps on target
>     - disable them
>     - create successors (anonimous children) only for enabled migrated
>       bitmaps
> 2. do nothing in precopy stage
> 3. just before target vm start: enable successors, created in (1)
> 4. migrate bitmap data
> 5. reclaime bitmaps (merge successors to their parents)
> 6. enable bitmaps (only bitmaps, which was enabled in source)
>
>
> Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
> (DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
> and I'll drop them in the following version.
>
> So, relatively to last DBMv7:
>
> 01-04: new patches, splitting common postcopy migration out of ram
>         postcopy migration
>     05: equal to DBMv7.05
>     06: new
>     07: equal to DBMv7.06
>     08: new
>     09: equal to DBMv7.07
>     10: new
>     11: derived from DBMv7.08, see below
> 12-15: equal to DBMv7.09-12
>     16: derived from DBMv7.13
>         - switch from fifo to socket, as postcopy don't work with fifo
>           for now
>         - change parameters: size, granularity, regions
>         - add time.sleep, to wait for postcopy migration phase (bad
>           temporary solution.
>         - drop Reviewed-by
>     17: new
>
>     11: the core patch of the series, it is derived from
>         [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
>         There are a lot of changes related to switching from precopy to
>         postcopy, but functions related to migration stream itself
>         (structs, send/load sequences) are mostly unchnaged.
>
>         So, changes, to switch from precopy to postcopy:
>         - removed all staff related to meta bitmaps and dirty phase!!!
>         - add dirty_bitmap_mig_enable_successors, and call it before
>           target vm start in loadvm_postcopy_handle_run
>         - add enabled_bitmaps list of bitmaps for
>           dirty_bitmap_mig_enable_successors
>
>         - enabled flag is send with start bitmap chunk instead of
>           completion chunk
>         - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
>           using meta bitmap granularity
>
>         - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
>           to postcopy stage
>         - dirty_bitmap_save_pending: remove dirty phase related pending,
>           switch pending to non-postcopyable
>         - dirty_bitmap_load_start: get enabled flag and prepare
>           successors for enabled bitmaps, also add them to
>           enabled_bitmaps list
>         - dirty_bitmap_load_complete: for enabled bitmaps: merge them
>           with successors and enable
>
>         - savevm handlers:
>           * remove separate savevm_dirty_bitmap_live_iterate_handlers state
>             (it was bad idea, any way), and move its save_live_iterate to
>             savevm_dirty_bitmap_handlers
>           * add is_active_iterate savevm handler, which allows iterations
>             only in postcopy stage (after stopping source vm)
>           * add has_postcopy savevm handler. (ofcourse, just returning true)
>           * use save_live_complete_postcopy instead of
>             save_live_complete_precopy
>
>         Other changes:
>         - some debug output changed
>         - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
>           was needed to omit iterations if bitmap data is small, possibly
>           this should be reimplemented)
>
> Vladimir Sementsov-Ogievskiy (18):
>    migration: add has_postcopy savevm handler
>    migration: fix ram_save_pending
>    migration: split common postcopy out of ram postcopy
>    migration: introduce postcopy-only pending
>    block: add bdrv_next_dirty_bitmap()
>    block: add bdrv_dirty_bitmap_enable_successor()
>    qapi: add dirty-bitmaps migration capability
>    block/dirty-bitmap: add bdrv_dirty_bitmap_release_successor
>    migration: include migrate_dirty_bitmaps in migrate_postcopy
>    migration/qemu-file: add qemu_put_counted_string()
>    migration: add is_active_iterate handler
>    migration: add postcopy migration of dirty bitmaps
>    iotests: maintain several vms in test
>    iotests: add add_incoming_migration to VM class
>    qapi: add md5 checksum of last dirty bitmap level to query-block
>    iotests: add default node-name
>    iotests: add dirty bitmap migration test 117
>    iotests: add dirty bitmap postcopy test
>
>   block/dirty-bitmap.c           |  26 ++
>   include/block/dirty-bitmap.h   |   6 +
>   include/migration/block.h      |   1 +
>   include/migration/migration.h  |   6 +
>   include/migration/qemu-file.h  |   2 +
>   include/migration/vmstate.h    |   7 +-
>   include/qemu/hbitmap.h         |   8 +
>   include/sysemu/sysemu.h        |   5 +-
>   migration/Makefile.objs        |   2 +-
>   migration/block-dirty-bitmap.c | 699 +++++++++++++++++++++++++++++++++++++++++
>   migration/block.c              |   7 +-
>   migration/migration.c          |  66 ++--
>   migration/postcopy-ram.c       |   4 +-
>   migration/qemu-file.c          |  13 +
>   migration/ram.c                |  19 +-
>   migration/savevm.c             |  56 +++-
>   migration/trace-events         |   2 +-
>   qapi-schema.json               |   4 +-
>   qapi/block-core.json           |   5 +-
>   tests/qemu-iotests/169         | 134 ++++++++
>   tests/qemu-iotests/169.out     |   5 +
>   tests/qemu-iotests/group       |   2 +
>   tests/qemu-iotests/iotests.py  |  21 +-
>   util/hbitmap.c                 |   8 +
>   vl.c                           |   1 +
>   25 files changed, 1055 insertions(+), 54 deletions(-)
>   create mode 100644 migration/block-dirty-bitmap.c
>   create mode 100755 tests/qemu-iotests/169
>   create mode 100644 tests/qemu-iotests/169.out
>


-- 
Best regards,
Vladimir

[Qemu-devel] ping Re: [PATCH 00/18] Dirty bitmaps postcopy migration

[Vladimir Sementsov-Ogievskiy]

ping

For now there are some notes mostly about accessory patches. What about 
migration itself? Is it ok? Has it a chance of being merged one day?

16.08.2016 13:25, Vladimir Sementsov-Ogievskiy wrote:
> v2:
> some bugs fixed, iotests a bit changed and merged into one test.
> based on block-next (https://github.com/XanClic/qemu/commits/block-next)
> clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
> online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2
>
> v1:
>
> These series are derived from my 'Dirty bitmaps migration' series. The
> core idea is switch to postcopy migration and drop usage of meta
> bitmaps.
>
> These patches provide dirty bitmap postcopy migration feature. Only
> named dirty bitmaps are to be migrated. Migration may be enabled using
> migration capabilities.
>
> The overall method (thanks to John Snow):
>
> 1. migrate bitmaps meta data in .save_live_setup
>     - create/find related bitmaps on target
>     - disable them
>     - create successors (anonimous children) only for enabled migrated
>       bitmaps
> 2. do nothing in precopy stage
> 3. just before target vm start: enable successors, created in (1)
> 4. migrate bitmap data
> 5. reclaime bitmaps (merge successors to their parents)
> 6. enable bitmaps (only bitmaps, which was enabled in source)
>
>
> Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
> (DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
> and I'll drop them in the following version.
>
> So, relatively to last DBMv7:
>
> 01-04: new patches, splitting common postcopy migration out of ram
>         postcopy migration
>     05: equal to DBMv7.05
>     06: new
>     07: equal to DBMv7.06
>     08: new
>     09: equal to DBMv7.07
>     10: new
>     11: derived from DBMv7.08, see below
> 12-15: equal to DBMv7.09-12
>     16: derived from DBMv7.13
>         - switch from fifo to socket, as postcopy don't work with fifo
>           for now
>         - change parameters: size, granularity, regions
>         - add time.sleep, to wait for postcopy migration phase (bad
>           temporary solution.
>         - drop Reviewed-by
>     17: new
>
>     11: the core patch of the series, it is derived from
>         [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
>         There are a lot of changes related to switching from precopy to
>         postcopy, but functions related to migration stream itself
>         (structs, send/load sequences) are mostly unchnaged.
>
>         So, changes, to switch from precopy to postcopy:
>         - removed all staff related to meta bitmaps and dirty phase!!!
>         - add dirty_bitmap_mig_enable_successors, and call it before
>           target vm start in loadvm_postcopy_handle_run
>         - add enabled_bitmaps list of bitmaps for
>           dirty_bitmap_mig_enable_successors
>
>         - enabled flag is send with start bitmap chunk instead of
>           completion chunk
>         - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
>           using meta bitmap granularity
>
>         - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
>           to postcopy stage
>         - dirty_bitmap_save_pending: remove dirty phase related pending,
>           switch pending to non-postcopyable
>         - dirty_bitmap_load_start: get enabled flag and prepare
>           successors for enabled bitmaps, also add them to
>           enabled_bitmaps list
>         - dirty_bitmap_load_complete: for enabled bitmaps: merge them
>           with successors and enable
>
>         - savevm handlers:
>           * remove separate savevm_dirty_bitmap_live_iterate_handlers state
>             (it was bad idea, any way), and move its save_live_iterate to
>             savevm_dirty_bitmap_handlers
>           * add is_active_iterate savevm handler, which allows iterations
>             only in postcopy stage (after stopping source vm)
>           * add has_postcopy savevm handler. (ofcourse, just returning true)
>           * use save_live_complete_postcopy instead of
>             save_live_complete_precopy
>
>         Other changes:
>         - some debug output changed
>         - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
>           was needed to omit iterations if bitmap data is small, possibly
>           this should be reimplemented)
>
> Vladimir Sementsov-Ogievskiy (18):
>    migration: add has_postcopy savevm handler
>    migration: fix ram_save_pending
>    migration: split common postcopy out of ram postcopy
>    migration: introduce postcopy-only pending
>    block: add bdrv_next_dirty_bitmap()
>    block: add bdrv_dirty_bitmap_enable_successor()
>    qapi: add dirty-bitmaps migration capability
>    block/dirty-bitmap: add bdrv_dirty_bitmap_release_successor
>    migration: include migrate_dirty_bitmaps in migrate_postcopy
>    migration/qemu-file: add qemu_put_counted_string()
>    migration: add is_active_iterate handler
>    migration: add postcopy migration of dirty bitmaps
>    iotests: maintain several vms in test
>    iotests: add add_incoming_migration to VM class
>    qapi: add md5 checksum of last dirty bitmap level to query-block
>    iotests: add default node-name
>    iotests: add dirty bitmap migration test 117
>    iotests: add dirty bitmap postcopy test
>
>   block/dirty-bitmap.c           |  26 ++
>   include/block/dirty-bitmap.h   |   6 +
>   include/migration/block.h      |   1 +
>   include/migration/migration.h  |   6 +
>   include/migration/qemu-file.h  |   2 +
>   include/migration/vmstate.h    |   7 +-
>   include/qemu/hbitmap.h         |   8 +
>   include/sysemu/sysemu.h        |   5 +-
>   migration/Makefile.objs        |   2 +-
>   migration/block-dirty-bitmap.c | 699 +++++++++++++++++++++++++++++++++++++++++
>   migration/block.c              |   7 +-
>   migration/migration.c          |  66 ++--
>   migration/postcopy-ram.c       |   4 +-
>   migration/qemu-file.c          |  13 +
>   migration/ram.c                |  19 +-
>   migration/savevm.c             |  56 +++-
>   migration/trace-events         |   2 +-
>   qapi-schema.json               |   4 +-
>   qapi/block-core.json           |   5 +-
>   tests/qemu-iotests/169         | 134 ++++++++
>   tests/qemu-iotests/169.out     |   5 +
>   tests/qemu-iotests/group       |   2 +
>   tests/qemu-iotests/iotests.py  |  21 +-
>   util/hbitmap.c                 |   8 +
>   vl.c                           |   1 +
>   25 files changed, 1055 insertions(+), 54 deletions(-)
>   create mode 100644 migration/block-dirty-bitmap.c
>   create mode 100755 tests/qemu-iotests/169
>   create mode 100644 tests/qemu-iotests/169.out
>


-- 
Best regards,
Vladimir

Re: [Qemu-devel] ping Re: [PATCH 00/18] Dirty bitmaps postcopy migration

[Stefan Hajnoczi]

[-- Attachment #1: Type: text/plain, Size: 7410 bytes --]

On Tue, Oct 25, 2016 at 04:04:35PM +0300, Vladimir Sementsov-Ogievskiy wrote:
> ping
> 
> For now there are some notes mostly about accessory patches. What about
> migration itself? Is it ok? Has it a chance of being merged one day?

This series mostly touches migration/ code.  Juan or DaveG?

> 16.08.2016 13:25, Vladimir Sementsov-Ogievskiy wrote:
> > v2:
> > some bugs fixed, iotests a bit changed and merged into one test.
> > based on block-next (https://github.com/XanClic/qemu/commits/block-next)
> > clone: tag postcopy-v2 from https://src.openvz.org/scm/~vsementsov/qemu.git
> > online: https://src.openvz.org/users/vsementsov/repos/qemu/browse?at=refs%2Ftags%2Fpostcopy-v2
> > 
> > v1:
> > 
> > These series are derived from my 'Dirty bitmaps migration' series. The
> > core idea is switch to postcopy migration and drop usage of meta
> > bitmaps.
> > 
> > These patches provide dirty bitmap postcopy migration feature. Only
> > named dirty bitmaps are to be migrated. Migration may be enabled using
> > migration capabilities.
> > 
> > The overall method (thanks to John Snow):
> > 
> > 1. migrate bitmaps meta data in .save_live_setup
> >     - create/find related bitmaps on target
> >     - disable them
> >     - create successors (anonimous children) only for enabled migrated
> >       bitmaps
> > 2. do nothing in precopy stage
> > 3. just before target vm start: enable successors, created in (1)
> > 4. migrate bitmap data
> > 5. reclaime bitmaps (merge successors to their parents)
> > 6. enable bitmaps (only bitmaps, which was enabled in source)
> > 
> > 
> > Some patches are unchnaged from (v7) of 'Dirty bitmaps migration'
> > (DBMv7). I've left Reviewed-by's for them, if you don't like it, say me
> > and I'll drop them in the following version.
> > 
> > So, relatively to last DBMv7:
> > 
> > 01-04: new patches, splitting common postcopy migration out of ram
> >         postcopy migration
> >     05: equal to DBMv7.05
> >     06: new
> >     07: equal to DBMv7.06
> >     08: new
> >     09: equal to DBMv7.07
> >     10: new
> >     11: derived from DBMv7.08, see below
> > 12-15: equal to DBMv7.09-12
> >     16: derived from DBMv7.13
> >         - switch from fifo to socket, as postcopy don't work with fifo
> >           for now
> >         - change parameters: size, granularity, regions
> >         - add time.sleep, to wait for postcopy migration phase (bad
> >           temporary solution.
> >         - drop Reviewed-by
> >     17: new
> > 
> >     11: the core patch of the series, it is derived from
> >         [DBMv7.08: migration: add migration_block-dirty-bitmap.c]
> >         There are a lot of changes related to switching from precopy to
> >         postcopy, but functions related to migration stream itself
> >         (structs, send/load sequences) are mostly unchnaged.
> > 
> >         So, changes, to switch from precopy to postcopy:
> >         - removed all staff related to meta bitmaps and dirty phase!!!
> >         - add dirty_bitmap_mig_enable_successors, and call it before
> >           target vm start in loadvm_postcopy_handle_run
> >         - add enabled_bitmaps list of bitmaps for
> >           dirty_bitmap_mig_enable_successors
> > 
> >         - enabled flag is send with start bitmap chunk instead of
> >           completion chunk
> >         - sectors_per_chunk is calculated directly from CHUNK_SIZE, not
> >           using meta bitmap granularity
> > 
> >         - dirty_bitmap_save_iterate: remove dirty_phase, move bulk_phase
> >           to postcopy stage
> >         - dirty_bitmap_save_pending: remove dirty phase related pending,
> >           switch pending to non-postcopyable
> >         - dirty_bitmap_load_start: get enabled flag and prepare
> >           successors for enabled bitmaps, also add them to
> >           enabled_bitmaps list
> >         - dirty_bitmap_load_complete: for enabled bitmaps: merge them
> >           with successors and enable
> > 
> >         - savevm handlers:
> >           * remove separate savevm_dirty_bitmap_live_iterate_handlers state
> >             (it was bad idea, any way), and move its save_live_iterate to
> >             savevm_dirty_bitmap_handlers
> >           * add is_active_iterate savevm handler, which allows iterations
> >             only in postcopy stage (after stopping source vm)
> >           * add has_postcopy savevm handler. (ofcourse, just returning true)
> >           * use save_live_complete_postcopy instead of
> >             save_live_complete_precopy
> > 
> >         Other changes:
> >         - some debug output changed
> >         - remove MIN_LIVE_SIZE, is_live_iterative and related staff (it
> >           was needed to omit iterations if bitmap data is small, possibly
> >           this should be reimplemented)
> > 
> > Vladimir Sementsov-Ogievskiy (18):
> >    migration: add has_postcopy savevm handler
> >    migration: fix ram_save_pending
> >    migration: split common postcopy out of ram postcopy
> >    migration: introduce postcopy-only pending
> >    block: add bdrv_next_dirty_bitmap()
> >    block: add bdrv_dirty_bitmap_enable_successor()
> >    qapi: add dirty-bitmaps migration capability
> >    block/dirty-bitmap: add bdrv_dirty_bitmap_release_successor
> >    migration: include migrate_dirty_bitmaps in migrate_postcopy
> >    migration/qemu-file: add qemu_put_counted_string()
> >    migration: add is_active_iterate handler
> >    migration: add postcopy migration of dirty bitmaps
> >    iotests: maintain several vms in test
> >    iotests: add add_incoming_migration to VM class
> >    qapi: add md5 checksum of last dirty bitmap level to query-block
> >    iotests: add default node-name
> >    iotests: add dirty bitmap migration test 117
> >    iotests: add dirty bitmap postcopy test
> > 
> >   block/dirty-bitmap.c           |  26 ++
> >   include/block/dirty-bitmap.h   |   6 +
> >   include/migration/block.h      |   1 +
> >   include/migration/migration.h  |   6 +
> >   include/migration/qemu-file.h  |   2 +
> >   include/migration/vmstate.h    |   7 +-
> >   include/qemu/hbitmap.h         |   8 +
> >   include/sysemu/sysemu.h        |   5 +-
> >   migration/Makefile.objs        |   2 +-
> >   migration/block-dirty-bitmap.c | 699 +++++++++++++++++++++++++++++++++++++++++
> >   migration/block.c              |   7 +-
> >   migration/migration.c          |  66 ++--
> >   migration/postcopy-ram.c       |   4 +-
> >   migration/qemu-file.c          |  13 +
> >   migration/ram.c                |  19 +-
> >   migration/savevm.c             |  56 +++-
> >   migration/trace-events         |   2 +-
> >   qapi-schema.json               |   4 +-
> >   qapi/block-core.json           |   5 +-
> >   tests/qemu-iotests/169         | 134 ++++++++
> >   tests/qemu-iotests/169.out     |   5 +
> >   tests/qemu-iotests/group       |   2 +
> >   tests/qemu-iotests/iotests.py  |  21 +-
> >   util/hbitmap.c                 |   8 +
> >   vl.c                           |   1 +
> >   25 files changed, 1055 insertions(+), 54 deletions(-)
> >   create mode 100644 migration/block-dirty-bitmap.c
> >   create mode 100755 tests/qemu-iotests/169
> >   create mode 100644 tests/qemu-iotests/169.out
> > 
> 
> 
> -- 
> Best regards,
> Vladimir
> 

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 455 bytes --]

Re: [Qemu-devel] ping Re: [PATCH 00/18] Dirty bitmaps postcopy migration

[Denis V. Lunev]

On 11/02/2016 02:13 PM, Stefan Hajnoczi wrote:
> On Tue, Oct 25, 2016 at 04:04:35PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>> ping
>>
>> For now there are some notes mostly about accessory patches. What about
>> migration itself? Is it ok? Has it a chance of being merged one day?
> This series mostly touches migration/ code.  Juan or DaveG?
yep. this is the problem :(

or they can just accept ;)

Re: [Qemu-devel] [PATCH 01/18] migration: add has_postcopy savevm handler

[Juan Quintela]

Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> wrote:
> Now postcopy-able states are recognized by not NULL
> save_live_complete_postcopy handler. But when we have several different
> postcopy-able states, it is not convenient. Ram postcopy may be
> disabled, while some other postcopy enabled, in this case Ram state
> should behave as it is not postcopy-able.
>
> This patch add separate has_postcopy handler to specify behaviour of
> savevm state.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

Re: [Qemu-devel] [PATCH 02/18] migration: fix ram_save_pending

[Juan Quintela]

Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> wrote:
> Fill postcopy-able pending only if ram postcopy is enabled.
> It is necessary because of there will be other postcopy-able states and
> when ram postcopy is disabled, it should not spoil common postcopy
> related pending.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

Re: [Qemu-devel] [PATCH 04/18] migration: introduce postcopy-only pending

[Juan Quintela]

Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> wrote:
> There would be savevm states (dirty-bitmap) which can migrate only in
> postcopy stage. The corresponding pending is introduced here.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> ---
>  include/migration/vmstate.h |  5 +++--
>  include/sysemu/sysemu.h     |  5 +++--
>  migration/block.c           |  7 ++++---
>  migration/migration.c       | 15 ++++++++-------
>  migration/ram.c             |  9 +++++----
>  migration/savevm.c          | 12 +++++++-----
>  migration/trace-events      |  2 +-
>  7 files changed, 31 insertions(+), 24 deletions(-)
>
> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
> index 5c30ef7..dc656be 100644
> --- a/include/migration/vmstate.h
> +++ b/include/migration/vmstate.h
> @@ -58,8 +58,9 @@ typedef struct SaveVMHandlers {
>      /* This runs outside the iothread lock!  */
>      int (*save_live_setup)(QEMUFile *f, void *opaque);
>      void (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size,
> -                              uint64_t *non_postcopiable_pending,
> -                              uint64_t *postcopiable_pending);
> +                              uint64_t *res_precopy_only,
> +                              uint64_t *res_compatible,
> +                              uint64_t *res_postcopy_only);

Ouch, we really need to move to use an struct here :p

But ugliness was already here.

BTW, with only this patch, it is not clear to me what res_compatible mean

> @@ -1133,7 +1135,7 @@ void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
>              }
>          }
>          se->ops->save_live_pending(f, se->opaque, max_size,
> -                                   res_non_postcopiable, res_postcopiable);
> +                res_precopy_only, res_compatible, res_postcopy_only);

Indentation is wrong.

Re: [Qemu-devel] [PATCH 03/18] migration: split common postcopy out of ram postcopy

[Juan Quintela]

Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> wrote:
> Split common postcopy staff from ram postcopy staff.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

> diff --git a/migration/savevm.c b/migration/savevm.c
> index d2efeeb..cce542f 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -73,7 +73,7 @@ static struct mig_cmd_args {
>      [MIG_CMD_INVALID]          = { .len = -1, .name = "INVALID" },
>      [MIG_CMD_OPEN_RETURN_PATH] = { .len =  0, .name = "OPEN_RETURN_PATH" },
>      [MIG_CMD_PING]             = { .len = sizeof(uint32_t), .name = "PING" },
> -    [MIG_CMD_POSTCOPY_ADVISE]  = { .len = 16, .name = "POSTCOPY_ADVISE" },
> +    [MIG_CMD_POSTCOPY_ADVISE]  = { .len = -1, .name = "POSTCOPY_ADVISE" },

Not sure if it is a good idea to change postcopy_advise or create a new
one.

Dave, can you comment?

Re: [Qemu-devel] [PATCH 12/18] migration: add postcopy migration of dirty bitmaps

[Juan Quintela]

Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> wrote:
> Postcopy migration of dirty bitmaps. Only named dirty bitmaps,
> associated with root nodes and non-root named nodes are migrated.
>
> If destination qemu is already containing a dirty bitmap with the same name
> as a migrated bitmap (for the same node), than, if their granularities are
> the same the migration will be done, otherwise the error will be generated.
>
> If destination qemu doesn't contain such bitmap it will be created.
>
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>

> diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
> new file mode 100644
> index 0000000..c668d02
> --- /dev/null
> +++ b/migration/block-dirty-bitmap.c
> @@ -0,0 +1,699 @@
> +/*
> + * QEMU dirty bitmap migration
> + *
> + * Postcopy migration of dirty bitmaps. Only named dirty bitmaps, associated
> + * with root nodes and non-root named nodes are migrated.
> + *
> + * If destination qemu is already containing a dirty bitmap with the same name
> + * as a migrated bitmap (for the same node), than, if their granularities are
> + * the same the migration will be done, otherwise the error will be generated.
> + *
> + * If destination qemu doesn't contain such bitmap it will be created.
> + *
> + * format of migration:
> + *
> + * # Header (shared for different chunk types)
> + * 1, 2 or 4 bytes: flags (see qemu_{put,put}_flags)
> + * [ 1 byte: node name size ] \  flags & DEVICE_NAME
> + * [ n bytes: node name     ] /
> + * [ 1 byte: bitmap name size ] \  flags & BITMAP_NAME
> + * [ n bytes: bitmap name     ] /
> + *
> + * # Start of bitmap migration (flags & START)
> + * header
> + * be64: granularity
> + * 1 byte: bitmap enabled flag
> + *
> + * # Complete of bitmap migration (flags & COMPLETE)
> + * header
> + *
> + * # Data chunk of bitmap migration
> + * header
> + * be64: start sector
> + * be32: number of sectors
> + * [ be64: buffer size  ] \ ! (flags & ZEROES)
> + * [ n bytes: buffer    ] /
> + *
> + * The last chunk in stream should contain flags & EOS. The chunk may skip
> + * device and/or bitmap names, assuming them to be the same with the previous
> + * chunk.
> + *
> + *
> + * This file is derived from migration/block.c
> + *
> + * Author:
> + * Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
> + *
> + * original copyright message:
> + * =====================================================================
> + * Copyright IBM, Corp. 2009
> + *
> + * Authors:
> + *  Liran Schour   <lirans@il.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2.  See
> + * the COPYING file in the top-level directory.
> + *
> + * Contributions after 2012-01-13 are licensed under the terms of the
> + * GNU GPL, version 2 or (at your option) any later version.
> + * =====================================================================
> + */

I think that the normal practice is putting first the copyright and then
the comment of the file.

> +static void qemu_put_bitmap_flags(QEMUFile *f, uint32_t flags)
> +{
> +    if (!(flags & 0xffffff00)) {
> +        qemu_put_byte(f, flags);
> +        return;
> +    }
> +
> +    if (!(flags & 0xffff0000)) {
> +        qemu_put_be16(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_16);
> +        return;
> +    }
> +
> +    qemu_put_be32(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_32);
> +}

Do need flags so many times to be a good idea to spend two flags and
make the code more complex?  Couldn't just sent always the 32bit word
and call it a day?

I have only looked at the stuff quickly from the migration point of
view, not about the bitmap stuff.

> +static void send_bitmap_complete(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
> +{
> +    send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_COMPLETE);
> +}
> +
> +static void send_bitmap_bits(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
> +                             uint64_t start_sector, uint32_t nr_sectors)
> +{
> +    /* align for buffer_is_zero() */
> +    uint64_t align = 4 * sizeof(long);
> +    uint64_t unaligned_size =
> +        bdrv_dirty_bitmap_serialization_size(dbms->bitmap,
> +                                             start_sector, nr_sectors);
> +    uint64_t buf_size = (unaligned_size + align - 1) & ~(align - 1);
> +    uint8_t *buf = g_malloc0(buf_size);
> +    uint32_t flags = DIRTY_BITMAP_MIG_FLAG_BITS;
> +
> +    bdrv_dirty_bitmap_serialize_part(dbms->bitmap, buf,
> +                                     start_sector, nr_sectors);
> +
> +    if (buffer_is_zero(buf, buf_size)) {
> +        g_free(buf);
> +        buf = NULL;
> +        flags |= DIRTY_BITMAP_MIG_FLAG_ZEROES;
> +    }
> +
> +    DPRINTF("parameters:"
> +            "\n   flags:        %x"
> +            "\n   start_sector: %" PRIu64
> +            "\n   nr_sectors:   %" PRIu32
> +            "\n   data_size:    %" PRIu64 "\n",
> +            flags, start_sector, nr_sectors, buf_size);

Now we are adding traces, not DPRINF's to new code in general.
Same for all DPRINTFs

> +
> +    send_bitmap_header(f, dbms, flags);
> +
> +    qemu_put_be64(f, start_sector);
> +    qemu_put_be32(f, nr_sectors);
> +
> +    /* if a block is zero we need to flush here since the network
> +     * bandwidth is now a lot higher than the storage device bandwidth.
> +     * thus if we queue zero blocks we slow down the migration.
> +     * also, skip writing block when migrate only dirty bitmaps. */
> +    if (flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
> +        qemu_fflush(f);

I thought that we were missing the g_free(buf) here, not sure if it is
better to put the return here, or do an else for the rest of the code.

Re: [Qemu-devel] ping Re: [PATCH 00/18] Dirty bitmaps postcopy migration

[Juan Quintela]

"Denis V. Lunev" <den@openvz.org> wrote:
> On 11/02/2016 02:13 PM, Stefan Hajnoczi wrote:
>> On Tue, Oct 25, 2016 at 04:04:35PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>> ping
>>>
>>> For now there are some notes mostly about accessory patches. What about
>>> migration itself? Is it ok? Has it a chance of being merged one day?
>> This series mostly touches migration/ code.  Juan or DaveG?
> yep. this is the problem :(
>
> or they can just accept ;)

I just commented on almost all the migration bits of the patches (and
noticed that fam/john snow commented on almost all the block ones).

Later, Juan.

Re: [Qemu-devel] ping Re: [PATCH 00/18] Dirty bitmaps postcopy migration

[Denis V. Lunev]

On 11/04/2016 04:12 PM, Juan Quintela wrote:
> "Denis V. Lunev" <den@openvz.org> wrote:
>> On 11/02/2016 02:13 PM, Stefan Hajnoczi wrote:
>>> On Tue, Oct 25, 2016 at 04:04:35PM +0300, Vladimir Sementsov-Ogievskiy wrote:
>>>> ping
>>>>
>>>> For now there are some notes mostly about accessory patches. What about
>>>> migration itself? Is it ok? Has it a chance of being merged one day?
>>> This series mostly touches migration/ code.  Juan or DaveG?
>> yep. this is the problem :(
>>
>> or they can just accept ;)
> I just commented on almost all the migration bits of the patches (and
> noticed that fam/john snow commented on almost all the block ones).
>
> Later, Juan.
great, then we will re-work this starting from Monday ;)

Thank you

Re: [Qemu-devel] [PATCH 10/18] migration/qemu-file: add qemu_put_counted_string()

[Vladimir Sementsov-Ogievskiy]

17.08.2016 12:09, Fam Zheng wrote:
> On Tue, 08/16 13:26, Vladimir Sementsov-Ogievskiy wrote:
>> Add function opposite to qemu_get_counted_string.
>> qemu_put_counted_string puts one-byte length of the string (string
>> should not be longer than 255 characters), and then it puts the string,
>> without last zero byte.
>>
>> Reviewed-by: John Snow <jsnow@redhat.com>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>>   include/migration/qemu-file.h |  2 ++
>>   migration/qemu-file.c         | 13 +++++++++++++
>>   2 files changed, 15 insertions(+)
>>
>> diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
>> index abedd46..d860c92 100644
>> --- a/include/migration/qemu-file.h
>> +++ b/include/migration/qemu-file.h
>> @@ -309,4 +309,6 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv)
>>   
>>   size_t qemu_get_counted_string(QEMUFile *f, char buf[256]);
>>   
>> +void qemu_put_counted_string(QEMUFile *f, const char *name);
>> +
>>   #endif
>> diff --git a/migration/qemu-file.c b/migration/qemu-file.c
>> index bbc565e..6fcdd68 100644
>> --- a/migration/qemu-file.c
>> +++ b/migration/qemu-file.c
>> @@ -690,6 +690,19 @@ size_t qemu_get_counted_string(QEMUFile *f, char buf[256])
>>   }
>>   
>>   /*
>> + * Put a string with one preceding byte containing its length. The length of
>> + * the string should be less than 256.
>> + */
>> +void qemu_put_counted_string(QEMUFile *f, const char *name)
> "Name" is a poor name. Perhaps call it "buf" like qemu_get_counted_string?

buf is poor too, as it say nothing about 0-ended string. Let it be 'str'.

>
> Fam
>
>> +{
>> +    size_t len = strlen(name);
>> +
>> +    assert(len < 256);
>> +    qemu_put_byte(f, len);
>> +    qemu_put_buffer(f, (const uint8_t *)name, len);
>> +}
>> +
>> +/*
>>    * Set the blocking state of the QEMUFile.
>>    * Note: On some transports the OS only keeps a single blocking state for
>>    *       both directions, and thus changing the blocking on the main
>> -- 
>> 1.8.3.1
>>
>>


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH 12/18] migration: add postcopy migration of dirty bitmaps

[Vladimir Sementsov-Ogievskiy]

04.11.2016 16:09, Juan Quintela wrote:
> Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> wrote:
>> Postcopy migration of dirty bitmaps. Only named dirty bitmaps,
>> associated with root nodes and non-root named nodes are migrated.
>>
>> If destination qemu is already containing a dirty bitmap with the same name
>> as a migrated bitmap (for the same node), than, if their granularities are
>> the same the migration will be done, otherwise the error will be generated.
>>
>> If destination qemu doesn't contain such bitmap it will be created.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
>> new file mode 100644
>> index 0000000..c668d02
>> --- /dev/null
>> +++ b/migration/block-dirty-bitmap.c
>> @@ -0,0 +1,699 @@
>> +/*
>> + * QEMU dirty bitmap migration
>> + *
>> + * Postcopy migration of dirty bitmaps. Only named dirty bitmaps, associated
>> + * with root nodes and non-root named nodes are migrated.
>> + *
>> + * If destination qemu is already containing a dirty bitmap with the same name
>> + * as a migrated bitmap (for the same node), than, if their granularities are
>> + * the same the migration will be done, otherwise the error will be generated.
>> + *
>> + * If destination qemu doesn't contain such bitmap it will be created.
>> + *
>> + * format of migration:
>> + *
>> + * # Header (shared for different chunk types)
>> + * 1, 2 or 4 bytes: flags (see qemu_{put,put}_flags)
>> + * [ 1 byte: node name size ] \  flags & DEVICE_NAME
>> + * [ n bytes: node name     ] /
>> + * [ 1 byte: bitmap name size ] \  flags & BITMAP_NAME
>> + * [ n bytes: bitmap name     ] /
>> + *
>> + * # Start of bitmap migration (flags & START)
>> + * header
>> + * be64: granularity
>> + * 1 byte: bitmap enabled flag
>> + *
>> + * # Complete of bitmap migration (flags & COMPLETE)
>> + * header
>> + *
>> + * # Data chunk of bitmap migration
>> + * header
>> + * be64: start sector
>> + * be32: number of sectors
>> + * [ be64: buffer size  ] \ ! (flags & ZEROES)
>> + * [ n bytes: buffer    ] /
>> + *
>> + * The last chunk in stream should contain flags & EOS. The chunk may skip
>> + * device and/or bitmap names, assuming them to be the same with the previous
>> + * chunk.
>> + *
>> + *
>> + * This file is derived from migration/block.c
>> + *
>> + * Author:
>> + * Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
>> + *
>> + * original copyright message:
>> + * =====================================================================
>> + * Copyright IBM, Corp. 2009
>> + *
>> + * Authors:
>> + *  Liran Schour   <lirans@il.ibm.com>
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2.  See
>> + * the COPYING file in the top-level directory.
>> + *
>> + * Contributions after 2012-01-13 are licensed under the terms of the
>> + * GNU GPL, version 2 or (at your option) any later version.
>> + * =====================================================================
>> + */
> I think that the normal practice is putting first the copyright and then
> the comment of the file.
>
>> +static void qemu_put_bitmap_flags(QEMUFile *f, uint32_t flags)
>> +{
>> +    if (!(flags & 0xffffff00)) {
>> +        qemu_put_byte(f, flags);
>> +        return;
>> +    }
>> +
>> +    if (!(flags & 0xffff0000)) {
>> +        qemu_put_be16(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_16);
>> +        return;
>> +    }
>> +
>> +    qemu_put_be32(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_32);
>> +}
> Do need flags so many times to be a good idea to spend two flags and
> make the code more complex?  Couldn't just sent always the 32bit word
> and call it a day?

I've started with one byte of flags (I don't need more) and I've 
implemented this after proposal by John Snow:
> I might recommend reserving the last bit of the second byte to be a 
> flag such as DIRTY_BITMAP_EXTRA_FLAGS that indicates the presence of 
> additional byte(s) of flags, to be determined later, if we ever need 
> them, but two bytes for now should be sufficient.

And I think it is not bad. Code is a bit more complex, yes, but why 
should we send 4 bytes with every chunk when (very likely) we will never 
use more than one?

>
> I have only looked at the stuff quickly from the migration point of
> view, not about the bitmap stuff.
>
>> +static void send_bitmap_complete(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
>> +{
>> +    send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_COMPLETE);
>> +}
>> +
>> +static void send_bitmap_bits(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
>> +                             uint64_t start_sector, uint32_t nr_sectors)
>> +{
>> +    /* align for buffer_is_zero() */
>> +    uint64_t align = 4 * sizeof(long);
>> +    uint64_t unaligned_size =
>> +        bdrv_dirty_bitmap_serialization_size(dbms->bitmap,
>> +                                             start_sector, nr_sectors);
>> +    uint64_t buf_size = (unaligned_size + align - 1) & ~(align - 1);
>> +    uint8_t *buf = g_malloc0(buf_size);
>> +    uint32_t flags = DIRTY_BITMAP_MIG_FLAG_BITS;
>> +
>> +    bdrv_dirty_bitmap_serialize_part(dbms->bitmap, buf,
>> +                                     start_sector, nr_sectors);
>> +
>> +    if (buffer_is_zero(buf, buf_size)) {
>> +        g_free(buf);
>> +        buf = NULL;
>> +        flags |= DIRTY_BITMAP_MIG_FLAG_ZEROES;
>> +    }
>> +
>> +    DPRINTF("parameters:"
>> +            "\n   flags:        %x"
>> +            "\n   start_sector: %" PRIu64
>> +            "\n   nr_sectors:   %" PRIu32
>> +            "\n   data_size:    %" PRIu64 "\n",
>> +            flags, start_sector, nr_sectors, buf_size);
> Now we are adding traces, not DPRINF's to new code in general.
> Same for all DPRINTFs
>
>> +
>> +    send_bitmap_header(f, dbms, flags);
>> +
>> +    qemu_put_be64(f, start_sector);
>> +    qemu_put_be32(f, nr_sectors);
>> +
>> +    /* if a block is zero we need to flush here since the network
>> +     * bandwidth is now a lot higher than the storage device bandwidth.
>> +     * thus if we queue zero blocks we slow down the migration.
>> +     * also, skip writing block when migrate only dirty bitmaps. */
>> +    if (flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
>> +        qemu_fflush(f);
> I thought that we were missing the g_free(buf) here, not sure if it is
> better to put the return here, or do an else for the rest of the code.
>


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH 04/18] migration: introduce postcopy-only pending

[Vladimir Sementsov-Ogievskiy]

04.11.2016 15:46, Juan Quintela wrote:
> Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com> wrote:
>> There would be savevm states (dirty-bitmap) which can migrate only in
>> postcopy stage. The corresponding pending is introduced here.
>>
>> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> ---
>>   include/migration/vmstate.h |  5 +++--
>>   include/sysemu/sysemu.h     |  5 +++--
>>   migration/block.c           |  7 ++++---
>>   migration/migration.c       | 15 ++++++++-------
>>   migration/ram.c             |  9 +++++----
>>   migration/savevm.c          | 12 +++++++-----
>>   migration/trace-events      |  2 +-
>>   7 files changed, 31 insertions(+), 24 deletions(-)
>>
>> diff --git a/include/migration/vmstate.h b/include/migration/vmstate.h
>> index 5c30ef7..dc656be 100644
>> --- a/include/migration/vmstate.h
>> +++ b/include/migration/vmstate.h
>> @@ -58,8 +58,9 @@ typedef struct SaveVMHandlers {
>>       /* This runs outside the iothread lock!  */
>>       int (*save_live_setup)(QEMUFile *f, void *opaque);
>>       void (*save_live_pending)(QEMUFile *f, void *opaque, uint64_t max_size,
>> -                              uint64_t *non_postcopiable_pending,
>> -                              uint64_t *postcopiable_pending);
>> +                              uint64_t *res_precopy_only,
>> +                              uint64_t *res_compatible,
>> +                              uint64_t *res_postcopy_only);
> Ouch, we really need to move to use an struct here :p
>
> But ugliness was already here.
>
> BTW, with only this patch, it is not clear to me what res_compatible mean

compatible mean data, which may be sent in both precopy/postcopy mode

>
>> @@ -1133,7 +1135,7 @@ void qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size,
>>               }
>>           }
>>           se->ops->save_live_pending(f, se->opaque, max_size,
>> -                                   res_non_postcopiable, res_postcopiable);
>> +                res_precopy_only, res_compatible, res_postcopy_only);
> Indentation is wrong.


-- 
Best regards,
Vladimir