[Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Vladimir Sementsov-Ogievskiy]

These patches provide dirty bitmap migration feature. Only named dirty
bitmaps are to be migrated. Migration may be enabled using migration
capabilities.

v5:
    - rebase on master
    - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
    - remove rfc, as incremental backup series by John Snow are in
      upstream
    

v4 significant changes:
 0001: tiny bugfix: out[i] -> out[i-start], same for 'in'
 0007: set chunk size to 1kb, disable live iteration for
       migrating data < 1mb size.

 tests: only one with md5 sum is here. used function event_wait
        by John Snow. (I hope, you don't mind me just adding this
        function with your 'Signed-off-by')

 rfc: This patch set is based on v13 of
      "block: incremental backup series" by John Snow, which are
      not pushed yet.

v3:
 based on v13 of "block: incremental backup series" by John Snow.

 changes from v2:
 removed patch for adding dirty parameter (migration capablities used
 instead).
 
 0001: printf's dropped, qapi used
 0002: part0 -> zeroes
 0003: part0 -> zeroes
 0005: dirty_dirty -> meta
       add comments about meta bitmap
       
 0006: the format is changed, nodes used instead of devices.

 other patches are new.

 rfc: there are two tests. They are the same but using different
 interfaces: md5 checksum of the bitmap last layer in query-block or
 separate query-block-dirty-bitmap with dirty bitmap regions.
 The second form is more appropriate for debugging, the first is more
 appropriate for simple regression control. Which should go to
 upstream?

v2:
 1. bug-fixes, that are already in upstream, and renaming of function
 bdrv_reset_dirty_bitmap (which is already in Snow's series) are
 dropped
 2. bitmap store/restore: the concept renamed to serialization, added
 function hbitmap_deserialize_part0, to not transfer zero blocks
 3. migration dirty parameter: added description comment
 4. Other patches are new.

v2.rfc:
Actually, in this version of the series I'm trying not use
migration/block.c at all. Instead a separate migration unit is added
in the new file migration/dirty-bitmap.c. Now bitmaps are migrated
like blocks in block migration, they have their "dirty-dirty" bitmaps,
for tracking set/unset changes during migration.

The advantages are:
  - no complications of migration/block.c
  - separate dirty-dirty bitmaps provide handling of "unset's"
  - more effective meta-data/data ratio - no tiny bitmap-blocks.



v1:
These patches provide dirty bitmap migration feature. Only named dirty
bitmaps are to be migrated. Migration is made as a part of block
migration in block-migration.c.

Dirty bitmap migration may be enabled by "dirty" parameter for qmp migrate
command. If "blk" and "inc" parameters are false when "dirty" is true
block migration is actually skipped: no allocatoions, no bdrv_read's,
no bdrv_write's, only bitmaps are migrated.

The patch set includes two my previous bug fixes, which are necessary
for it. The patch set is based on Incremental backup series by John
Snow.
Vladimir Sementsov-Ogievskiy (12):
  hbitmap: serialization
  block: BdrvDirtyBitmap serialization interface
  block: tiny refactoring: minimize hbitmap_(set/reset) usage
  block: add meta bitmaps
  block: add bdrv_next_dirty_bitmap()
  qapi: add dirty-bitmaps migration capability
  migration: add migration/block-dirty-bitmap.c
  iotests: maintain several vms in test
  iotests: add add_incoming_migration to VM class
  qapi: add md5 checksum of last dirty bitmap level to query-block
  iotests: add dirty bitmap migration test
  migration/qemu-file: make functions qemu_(get/put)_string public

 block.c                        |  98 +++++-
 include/block/block.h          |  22 ++
 include/migration/block.h      |   1 +
 include/migration/migration.h  |   1 +
 include/migration/qemu-file.h  |  17 +
 include/qemu/hbitmap.h         |  67 ++++
 migration/Makefile.objs        |   2 +-
 migration/block-dirty-bitmap.c | 693 +++++++++++++++++++++++++++++++++++++++++
 migration/migration.c          |   9 +
 migration/qemu-file.c          |  18 ++
 qapi-schema.json               |   4 +-
 qapi/block-core.json           |   4 +-
 tests/qemu-iotests/117         |  84 +++++
 tests/qemu-iotests/117.out     |   5 +
 tests/qemu-iotests/group       |   1 +
 tests/qemu-iotests/iotests.py  |  14 +-
 util/hbitmap.c                 | 106 +++++++
 vl.c                           |   1 +
 18 files changed, 1137 insertions(+), 10 deletions(-)
 create mode 100644 migration/block-dirty-bitmap.c
 create mode 100755 tests/qemu-iotests/117
 create mode 100644 tests/qemu-iotests/117.out

-- 
1.9.1

[Qemu-devel] [PATCH 01/12] hbitmap: serialization

[Vladimir Sementsov-Ogievskiy]

Functions to serialize / deserialize(restore) HBitmap. HBitmap should be
saved to linear sequence of bits independently of endianness and bitmap
array element (unsigned long) size. Therefore Little Endian is chosen.

These functions are appropriate for dirty bitmap migration, restoring
the bitmap in several steps is available. To save performance, every
step writes only the last level of the bitmap. All other levels are
restored by hbitmap_deserialize_finish() as a last step of restoring.
So, HBitmap is inconsistent while restoring.

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 include/qemu/hbitmap.h | 59 ++++++++++++++++++++++++++++++
 util/hbitmap.c         | 98 ++++++++++++++++++++++++++++++++++++++++++++++++++
 2 files changed, 157 insertions(+)

diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index f0a85f8..4c8e385 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -141,6 +141,65 @@ void hbitmap_reset(HBitmap *hb, uint64_t start, uint64_t count);
 bool hbitmap_get(const HBitmap *hb, uint64_t item);
 
 /**
+ * hbitmap_data_size:
+ * @hb: HBitmap to operate on.
+ * @count: Number of bits
+ *
+ * Return amount of bytes hbitmap_serialize_part needs
+ */
+uint64_t hbitmap_data_size(const HBitmap *hb, uint64_t count);
+
+/**
+ * hbitmap_serialize_part
+ * @hb: HBitmap to oprate on.
+ * @buf: Buffer to store serialized bitmap.
+ * @start: First bit to store.
+ * @count: Number of bits to store.
+ *
+ * Stores HBitmap data corresponding to given region. The format of saved data
+ * is linear sequence of bits, so it can be used by hbitmap_deserialize_part
+ * independently of endianness and size of HBitmap level array elements
+ */
+void hbitmap_serialize_part(const HBitmap *hb, uint8_t *buf,
+                            uint64_t start, uint64_t count);
+
+/**
+ * hbitmap_deserialize_part
+ * @hb: HBitmap to operate on.
+ * @buf: Buffer to restore bitmap data from.
+ * @start: First bit to restore.
+ * @count: Number of bits to restore.
+ *
+ * Retores HBitmap data corresponding to given region. The format is the same
+ * as for hbitmap_serialize_part.
+ *
+ * ! The bitmap becomes inconsistent after this operation.
+ * hbitmap_serialize_finish should be called before using the bitmap after
+ * data restoring.
+ */
+void hbitmap_deserialize_part(HBitmap *hb, uint8_t *buf,
+                              uint64_t start, uint64_t count);
+
+/**
+ * hbitmap_deserialize_zeroes
+ * @hb: HBitmap to operate on.
+ * @start: First bit to restore.
+ * @count: Number of bits to restore.
+ *
+ * Same as hbitmap_serialize_part, but fills the bitmap with zeroes.
+ */
+void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count);
+
+/**
+ * hbitmap_deserialize_finish
+ * @hb: HBitmap to operate on.
+ *
+ * Repair HBitmap after calling hbitmap_deserialize_data. Actually, all HBitmap
+ * layers are restored here.
+ */
+void hbitmap_deserialize_finish(HBitmap *hb);
+
+/**
  * hbitmap_free:
  * @hb: HBitmap to operate on.
  *
diff --git a/util/hbitmap.c b/util/hbitmap.c
index a10c7ae..46296bc 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -365,6 +365,104 @@ bool hbitmap_get(const HBitmap *hb, uint64_t item)
     return (hb->levels[HBITMAP_LEVELS - 1][pos >> BITS_PER_LEVEL] & bit) != 0;
 }
 
+uint64_t hbitmap_data_size(const HBitmap *hb, uint64_t count)
+{
+    uint64_t size, gran;
+
+    if (count == 0) {
+        return 0;
+    }
+
+    gran = 1ll << hb->granularity;
+    size = (((gran + count - 2) >> hb->granularity) >> BITS_PER_LEVEL) + 1;
+
+    return size * sizeof(unsigned long);
+}
+
+void hbitmap_serialize_part(const HBitmap *hb, uint8_t *buf,
+                            uint64_t start, uint64_t count)
+{
+    uint64_t i;
+    uint64_t last = start + count - 1;
+    unsigned long *out = (unsigned long *)buf;
+
+    if (count == 0) {
+        return;
+    }
+
+    start = (start >> hb->granularity) >> BITS_PER_LEVEL;
+    last = (last >> hb->granularity) >> BITS_PER_LEVEL;
+    count = last - start + 1;
+
+    for (i = start; i <= last; ++i) {
+        unsigned long el = hb->levels[HBITMAP_LEVELS - 1][i];
+        out[i - start] =
+            (BITS_PER_LONG == 32 ? cpu_to_le32(el) : cpu_to_le64(el));
+    }
+}
+
+void hbitmap_deserialize_part(HBitmap *hb, uint8_t *buf,
+                              uint64_t start, uint64_t count)
+{
+    uint64_t i;
+    uint64_t last = start + count - 1;
+    unsigned long *in = (unsigned long *)buf;
+
+    if (count == 0) {
+        return;
+    }
+
+    start = (start >> hb->granularity) >> BITS_PER_LEVEL;
+    last = (last >> hb->granularity) >> BITS_PER_LEVEL;
+    count = last - start + 1;
+
+    for (i = start; i <= last; ++i) {
+        hb->levels[HBITMAP_LEVELS - 1][i] =
+            (BITS_PER_LONG == 32 ? le32_to_cpu(in[i - start]) :
+                                   le64_to_cpu(in[i - start]));
+    }
+}
+
+void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count)
+{
+    uint64_t last = start + count - 1;
+
+    if (count == 0) {
+        return;
+    }
+
+    start = (start >> hb->granularity) >> BITS_PER_LEVEL;
+    last = (last >> hb->granularity) >> BITS_PER_LEVEL;
+    count = last - start + 1;
+
+    memset(hb->levels[HBITMAP_LEVELS - 1] + start, 0,
+           count * sizeof(unsigned long));
+}
+
+void hbitmap_deserialize_finish(HBitmap *bitmap)
+{
+    int64_t i, size, prev_size;
+    int lev;
+
+    /* restore levels starting from penultimate to zero level, assuming
+     * that the last level is ok */
+    size = MAX((bitmap->size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
+    for (lev = HBITMAP_LEVELS - 1; lev-- > 0; ) {
+        prev_size = size;
+        size = MAX((size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
+        memset(bitmap->levels[lev], 0, size * sizeof(unsigned long));
+
+        for (i = 0; i < prev_size; ++i) {
+            if (bitmap->levels[lev + 1][i]) {
+                bitmap->levels[lev][i >> BITS_PER_LEVEL] |=
+                    1 << (i & (BITS_PER_LONG - 1));
+            }
+        }
+    }
+
+    bitmap->levels[0][0] |= 1UL << (BITS_PER_LONG - 1);
+}
+
 void hbitmap_free(HBitmap *hb)
 {
     unsigned i;
-- 
1.9.1

[Qemu-devel] [PATCH 02/12] block: BdrvDirtyBitmap serialization interface

[Vladimir Sementsov-Ogievskiy]

Several functions to provide necessary access to BdrvDirtyBitmap for
block-migration.c

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 block.c               | 36 ++++++++++++++++++++++++++++++++++++
 include/block/block.h | 13 +++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/block.c b/block.c
index 7904098..fede5e2 100644
--- a/block.c
+++ b/block.c
@@ -3312,6 +3312,42 @@ void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap)
     hbitmap_reset(bitmap->bitmap, 0, bitmap->size);
 }
 
+const char *bdrv_dirty_bitmap_name(const BdrvDirtyBitmap *bitmap)
+{
+    return bitmap->name;
+}
+
+uint64_t bdrv_dirty_bitmap_data_size(const BdrvDirtyBitmap *bitmap,
+                                     uint64_t count)
+{
+    return hbitmap_data_size(bitmap->bitmap, count);
+}
+
+void bdrv_dirty_bitmap_serialize_part(const BdrvDirtyBitmap *bitmap,
+                                      uint8_t *buf, uint64_t start,
+                                      uint64_t count)
+{
+    hbitmap_serialize_part(bitmap->bitmap, buf, start, count);
+}
+
+void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap,
+                                        uint8_t *buf, uint64_t start,
+                                        uint64_t count)
+{
+    hbitmap_deserialize_part(bitmap->bitmap, buf, start, count);
+}
+
+void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
+                                          uint64_t start, uint64_t count)
+{
+    hbitmap_deserialize_zeroes(bitmap->bitmap, start, count);
+}
+
+void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap)
+{
+    hbitmap_deserialize_finish(bitmap->bitmap);
+}
+
 void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
                     int nr_sectors)
 {
diff --git a/include/block/block.h b/include/block/block.h
index 7d1a717..1c16906 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -484,6 +484,19 @@ void bdrv_dirty_iter_init(BdrvDirtyBitmap *bitmap, struct HBitmapIter *hbi);
 void bdrv_set_dirty_iter(struct HBitmapIter *hbi, int64_t offset);
 int64_t bdrv_get_dirty_count(BdrvDirtyBitmap *bitmap);
 
+const char *bdrv_dirty_bitmap_name(const BdrvDirtyBitmap *bitmap);
+uint64_t bdrv_dirty_bitmap_data_size(const BdrvDirtyBitmap *bitmap,
+                                     uint64_t count);
+void bdrv_dirty_bitmap_serialize_part(const BdrvDirtyBitmap *bitmap,
+                                      uint8_t *buf, uint64_t start,
+                                      uint64_t count);
+void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap,
+                                        uint8_t *buf, uint64_t start,
+                                        uint64_t count);
+void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
+                                          uint64_t start, uint64_t count);
+void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
+
 void bdrv_enable_copy_on_read(BlockDriverState *bs);
 void bdrv_disable_copy_on_read(BlockDriverState *bs);
 
-- 
1.9.1

[Qemu-devel] [PATCH 03/12] block: tiny refactoring: minimize hbitmap_(set/reset) usage

[Vladimir Sementsov-Ogievskiy]

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 block.c | 7 +++----
 1 file changed, 3 insertions(+), 4 deletions(-)

diff --git a/block.c b/block.c
index fede5e2..8376e8d 100644
--- a/block.c
+++ b/block.c
@@ -3308,8 +3308,7 @@ void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
 
 void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap)
 {
-    assert(bdrv_dirty_bitmap_enabled(bitmap));
-    hbitmap_reset(bitmap->bitmap, 0, bitmap->size);
+    bdrv_reset_dirty_bitmap(bitmap, 0, bitmap->size);
 }
 
 const char *bdrv_dirty_bitmap_name(const BdrvDirtyBitmap *bitmap)
@@ -3356,7 +3355,7 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
         if (!bdrv_dirty_bitmap_enabled(bitmap)) {
             continue;
         }
-        hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
+        bdrv_set_dirty_bitmap(bitmap, cur_sector, nr_sectors);
     }
 }
 
@@ -3368,7 +3367,7 @@ void bdrv_reset_dirty(BlockDriverState *bs, int64_t cur_sector,
         if (!bdrv_dirty_bitmap_enabled(bitmap)) {
             continue;
         }
-        hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
+        bdrv_reset_dirty_bitmap(bitmap, cur_sector, nr_sectors);
     }
 }
 
-- 
1.9.1

[Qemu-devel] [PATCH 04/12] block: add meta bitmaps

[Vladimir Sementsov-Ogievskiy]

Meta bitmap is a 'dirty bitmap' for the BdrvDirtyBitmap. It tracks
changes (set/unset) of this BdrvDirtyBitmap. It is needed for live
migration of block dirty bitmaps.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 block.c               | 44 ++++++++++++++++++++++++++++++++++++++++++++
 include/block/block.h |  7 +++++++
 2 files changed, 51 insertions(+)

diff --git a/block.c b/block.c
index 8376e8d..a167eb6 100644
--- a/block.c
+++ b/block.c
@@ -58,9 +58,15 @@
  * (3) successor is set: frozen mode.
  *     A frozen bitmap cannot be renamed, deleted, anonymized, cleared, set,
  *     or enabled. A frozen bitmap can only abdicate() or reclaim().
+ *
+ * Meta bitmap:
+ * Meta bitmap is a 'dirty bitmap' for the BdrvDirtyBitmap. It tracks changes
+ * (set/unset) of this BdrvDirtyBitmap. It is needed for live migration of
+ * block dirty bitmaps.
  */
 struct BdrvDirtyBitmap {
     HBitmap *bitmap;            /* Dirty sector bitmap implementation */
+    HBitmap *meta_bitmap;       /* Meta bitmap */
     BdrvDirtyBitmap *successor; /* Anonymous child; implies frozen status */
     char *name;                 /* Optional non-empty unique ID */
     int64_t size;               /* Size of the bitmap (Number of sectors) */
@@ -3062,6 +3068,35 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)
     bitmap->name = NULL;
 }
 
+HBitmap *bdrv_create_meta_bitmap(BdrvDirtyBitmap *bitmap,
+                                 uint64_t chunk_size)
+{
+    uint64_t sector_granularity;
+
+    assert((chunk_size & (chunk_size - 1)) == 0);
+
+    /* one chunk is corresponding to one bit of the meta bitmap, and each bit
+     * of the chunk is corresponding to 'bdrv_dirty_bitmap_granularity(bitmap)'
+     * bytes of the node */
+    sector_granularity =
+        (chunk_size * 8 * bdrv_dirty_bitmap_granularity(bitmap))
+        >> BDRV_SECTOR_BITS;
+    assert(sector_granularity);
+
+    bitmap->meta_bitmap =
+        hbitmap_alloc(bitmap->size, ffsll(sector_granularity) - 1);
+
+    return bitmap->meta_bitmap;
+}
+
+void bdrv_release_meta_bitmap(BdrvDirtyBitmap *bitmap)
+{
+    if (bitmap->meta_bitmap) {
+        hbitmap_free(bitmap->meta_bitmap);
+        bitmap->meta_bitmap = NULL;
+    }
+}
+
 BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
                                           uint32_t granularity,
                                           const char *name,
@@ -3212,6 +3247,9 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
             assert(!bdrv_dirty_bitmap_frozen(bm));
             QLIST_REMOVE(bitmap, list);
             hbitmap_free(bitmap->bitmap);
+            if (bitmap->meta_bitmap) {
+                hbitmap_free(bitmap->meta_bitmap);
+            }
             g_free(bitmap->name);
             g_free(bitmap);
             return;
@@ -3297,6 +3335,9 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
 {
     assert(bdrv_dirty_bitmap_enabled(bitmap));
     hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
+    if (bitmap->meta_bitmap) {
+        hbitmap_set(bitmap->meta_bitmap, cur_sector, nr_sectors);
+    }
 }
 
 void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
@@ -3304,6 +3345,9 @@ void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
 {
     assert(bdrv_dirty_bitmap_enabled(bitmap));
     hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
+    if (bitmap->meta_bitmap) {
+        hbitmap_set(bitmap->meta_bitmap, cur_sector, nr_sectors);
+    }
 }
 
 void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap)
diff --git a/include/block/block.h b/include/block/block.h
index 1c16906..2d91161 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -4,6 +4,7 @@
 #include "block/aio.h"
 #include "qemu-common.h"
 #include "qemu/option.h"
+#include "qemu/hbitmap.h"
 #include "block/coroutine.h"
 #include "block/accounting.h"
 #include "qapi/qmp/qobject.h"
@@ -497,6 +498,12 @@ void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
                                           uint64_t start, uint64_t count);
 void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
 
+/* chunk size here is number of bytes of the @bitmap data per one bit of the
+ * meta bitmap being created */
+HBitmap *bdrv_create_meta_bitmap(BdrvDirtyBitmap *bitmap,
+                                 uint64_t granularity);
+void bdrv_release_meta_bitmap(BdrvDirtyBitmap *bitmap);
+
 void bdrv_enable_copy_on_read(BlockDriverState *bs);
 void bdrv_disable_copy_on_read(BlockDriverState *bs);
 
-- 
1.9.1

[Qemu-devel] [PATCH 05/12] block: add bdrv_next_dirty_bitmap()

[Vladimir Sementsov-Ogievskiy]

Like bdrv_next()  - bdrv_next_dirty_bitmap() is a function to provide
access to private dirty bitmaps list.

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 block.c               | 10 ++++++++++
 include/block/block.h |  2 ++
 2 files changed, 12 insertions(+)

diff --git a/block.c b/block.c
index a167eb6..4098a06 100644
--- a/block.c
+++ b/block.c
@@ -3291,6 +3291,16 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
     return list;
 }
 
+BdrvDirtyBitmap *bdrv_next_dirty_bitmap(BlockDriverState *bs,
+                                        BdrvDirtyBitmap *bitmap)
+{
+    if (bitmap == NULL) {
+        return QLIST_FIRST(&bs->dirty_bitmaps);
+    }
+
+    return QLIST_NEXT(bitmap, list);
+}
+
 int bdrv_get_dirty(BlockDriverState *bs, BdrvDirtyBitmap *bitmap, int64_t sector)
 {
     if (bitmap) {
diff --git a/include/block/block.h b/include/block/block.h
index 2d91161..593c29e 100644
--- a/include/block/block.h
+++ b/include/block/block.h
@@ -497,6 +497,8 @@ void bdrv_dirty_bitmap_deserialize_part(BdrvDirtyBitmap *bitmap,
 void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
                                           uint64_t start, uint64_t count);
 void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
+BdrvDirtyBitmap *bdrv_next_dirty_bitmap(BlockDriverState *bs,
+                                        BdrvDirtyBitmap *bitmap);
 
 /* chunk size here is number of bytes of the @bitmap data per one bit of the
  * meta bitmap being created */
-- 
1.9.1

[Qemu-devel] [PATCH 06/12] qapi: add dirty-bitmaps migration capability

[Vladimir Sementsov-Ogievskiy]

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 include/migration/migration.h | 1 +
 migration/migration.c         | 9 +++++++++
 qapi-schema.json              | 4 +++-
 3 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index a6e025a..e3e213e 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -145,6 +145,7 @@ void migrate_add_blocker(Error *reason);
 void migrate_del_blocker(Error *reason);
 
 bool migrate_zero_blocks(void);
+bool migrate_dirty_bitmaps(void);
 
 bool migrate_auto_converge(void);
 
diff --git a/migration/migration.c b/migration/migration.c
index 732d229..44d458b 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -708,6 +708,15 @@ int migrate_decompress_threads(void)
     return s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS];
 }
 
+bool migrate_dirty_bitmaps(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS];
+}
+
 int migrate_use_xbzrle(void)
 {
     MigrationState *s;
diff --git a/qapi-schema.json b/qapi-schema.json
index 9c92482..ec2c464 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -526,11 +526,13 @@
 # @auto-converge: If enabled, QEMU will automatically throttle down the guest
 #          to speed up convergence of RAM migration. (since 1.6)
 #
+# @dirty-bitmaps: If enabled, QEMU will migrate named dirty bitmaps. (since 2.3)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
   'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
-           'compress'] }
+           'compress', 'dirty-bitmaps'] }
 
 ##
 # @MigrationCapabilityStatus
-- 
1.9.1

[Qemu-devel] [PATCH 07/12] migration: add migration/block-dirty-bitmap.c

[Vladimir Sementsov-Ogievskiy]

Live migration of dirty bitmaps. Only named dirty bitmaps, associated with
root nodes and non-root named nodes are migrated.

If destination qemu is already containing a dirty bitmap with the same name
as a migrated bitmap (for the same node), than, if their granularities are
the same the migration will be done, otherwise the error will be generated.

If destination qemu doesn't contain such bitmap it will be created.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 include/migration/block.h      |   1 +
 migration/Makefile.objs        |   2 +-
 migration/block-dirty-bitmap.c | 728 +++++++++++++++++++++++++++++++++++++++++
 vl.c                           |   1 +
 4 files changed, 731 insertions(+), 1 deletion(-)
 create mode 100644 migration/block-dirty-bitmap.c

diff --git a/include/migration/block.h b/include/migration/block.h
index ffa8ac0..566bb9f 100644
--- a/include/migration/block.h
+++ b/include/migration/block.h
@@ -14,6 +14,7 @@
 #ifndef BLOCK_MIGRATION_H
 #define BLOCK_MIGRATION_H
 
+void dirty_bitmap_mig_init(void);
 void blk_mig_init(void);
 int blk_mig_active(void);
 uint64_t blk_mig_bytes_transferred(void);
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index d929e96..128612d 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -6,5 +6,5 @@ common-obj-y += xbzrle.o
 common-obj-$(CONFIG_RDMA) += rdma.o
 common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o
 
-common-obj-y += block.o
+common-obj-y += block.o block-dirty-bitmap.o
 
diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
new file mode 100644
index 0000000..2d3ba23
--- /dev/null
+++ b/migration/block-dirty-bitmap.c
@@ -0,0 +1,728 @@
+/*
+ * QEMU dirty bitmap migration
+ *
+ * Live migration of dirty bitmaps. Only named dirty bitmaps, associated with
+ * root nodes and non-root named nodes are migrated. Live iteration is disabled
+ * for small data amount (see MIN_LIVE_SIZE).
+ *
+ * If destination qemu is already containing a dirty bitmap with the same name
+ * as a migrated bitmap (for the same node), than, if their granularities are
+ * the same the migration will be done, otherwise the error will be generated.
+ *
+ * If destination qemu doesn't contain such bitmap it will be created.
+ *
+ * format of migration:
+ *
+ * # Header (shared for different chunk types)
+ * 1, 2 or 4 bytes: flags (see qemu_{put,put}_flags)
+ * [ 1 byte: node name size ] \  flags & DEVICE_NAME
+ * [ n bytes: node name     ] /
+ * [ 1 byte: bitmap name size ] \  flags & BITMAP_NAME
+ * [ n bytes: bitmap name     ] /
+ *
+ * # Start of bitmap migration (flags & START)
+ * header
+ * be64: granularity
+ *
+ * # Complete of bitmap migration (flags & COMPLETE)
+ * header
+ * 1 byte: bitmap enabled flag
+ *
+ * # Data chunk of bitmap migration
+ * header
+ * be64: start sector
+ * be32: number of sectors
+ * [ be64: buffer size  ] \ ! (flags & ZEROES)
+ * [ n bytes: buffer    ] /
+ *
+ * The last chunk in stream should contain flags & EOS. The chunk may skip
+ * device and/or bitmap names, assuming them to be the same with the previous
+ * chunk.
+ *
+ *
+ * This file is derived from migration/block.c
+ *
+ * Author:
+ * Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
+ *
+ * original copyright message:
+ * =====================================================================
+ * Copyright IBM, Corp. 2009
+ *
+ * Authors:
+ *  Liran Schour   <lirans@il.ibm.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2.  See
+ * the COPYING file in the top-level directory.
+ *
+ * Contributions after 2012-01-13 are licensed under the terms of the
+ * GNU GPL, version 2 or (at your option) any later version.
+ * =====================================================================
+ */
+
+#include "block/block.h"
+#include "block/block_int.h"
+#include "sysemu/block-backend.h"
+#include "qemu/main-loop.h"
+#include "qemu/error-report.h"
+#include "migration/block.h"
+#include "migration/migration.h"
+#include "qemu/hbitmap.h"
+#include <assert.h>
+
+#define CHUNK_SIZE     (1 << 10)
+#define MIN_LIVE_SIZE  (1 << 20)
+
+/* Flags occupy from one to four bytes. In all but one the 7-th (EXTRA_FLAGS)
+ * bit should be set. */
+#define DIRTY_BITMAP_MIG_FLAG_EOS           0x01
+#define DIRTY_BITMAP_MIG_FLAG_ZEROES        0x02
+#define DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME   0x04
+#define DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME   0x08
+#define DIRTY_BITMAP_MIG_FLAG_START         0x10
+#define DIRTY_BITMAP_MIG_FLAG_COMPLETE      0x20
+#define DIRTY_BITMAP_MIG_FLAG_BITS          0x40
+
+#define DIRTY_BITMAP_MIG_EXTRA_FLAGS        0x80
+#define DIRTY_BITMAP_MIG_FLAGS_SIZE_16      0x8000
+#define DIRTY_BITMAP_MIG_FLAGS_SIZE_32      0x8080
+
+#define DEBUG_DIRTY_BITMAP_MIGRATION 1
+
+#define DPRINTF(fmt, args...) \
+    do { \
+        if (DEBUG_DIRTY_BITMAP_MIGRATION) { \
+            printf("DMIG %s:%d", __func__, __LINE__); \
+            printf(fmt, ##args); \
+        } \
+    } while (0)
+
+typedef struct DirtyBitmapMigBitmapState {
+    /* Written during setup phase. */
+    BlockDriverState *bs;
+    const char *node_name;
+    BdrvDirtyBitmap *bitmap;
+    HBitmap *meta_bitmap;
+    uint64_t total_sectors;
+    uint64_t sectors_per_chunk;
+    QSIMPLEQ_ENTRY(DirtyBitmapMigBitmapState) entry;
+
+    /* For bulk phase. */
+    bool bulk_completed;
+    uint64_t cur_sector;
+
+    /* For dirty phase. */
+    HBitmapIter iter_dirty;
+} DirtyBitmapMigBitmapState;
+
+typedef struct DirtyBitmapMigState {
+    QSIMPLEQ_HEAD(dbms_list, DirtyBitmapMigBitmapState) dbms_list;
+
+    bool bulk_completed;
+    bool is_live_iterative;
+
+    /* for send_bitmap_bits() */
+    BlockDriverState *prev_bs;
+    BdrvDirtyBitmap *prev_bitmap;
+} DirtyBitmapMigState;
+
+typedef struct DirtyBitmapLoadState {
+    uint32_t flags;
+    char node_name[256];
+    char bitmap_name[256];
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *bitmap;
+} DirtyBitmapLoadState;
+
+static DirtyBitmapMigState dirty_bitmap_mig_state;
+
+static uint32_t qemu_get_flags(QEMUFile *f)
+{
+    uint8_t flags = qemu_get_byte(f);
+    if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
+        flags = flags << 8 | qemu_get_byte(f);
+        if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
+            flags = flags << 16 | qemu_get_be16(f);
+        }
+    }
+
+    return flags;
+}
+
+static void qemu_put_flags(QEMUFile *f, uint32_t flags)
+{
+    if (!(flags & 0xffffff00)) {
+        qemu_put_byte(f, flags);
+        return;
+    }
+
+    if (!(flags & 0xffff0000)) {
+        qemu_put_be16(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_16);
+        return;
+    }
+
+    qemu_put_be32(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_32);
+}
+
+/* read name from qemu file:
+ * format:
+ * 1 byte : len = name length (<256)
+ * len bytes : name without last zero byte
+ *
+ * name should point to the buffer >= 256 bytes length
+ */
+static char *qemu_get_string(QEMUFile *f, char *name)
+{
+    int len = qemu_get_byte(f);
+    qemu_get_buffer(f, (uint8_t *)name, len);
+    name[len] = '\0';
+
+    DPRINTF("get name: %d %s\n", len, name);
+
+    return name;
+}
+
+/* write name to qemu file:
+ * format:
+ * same as for qemu_get_string
+ *
+ * maximum name length is 255
+ */
+static void qemu_put_string(QEMUFile *f, const char *name)
+{
+    int len = strlen(name);
+
+    DPRINTF("put name: %d %s\n", len, name);
+
+    assert(len < 256);
+    qemu_put_byte(f, len);
+    qemu_put_buffer(f, (const uint8_t *)name, len);
+}
+
+static void send_bitmap_header(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
+                               uint32_t additional_flags)
+{
+    BlockDriverState *bs = dbms->bs;
+    BdrvDirtyBitmap *bitmap = dbms->bitmap;
+    uint32_t flags = additional_flags;
+
+    if (bs != dirty_bitmap_mig_state.prev_bs) {
+        dirty_bitmap_mig_state.prev_bs = bs;
+        flags |= DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME;
+    }
+
+    if (bitmap != dirty_bitmap_mig_state.prev_bitmap) {
+        dirty_bitmap_mig_state.prev_bitmap = bitmap;
+        flags |= DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME;
+    }
+
+    qemu_put_flags(f, flags);
+
+    if (flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
+        qemu_put_string(f, dbms->node_name);
+    }
+
+    if (flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
+        qemu_put_string(f, bdrv_dirty_bitmap_name(bitmap));
+    }
+}
+
+static void send_bitmap_start(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
+{
+    send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_START);
+    qemu_put_be32(f, bdrv_dirty_bitmap_granularity(dbms->bitmap));
+}
+
+static void send_bitmap_complete(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
+{
+    send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_COMPLETE);
+    qemu_put_byte(f, bdrv_dirty_bitmap_enabled(dbms->bitmap));
+}
+
+static void send_bitmap_bits(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
+                             uint64_t start_sector, uint32_t nr_sectors)
+{
+    /* align for buffer_is_zero() */
+    uint64_t align = 4 * sizeof(long);
+    uint64_t buf_size =
+        (bdrv_dirty_bitmap_data_size(dbms->bitmap, nr_sectors) + align - 1) &
+        ~(align - 1);
+    uint8_t *buf = g_malloc0(buf_size);
+    uint32_t flags = DIRTY_BITMAP_MIG_FLAG_BITS;
+
+    bdrv_dirty_bitmap_serialize_part(dbms->bitmap, buf,
+                                     start_sector, nr_sectors);
+
+    if (buffer_is_zero(buf, buf_size)) {
+        g_free(buf);
+        buf = NULL;
+        flags |= DIRTY_BITMAP_MIG_FLAG_ZEROES;
+    }
+
+    DPRINTF("parameters:"
+            "\n   flags:        %x"
+            "\n   start_sector: %" PRIu64
+            "\n   nr_sectors:   %" PRIu32
+            "\n   data_size:    %" PRIu64 "\n",
+            flags, start_sector, nr_sectors, buf_size);
+
+    send_bitmap_header(f, dbms, flags);
+
+    qemu_put_be64(f, start_sector);
+    qemu_put_be32(f, nr_sectors);
+
+    /* if a block is zero we need to flush here since the network
+     * bandwidth is now a lot higher than the storage device bandwidth.
+     * thus if we queue zero blocks we slow down the migration.
+     * also, skip writing block when migrate only dirty bitmaps. */
+    if (flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
+        qemu_fflush(f);
+        return;
+    }
+
+    qemu_put_be64(f, buf_size);
+    qemu_put_buffer(f, buf, buf_size);
+    g_free(buf);
+}
+
+
+/* Called with iothread lock taken.  */
+
+static void set_dirty_tracking(void)
+{
+    DirtyBitmapMigBitmapState *dbms;
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        dbms->meta_bitmap =
+            bdrv_create_meta_bitmap(dbms->bitmap, CHUNK_SIZE);
+        dbms->sectors_per_chunk =
+            UINT64_C(1) << hbitmap_granularity(dbms->meta_bitmap);
+    }
+}
+
+static void unset_dirty_tracking(void)
+{
+    DirtyBitmapMigBitmapState *dbms;
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        bdrv_release_meta_bitmap(dbms->bitmap);
+    }
+}
+
+static void init_dirty_bitmap_migration(void)
+{
+    BlockDriverState *bs;
+    BdrvDirtyBitmap *bitmap;
+    DirtyBitmapMigBitmapState *dbms;
+    uint64_t total_bytes = 0;
+
+    dirty_bitmap_mig_state.bulk_completed = false;
+    dirty_bitmap_mig_state.prev_bs = NULL;
+    dirty_bitmap_mig_state.prev_bitmap = NULL;
+
+    for (bs = bdrv_next(NULL); bs; bs = bdrv_next(bs)) {
+        for (bitmap = bdrv_next_dirty_bitmap(bs, NULL); bitmap;
+             bitmap = bdrv_next_dirty_bitmap(bs, bitmap)) {
+            if (!bdrv_dirty_bitmap_name(bitmap)) {
+                continue;
+            }
+
+            if (!bdrv_get_node_name(bs) && blk_bs(bs->blk) != bs) {
+                /* not named non-root node */
+                continue;
+            }
+
+            dbms = g_new0(DirtyBitmapMigBitmapState, 1);
+            dbms->bs = bs;
+            dbms->node_name = bdrv_get_node_name(bs);
+            if (!dbms->node_name || dbms->node_name[0] == '\0') {
+                dbms->node_name = bdrv_get_device_name(bs);
+            }
+            dbms->bitmap = bitmap;
+            dbms->total_sectors = bdrv_nb_sectors(bs);
+            total_bytes +=
+                bdrv_dirty_bitmap_data_size(bitmap, dbms->total_sectors);
+
+            QSIMPLEQ_INSERT_TAIL(&dirty_bitmap_mig_state.dbms_list,
+                                 dbms, entry);
+        }
+    }
+
+    dirty_bitmap_mig_state.is_live_iterative = total_bytes > MIN_LIVE_SIZE;
+}
+
+/* Called with no lock taken.  */
+static void bulk_phase_send_chunk(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
+{
+    uint32_t nr_sectors = MIN(dbms->total_sectors - dbms->cur_sector,
+                             dbms->sectors_per_chunk);
+
+    send_bitmap_bits(f, dbms, dbms->cur_sector, nr_sectors);
+
+    dbms->cur_sector += nr_sectors;
+    if (dbms->cur_sector >= dbms->total_sectors) {
+        dbms->bulk_completed = true;
+    }
+}
+
+/* Called with no lock taken.  */
+static void bulk_phase(QEMUFile *f, bool limit)
+{
+    DirtyBitmapMigBitmapState *dbms;
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        while (!dbms->bulk_completed) {
+            bulk_phase_send_chunk(f, dbms);
+            if (limit && qemu_file_rate_limit(f)) {
+                return;
+            }
+        }
+    }
+
+    dirty_bitmap_mig_state.bulk_completed = true;
+}
+
+static void blk_mig_reset_dirty_cursor(void)
+{
+    DirtyBitmapMigBitmapState *dbms;
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        hbitmap_iter_init(&dbms->iter_dirty, dbms->meta_bitmap, 0);
+    }
+}
+
+/* Called with iothread lock taken. */
+static bool dirty_phase_send_chunk(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
+{
+    uint32_t nr_sectors;
+    size_t old_pos = dbms->iter_dirty.pos;
+    int64_t cur = hbitmap_iter_next(&dbms->iter_dirty);
+
+    /* restart search from the beginning */
+    if (old_pos && cur == -1) {
+        hbitmap_iter_init(&dbms->iter_dirty, dbms->meta_bitmap, 0);
+        cur = hbitmap_iter_next(&dbms->iter_dirty);
+    }
+
+    if (cur == -1) {
+        hbitmap_iter_init(&dbms->iter_dirty, dbms->meta_bitmap, 0);
+        return false;
+    }
+
+    nr_sectors = MIN(dbms->total_sectors - cur, dbms->sectors_per_chunk);
+    send_bitmap_bits(f, dbms, cur, nr_sectors);
+    hbitmap_reset(dbms->meta_bitmap, cur, dbms->sectors_per_chunk);
+
+    return true;
+}
+
+/* Called with iothread lock taken. */
+static void dirty_phase(QEMUFile *f, bool limit)
+{
+    DirtyBitmapMigBitmapState *dbms;
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        while (dirty_phase_send_chunk(f, dbms)) {
+            if (limit && qemu_file_rate_limit(f)) {
+                return;
+            }
+        }
+    }
+}
+
+
+/* Called with iothread lock taken.  */
+static void dirty_bitmap_mig_cleanup(void)
+{
+    DirtyBitmapMigBitmapState *dbms;
+
+    unset_dirty_tracking();
+
+    while ((dbms = QSIMPLEQ_FIRST(&dirty_bitmap_mig_state.dbms_list)) != NULL) {
+        QSIMPLEQ_REMOVE_HEAD(&dirty_bitmap_mig_state.dbms_list, entry);
+        g_free(dbms);
+    }
+}
+
+static void dirty_bitmap_migration_cancel(void *opaque)
+{
+    dirty_bitmap_mig_cleanup();
+}
+
+static int dirty_bitmap_save_iterate(QEMUFile *f, void *opaque)
+{
+    DPRINTF("enter\n");
+
+    if (dirty_bitmap_mig_state.bulk_completed) {
+        qemu_mutex_lock_iothread();
+        dirty_phase(f, true);
+        qemu_mutex_unlock_iothread();
+    } else {
+        bulk_phase(f, true);
+    }
+
+    qemu_put_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
+
+    return dirty_bitmap_mig_state.bulk_completed;
+}
+
+/* Called with iothread lock taken.  */
+
+static int dirty_bitmap_save_complete(QEMUFile *f, void *opaque)
+{
+    DirtyBitmapMigBitmapState *dbms;
+    DPRINTF("enter\n");
+
+    if (!dirty_bitmap_mig_state.bulk_completed) {
+        bulk_phase(f, false);
+    }
+
+    blk_mig_reset_dirty_cursor();
+    dirty_phase(f, false);
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        send_bitmap_complete(f, dbms);
+    }
+
+    qemu_put_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
+
+    DPRINTF("Dirty bitmaps migration completed\n");
+
+    dirty_bitmap_mig_cleanup();
+    return 0;
+}
+
+static uint64_t dirty_bitmap_save_pending(QEMUFile *f, void *opaque,
+                                          uint64_t max_size)
+{
+    DirtyBitmapMigBitmapState *dbms;
+    uint64_t pending = 0;
+
+    qemu_mutex_lock_iothread();
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        uint64_t sectors = hbitmap_count(dbms->meta_bitmap);
+        if (!dbms->bulk_completed) {
+            sectors += dbms->total_sectors - dbms->cur_sector;
+        }
+        pending += bdrv_dirty_bitmap_data_size(dbms->bitmap, sectors);
+    }
+
+    qemu_mutex_unlock_iothread();
+
+    DPRINTF("pending %" PRIu64 ", max: %" PRIu64 "\n",
+            pending, max_size);
+    return pending;
+}
+
+/* First occurrence of this bitmap. It should be created if doesn't exist */
+static int dirty_bitmap_load_start(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+    uint32_t granularity = qemu_get_be32(f);
+    if (!s->bitmap) {
+        Error *local_err = NULL;
+        s->bitmap = bdrv_create_dirty_bitmap(s->bs, granularity,
+                                             s->bitmap_name, &local_err);
+        if (!s->bitmap) {
+            error_report("%s", error_get_pretty(local_err));
+            error_free(local_err);
+            return -EINVAL;
+        }
+    } else {
+        uint32_t dest_granularity =
+            bdrv_dirty_bitmap_granularity(s->bitmap);
+        if (dest_granularity != granularity) {
+            fprintf(stderr,
+                    "Error: "
+                    "Migrated bitmap granularity (%" PRIu32 ") "
+                    "doesn't match the destination bitmap '%s' "
+                    "granularity (%" PRIu32 ")\n",
+                    granularity,
+                    bdrv_dirty_bitmap_name(s->bitmap),
+                    dest_granularity);
+            return -EINVAL;
+        }
+    }
+
+    bdrv_disable_dirty_bitmap(s->bitmap);
+
+    return 0;
+}
+
+static void dirty_bitmap_load_complete(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+    bool enabled;
+
+    bdrv_dirty_bitmap_deserialize_finish(s->bitmap);
+
+    enabled = qemu_get_byte(f);
+    if (enabled) {
+        bdrv_enable_dirty_bitmap(s->bitmap);
+    }
+}
+
+static int dirty_bitmap_load_bits(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+    uint64_t first_sector = qemu_get_be64(f);
+    uint32_t nr_sectors = qemu_get_be32(f);
+    DPRINTF("chunk: %lu %u\n", first_sector, nr_sectors);
+
+
+    if (s->flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
+        DPRINTF("   - zeroes\n");
+        bdrv_dirty_bitmap_deserialize_zeroes(s->bitmap, first_sector,
+                                             nr_sectors);
+    } else {
+        uint8_t *buf;
+        uint64_t buf_size = qemu_get_be64(f);
+        uint64_t needed_size =
+            bdrv_dirty_bitmap_data_size(s->bitmap, nr_sectors);
+
+        if (needed_size > buf_size) {
+            fprintf(stderr,
+                    "Error: Migrated bitmap granularity doesn't "
+                    "match the destination bitmap '%s' granularity\n",
+                    bdrv_dirty_bitmap_name(s->bitmap));
+            return -EINVAL;
+        }
+
+        buf = g_malloc(buf_size);
+        qemu_get_buffer(f, buf, buf_size);
+        bdrv_dirty_bitmap_deserialize_part(s->bitmap, buf,
+                                           first_sector,
+                                           nr_sectors);
+        g_free(buf);
+    }
+
+    return 0;
+}
+
+static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s)
+{
+    Error *local_err = NULL;
+    s->flags = qemu_get_flags(f);
+    DPRINTF("flags: %x\n", s->flags);
+
+    if (s->flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
+        qemu_get_string(f, s->node_name);
+        s->bs = bdrv_lookup_bs(s->node_name, s->node_name, &local_err);
+        if (!s->bs) {
+            error_report("%s", error_get_pretty(local_err));
+            error_free(local_err);
+            return -EINVAL;
+        }
+    } else if (!s->bs) {
+        fprintf(stderr, "Error: block device name is not set\n");
+        return -EINVAL;
+    }
+
+    if (s->flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
+        qemu_get_string(f, s->bitmap_name);
+        s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);
+
+        /* bitmap may be NULL here, it wouldn't be an error if it is the
+         * first occurrence of the bitmap */
+        if (!s->bitmap && !(s->flags & DIRTY_BITMAP_MIG_FLAG_START)) {
+            fprintf(stderr, "Error: unknown dirty bitmap "
+                    "'%s' for block device '%s'\n",
+                    s->bitmap_name, s->node_name);
+            return -EINVAL;
+        }
+    } else if (!s->bitmap) {
+        fprintf(stderr, "Error: block device name is not set\n");
+        return -EINVAL;
+    }
+
+    return 0;
+}
+
+static int dirty_bitmap_load(QEMUFile *f, void *opaque, int version_id)
+{
+    static DirtyBitmapLoadState s;
+
+    int ret = 0;
+
+    DPRINTF("load start\n");
+
+    do {
+        dirty_bitmap_load_header(f, &s);
+
+        if (s.flags & DIRTY_BITMAP_MIG_FLAG_START) {
+            ret = dirty_bitmap_load_start(f, &s);
+        } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_COMPLETE) {
+            dirty_bitmap_load_complete(f, &s);
+        } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_BITS) {
+            ret = dirty_bitmap_load_bits(f, &s);
+        }
+
+        DPRINTF("ret: %d\n", ret);
+        if (!ret) {
+            ret = qemu_file_get_error(f);
+        }
+
+        DPRINTF("ret: %d\n", ret);
+        if (ret) {
+            return ret;
+        }
+    } while (!(s.flags & DIRTY_BITMAP_MIG_FLAG_EOS));
+
+    DPRINTF("load finish\n");
+    return 0;
+}
+
+static int dirty_bitmap_save_setup(QEMUFile *f, void *opaque)
+{
+    DirtyBitmapMigBitmapState *dbms = NULL;
+    init_dirty_bitmap_migration();
+
+    qemu_mutex_lock_iothread();
+    /* start track dirtiness of dirty bitmaps */
+    set_dirty_tracking();
+    qemu_mutex_unlock_iothread();
+
+    blk_mig_reset_dirty_cursor();
+
+    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
+        send_bitmap_start(f, dbms);
+    }
+    qemu_put_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
+
+    return 0;
+}
+
+static bool dirty_bitmap_is_active(void *opaque)
+{
+    return migrate_dirty_bitmaps();
+}
+
+static bool dirty_bitmap_live_iterate_is_active(void *opaque)
+{
+    return migrate_dirty_bitmaps() && dirty_bitmap_mig_state.is_live_iterative;
+}
+
+static SaveVMHandlers savevm_dirty_bitmap_handlers = {
+    .save_live_setup = dirty_bitmap_save_setup,
+    .save_live_complete = dirty_bitmap_save_complete,
+    .save_live_pending = dirty_bitmap_save_pending,
+    .load_state = dirty_bitmap_load,
+    .cancel = dirty_bitmap_migration_cancel,
+    .is_active = dirty_bitmap_is_active,
+};
+
+static SaveVMHandlers savevm_dirty_bitmap_live_iterate_handlers = {
+    .save_live_iterate = dirty_bitmap_save_iterate,
+    .load_state = dirty_bitmap_load,
+    .is_active = dirty_bitmap_live_iterate_is_active,
+};
+
+void dirty_bitmap_mig_init(void)
+{
+    QSIMPLEQ_INIT(&dirty_bitmap_mig_state.dbms_list);
+
+    register_savevm_live(NULL, "dirty-bitmap", 0, 1,
+                         &savevm_dirty_bitmap_handlers,
+                         &dirty_bitmap_mig_state);
+    register_savevm_live(NULL, "dirty-bitmap-live-iterate", 0, 1,
+                         &savevm_dirty_bitmap_live_iterate_handlers,
+                         &dirty_bitmap_mig_state);
+}
diff --git a/vl.c b/vl.c
index 15bccc4..83871f5 100644
--- a/vl.c
+++ b/vl.c
@@ -4174,6 +4174,7 @@ int main(int argc, char **argv, char **envp)
 
     blk_mig_init();
     ram_mig_init();
+    dirty_bitmap_mig_init();
 
     /* If the currently selected machine wishes to override the units-per-bus
      * property of its default HBA interface type, do so now. */
-- 
1.9.1

[Qemu-devel] [PATCH 08/12] iotests: maintain several vms in test

[Vladimir Sementsov-Ogievskiy]

The only problem with it is the same qmp socket name (which is
vm._monitor_path) for all vms. And because of this second vm couldn't be
lauched (vm.launch() fails because of socket is already in use).
This patch adds a number of vm into vm._monitor_path

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 tests/qemu-iotests/iotests.py | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index e93e623..0579849 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -97,11 +97,12 @@ def event_match(event, match=None):
 
 class VM(object):
     '''A QEMU VM'''
+    nb_vms = 0
 
     def __init__(self):
-        self._monitor_path = os.path.join(test_dir, 'qemu-mon.%d' % os.getpid())
-        self._qemu_log_path = os.path.join(test_dir, 'qemu-log.%d' % os.getpid())
-        self._qtest_path = os.path.join(test_dir, 'qemu-qtest.%d' % os.getpid())
+        self._monitor_path = os.path.join(test_dir, 'qemu-mon.%d.%d' % (os.getpid(), VM.nb_vms))
+        self._qemu_log_path = os.path.join(test_dir, 'qemu-log.%d.%d' % (os.getpid(), VM.nb_vms))
+        self._qtest_path = os.path.join(test_dir, 'qemu-qtest.%d.%d' % (os.getpid(), VM.nb_vms))
         self._args = qemu_args + ['-chardev',
                      'socket,id=mon,path=' + self._monitor_path,
                      '-mon', 'chardev=mon,mode=control',
@@ -110,6 +111,7 @@ class VM(object):
                      '-display', 'none', '-vga', 'none']
         self._num_drives = 0
         self._events = []
+        VM.nb_vms += 1
 
     # This can be used to add an unused monitor instance.
     def add_monitor_telnet(self, ip, port):
-- 
1.9.1

[Qemu-devel] [PATCH 09/12] iotests: add add_incoming_migration to VM class

[Vladimir Sementsov-Ogievskiy]

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 tests/qemu-iotests/iotests.py | 6 ++++++
 1 file changed, 6 insertions(+)

diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 0579849..945f2a2 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -134,6 +134,12 @@ class VM(object):
         self._num_drives += 1
         return self
 
+    def add_incoming_migration(self, desc):
+        '''Add an incoming migration to the VM'''
+        self._args.append('-incoming')
+        self._args.append(desc)
+        return self
+
     def pause_drive(self, drive, event=None):
         '''Pause drive r/w operations'''
         if not event:
-- 
1.9.1

[Qemu-devel] [PATCH 10/12] qapi: add md5 checksum of last dirty bitmap level to query-block

[Vladimir Sementsov-Ogievskiy]

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 block.c                | 1 +
 include/qemu/hbitmap.h | 8 ++++++++
 qapi/block-core.json   | 4 +++-
 util/hbitmap.c         | 8 ++++++++
 4 files changed, 20 insertions(+), 1 deletion(-)

diff --git a/block.c b/block.c
index 4098a06..575584d 100644
--- a/block.c
+++ b/block.c
@@ -3283,6 +3283,7 @@ BlockDirtyInfoList *bdrv_query_dirty_bitmaps(BlockDriverState *bs)
         info->has_name = !!bm->name;
         info->name = g_strdup(bm->name);
         info->frozen = bdrv_dirty_bitmap_frozen(bm);
+        info->md5 = hbitmap_md5(bm->bitmap);
         entry->value = info;
         *plist = entry;
         plist = &entry->next;
diff --git a/include/qemu/hbitmap.h b/include/qemu/hbitmap.h
index 4c8e385..ba1a1c6 100644
--- a/include/qemu/hbitmap.h
+++ b/include/qemu/hbitmap.h
@@ -200,6 +200,14 @@ void hbitmap_deserialize_zeroes(HBitmap *hb, uint64_t start, uint64_t count);
 void hbitmap_deserialize_finish(HBitmap *hb);
 
 /**
+ * hbitmap_md5:
+ * @bitmap: HBitmap to operate on.
+ *
+ * Returns md5 checksum of the last level.
+ */
+char *hbitmap_md5(const HBitmap *bitmap);
+
+/**
  * hbitmap_free:
  * @hb: HBitmap to operate on.
  *
diff --git a/qapi/block-core.json b/qapi/block-core.json
index 863ffea..eb647e8 100644
--- a/qapi/block-core.json
+++ b/qapi/block-core.json
@@ -338,11 +338,13 @@
 #
 # @frozen: whether the dirty bitmap is frozen (Since 2.4)
 #
+# @md5: md5 checksum of the last bitmap level (since 2.3)
+#
 # Since: 1.3
 ##
 { 'struct': 'BlockDirtyInfo',
   'data': {'*name': 'str', 'count': 'int', 'granularity': 'uint32',
-           'frozen': 'bool'} }
+           'frozen': 'bool', 'md5': 'str'} }
 
 ##
 # @BlockInfo:
diff --git a/util/hbitmap.c b/util/hbitmap.c
index 46296bc..83e596f 100644
--- a/util/hbitmap.c
+++ b/util/hbitmap.c
@@ -578,3 +578,11 @@ bool hbitmap_merge(HBitmap *a, const HBitmap *b)
 
     return true;
 }
+
+char *hbitmap_md5(const HBitmap *bitmap)
+{
+    uint64_t size =
+        MAX((bitmap->size + BITS_PER_LONG - 1) >> BITS_PER_LEVEL, 1);
+    const guchar *data = (const guchar *)bitmap->levels[HBITMAP_LEVELS - 1];
+    return g_compute_checksum_for_data(G_CHECKSUM_MD5, data, size);
+}
-- 
1.9.1

[Qemu-devel] [PATCH 11/12] iotests: add dirty bitmap migration test

[Vladimir Sementsov-Ogievskiy]

The test starts two vms (vm_a, vm_b), create dirty bitmap in
the first one, do several writes to corresponding device and
then migrate vm_a to vm_b with dirty bitmaps.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 tests/qemu-iotests/117     | 84 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/qemu-iotests/117.out |  5 +++
 tests/qemu-iotests/group   |  1 +
 3 files changed, 90 insertions(+)
 create mode 100755 tests/qemu-iotests/117
 create mode 100644 tests/qemu-iotests/117.out

diff --git a/tests/qemu-iotests/117 b/tests/qemu-iotests/117
new file mode 100755
index 0000000..9fab5d0
--- /dev/null
+++ b/tests/qemu-iotests/117
@@ -0,0 +1,84 @@
+#!/usr/bin/env python
+#
+# Tests for dirty bitmaps migration.
+#
+# (C) Vladimir Sementsov-Ogievskiy 2015
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program.  If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import iotests
+import time
+from iotests import qemu_img
+
+disk_a = os.path.join(iotests.test_dir, 'disk_a')
+disk_b = os.path.join(iotests.test_dir, 'disk_b')
+fifo   = os.path.join(iotests.test_dir, 'fifo')
+
+size   = 0x40000000 # 1G
+sector_size = 512
+granularity = 0x10000
+regions = [
+    { 'start': 0,          'count': 0x100000 },
+    { 'start': 0x10000000, 'count': 0x20000  },
+    { 'start': 0x39990000, 'count': 0x10000  }
+    ]
+
+class TestDirtyBitmapMigration(iotests.QMPTestCase):
+
+    def setUp(self):
+        os.mkfifo(fifo)
+        qemu_img('create', '-f', iotests.imgfmt, disk_a, str(size))
+        qemu_img('create', '-f', iotests.imgfmt, disk_b, str(size))
+        self.vm_a = iotests.VM().add_drive(disk_a)
+        self.vm_b = iotests.VM().add_drive(disk_b)
+        self.vm_b.add_incoming_migration("exec: cat " + fifo)
+        self.vm_a.launch()
+        self.vm_b.launch()
+
+    def tearDown(self):
+        self.vm_a.shutdown()
+        self.vm_b.shutdown()
+        os.remove(disk_a)
+        os.remove(disk_b)
+        os.remove(fifo)
+
+    def test_migration(self):
+        result = self.vm_a.qmp('block-dirty-bitmap-add', node='drive0',
+                               name='bitmap', granularity=granularity)
+        self.assert_qmp(result, 'return', {});
+
+        for r in regions:
+          self.vm_a.hmp_qemu_io('drive0',
+                                'write %d %d' % (r['start'], r['count']))
+
+        result = self.vm_a.qmp('query-block');
+        md5 = result['return'][0]['dirty-bitmaps'][0]['md5']
+
+        result = self.vm_a.qmp('migrate-set-capabilities',
+                               capabilities=[{'capability': 'dirty-bitmaps',
+                                              'state': True}])
+        self.assert_qmp(result, 'return', {})
+
+        result = self.vm_a.qmp('migrate', uri='exec:cat>' + fifo)
+        self.assertIsNotNone(self.vm_a.event_wait("STOP"))
+        self.assertIsNotNone(self.vm_b.event_wait("RESUME"))
+
+        result = self.vm_b.qmp('query-block');
+        self.assert_qmp(result, 'return[0]/dirty-bitmaps[0]/md5', md5);
+
+
+if __name__ == '__main__':
+    iotests.main()
diff --git a/tests/qemu-iotests/117.out b/tests/qemu-iotests/117.out
new file mode 100644
index 0000000..ae1213e
--- /dev/null
+++ b/tests/qemu-iotests/117.out
@@ -0,0 +1,5 @@
+.
+----------------------------------------------------------------------
+Ran 1 tests
+
+OK
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index 6ca3466..6812681 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -121,6 +121,7 @@
 114 rw auto quick
 115 rw auto
 116 rw auto quick
+117 rw auto quick
 121 rw auto
 122 rw auto
 123 rw auto quick
-- 
1.9.1

[Qemu-devel] [PATCH 12/12] migration/qemu-file: make functions qemu_(get/put)_string public

[Vladimir Sementsov-Ogievskiy]

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
---
 include/migration/qemu-file.h  | 17 +++++++++++++++++
 migration/block-dirty-bitmap.c | 35 -----------------------------------
 migration/qemu-file.c          | 18 ++++++++++++++++++
 3 files changed, 35 insertions(+), 35 deletions(-)

diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index a01c5b8..cc9f29d 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -312,4 +312,21 @@ static inline void qemu_get_sbe64s(QEMUFile *f, int64_t *pv)
 {
     qemu_get_be64s(f, (uint64_t *)pv);
 }
+
+/* read name from qemu file:
+ * format:
+ * 1 byte : len = name length (<256)
+ * len bytes : name without last zero byte
+ *
+ * name should point to the buffer >= 256 bytes length
+ */
+char *qemu_get_string(QEMUFile *f, char *name);
+
+/* write name to qemu file:
+ * format:
+ * same as for qemu_get_string
+ *
+ * maximum name length is 255
+ */
+void qemu_put_string(QEMUFile *f, const char *name);
 #endif
diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
index 2d3ba23..3c610af 100644
--- a/migration/block-dirty-bitmap.c
+++ b/migration/block-dirty-bitmap.c
@@ -164,41 +164,6 @@ static void qemu_put_flags(QEMUFile *f, uint32_t flags)
     qemu_put_be32(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_32);
 }
 
-/* read name from qemu file:
- * format:
- * 1 byte : len = name length (<256)
- * len bytes : name without last zero byte
- *
- * name should point to the buffer >= 256 bytes length
- */
-static char *qemu_get_string(QEMUFile *f, char *name)
-{
-    int len = qemu_get_byte(f);
-    qemu_get_buffer(f, (uint8_t *)name, len);
-    name[len] = '\0';
-
-    DPRINTF("get name: %d %s\n", len, name);
-
-    return name;
-}
-
-/* write name to qemu file:
- * format:
- * same as for qemu_get_string
- *
- * maximum name length is 255
- */
-static void qemu_put_string(QEMUFile *f, const char *name)
-{
-    int len = strlen(name);
-
-    DPRINTF("put name: %d %s\n", len, name);
-
-    assert(len < 256);
-    qemu_put_byte(f, len);
-    qemu_put_buffer(f, (const uint8_t *)name, len);
-}
-
 static void send_bitmap_header(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
                                uint32_t additional_flags)
 {
diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 2750365..68da16f 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -585,3 +585,21 @@ int qemu_put_qemu_file(QEMUFile *f_des, QEMUFile *f_src)
     }
     return len;
 }
+
+char *qemu_get_string(QEMUFile *f, char *name)
+{
+    int len = qemu_get_byte(f);
+    qemu_get_buffer(f, (uint8_t *)name, len);
+    name[len] = '\0';
+
+    return name;
+}
+
+void qemu_put_string(QEMUFile *f, const char *name)
+{
+    int len = strlen(name);
+
+    assert(len < 256);
+    qemu_put_byte(f, len);
+    qemu_put_buffer(f, (const uint8_t *)name, len);
+}
-- 
1.9.1

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Vladimir Sementsov-Ogievskiy]

Hi all.

Hmm. There is an interesting suggestion from Denis Lunev (in CC) about 
how to drop meta bitmaps and make things easer.

method:

 > start migration
disk and memory are migrated, but not dirty bitmaps.
 > stop vm
create all necessary bitmaps in destination vm (empty, but with same 
names and granularities and enabled flag)
 > start destination vm
empty bitmaps are tracking now
 > start migrating dirty bitmaps. merge them to corresponding bitmaps in 
destination
while bitmaps are migrating, they should be in some kind of 
'inconsistent' state.
so, we can't start backup or other migration while bitmaps are 
migrating, but vm is already _running_ on destination.

what do you think about it?

On 13.05.2015 18:29, Vladimir Sementsov-Ogievskiy wrote:
> These patches provide dirty bitmap migration feature. Only named dirty
> bitmaps are to be migrated. Migration may be enabled using migration
> capabilities.
>
> v5:
>      - rebase on master
>      - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
>      - remove rfc, as incremental backup series by John Snow are in
>        upstream
>      
>
> v4 significant changes:
>   0001: tiny bugfix: out[i] -> out[i-start], same for 'in'
>   0007: set chunk size to 1kb, disable live iteration for
>         migrating data < 1mb size.
>
>   tests: only one with md5 sum is here. used function event_wait
>          by John Snow. (I hope, you don't mind me just adding this
>          function with your 'Signed-off-by')
>
>   rfc: This patch set is based on v13 of
>        "block: incremental backup series" by John Snow, which are
>        not pushed yet.
>
> v3:
>   based on v13 of "block: incremental backup series" by John Snow.
>
>   changes from v2:
>   removed patch for adding dirty parameter (migration capablities used
>   instead).
>   
>   0001: printf's dropped, qapi used
>   0002: part0 -> zeroes
>   0003: part0 -> zeroes
>   0005: dirty_dirty -> meta
>         add comments about meta bitmap
>         
>   0006: the format is changed, nodes used instead of devices.
>
>   other patches are new.
>
>   rfc: there are two tests. They are the same but using different
>   interfaces: md5 checksum of the bitmap last layer in query-block or
>   separate query-block-dirty-bitmap with dirty bitmap regions.
>   The second form is more appropriate for debugging, the first is more
>   appropriate for simple regression control. Which should go to
>   upstream?
>
> v2:
>   1. bug-fixes, that are already in upstream, and renaming of function
>   bdrv_reset_dirty_bitmap (which is already in Snow's series) are
>   dropped
>   2. bitmap store/restore: the concept renamed to serialization, added
>   function hbitmap_deserialize_part0, to not transfer zero blocks
>   3. migration dirty parameter: added description comment
>   4. Other patches are new.
>
> v2.rfc:
> Actually, in this version of the series I'm trying not use
> migration/block.c at all. Instead a separate migration unit is added
> in the new file migration/dirty-bitmap.c. Now bitmaps are migrated
> like blocks in block migration, they have their "dirty-dirty" bitmaps,
> for tracking set/unset changes during migration.
>
> The advantages are:
>    - no complications of migration/block.c
>    - separate dirty-dirty bitmaps provide handling of "unset's"
>    - more effective meta-data/data ratio - no tiny bitmap-blocks.
>
>
>
> v1:
> These patches provide dirty bitmap migration feature. Only named dirty
> bitmaps are to be migrated. Migration is made as a part of block
> migration in block-migration.c.
>
> Dirty bitmap migration may be enabled by "dirty" parameter for qmp migrate
> command. If "blk" and "inc" parameters are false when "dirty" is true
> block migration is actually skipped: no allocatoions, no bdrv_read's,
> no bdrv_write's, only bitmaps are migrated.
>
> The patch set includes two my previous bug fixes, which are necessary
> for it. The patch set is based on Incremental backup series by John
> Snow.
> Vladimir Sementsov-Ogievskiy (12):
>    hbitmap: serialization
>    block: BdrvDirtyBitmap serialization interface
>    block: tiny refactoring: minimize hbitmap_(set/reset) usage
>    block: add meta bitmaps
>    block: add bdrv_next_dirty_bitmap()
>    qapi: add dirty-bitmaps migration capability
>    migration: add migration/block-dirty-bitmap.c
>    iotests: maintain several vms in test
>    iotests: add add_incoming_migration to VM class
>    qapi: add md5 checksum of last dirty bitmap level to query-block
>    iotests: add dirty bitmap migration test
>    migration/qemu-file: make functions qemu_(get/put)_string public
>
>   block.c                        |  98 +++++-
>   include/block/block.h          |  22 ++
>   include/migration/block.h      |   1 +
>   include/migration/migration.h  |   1 +
>   include/migration/qemu-file.h  |  17 +
>   include/qemu/hbitmap.h         |  67 ++++
>   migration/Makefile.objs        |   2 +-
>   migration/block-dirty-bitmap.c | 693 +++++++++++++++++++++++++++++++++++++++++
>   migration/migration.c          |   9 +
>   migration/qemu-file.c          |  18 ++
>   qapi-schema.json               |   4 +-
>   qapi/block-core.json           |   4 +-
>   tests/qemu-iotests/117         |  84 +++++
>   tests/qemu-iotests/117.out     |   5 +
>   tests/qemu-iotests/group       |   1 +
>   tests/qemu-iotests/iotests.py  |  14 +-
>   util/hbitmap.c                 | 106 +++++++
>   vl.c                           |   1 +
>   18 files changed, 1137 insertions(+), 10 deletions(-)
>   create mode 100644 migration/block-dirty-bitmap.c
>   create mode 100755 tests/qemu-iotests/117
>   create mode 100644 tests/qemu-iotests/117.out
>


-- 
Best regards,
Vladimir

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Denis V. Lunev]

On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
> Hi all.
>
> Hmm. There is an interesting suggestion from Denis Lunev (in CC) about 
> how to drop meta bitmaps and make things easer.
>
> method:
>
> > start migration
> disk and memory are migrated, but not dirty bitmaps.
> > stop vm
> create all necessary bitmaps in destination vm (empty, but with same 
> names and granularities and enabled flag)
> > start destination vm
> empty bitmaps are tracking now
> > start migrating dirty bitmaps. merge them to corresponding bitmaps 
> in destination
> while bitmaps are migrating, they should be in some kind of 
> 'inconsistent' state.
> so, we can't start backup or other migration while bitmaps are 
> migrating, but vm is already _running_ on destination.
>
> what do you think about it?
>

the description is a bit incorrect

- start migration process, perform memory and disk migration
    as usual. VM is still executed at source
- start VM on target. VM on source should be on pause as usual,
    do not finish migration process. Running VM on target "writes"
    normally setting dirty bits as usual
- copy active dirty bitmaps from source to target. This is safe
    as VM on source is not running
- "OR" copied bitmaps with ones running on target
- finish migration process (stop source VM).

Downtime will not be increased due to dirty bitmaps with this
approach, migration process is very simple - plain data copy.

Regards,
     Den

> On 13.05.2015 18:29, Vladimir Sementsov-Ogievskiy wrote:
>> These patches provide dirty bitmap migration feature. Only named dirty
>> bitmaps are to be migrated. Migration may be enabled using migration
>> capabilities.
>>
>> v5:
>>      - rebase on master
>>      - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
>>      - remove rfc, as incremental backup series by John Snow are in
>>        upstream
>>
>> v4 significant changes:
>>   0001: tiny bugfix: out[i] -> out[i-start], same for 'in'
>>   0007: set chunk size to 1kb, disable live iteration for
>>         migrating data < 1mb size.
>>
>>   tests: only one with md5 sum is here. used function event_wait
>>          by John Snow. (I hope, you don't mind me just adding this
>>          function with your 'Signed-off-by')
>>
>>   rfc: This patch set is based on v13 of
>>        "block: incremental backup series" by John Snow, which are
>>        not pushed yet.
>>
>> v3:
>>   based on v13 of "block: incremental backup series" by John Snow.
>>
>>   changes from v2:
>>   removed patch for adding dirty parameter (migration capablities used
>>   instead).
>>     0001: printf's dropped, qapi used
>>   0002: part0 -> zeroes
>>   0003: part0 -> zeroes
>>   0005: dirty_dirty -> meta
>>         add comments about meta bitmap
>>           0006: the format is changed, nodes used instead of devices.
>>
>>   other patches are new.
>>
>>   rfc: there are two tests. They are the same but using different
>>   interfaces: md5 checksum of the bitmap last layer in query-block or
>>   separate query-block-dirty-bitmap with dirty bitmap regions.
>>   The second form is more appropriate for debugging, the first is more
>>   appropriate for simple regression control. Which should go to
>>   upstream?
>>
>> v2:
>>   1. bug-fixes, that are already in upstream, and renaming of function
>>   bdrv_reset_dirty_bitmap (which is already in Snow's series) are
>>   dropped
>>   2. bitmap store/restore: the concept renamed to serialization, added
>>   function hbitmap_deserialize_part0, to not transfer zero blocks
>>   3. migration dirty parameter: added description comment
>>   4. Other patches are new.
>>
>> v2.rfc:
>> Actually, in this version of the series I'm trying not use
>> migration/block.c at all. Instead a separate migration unit is added
>> in the new file migration/dirty-bitmap.c. Now bitmaps are migrated
>> like blocks in block migration, they have their "dirty-dirty" bitmaps,
>> for tracking set/unset changes during migration.
>>
>> The advantages are:
>>    - no complications of migration/block.c
>>    - separate dirty-dirty bitmaps provide handling of "unset's"
>>    - more effective meta-data/data ratio - no tiny bitmap-blocks.
>>
>>
>>
>> v1:
>> These patches provide dirty bitmap migration feature. Only named dirty
>> bitmaps are to be migrated. Migration is made as a part of block
>> migration in block-migration.c.
>>
>> Dirty bitmap migration may be enabled by "dirty" parameter for qmp 
>> migrate
>> command. If "blk" and "inc" parameters are false when "dirty" is true
>> block migration is actually skipped: no allocatoions, no bdrv_read's,
>> no bdrv_write's, only bitmaps are migrated.
>>
>> The patch set includes two my previous bug fixes, which are necessary
>> for it. The patch set is based on Incremental backup series by John
>> Snow.
>> Vladimir Sementsov-Ogievskiy (12):
>>    hbitmap: serialization
>>    block: BdrvDirtyBitmap serialization interface
>>    block: tiny refactoring: minimize hbitmap_(set/reset) usage
>>    block: add meta bitmaps
>>    block: add bdrv_next_dirty_bitmap()
>>    qapi: add dirty-bitmaps migration capability
>>    migration: add migration/block-dirty-bitmap.c
>>    iotests: maintain several vms in test
>>    iotests: add add_incoming_migration to VM class
>>    qapi: add md5 checksum of last dirty bitmap level to query-block
>>    iotests: add dirty bitmap migration test
>>    migration/qemu-file: make functions qemu_(get/put)_string public
>>
>>   block.c                        |  98 +++++-
>>   include/block/block.h          |  22 ++
>>   include/migration/block.h      |   1 +
>>   include/migration/migration.h  |   1 +
>>   include/migration/qemu-file.h  |  17 +
>>   include/qemu/hbitmap.h         |  67 ++++
>>   migration/Makefile.objs        |   2 +-
>>   migration/block-dirty-bitmap.c | 693 
>> +++++++++++++++++++++++++++++++++++++++++
>>   migration/migration.c          |   9 +
>>   migration/qemu-file.c          |  18 ++
>>   qapi-schema.json               |   4 +-
>>   qapi/block-core.json           |   4 +-
>>   tests/qemu-iotests/117         |  84 +++++
>>   tests/qemu-iotests/117.out     |   5 +
>>   tests/qemu-iotests/group       |   1 +
>>   tests/qemu-iotests/iotests.py  |  14 +-
>>   util/hbitmap.c                 | 106 +++++++
>>   vl.c                           |   1 +
>>   18 files changed, 1137 insertions(+), 10 deletions(-)
>>   create mode 100644 migration/block-dirty-bitmap.c
>>   create mode 100755 tests/qemu-iotests/117
>>   create mode 100644 tests/qemu-iotests/117.out
>>
>
>

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[John Snow]

On 05/21/2015 09:57 AM, Denis V. Lunev wrote:
> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
>> Hi all.
>>
>> Hmm. There is an interesting suggestion from Denis Lunev (in CC) about
>> how to drop meta bitmaps and make things easer.
>>
>> method:
>>
>> > start migration
>> disk and memory are migrated, but not dirty bitmaps.
>> > stop vm
>> create all necessary bitmaps in destination vm (empty, but with same
>> names and granularities and enabled flag)
>> > start destination vm
>> empty bitmaps are tracking now
>> > start migrating dirty bitmaps. merge them to corresponding bitmaps
>> in destination
>> while bitmaps are migrating, they should be in some kind of
>> 'inconsistent' state.
>> so, we can't start backup or other migration while bitmaps are
>> migrating, but vm is already _running_ on destination.
>>
>> what do you think about it?
>>
> 
> the description is a bit incorrect
> 
> - start migration process, perform memory and disk migration
>    as usual. VM is still executed at source
> - start VM on target. VM on source should be on pause as usual,
>    do not finish migration process. Running VM on target "writes"
>    normally setting dirty bits as usual
> - copy active dirty bitmaps from source to target. This is safe
>    as VM on source is not running
> - "OR" copied bitmaps with ones running on target
> - finish migration process (stop source VM).
> 
> Downtime will not be increased due to dirty bitmaps with this
> approach, migration process is very simple - plain data copy.
> 
> Regards,
>     Den
> 

I was actually just discussing the live migration approach a little bit
ago with Stefan, trying to decide on the "right" packet format (The only
two patches I haven't ACKed yet are ones in which we need to choose a
send size) and we decided that 1KiB chunk sends would be appropriate for
live migration.

I think I'm okay with that method, but obviously this approach outlined
here would also work very well and would avoid meta bitmaps, chunk
sizes, migration tuning, convergence questions, etc etc etc.

You'd need to add a new status to the bitmap on the target (maybe
"INCOMPLETE" or "MIGRATING") that prevents it from being used for a
backup operation without preventing it from recording new writes.

My only concern is how easy it will be to work this into the migration
workflow.

It would require some sort of "post-migration" ternary phase, I suppose,
for devices/data that can be transferred after the VM starts -- and I
suspect we'll be the only use of that phase for now.

David, what are your thoughts, here? Would you prefer Vladimir and I
push forward on the live migration approach, or add a new post-hoc
phase? This approach might be simpler on the block layer, but I would be
rather upset if he scrapped his entire series for the second time for
another approach that also didn't get accepted.

--js

>> On 13.05.2015 18:29, Vladimir Sementsov-Ogievskiy wrote:
>>> These patches provide dirty bitmap migration feature. Only named dirty
>>> bitmaps are to be migrated. Migration may be enabled using migration
>>> capabilities.
>>>
>>> v5:
>>>      - rebase on master
>>>      - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
>>>      - remove rfc, as incremental backup series by John Snow are in
>>>        upstream
>>>
>>> v4 significant changes:
>>>   0001: tiny bugfix: out[i] -> out[i-start], same for 'in'
>>>   0007: set chunk size to 1kb, disable live iteration for
>>>         migrating data < 1mb size.
>>>
>>>   tests: only one with md5 sum is here. used function event_wait
>>>          by John Snow. (I hope, you don't mind me just adding this
>>>          function with your 'Signed-off-by')
>>>
>>>   rfc: This patch set is based on v13 of
>>>        "block: incremental backup series" by John Snow, which are
>>>        not pushed yet.
>>>
>>> v3:
>>>   based on v13 of "block: incremental backup series" by John Snow.
>>>
>>>   changes from v2:
>>>   removed patch for adding dirty parameter (migration capablities used
>>>   instead).
>>>     0001: printf's dropped, qapi used
>>>   0002: part0 -> zeroes
>>>   0003: part0 -> zeroes
>>>   0005: dirty_dirty -> meta
>>>         add comments about meta bitmap
>>>           0006: the format is changed, nodes used instead of devices.
>>>
>>>   other patches are new.
>>>
>>>   rfc: there are two tests. They are the same but using different
>>>   interfaces: md5 checksum of the bitmap last layer in query-block or
>>>   separate query-block-dirty-bitmap with dirty bitmap regions.
>>>   The second form is more appropriate for debugging, the first is more
>>>   appropriate for simple regression control. Which should go to
>>>   upstream?
>>>
>>> v2:
>>>   1. bug-fixes, that are already in upstream, and renaming of function
>>>   bdrv_reset_dirty_bitmap (which is already in Snow's series) are
>>>   dropped
>>>   2. bitmap store/restore: the concept renamed to serialization, added
>>>   function hbitmap_deserialize_part0, to not transfer zero blocks
>>>   3. migration dirty parameter: added description comment
>>>   4. Other patches are new.
>>>
>>> v2.rfc:
>>> Actually, in this version of the series I'm trying not use
>>> migration/block.c at all. Instead a separate migration unit is added
>>> in the new file migration/dirty-bitmap.c. Now bitmaps are migrated
>>> like blocks in block migration, they have their "dirty-dirty" bitmaps,
>>> for tracking set/unset changes during migration.
>>>
>>> The advantages are:
>>>    - no complications of migration/block.c
>>>    - separate dirty-dirty bitmaps provide handling of "unset's"
>>>    - more effective meta-data/data ratio - no tiny bitmap-blocks.
>>>
>>>
>>>
>>> v1:
>>> These patches provide dirty bitmap migration feature. Only named dirty
>>> bitmaps are to be migrated. Migration is made as a part of block
>>> migration in block-migration.c.
>>>
>>> Dirty bitmap migration may be enabled by "dirty" parameter for qmp
>>> migrate
>>> command. If "blk" and "inc" parameters are false when "dirty" is true
>>> block migration is actually skipped: no allocatoions, no bdrv_read's,
>>> no bdrv_write's, only bitmaps are migrated.
>>>
>>> The patch set includes two my previous bug fixes, which are necessary
>>> for it. The patch set is based on Incremental backup series by John
>>> Snow.
>>> Vladimir Sementsov-Ogievskiy (12):
>>>    hbitmap: serialization
>>>    block: BdrvDirtyBitmap serialization interface
>>>    block: tiny refactoring: minimize hbitmap_(set/reset) usage
>>>    block: add meta bitmaps
>>>    block: add bdrv_next_dirty_bitmap()
>>>    qapi: add dirty-bitmaps migration capability
>>>    migration: add migration/block-dirty-bitmap.c
>>>    iotests: maintain several vms in test
>>>    iotests: add add_incoming_migration to VM class
>>>    qapi: add md5 checksum of last dirty bitmap level to query-block
>>>    iotests: add dirty bitmap migration test
>>>    migration/qemu-file: make functions qemu_(get/put)_string public
>>>
>>>   block.c                        |  98 +++++-
>>>   include/block/block.h          |  22 ++
>>>   include/migration/block.h      |   1 +
>>>   include/migration/migration.h  |   1 +
>>>   include/migration/qemu-file.h  |  17 +
>>>   include/qemu/hbitmap.h         |  67 ++++
>>>   migration/Makefile.objs        |   2 +-
>>>   migration/block-dirty-bitmap.c | 693
>>> +++++++++++++++++++++++++++++++++++++++++
>>>   migration/migration.c          |   9 +
>>>   migration/qemu-file.c          |  18 ++
>>>   qapi-schema.json               |   4 +-
>>>   qapi/block-core.json           |   4 +-
>>>   tests/qemu-iotests/117         |  84 +++++
>>>   tests/qemu-iotests/117.out     |   5 +
>>>   tests/qemu-iotests/group       |   1 +
>>>   tests/qemu-iotests/iotests.py  |  14 +-
>>>   util/hbitmap.c                 | 106 +++++++
>>>   vl.c                           |   1 +
>>>   18 files changed, 1137 insertions(+), 10 deletions(-)
>>>   create mode 100644 migration/block-dirty-bitmap.c
>>>   create mode 100755 tests/qemu-iotests/117
>>>   create mode 100644 tests/qemu-iotests/117.out
>>>
>>
>>
> 

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Denis V. Lunev]

On 21/05/15 19:44, John Snow wrote:
>
> On 05/21/2015 09:57 AM, Denis V. Lunev wrote:
>> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
>>> Hi all.
>>>
>>> Hmm. There is an interesting suggestion from Denis Lunev (in CC) about
>>> how to drop meta bitmaps and make things easer.
>>>
>>> method:
>>>
>>>> start migration
>>> disk and memory are migrated, but not dirty bitmaps.
>>>> stop vm
>>> create all necessary bitmaps in destination vm (empty, but with same
>>> names and granularities and enabled flag)
>>>> start destination vm
>>> empty bitmaps are tracking now
>>>> start migrating dirty bitmaps. merge them to corresponding bitmaps
>>> in destination
>>> while bitmaps are migrating, they should be in some kind of
>>> 'inconsistent' state.
>>> so, we can't start backup or other migration while bitmaps are
>>> migrating, but vm is already _running_ on destination.
>>>
>>> what do you think about it?
>>>
>> the description is a bit incorrect
>>
>> - start migration process, perform memory and disk migration
>>     as usual. VM is still executed at source
>> - start VM on target. VM on source should be on pause as usual,
>>     do not finish migration process. Running VM on target "writes"
>>     normally setting dirty bits as usual
>> - copy active dirty bitmaps from source to target. This is safe
>>     as VM on source is not running
>> - "OR" copied bitmaps with ones running on target
>> - finish migration process (stop source VM).
>>
>> Downtime will not be increased due to dirty bitmaps with this
>> approach, migration process is very simple - plain data copy.
>>
>> Regards,
>>      Den
>>
> I was actually just discussing the live migration approach a little bit
> ago with Stefan, trying to decide on the "right" packet format (The only
> two patches I haven't ACKed yet are ones in which we need to choose a
> send size) and we decided that 1KiB chunk sends would be appropriate for
> live migration.
>
> I think I'm okay with that method, but obviously this approach outlined
> here would also work very well and would avoid meta bitmaps, chunk
> sizes, migration tuning, convergence questions, etc etc etc.
>
> You'd need to add a new status to the bitmap on the target (maybe
> "INCOMPLETE" or "MIGRATING") that prevents it from being used for a
> backup operation without preventing it from recording new writes.
>
> My only concern is how easy it will be to work this into the migration
> workflow.
>
> It would require some sort of "post-migration" ternary phase, I suppose,
> for devices/data that can be transferred after the VM starts -- and I
> suspect we'll be the only use of that phase for now.
>
> David, what are your thoughts, here? Would you prefer Vladimir and I
> push forward on the live migration approach, or add a new post-hoc
> phase? This approach might be simpler on the block layer, but I would be
> rather upset if he scrapped his entire series for the second time for
> another approach that also didn't get accepted.
>
> --js

hmmm.... It looks like we should proceed with this to fit 2.4 dates.
There is not much interest at the moment. I think that we could
implement this later in 2.5 etc...

Regards,
     Den

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Denis V. Lunev]

On 26/05/15 17:48, Denis V. Lunev wrote:
> On 21/05/15 19:44, John Snow wrote:
>>
>> On 05/21/2015 09:57 AM, Denis V. Lunev wrote:
>>> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
>>>> Hi all.
>>>>
>>>> Hmm. There is an interesting suggestion from Denis Lunev (in CC) about
>>>> how to drop meta bitmaps and make things easer.
>>>>
>>>> method:
>>>>
>>>>> start migration
>>>> disk and memory are migrated, but not dirty bitmaps.
>>>>> stop vm
>>>> create all necessary bitmaps in destination vm (empty, but with same
>>>> names and granularities and enabled flag)
>>>>> start destination vm
>>>> empty bitmaps are tracking now
>>>>> start migrating dirty bitmaps. merge them to corresponding bitmaps
>>>> in destination
>>>> while bitmaps are migrating, they should be in some kind of
>>>> 'inconsistent' state.
>>>> so, we can't start backup or other migration while bitmaps are
>>>> migrating, but vm is already _running_ on destination.
>>>>
>>>> what do you think about it?
>>>>
>>> the description is a bit incorrect
>>>
>>> - start migration process, perform memory and disk migration
>>>     as usual. VM is still executed at source
>>> - start VM on target. VM on source should be on pause as usual,
>>>     do not finish migration process. Running VM on target "writes"
>>>     normally setting dirty bits as usual
>>> - copy active dirty bitmaps from source to target. This is safe
>>>     as VM on source is not running
>>> - "OR" copied bitmaps with ones running on target
>>> - finish migration process (stop source VM).
>>>
>>> Downtime will not be increased due to dirty bitmaps with this
>>> approach, migration process is very simple - plain data copy.
>>>
>>> Regards,
>>>      Den
>>>
>> I was actually just discussing the live migration approach a little bit
>> ago with Stefan, trying to decide on the "right" packet format (The only
>> two patches I haven't ACKed yet are ones in which we need to choose a
>> send size) and we decided that 1KiB chunk sends would be appropriate for
>> live migration.
>>
>> I think I'm okay with that method, but obviously this approach outlined
>> here would also work very well and would avoid meta bitmaps, chunk
>> sizes, migration tuning, convergence questions, etc etc etc.
>>
>> You'd need to add a new status to the bitmap on the target (maybe
>> "INCOMPLETE" or "MIGRATING") that prevents it from being used for a
>> backup operation without preventing it from recording new writes.
>>
>> My only concern is how easy it will be to work this into the migration
>> workflow.
>>
>> It would require some sort of "post-migration" ternary phase, I suppose,
>> for devices/data that can be transferred after the VM starts -- and I
>> suspect we'll be the only use of that phase for now.
>>
>> David, what are your thoughts, here? Would you prefer Vladimir and I
>> push forward on the live migration approach, or add a new post-hoc
>> phase? This approach might be simpler on the block layer, but I would be
>> rather upset if he scrapped his entire series for the second time for
>> another approach that also didn't get accepted.
>>
>> --js
>
> hmmm.... It looks like we should proceed with this to fit 2.4 dates.
> There is not much interest at the moment. I think that we could
> implement this later in 2.5 etc...
>
> Regards,
>     Den

oops. I have written something strange. Anyway, I think that for
now we should proceed with this patchset to fit QEMU 2.4 dates.
The implementation with additional stage (my proposal) could be
added later, f.e. in 2.5 as I do not see much interest from migration
gurus.

In this case the review will take a ... lot of time.

Regards,
     Den

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[John Snow]

On 05/26/2015 10:51 AM, Denis V. Lunev wrote:
> On 26/05/15 17:48, Denis V. Lunev wrote:
>> On 21/05/15 19:44, John Snow wrote:
>>>
>>> On 05/21/2015 09:57 AM, Denis V. Lunev wrote:
>>>> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
>>>>> Hi all.
>>>>>
>>>>> Hmm. There is an interesting suggestion from Denis Lunev (in CC) about
>>>>> how to drop meta bitmaps and make things easer.
>>>>>
>>>>> method:
>>>>>
>>>>>> start migration
>>>>> disk and memory are migrated, but not dirty bitmaps.
>>>>>> stop vm
>>>>> create all necessary bitmaps in destination vm (empty, but with same
>>>>> names and granularities and enabled flag)
>>>>>> start destination vm
>>>>> empty bitmaps are tracking now
>>>>>> start migrating dirty bitmaps. merge them to corresponding bitmaps
>>>>> in destination
>>>>> while bitmaps are migrating, they should be in some kind of
>>>>> 'inconsistent' state.
>>>>> so, we can't start backup or other migration while bitmaps are
>>>>> migrating, but vm is already _running_ on destination.
>>>>>
>>>>> what do you think about it?
>>>>>
>>>> the description is a bit incorrect
>>>>
>>>> - start migration process, perform memory and disk migration
>>>>     as usual. VM is still executed at source
>>>> - start VM on target. VM on source should be on pause as usual,
>>>>     do not finish migration process. Running VM on target "writes"
>>>>     normally setting dirty bits as usual
>>>> - copy active dirty bitmaps from source to target. This is safe
>>>>     as VM on source is not running
>>>> - "OR" copied bitmaps with ones running on target
>>>> - finish migration process (stop source VM).
>>>>
>>>> Downtime will not be increased due to dirty bitmaps with this
>>>> approach, migration process is very simple - plain data copy.
>>>>
>>>> Regards,
>>>>      Den
>>>>
>>> I was actually just discussing the live migration approach a little bit
>>> ago with Stefan, trying to decide on the "right" packet format (The only
>>> two patches I haven't ACKed yet are ones in which we need to choose a
>>> send size) and we decided that 1KiB chunk sends would be appropriate for
>>> live migration.
>>>
>>> I think I'm okay with that method, but obviously this approach outlined
>>> here would also work very well and would avoid meta bitmaps, chunk
>>> sizes, migration tuning, convergence questions, etc etc etc.
>>>
>>> You'd need to add a new status to the bitmap on the target (maybe
>>> "INCOMPLETE" or "MIGRATING") that prevents it from being used for a
>>> backup operation without preventing it from recording new writes.
>>>
>>> My only concern is how easy it will be to work this into the migration
>>> workflow.
>>>
>>> It would require some sort of "post-migration" ternary phase, I suppose,
>>> for devices/data that can be transferred after the VM starts -- and I
>>> suspect we'll be the only use of that phase for now.
>>>
>>> David, what are your thoughts, here? Would you prefer Vladimir and I
>>> push forward on the live migration approach, or add a new post-hoc
>>> phase? This approach might be simpler on the block layer, but I would be
>>> rather upset if he scrapped his entire series for the second time for
>>> another approach that also didn't get accepted.
>>>
>>> --js
>>
>> hmmm.... It looks like we should proceed with this to fit 2.4 dates.
>> There is not much interest at the moment. I think that we could
>> implement this later in 2.5 etc...
>>
>> Regards,
>>     Den
> 
> oops. I have written something strange. Anyway, I think that for
> now we should proceed with this patchset to fit QEMU 2.4 dates.
> The implementation with additional stage (my proposal) could be
> added later, f.e. in 2.5 as I do not see much interest from migration
> gurus.
> 
> In this case the review will take a ... lot of time.
> 
> Regards,
>     Den
> 

That sounds good to me. I think this solution is workable for 2.4, and
we can begin working on a post-migration phase for the future to help
simplify our cases a lot.

I have been out sick much of this week, so apologies in my lack of
fervor getting this series upstream recently.

--js

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Denis V. Lunev]

On 28/05/15 23:09, John Snow wrote:
>
> On 05/26/2015 10:51 AM, Denis V. Lunev wrote:
>> On 26/05/15 17:48, Denis V. Lunev wrote:
>>> On 21/05/15 19:44, John Snow wrote:
>>>> On 05/21/2015 09:57 AM, Denis V. Lunev wrote:
>>>>> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
>>>>>> Hi all.
>>>>>>
>>>>>> Hmm. There is an interesting suggestion from Denis Lunev (in CC) about
>>>>>> how to drop meta bitmaps and make things easer.
>>>>>>
>>>>>> method:
>>>>>>
>>>>>>> start migration
>>>>>> disk and memory are migrated, but not dirty bitmaps.
>>>>>>> stop vm
>>>>>> create all necessary bitmaps in destination vm (empty, but with same
>>>>>> names and granularities and enabled flag)
>>>>>>> start destination vm
>>>>>> empty bitmaps are tracking now
>>>>>>> start migrating dirty bitmaps. merge them to corresponding bitmaps
>>>>>> in destination
>>>>>> while bitmaps are migrating, they should be in some kind of
>>>>>> 'inconsistent' state.
>>>>>> so, we can't start backup or other migration while bitmaps are
>>>>>> migrating, but vm is already _running_ on destination.
>>>>>>
>>>>>> what do you think about it?
>>>>>>
>>>>> the description is a bit incorrect
>>>>>
>>>>> - start migration process, perform memory and disk migration
>>>>>      as usual. VM is still executed at source
>>>>> - start VM on target. VM on source should be on pause as usual,
>>>>>      do not finish migration process. Running VM on target "writes"
>>>>>      normally setting dirty bits as usual
>>>>> - copy active dirty bitmaps from source to target. This is safe
>>>>>      as VM on source is not running
>>>>> - "OR" copied bitmaps with ones running on target
>>>>> - finish migration process (stop source VM).
>>>>>
>>>>> Downtime will not be increased due to dirty bitmaps with this
>>>>> approach, migration process is very simple - plain data copy.
>>>>>
>>>>> Regards,
>>>>>       Den
>>>>>
>>>> I was actually just discussing the live migration approach a little bit
>>>> ago with Stefan, trying to decide on the "right" packet format (The only
>>>> two patches I haven't ACKed yet are ones in which we need to choose a
>>>> send size) and we decided that 1KiB chunk sends would be appropriate for
>>>> live migration.
>>>>
>>>> I think I'm okay with that method, but obviously this approach outlined
>>>> here would also work very well and would avoid meta bitmaps, chunk
>>>> sizes, migration tuning, convergence questions, etc etc etc.
>>>>
>>>> You'd need to add a new status to the bitmap on the target (maybe
>>>> "INCOMPLETE" or "MIGRATING") that prevents it from being used for a
>>>> backup operation without preventing it from recording new writes.
>>>>
>>>> My only concern is how easy it will be to work this into the migration
>>>> workflow.
>>>>
>>>> It would require some sort of "post-migration" ternary phase, I suppose,
>>>> for devices/data that can be transferred after the VM starts -- and I
>>>> suspect we'll be the only use of that phase for now.
>>>>
>>>> David, what are your thoughts, here? Would you prefer Vladimir and I
>>>> push forward on the live migration approach, or add a new post-hoc
>>>> phase? This approach might be simpler on the block layer, but I would be
>>>> rather upset if he scrapped his entire series for the second time for
>>>> another approach that also didn't get accepted.
>>>>
>>>> --js
>>> hmmm.... It looks like we should proceed with this to fit 2.4 dates.
>>> There is not much interest at the moment. I think that we could
>>> implement this later in 2.5 etc...
>>>
>>> Regards,
>>>      Den
>> oops. I have written something strange. Anyway, I think that for
>> now we should proceed with this patchset to fit QEMU 2.4 dates.
>> The implementation with additional stage (my proposal) could be
>> added later, f.e. in 2.5 as I do not see much interest from migration
>> gurus.
>>
>> In this case the review will take a ... lot of time.
>>
>> Regards,
>>      Den
>>
> That sounds good to me. I think this solution is workable for 2.4, and
> we can begin working on a post-migration phase for the future to help
> simplify our cases a lot.
>
> I have been out sick much of this week, so apologies in my lack of
> fervor getting this series upstream recently.
>
> --js
no prob :)

Re: [Qemu-devel] [PATCH 04/12] block: add meta bitmaps

[John Snow]

On 05/13/2015 11:29 AM, Vladimir Sementsov-Ogievskiy wrote:
> Meta bitmap is a 'dirty bitmap' for the BdrvDirtyBitmap. It tracks
> changes (set/unset) of this BdrvDirtyBitmap. It is needed for live
> migration of block dirty bitmaps.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
> ---
>  block.c               | 44 ++++++++++++++++++++++++++++++++++++++++++++
>  include/block/block.h |  7 +++++++
>  2 files changed, 51 insertions(+)
> 
> diff --git a/block.c b/block.c
> index 8376e8d..a167eb6 100644
> --- a/block.c
> +++ b/block.c
> @@ -58,9 +58,15 @@
>   * (3) successor is set: frozen mode.
>   *     A frozen bitmap cannot be renamed, deleted, anonymized, cleared, set,
>   *     or enabled. A frozen bitmap can only abdicate() or reclaim().
> + *
> + * Meta bitmap:
> + * Meta bitmap is a 'dirty bitmap' for the BdrvDirtyBitmap. It tracks changes
> + * (set/unset) of this BdrvDirtyBitmap. It is needed for live migration of
> + * block dirty bitmaps.
>   */
>  struct BdrvDirtyBitmap {
>      HBitmap *bitmap;            /* Dirty sector bitmap implementation */
> +    HBitmap *meta_bitmap;       /* Meta bitmap */
>      BdrvDirtyBitmap *successor; /* Anonymous child; implies frozen status */
>      char *name;                 /* Optional non-empty unique ID */
>      int64_t size;               /* Size of the bitmap (Number of sectors) */
> @@ -3062,6 +3068,35 @@ void bdrv_dirty_bitmap_make_anon(BdrvDirtyBitmap *bitmap)
>      bitmap->name = NULL;
>  }
>  
> +HBitmap *bdrv_create_meta_bitmap(BdrvDirtyBitmap *bitmap,
> +                                 uint64_t chunk_size)
> +{
> +    uint64_t sector_granularity;
> +
> +    assert((chunk_size & (chunk_size - 1)) == 0);
> +
> +    /* one chunk is corresponding to one bit of the meta bitmap, and each bit
> +     * of the chunk is corresponding to 'bdrv_dirty_bitmap_granularity(bitmap)'
> +     * bytes of the node */
> +    sector_granularity =
> +        (chunk_size * 8 * bdrv_dirty_bitmap_granularity(bitmap))
> +        >> BDRV_SECTOR_BITS;
> +    assert(sector_granularity);
> +
> +    bitmap->meta_bitmap =
> +        hbitmap_alloc(bitmap->size, ffsll(sector_granularity) - 1);
> +
> +    return bitmap->meta_bitmap;
> +}
> +
> +void bdrv_release_meta_bitmap(BdrvDirtyBitmap *bitmap)
> +{
> +    if (bitmap->meta_bitmap) {
> +        hbitmap_free(bitmap->meta_bitmap);
> +        bitmap->meta_bitmap = NULL;
> +    }
> +}
> +
>  BdrvDirtyBitmap *bdrv_create_dirty_bitmap(BlockDriverState *bs,
>                                            uint32_t granularity,
>                                            const char *name,
> @@ -3212,6 +3247,9 @@ void bdrv_release_dirty_bitmap(BlockDriverState *bs, BdrvDirtyBitmap *bitmap)
>              assert(!bdrv_dirty_bitmap_frozen(bm));
>              QLIST_REMOVE(bitmap, list);
>              hbitmap_free(bitmap->bitmap);
> +            if (bitmap->meta_bitmap) {
> +                hbitmap_free(bitmap->meta_bitmap);
> +            }
>              g_free(bitmap->name);
>              g_free(bitmap);
>              return;
> @@ -3297,6 +3335,9 @@ void bdrv_set_dirty_bitmap(BdrvDirtyBitmap *bitmap,
>  {
>      assert(bdrv_dirty_bitmap_enabled(bitmap));
>      hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
> +    if (bitmap->meta_bitmap) {
> +        hbitmap_set(bitmap->meta_bitmap, cur_sector, nr_sectors);
> +    }
>  }
>  
>  void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
> @@ -3304,6 +3345,9 @@ void bdrv_reset_dirty_bitmap(BdrvDirtyBitmap *bitmap,
>  {
>      assert(bdrv_dirty_bitmap_enabled(bitmap));
>      hbitmap_reset(bitmap->bitmap, cur_sector, nr_sectors);
> +    if (bitmap->meta_bitmap) {
> +        hbitmap_set(bitmap->meta_bitmap, cur_sector, nr_sectors);
> +    }
>  }
>  
>  void bdrv_clear_dirty_bitmap(BdrvDirtyBitmap *bitmap)
> diff --git a/include/block/block.h b/include/block/block.h
> index 1c16906..2d91161 100644
> --- a/include/block/block.h
> +++ b/include/block/block.h
> @@ -4,6 +4,7 @@
>  #include "block/aio.h"
>  #include "qemu-common.h"
>  #include "qemu/option.h"
> +#include "qemu/hbitmap.h"
>  #include "block/coroutine.h"
>  #include "block/accounting.h"
>  #include "qapi/qmp/qobject.h"
> @@ -497,6 +498,12 @@ void bdrv_dirty_bitmap_deserialize_zeroes(BdrvDirtyBitmap *bitmap,
>                                            uint64_t start, uint64_t count);
>  void bdrv_dirty_bitmap_deserialize_finish(BdrvDirtyBitmap *bitmap);
>  
> +/* chunk size here is number of bytes of the @bitmap data per one bit of the
> + * meta bitmap being created */
> +HBitmap *bdrv_create_meta_bitmap(BdrvDirtyBitmap *bitmap,
> +                                 uint64_t granularity);
> +void bdrv_release_meta_bitmap(BdrvDirtyBitmap *bitmap);
> +
>  void bdrv_enable_copy_on_read(BlockDriverState *bs);
>  void bdrv_disable_copy_on_read(BlockDriverState *bs);
>  
> 

Reviewed-by: John Snow <jsnow@redhat.com>

Re: [Qemu-devel] [PATCH 07/12] migration: add migration/block-dirty-bitmap.c

[John Snow]

On 05/13/2015 11:30 AM, Vladimir Sementsov-Ogievskiy wrote:
> Live migration of dirty bitmaps. Only named dirty bitmaps, associated with
> root nodes and non-root named nodes are migrated.
> 
> If destination qemu is already containing a dirty bitmap with the same name
> as a migrated bitmap (for the same node), than, if their granularities are
> the same the migration will be done, otherwise the error will be generated.
> 
> If destination qemu doesn't contain such bitmap it will be created.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
> ---
>  include/migration/block.h      |   1 +
>  migration/Makefile.objs        |   2 +-
>  migration/block-dirty-bitmap.c | 728 +++++++++++++++++++++++++++++++++++++++++
>  vl.c                           |   1 +
>  4 files changed, 731 insertions(+), 1 deletion(-)
>  create mode 100644 migration/block-dirty-bitmap.c
> 
> diff --git a/include/migration/block.h b/include/migration/block.h
> index ffa8ac0..566bb9f 100644
> --- a/include/migration/block.h
> +++ b/include/migration/block.h
> @@ -14,6 +14,7 @@
>  #ifndef BLOCK_MIGRATION_H
>  #define BLOCK_MIGRATION_H
>  
> +void dirty_bitmap_mig_init(void);
>  void blk_mig_init(void);
>  int blk_mig_active(void);
>  uint64_t blk_mig_bytes_transferred(void);
> diff --git a/migration/Makefile.objs b/migration/Makefile.objs
> index d929e96..128612d 100644
> --- a/migration/Makefile.objs
> +++ b/migration/Makefile.objs
> @@ -6,5 +6,5 @@ common-obj-y += xbzrle.o
>  common-obj-$(CONFIG_RDMA) += rdma.o
>  common-obj-$(CONFIG_POSIX) += exec.o unix.o fd.o
>  
> -common-obj-y += block.o
> +common-obj-y += block.o block-dirty-bitmap.o
>  
> diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
> new file mode 100644
> index 0000000..2d3ba23
> --- /dev/null
> +++ b/migration/block-dirty-bitmap.c
> @@ -0,0 +1,728 @@
> +/*
> + * QEMU dirty bitmap migration
> + *
> + * Live migration of dirty bitmaps. Only named dirty bitmaps, associated with
> + * root nodes and non-root named nodes are migrated. Live iteration is disabled
> + * for small data amount (see MIN_LIVE_SIZE).
> + *
> + * If destination qemu is already containing a dirty bitmap with the same name
> + * as a migrated bitmap (for the same node), than, if their granularities are
> + * the same the migration will be done, otherwise the error will be generated.
> + *
> + * If destination qemu doesn't contain such bitmap it will be created.
> + *
> + * format of migration:
> + *
> + * # Header (shared for different chunk types)
> + * 1, 2 or 4 bytes: flags (see qemu_{put,put}_flags)
> + * [ 1 byte: node name size ] \  flags & DEVICE_NAME
> + * [ n bytes: node name     ] /
> + * [ 1 byte: bitmap name size ] \  flags & BITMAP_NAME
> + * [ n bytes: bitmap name     ] /
> + *
> + * # Start of bitmap migration (flags & START)
> + * header
> + * be64: granularity
> + *
> + * # Complete of bitmap migration (flags & COMPLETE)
> + * header
> + * 1 byte: bitmap enabled flag
> + *
> + * # Data chunk of bitmap migration
> + * header
> + * be64: start sector
> + * be32: number of sectors
> + * [ be64: buffer size  ] \ ! (flags & ZEROES)
> + * [ n bytes: buffer    ] /
> + *
> + * The last chunk in stream should contain flags & EOS. The chunk may skip
> + * device and/or bitmap names, assuming them to be the same with the previous
> + * chunk.
> + *
> + *
> + * This file is derived from migration/block.c
> + *
> + * Author:
> + * Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
> + *
> + * original copyright message:
> + * =====================================================================
> + * Copyright IBM, Corp. 2009
> + *
> + * Authors:
> + *  Liran Schour   <lirans@il.ibm.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2.  See
> + * the COPYING file in the top-level directory.
> + *
> + * Contributions after 2012-01-13 are licensed under the terms of the
> + * GNU GPL, version 2 or (at your option) any later version.
> + * =====================================================================
> + */
> +
> +#include "block/block.h"
> +#include "block/block_int.h"
> +#include "sysemu/block-backend.h"
> +#include "qemu/main-loop.h"
> +#include "qemu/error-report.h"
> +#include "migration/block.h"
> +#include "migration/migration.h"
> +#include "qemu/hbitmap.h"
> +#include <assert.h>
> +
> +#define CHUNK_SIZE     (1 << 10)
> +#define MIN_LIVE_SIZE  (1 << 20)
> +
> +/* Flags occupy from one to four bytes. In all but one the 7-th (EXTRA_FLAGS)
> + * bit should be set. */
> +#define DIRTY_BITMAP_MIG_FLAG_EOS           0x01
> +#define DIRTY_BITMAP_MIG_FLAG_ZEROES        0x02
> +#define DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME   0x04
> +#define DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME   0x08
> +#define DIRTY_BITMAP_MIG_FLAG_START         0x10
> +#define DIRTY_BITMAP_MIG_FLAG_COMPLETE      0x20
> +#define DIRTY_BITMAP_MIG_FLAG_BITS          0x40
> +
> +#define DIRTY_BITMAP_MIG_EXTRA_FLAGS        0x80
> +#define DIRTY_BITMAP_MIG_FLAGS_SIZE_16      0x8000
> +#define DIRTY_BITMAP_MIG_FLAGS_SIZE_32      0x8080
> +
> +#define DEBUG_DIRTY_BITMAP_MIGRATION 1
> +

Should be 0 for the final check-in.

> +#define DPRINTF(fmt, args...) \
> +    do { \
> +        if (DEBUG_DIRTY_BITMAP_MIGRATION) { \
> +            printf("DMIG %s:%d", __func__, __LINE__); \
> +            printf(fmt, ##args); \
> +        } \
> +    } while (0)
> +
> +typedef struct DirtyBitmapMigBitmapState {
> +    /* Written during setup phase. */
> +    BlockDriverState *bs;
> +    const char *node_name;
> +    BdrvDirtyBitmap *bitmap;
> +    HBitmap *meta_bitmap;
> +    uint64_t total_sectors;
> +    uint64_t sectors_per_chunk;
> +    QSIMPLEQ_ENTRY(DirtyBitmapMigBitmapState) entry;
> +
> +    /* For bulk phase. */
> +    bool bulk_completed;
> +    uint64_t cur_sector;
> +
> +    /* For dirty phase. */
> +    HBitmapIter iter_dirty;
> +} DirtyBitmapMigBitmapState;
> +
> +typedef struct DirtyBitmapMigState {
> +    QSIMPLEQ_HEAD(dbms_list, DirtyBitmapMigBitmapState) dbms_list;
> +
> +    bool bulk_completed;
> +    bool is_live_iterative;
> +
> +    /* for send_bitmap_bits() */
> +    BlockDriverState *prev_bs;
> +    BdrvDirtyBitmap *prev_bitmap;
> +} DirtyBitmapMigState;
> +
> +typedef struct DirtyBitmapLoadState {
> +    uint32_t flags;
> +    char node_name[256];
> +    char bitmap_name[256];
> +    BlockDriverState *bs;
> +    BdrvDirtyBitmap *bitmap;
> +} DirtyBitmapLoadState;
> +
> +static DirtyBitmapMigState dirty_bitmap_mig_state;
> +
> +static uint32_t qemu_get_flags(QEMUFile *f)
> +{
> +    uint8_t flags = qemu_get_byte(f);
> +    if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
> +        flags = flags << 8 | qemu_get_byte(f);
> +        if (flags & DIRTY_BITMAP_MIG_EXTRA_FLAGS) {
> +            flags = flags << 16 | qemu_get_be16(f);
> +        }
> +    }
> +
> +    return flags;
> +}
> +
> +static void qemu_put_flags(QEMUFile *f, uint32_t flags)
> +{
> +    if (!(flags & 0xffffff00)) {
> +        qemu_put_byte(f, flags);
> +        return;
> +    }
> +
> +    if (!(flags & 0xffff0000)) {
> +        qemu_put_be16(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_16);
> +        return;
> +    }
> +
> +    qemu_put_be32(f, flags | DIRTY_BITMAP_MIG_FLAGS_SIZE_32);
> +}
> +
> +/* read name from qemu file:
> + * format:
> + * 1 byte : len = name length (<256)
> + * len bytes : name without last zero byte
> + *
> + * name should point to the buffer >= 256 bytes length
> + */
> +static char *qemu_get_string(QEMUFile *f, char *name)
> +{
> +    int len = qemu_get_byte(f);
> +    qemu_get_buffer(f, (uint8_t *)name, len);
> +    name[len] = '\0';
> +
> +    DPRINTF("get name: %d %s\n", len, name);
> +
> +    return name;
> +}
> +
> +/* write name to qemu file:
> + * format:
> + * same as for qemu_get_string
> + *
> + * maximum name length is 255
> + */
> +static void qemu_put_string(QEMUFile *f, const char *name)
> +{
> +    int len = strlen(name);
> +
> +    DPRINTF("put name: %d %s\n", len, name);
> +
> +    assert(len < 256);
> +    qemu_put_byte(f, len);
> +    qemu_put_buffer(f, (const uint8_t *)name, len);
> +}
> +
> +static void send_bitmap_header(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
> +                               uint32_t additional_flags)
> +{
> +    BlockDriverState *bs = dbms->bs;
> +    BdrvDirtyBitmap *bitmap = dbms->bitmap;
> +    uint32_t flags = additional_flags;
> +
> +    if (bs != dirty_bitmap_mig_state.prev_bs) {
> +        dirty_bitmap_mig_state.prev_bs = bs;
> +        flags |= DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME;
> +    }
> +
> +    if (bitmap != dirty_bitmap_mig_state.prev_bitmap) {
> +        dirty_bitmap_mig_state.prev_bitmap = bitmap;
> +        flags |= DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME;
> +    }
> +
> +    qemu_put_flags(f, flags);
> +
> +    if (flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
> +        qemu_put_string(f, dbms->node_name);
> +    }
> +
> +    if (flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
> +        qemu_put_string(f, bdrv_dirty_bitmap_name(bitmap));
> +    }
> +}
> +
> +static void send_bitmap_start(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
> +{
> +    send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_START);
> +    qemu_put_be32(f, bdrv_dirty_bitmap_granularity(dbms->bitmap));
> +}
> +
> +static void send_bitmap_complete(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
> +{
> +    send_bitmap_header(f, dbms, DIRTY_BITMAP_MIG_FLAG_COMPLETE);
> +    qemu_put_byte(f, bdrv_dirty_bitmap_enabled(dbms->bitmap));
> +}
> +
> +static void send_bitmap_bits(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
> +                             uint64_t start_sector, uint32_t nr_sectors)
> +{
> +    /* align for buffer_is_zero() */
> +    uint64_t align = 4 * sizeof(long);
> +    uint64_t buf_size =
> +        (bdrv_dirty_bitmap_data_size(dbms->bitmap, nr_sectors) + align - 1) &
> +        ~(align - 1);
> +    uint8_t *buf = g_malloc0(buf_size);
> +    uint32_t flags = DIRTY_BITMAP_MIG_FLAG_BITS;
> +
> +    bdrv_dirty_bitmap_serialize_part(dbms->bitmap, buf,
> +                                     start_sector, nr_sectors);
> +
> +    if (buffer_is_zero(buf, buf_size)) {
> +        g_free(buf);
> +        buf = NULL;
> +        flags |= DIRTY_BITMAP_MIG_FLAG_ZEROES;
> +    }
> +
> +    DPRINTF("parameters:"
> +            "\n   flags:        %x"
> +            "\n   start_sector: %" PRIu64
> +            "\n   nr_sectors:   %" PRIu32
> +            "\n   data_size:    %" PRIu64 "\n",
> +            flags, start_sector, nr_sectors, buf_size);
> +
> +    send_bitmap_header(f, dbms, flags);
> +
> +    qemu_put_be64(f, start_sector);
> +    qemu_put_be32(f, nr_sectors);
> +
> +    /* if a block is zero we need to flush here since the network
> +     * bandwidth is now a lot higher than the storage device bandwidth.
> +     * thus if we queue zero blocks we slow down the migration.
> +     * also, skip writing block when migrate only dirty bitmaps. */
> +    if (flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
> +        qemu_fflush(f);
> +        return;
> +    }
> +
> +    qemu_put_be64(f, buf_size);
> +    qemu_put_buffer(f, buf, buf_size);
> +    g_free(buf);
> +}
> +
> +
> +/* Called with iothread lock taken.  */
> +
> +static void set_dirty_tracking(void)
> +{
> +    DirtyBitmapMigBitmapState *dbms;
> +
> +    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> +        dbms->meta_bitmap =
> +            bdrv_create_meta_bitmap(dbms->bitmap, CHUNK_SIZE);
> +        dbms->sectors_per_chunk =
> +            UINT64_C(1) << hbitmap_granularity(dbms->meta_bitmap);
> +    }
> +}
> +
> +static void unset_dirty_tracking(void)
> +{
> +    DirtyBitmapMigBitmapState *dbms;
> +
> +    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> +        bdrv_release_meta_bitmap(dbms->bitmap);
> +    }
> +}
> +
> +static void init_dirty_bitmap_migration(void)
> +{
> +    BlockDriverState *bs;
> +    BdrvDirtyBitmap *bitmap;
> +    DirtyBitmapMigBitmapState *dbms;
> +    uint64_t total_bytes = 0;
> +
> +    dirty_bitmap_mig_state.bulk_completed = false;
> +    dirty_bitmap_mig_state.prev_bs = NULL;
> +    dirty_bitmap_mig_state.prev_bitmap = NULL;
> +
> +    for (bs = bdrv_next(NULL); bs; bs = bdrv_next(bs)) {
> +        for (bitmap = bdrv_next_dirty_bitmap(bs, NULL); bitmap;
> +             bitmap = bdrv_next_dirty_bitmap(bs, bitmap)) {
> +            if (!bdrv_dirty_bitmap_name(bitmap)) {
> +                continue;
> +            }
> +
> +            if (!bdrv_get_node_name(bs) && blk_bs(bs->blk) != bs) {
> +                /* not named non-root node */
> +                continue;
> +            }
> +
> +            dbms = g_new0(DirtyBitmapMigBitmapState, 1);
> +            dbms->bs = bs;
> +            dbms->node_name = bdrv_get_node_name(bs);
> +            if (!dbms->node_name || dbms->node_name[0] == '\0') {
> +                dbms->node_name = bdrv_get_device_name(bs);
> +            }
> +            dbms->bitmap = bitmap;
> +            dbms->total_sectors = bdrv_nb_sectors(bs);
> +            total_bytes +=
> +                bdrv_dirty_bitmap_data_size(bitmap, dbms->total_sectors);
> +
> +            QSIMPLEQ_INSERT_TAIL(&dirty_bitmap_mig_state.dbms_list,
> +                                 dbms, entry);
> +        }
> +    }
> +
> +    dirty_bitmap_mig_state.is_live_iterative = total_bytes > MIN_LIVE_SIZE;
> +}
> +
> +/* Called with no lock taken.  */
> +static void bulk_phase_send_chunk(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
> +{
> +    uint32_t nr_sectors = MIN(dbms->total_sectors - dbms->cur_sector,
> +                             dbms->sectors_per_chunk);
> +
> +    send_bitmap_bits(f, dbms, dbms->cur_sector, nr_sectors);
> +
> +    dbms->cur_sector += nr_sectors;
> +    if (dbms->cur_sector >= dbms->total_sectors) {
> +        dbms->bulk_completed = true;
> +    }
> +}
> +
> +/* Called with no lock taken.  */
> +static void bulk_phase(QEMUFile *f, bool limit)
> +{
> +    DirtyBitmapMigBitmapState *dbms;
> +
> +    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> +        while (!dbms->bulk_completed) {
> +            bulk_phase_send_chunk(f, dbms);
> +            if (limit && qemu_file_rate_limit(f)) {
> +                return;
> +            }
> +        }
> +    }
> +
> +    dirty_bitmap_mig_state.bulk_completed = true;
> +}
> +
> +static void blk_mig_reset_dirty_cursor(void)
> +{
> +    DirtyBitmapMigBitmapState *dbms;
> +
> +    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> +        hbitmap_iter_init(&dbms->iter_dirty, dbms->meta_bitmap, 0);
> +    }
> +}
> +
> +/* Called with iothread lock taken. */
> +static bool dirty_phase_send_chunk(QEMUFile *f, DirtyBitmapMigBitmapState *dbms)
> +{
> +    uint32_t nr_sectors;
> +    size_t old_pos = dbms->iter_dirty.pos;
> +    int64_t cur = hbitmap_iter_next(&dbms->iter_dirty);
> +
> +    /* restart search from the beginning */
> +    if (old_pos && cur == -1) {
> +        hbitmap_iter_init(&dbms->iter_dirty, dbms->meta_bitmap, 0);
> +        cur = hbitmap_iter_next(&dbms->iter_dirty);
> +    }
> +
> +    if (cur == -1) {
> +        hbitmap_iter_init(&dbms->iter_dirty, dbms->meta_bitmap, 0);
> +        return false;
> +    }
> +
> +    nr_sectors = MIN(dbms->total_sectors - cur, dbms->sectors_per_chunk);
> +    send_bitmap_bits(f, dbms, cur, nr_sectors);
> +    hbitmap_reset(dbms->meta_bitmap, cur, dbms->sectors_per_chunk);
> +
> +    return true;
> +}
> +
> +/* Called with iothread lock taken. */
> +static void dirty_phase(QEMUFile *f, bool limit)
> +{
> +    DirtyBitmapMigBitmapState *dbms;
> +
> +    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> +        while (dirty_phase_send_chunk(f, dbms)) {
> +            if (limit && qemu_file_rate_limit(f)) {
> +                return;
> +            }
> +        }
> +    }
> +}
> +
> +
> +/* Called with iothread lock taken.  */
> +static void dirty_bitmap_mig_cleanup(void)
> +{
> +    DirtyBitmapMigBitmapState *dbms;
> +
> +    unset_dirty_tracking();
> +
> +    while ((dbms = QSIMPLEQ_FIRST(&dirty_bitmap_mig_state.dbms_list)) != NULL) {
> +        QSIMPLEQ_REMOVE_HEAD(&dirty_bitmap_mig_state.dbms_list, entry);
> +        g_free(dbms);
> +    }
> +}
> +
> +static void dirty_bitmap_migration_cancel(void *opaque)
> +{
> +    dirty_bitmap_mig_cleanup();
> +}
> +
> +static int dirty_bitmap_save_iterate(QEMUFile *f, void *opaque)
> +{
> +    DPRINTF("enter\n");
> +
> +    if (dirty_bitmap_mig_state.bulk_completed) {
> +        qemu_mutex_lock_iothread();
> +        dirty_phase(f, true);
> +        qemu_mutex_unlock_iothread();
> +    } else {
> +        bulk_phase(f, true);
> +    }
> +
> +    qemu_put_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
> +
> +    return dirty_bitmap_mig_state.bulk_completed;
> +}
> +
> +/* Called with iothread lock taken.  */
> +
> +static int dirty_bitmap_save_complete(QEMUFile *f, void *opaque)
> +{
> +    DirtyBitmapMigBitmapState *dbms;
> +    DPRINTF("enter\n");
> +
> +    if (!dirty_bitmap_mig_state.bulk_completed) {
> +        bulk_phase(f, false);
> +    }
> +
> +    blk_mig_reset_dirty_cursor();
> +    dirty_phase(f, false);
> +
> +    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> +        send_bitmap_complete(f, dbms);
> +    }
> +
> +    qemu_put_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
> +
> +    DPRINTF("Dirty bitmaps migration completed\n");
> +
> +    dirty_bitmap_mig_cleanup();
> +    return 0;
> +}
> +
> +static uint64_t dirty_bitmap_save_pending(QEMUFile *f, void *opaque,
> +                                          uint64_t max_size)
> +{
> +    DirtyBitmapMigBitmapState *dbms;
> +    uint64_t pending = 0;
> +
> +    qemu_mutex_lock_iothread();
> +
> +    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> +        uint64_t sectors = hbitmap_count(dbms->meta_bitmap);
> +        if (!dbms->bulk_completed) {
> +            sectors += dbms->total_sectors - dbms->cur_sector;
> +        }
> +        pending += bdrv_dirty_bitmap_data_size(dbms->bitmap, sectors);
> +    }
> +
> +    qemu_mutex_unlock_iothread();
> +
> +    DPRINTF("pending %" PRIu64 ", max: %" PRIu64 "\n",
> +            pending, max_size);
> +    return pending;
> +}
> +
> +/* First occurrence of this bitmap. It should be created if doesn't exist */
> +static int dirty_bitmap_load_start(QEMUFile *f, DirtyBitmapLoadState *s)
> +{
> +    uint32_t granularity = qemu_get_be32(f);
> +    if (!s->bitmap) {
> +        Error *local_err = NULL;
> +        s->bitmap = bdrv_create_dirty_bitmap(s->bs, granularity,
> +                                             s->bitmap_name, &local_err);
> +        if (!s->bitmap) {
> +            error_report("%s", error_get_pretty(local_err));
> +            error_free(local_err);
> +            return -EINVAL;
> +        }
> +    } else {
> +        uint32_t dest_granularity =
> +            bdrv_dirty_bitmap_granularity(s->bitmap);
> +        if (dest_granularity != granularity) {
> +            fprintf(stderr,
> +                    "Error: "
> +                    "Migrated bitmap granularity (%" PRIu32 ") "
> +                    "doesn't match the destination bitmap '%s' "
> +                    "granularity (%" PRIu32 ")\n",
> +                    granularity,
> +                    bdrv_dirty_bitmap_name(s->bitmap),
> +                    dest_granularity);
> +            return -EINVAL;
> +        }
> +    }
> +
> +    bdrv_disable_dirty_bitmap(s->bitmap);
> +
> +    return 0;
> +}
> +
> +static void dirty_bitmap_load_complete(QEMUFile *f, DirtyBitmapLoadState *s)
> +{
> +    bool enabled;
> +
> +    bdrv_dirty_bitmap_deserialize_finish(s->bitmap);
> +
> +    enabled = qemu_get_byte(f);
> +    if (enabled) {
> +        bdrv_enable_dirty_bitmap(s->bitmap);
> +    }
> +}
> +
> +static int dirty_bitmap_load_bits(QEMUFile *f, DirtyBitmapLoadState *s)
> +{
> +    uint64_t first_sector = qemu_get_be64(f);
> +    uint32_t nr_sectors = qemu_get_be32(f);
> +    DPRINTF("chunk: %lu %u\n", first_sector, nr_sectors);

This will break 32 bit compilation, try "%"PRIu64 for first_sector instead.

> +
> +
> +    if (s->flags & DIRTY_BITMAP_MIG_FLAG_ZEROES) {
> +        DPRINTF("   - zeroes\n");
> +        bdrv_dirty_bitmap_deserialize_zeroes(s->bitmap, first_sector,
> +                                             nr_sectors);
> +    } else {
> +        uint8_t *buf;
> +        uint64_t buf_size = qemu_get_be64(f);
> +        uint64_t needed_size =
> +            bdrv_dirty_bitmap_data_size(s->bitmap, nr_sectors);
> +
> +        if (needed_size > buf_size) {
> +            fprintf(stderr,
> +                    "Error: Migrated bitmap granularity doesn't "
> +                    "match the destination bitmap '%s' granularity\n",
> +                    bdrv_dirty_bitmap_name(s->bitmap));
> +            return -EINVAL;
> +        }
> +
> +        buf = g_malloc(buf_size);
> +        qemu_get_buffer(f, buf, buf_size);
> +        bdrv_dirty_bitmap_deserialize_part(s->bitmap, buf,
> +                                           first_sector,
> +                                           nr_sectors);
> +        g_free(buf);
> +    }
> +
> +    return 0;
> +}
> +
> +static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s)
> +{
> +    Error *local_err = NULL;
> +    s->flags = qemu_get_flags(f);
> +    DPRINTF("flags: %x\n", s->flags);
> +
> +    if (s->flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
> +        qemu_get_string(f, s->node_name);
> +        s->bs = bdrv_lookup_bs(s->node_name, s->node_name, &local_err);
> +        if (!s->bs) {
> +            error_report("%s", error_get_pretty(local_err));
> +            error_free(local_err);
> +            return -EINVAL;
> +        }
> +    } else if (!s->bs) {
> +        fprintf(stderr, "Error: block device name is not set\n");
> +        return -EINVAL;
> +    }
> +
> +    if (s->flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
> +        qemu_get_string(f, s->bitmap_name);
> +        s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);
> +
> +        /* bitmap may be NULL here, it wouldn't be an error if it is the
> +         * first occurrence of the bitmap */
> +        if (!s->bitmap && !(s->flags & DIRTY_BITMAP_MIG_FLAG_START)) {
> +            fprintf(stderr, "Error: unknown dirty bitmap "
> +                    "'%s' for block device '%s'\n",
> +                    s->bitmap_name, s->node_name);
> +            return -EINVAL;
> +        }
> +    } else if (!s->bitmap) {
> +        fprintf(stderr, "Error: block device name is not set\n");
> +        return -EINVAL;
> +    }
> +
> +    return 0;
> +}
> +
> +static int dirty_bitmap_load(QEMUFile *f, void *opaque, int version_id)
> +{
> +    static DirtyBitmapLoadState s;
> +
> +    int ret = 0;
> +
> +    DPRINTF("load start\n");
> +
> +    do {
> +        dirty_bitmap_load_header(f, &s);
> +
> +        if (s.flags & DIRTY_BITMAP_MIG_FLAG_START) {
> +            ret = dirty_bitmap_load_start(f, &s);
> +        } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_COMPLETE) {
> +            dirty_bitmap_load_complete(f, &s);
> +        } else if (s.flags & DIRTY_BITMAP_MIG_FLAG_BITS) {
> +            ret = dirty_bitmap_load_bits(f, &s);
> +        }
> +
> +        DPRINTF("ret: %d\n", ret);
> +        if (!ret) {
> +            ret = qemu_file_get_error(f);
> +        }
> +
> +        DPRINTF("ret: %d\n", ret);
> +        if (ret) {
> +            return ret;
> +        }
> +    } while (!(s.flags & DIRTY_BITMAP_MIG_FLAG_EOS));
> +
> +    DPRINTF("load finish\n");
> +    return 0;
> +}
> +
> +static int dirty_bitmap_save_setup(QEMUFile *f, void *opaque)
> +{
> +    DirtyBitmapMigBitmapState *dbms = NULL;
> +    init_dirty_bitmap_migration();
> +
> +    qemu_mutex_lock_iothread();
> +    /* start track dirtiness of dirty bitmaps */
> +    set_dirty_tracking();
> +    qemu_mutex_unlock_iothread();
> +
> +    blk_mig_reset_dirty_cursor();
> +
> +    QSIMPLEQ_FOREACH(dbms, &dirty_bitmap_mig_state.dbms_list, entry) {
> +        send_bitmap_start(f, dbms);
> +    }
> +    qemu_put_flags(f, DIRTY_BITMAP_MIG_FLAG_EOS);
> +
> +    return 0;
> +}
> +
> +static bool dirty_bitmap_is_active(void *opaque)
> +{
> +    return migrate_dirty_bitmaps();
> +}
> +
> +static bool dirty_bitmap_live_iterate_is_active(void *opaque)
> +{
> +    return migrate_dirty_bitmaps() && dirty_bitmap_mig_state.is_live_iterative;
> +}

Why not swap the two conditions to avoid the function call?

> +
> +static SaveVMHandlers savevm_dirty_bitmap_handlers = {
> +    .save_live_setup = dirty_bitmap_save_setup,
> +    .save_live_complete = dirty_bitmap_save_complete,
> +    .save_live_pending = dirty_bitmap_save_pending,
> +    .load_state = dirty_bitmap_load,
> +    .cancel = dirty_bitmap_migration_cancel,
> +    .is_active = dirty_bitmap_is_active,
> +};
> +
> +static SaveVMHandlers savevm_dirty_bitmap_live_iterate_handlers = {
> +    .save_live_iterate = dirty_bitmap_save_iterate,
> +    .load_state = dirty_bitmap_load,
> +    .is_active = dirty_bitmap_live_iterate_is_active,
> +};
> +
> +void dirty_bitmap_mig_init(void)
> +{
> +    QSIMPLEQ_INIT(&dirty_bitmap_mig_state.dbms_list);
> +
> +    register_savevm_live(NULL, "dirty-bitmap", 0, 1,
> +                         &savevm_dirty_bitmap_handlers,
> +                         &dirty_bitmap_mig_state);
> +    register_savevm_live(NULL, "dirty-bitmap-live-iterate", 0, 1,
> +                         &savevm_dirty_bitmap_live_iterate_handlers,
> +                         &dirty_bitmap_mig_state);
> +}
> diff --git a/vl.c b/vl.c
> index 15bccc4..83871f5 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -4174,6 +4174,7 @@ int main(int argc, char **argv, char **envp)
>  
>      blk_mig_init();
>      ram_mig_init();
> +    dirty_bitmap_mig_init();
>  
>      /* If the currently selected machine wishes to override the units-per-bus
>       * property of its default HBA interface type, do so now. */
> 

With the printf format fixed and the DEBUG definition set back to 0:

Reviewed-by: John Snow <jsnow@redhat.com>

Re: [Qemu-devel] [PATCH 11/12] iotests: add dirty bitmap migration test

[John Snow]

On 05/13/2015 11:30 AM, Vladimir Sementsov-Ogievskiy wrote:
> The test starts two vms (vm_a, vm_b), create dirty bitmap in
> the first one, do several writes to corresponding device and
> then migrate vm_a to vm_b with dirty bitmaps.
> 
> Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>
> ---
>  tests/qemu-iotests/117     | 84 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/qemu-iotests/117.out |  5 +++
>  tests/qemu-iotests/group   |  1 +
>  3 files changed, 90 insertions(+)
>  create mode 100755 tests/qemu-iotests/117
>  create mode 100644 tests/qemu-iotests/117.out
> 
> diff --git a/tests/qemu-iotests/117 b/tests/qemu-iotests/117
> new file mode 100755
> index 0000000..9fab5d0
> --- /dev/null
> +++ b/tests/qemu-iotests/117
> @@ -0,0 +1,84 @@
> +#!/usr/bin/env python
> +#
> +# Tests for dirty bitmaps migration.
> +#
> +# (C) Vladimir Sementsov-Ogievskiy 2015
> +#
> +# This program is free software; you can redistribute it and/or modify
> +# it under the terms of the GNU General Public License as published by
> +# the Free Software Foundation; either version 2 of the License, or
> +# (at your option) any later version.
> +#
> +# This program is distributed in the hope that it will be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program.  If not, see <http://www.gnu.org/licenses/>.
> +#
> +
> +import os
> +import iotests
> +import time
> +from iotests import qemu_img
> +
> +disk_a = os.path.join(iotests.test_dir, 'disk_a')
> +disk_b = os.path.join(iotests.test_dir, 'disk_b')
> +fifo   = os.path.join(iotests.test_dir, 'fifo')
> +
> +size   = 0x40000000 # 1G
> +sector_size = 512
> +granularity = 0x10000
> +regions = [
> +    { 'start': 0,          'count': 0x100000 },
> +    { 'start': 0x10000000, 'count': 0x20000  },
> +    { 'start': 0x39990000, 'count': 0x10000  }
> +    ]
> +
> +class TestDirtyBitmapMigration(iotests.QMPTestCase):
> +
> +    def setUp(self):
> +        os.mkfifo(fifo)
> +        qemu_img('create', '-f', iotests.imgfmt, disk_a, str(size))
> +        qemu_img('create', '-f', iotests.imgfmt, disk_b, str(size))
> +        self.vm_a = iotests.VM().add_drive(disk_a)
> +        self.vm_b = iotests.VM().add_drive(disk_b)
> +        self.vm_b.add_incoming_migration("exec: cat " + fifo)
> +        self.vm_a.launch()
> +        self.vm_b.launch()
> +
> +    def tearDown(self):
> +        self.vm_a.shutdown()
> +        self.vm_b.shutdown()
> +        os.remove(disk_a)
> +        os.remove(disk_b)
> +        os.remove(fifo)
> +
> +    def test_migration(self):
> +        result = self.vm_a.qmp('block-dirty-bitmap-add', node='drive0',
> +                               name='bitmap', granularity=granularity)
> +        self.assert_qmp(result, 'return', {});
> +
> +        for r in regions:
> +          self.vm_a.hmp_qemu_io('drive0',
> +                                'write %d %d' % (r['start'], r['count']))
> +
> +        result = self.vm_a.qmp('query-block');
> +        md5 = result['return'][0]['dirty-bitmaps'][0]['md5']
> +
> +        result = self.vm_a.qmp('migrate-set-capabilities',
> +                               capabilities=[{'capability': 'dirty-bitmaps',
> +                                              'state': True}])
> +        self.assert_qmp(result, 'return', {})
> +
> +        result = self.vm_a.qmp('migrate', uri='exec:cat>' + fifo)
> +        self.assertIsNotNone(self.vm_a.event_wait("STOP"))
> +        self.assertIsNotNone(self.vm_b.event_wait("RESUME"))

I recently got stung for this myself, but apparently certain versions we
target with QEMU don't have Python 2.7, which is when the
assertIsNotNone helper was introduced.

We can use e.g. assertNotEqual(self.vm_a.event_wait("STOP"), None) instead.

> +
> +        result = self.vm_b.qmp('query-block');
> +        self.assert_qmp(result, 'return[0]/dirty-bitmaps[0]/md5', md5);
> +
> +
> +if __name__ == '__main__':
> +    iotests.main()
> diff --git a/tests/qemu-iotests/117.out b/tests/qemu-iotests/117.out
> new file mode 100644
> index 0000000..ae1213e
> --- /dev/null
> +++ b/tests/qemu-iotests/117.out
> @@ -0,0 +1,5 @@
> +.
> +----------------------------------------------------------------------
> +Ran 1 tests
> +
> +OK
> diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
> index 6ca3466..6812681 100644
> --- a/tests/qemu-iotests/group
> +++ b/tests/qemu-iotests/group
> @@ -121,6 +121,7 @@
>  114 rw auto quick
>  115 rw auto
>  116 rw auto quick
> +117 rw auto quick
>  121 rw auto
>  122 rw auto
>  123 rw auto quick
> 

With that edit:

Reviewed-by: John Snow <jsnow@redhat.com>

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[John Snow]

On 05/13/2015 11:29 AM, Vladimir Sementsov-Ogievskiy wrote:
> These patches provide dirty bitmap migration feature. Only named dirty
> bitmaps are to be migrated. Migration may be enabled using migration
> capabilities.
> 
> v5:
>     - rebase on master
>     - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
>     - remove rfc, as incremental backup series by John Snow are in
>       upstream
>     
> 

[...]

I believe as of now I've reviewed all of these patches. One more respin
may be warranted, but I'd like to start pressing for this to be merged
into Kevin's tree (iotests, core) or Stefan's (migration).

My hunch if Stefan's, since we touch those bits more, and he's already
reviewed the core functionality we're modifying, here.

It might be nice to have a maintainer review prior to Vladimir
respinning the series for my minor nitpicks on v5 so that we can spin up
a v6 expecting it to be the final iteration.

Vladimir: With this hopefully out of the way soon, would you like to
rebase your persistence series now so we can begin picking through that
on-list so we can push that through for 2.4?

Thanks!

--js

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[John Snow]

On 05/28/2015 04:56 PM, Denis V. Lunev wrote:
> On 28/05/15 23:09, John Snow wrote:
>>
>> On 05/26/2015 10:51 AM, Denis V. Lunev wrote:
>>> On 26/05/15 17:48, Denis V. Lunev wrote:
>>>> On 21/05/15 19:44, John Snow wrote:
>>>>> On 05/21/2015 09:57 AM, Denis V. Lunev wrote:
>>>>>> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>> Hi all.
>>>>>>>
>>>>>>> Hmm. There is an interesting suggestion from Denis Lunev (in CC)
>>>>>>> about
>>>>>>> how to drop meta bitmaps and make things easer.
>>>>>>>
>>>>>>> method:
>>>>>>>
>>>>>>>> start migration
>>>>>>> disk and memory are migrated, but not dirty bitmaps.
>>>>>>>> stop vm
>>>>>>> create all necessary bitmaps in destination vm (empty, but with same
>>>>>>> names and granularities and enabled flag)
>>>>>>>> start destination vm
>>>>>>> empty bitmaps are tracking now
>>>>>>>> start migrating dirty bitmaps. merge them to corresponding bitmaps
>>>>>>> in destination
>>>>>>> while bitmaps are migrating, they should be in some kind of
>>>>>>> 'inconsistent' state.
>>>>>>> so, we can't start backup or other migration while bitmaps are
>>>>>>> migrating, but vm is already _running_ on destination.
>>>>>>>
>>>>>>> what do you think about it?
>>>>>>>
>>>>>> the description is a bit incorrect
>>>>>>
>>>>>> - start migration process, perform memory and disk migration
>>>>>>      as usual. VM is still executed at source
>>>>>> - start VM on target. VM on source should be on pause as usual,
>>>>>>      do not finish migration process. Running VM on target "writes"
>>>>>>      normally setting dirty bits as usual
>>>>>> - copy active dirty bitmaps from source to target. This is safe
>>>>>>      as VM on source is not running
>>>>>> - "OR" copied bitmaps with ones running on target
>>>>>> - finish migration process (stop source VM).
>>>>>>
>>>>>> Downtime will not be increased due to dirty bitmaps with this
>>>>>> approach, migration process is very simple - plain data copy.
>>>>>>
>>>>>> Regards,
>>>>>>       Den
>>>>>>
>>>>> I was actually just discussing the live migration approach a little
>>>>> bit
>>>>> ago with Stefan, trying to decide on the "right" packet format (The
>>>>> only
>>>>> two patches I haven't ACKed yet are ones in which we need to choose a
>>>>> send size) and we decided that 1KiB chunk sends would be
>>>>> appropriate for
>>>>> live migration.
>>>>>
>>>>> I think I'm okay with that method, but obviously this approach
>>>>> outlined
>>>>> here would also work very well and would avoid meta bitmaps, chunk
>>>>> sizes, migration tuning, convergence questions, etc etc etc.
>>>>>
>>>>> You'd need to add a new status to the bitmap on the target (maybe
>>>>> "INCOMPLETE" or "MIGRATING") that prevents it from being used for a
>>>>> backup operation without preventing it from recording new writes.
>>>>>
>>>>> My only concern is how easy it will be to work this into the migration
>>>>> workflow.
>>>>>
>>>>> It would require some sort of "post-migration" ternary phase, I
>>>>> suppose,
>>>>> for devices/data that can be transferred after the VM starts -- and I
>>>>> suspect we'll be the only use of that phase for now.
>>>>>
>>>>> David, what are your thoughts, here? Would you prefer Vladimir and I
>>>>> push forward on the live migration approach, or add a new post-hoc
>>>>> phase? This approach might be simpler on the block layer, but I
>>>>> would be
>>>>> rather upset if he scrapped his entire series for the second time for
>>>>> another approach that also didn't get accepted.
>>>>>
>>>>> --js
>>>> hmmm.... It looks like we should proceed with this to fit 2.4 dates.
>>>> There is not much interest at the moment. I think that we could
>>>> implement this later in 2.5 etc...
>>>>
>>>> Regards,
>>>>      Den
>>> oops. I have written something strange. Anyway, I think that for
>>> now we should proceed with this patchset to fit QEMU 2.4 dates.
>>> The implementation with additional stage (my proposal) could be
>>> added later, f.e. in 2.5 as I do not see much interest from migration
>>> gurus.
>>>
>>> In this case the review will take a ... lot of time.
>>>
>>> Regards,
>>>      Den
>>>
>> That sounds good to me. I think this solution is workable for 2.4, and
>> we can begin working on a post-migration phase for the future to help
>> simplify our cases a lot.
>>
>> I have been out sick much of this week, so apologies in my lack of
>> fervor getting this series upstream recently.
>>
>> --js
> no prob :)

Had a chat with Stefan about this approach and apparently that's what
the postcopy migration patches on-list are all about.

Stefan brought up the point of post-hoc reliability: It's possible to
transfer control to the new VM and then lose your link, making migration
completion impossible. Adding a post-copy phase to our existing live
migration is a non-starter, because it introduces unfairly this
unreliability to the existing system.

However, we can make this idea work for migrations started via the
post-copy mechanism, because the entire migration already carries that
known risk of completion failure.

It seems like the likely outcome though is that migrations will be able
to be completed with either mechanism in the future: either up-front
migration or post-copy migration. In that light, it seems we won't be
able to fully rid ourselves of the meta_bitmap idea, making the
post-copy idea here not too useful in culling our complexity, since
we'll have to support the current standard live migration anyway.

So I have reviewed the current set of patches under the assumption that
it seems like the right way to go for 2.4 and beyond.

Thank you!
--js

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Vladimir Sementsov-Ogievskiy]

On 03.06.2015 01:12, John Snow wrote:
>
> On 05/13/2015 11:29 AM, Vladimir Sementsov-Ogievskiy wrote:
>> These patches provide dirty bitmap migration feature. Only named dirty
>> bitmaps are to be migrated. Migration may be enabled using migration
>> capabilities.
>>
>> v5:
>>      - rebase on master
>>      - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
>>      - remove rfc, as incremental backup series by John Snow are in
>>        upstream
>>      
>>
> [...]
>
> I believe as of now I've reviewed all of these patches. One more respin
> may be warranted, but I'd like to start pressing for this to be merged
> into Kevin's tree (iotests, core) or Stefan's (migration).
>
> My hunch if Stefan's, since we touch those bits more, and he's already
> reviewed the core functionality we're modifying, here.
>
> It might be nice to have a maintainer review prior to Vladimir
> respinning the series for my minor nitpicks on v5 so that we can spin up
> a v6 expecting it to be the final iteration.
>
> Vladimir: With this hopefully out of the way soon, would you like to
> rebase your persistence series now so we can begin picking through that
> on-list so we can push that through for 2.4?
Ok)
>
> Thanks!
>
> --js


-- 
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Denis V. Lunev]

On 03/06/15 01:17, John Snow wrote:
>
> On 05/28/2015 04:56 PM, Denis V. Lunev wrote:
>> On 28/05/15 23:09, John Snow wrote:
>>> On 05/26/2015 10:51 AM, Denis V. Lunev wrote:
>>>> On 26/05/15 17:48, Denis V. Lunev wrote:
>>>>> On 21/05/15 19:44, John Snow wrote:
>>>>>> On 05/21/2015 09:57 AM, Denis V. Lunev wrote:
>>>>>>> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> Hmm. There is an interesting suggestion from Denis Lunev (in CC)
>>>>>>>> about
>>>>>>>> how to drop meta bitmaps and make things easer.
>>>>>>>>
>>>>>>>> method:
>>>>>>>>
>>>>>>>>> start migration
>>>>>>>> disk and memory are migrated, but not dirty bitmaps.
>>>>>>>>> stop vm
>>>>>>>> create all necessary bitmaps in destination vm (empty, but with same
>>>>>>>> names and granularities and enabled flag)
>>>>>>>>> start destination vm
>>>>>>>> empty bitmaps are tracking now
>>>>>>>>> start migrating dirty bitmaps. merge them to corresponding bitmaps
>>>>>>>> in destination
>>>>>>>> while bitmaps are migrating, they should be in some kind of
>>>>>>>> 'inconsistent' state.
>>>>>>>> so, we can't start backup or other migration while bitmaps are
>>>>>>>> migrating, but vm is already _running_ on destination.
>>>>>>>>
>>>>>>>> what do you think about it?
>>>>>>>>
>>>>>>> the description is a bit incorrect
>>>>>>>
>>>>>>> - start migration process, perform memory and disk migration
>>>>>>>       as usual. VM is still executed at source
>>>>>>> - start VM on target. VM on source should be on pause as usual,
>>>>>>>       do not finish migration process. Running VM on target "writes"
>>>>>>>       normally setting dirty bits as usual
>>>>>>> - copy active dirty bitmaps from source to target. This is safe
>>>>>>>       as VM on source is not running
>>>>>>> - "OR" copied bitmaps with ones running on target
>>>>>>> - finish migration process (stop source VM).
>>>>>>>
>>>>>>> Downtime will not be increased due to dirty bitmaps with this
>>>>>>> approach, migration process is very simple - plain data copy.
>>>>>>>
>>>>>>> Regards,
>>>>>>>        Den
>>>>>>>
>>>>>> I was actually just discussing the live migration approach a little
>>>>>> bit
>>>>>> ago with Stefan, trying to decide on the "right" packet format (The
>>>>>> only
>>>>>> two patches I haven't ACKed yet are ones in which we need to choose a
>>>>>> send size) and we decided that 1KiB chunk sends would be
>>>>>> appropriate for
>>>>>> live migration.
>>>>>>
>>>>>> I think I'm okay with that method, but obviously this approach
>>>>>> outlined
>>>>>> here would also work very well and would avoid meta bitmaps, chunk
>>>>>> sizes, migration tuning, convergence questions, etc etc etc.
>>>>>>
>>>>>> You'd need to add a new status to the bitmap on the target (maybe
>>>>>> "INCOMPLETE" or "MIGRATING") that prevents it from being used for a
>>>>>> backup operation without preventing it from recording new writes.
>>>>>>
>>>>>> My only concern is how easy it will be to work this into the migration
>>>>>> workflow.
>>>>>>
>>>>>> It would require some sort of "post-migration" ternary phase, I
>>>>>> suppose,
>>>>>> for devices/data that can be transferred after the VM starts -- and I
>>>>>> suspect we'll be the only use of that phase for now.
>>>>>>
>>>>>> David, what are your thoughts, here? Would you prefer Vladimir and I
>>>>>> push forward on the live migration approach, or add a new post-hoc
>>>>>> phase? This approach might be simpler on the block layer, but I
>>>>>> would be
>>>>>> rather upset if he scrapped his entire series for the second time for
>>>>>> another approach that also didn't get accepted.
>>>>>>
>>>>>> --js
>>>>> hmmm.... It looks like we should proceed with this to fit 2.4 dates.
>>>>> There is not much interest at the moment. I think that we could
>>>>> implement this later in 2.5 etc...
>>>>>
>>>>> Regards,
>>>>>       Den
>>>> oops. I have written something strange. Anyway, I think that for
>>>> now we should proceed with this patchset to fit QEMU 2.4 dates.
>>>> The implementation with additional stage (my proposal) could be
>>>> added later, f.e. in 2.5 as I do not see much interest from migration
>>>> gurus.
>>>>
>>>> In this case the review will take a ... lot of time.
>>>>
>>>> Regards,
>>>>       Den
>>>>
>>> That sounds good to me. I think this solution is workable for 2.4, and
>>> we can begin working on a post-migration phase for the future to help
>>> simplify our cases a lot.
>>>
>>> I have been out sick much of this week, so apologies in my lack of
>>> fervor getting this series upstream recently.
>>>
>>> --js
>> no prob :)
> Had a chat with Stefan about this approach and apparently that's what
> the postcopy migration patches on-list are all about.
>
> Stefan brought up the point of post-hoc reliability: It's possible to
> transfer control to the new VM and then lose your link, making migration
> completion impossible. Adding a post-copy phase to our existing live
> migration is a non-starter, because it introduces unfairly this
> unreliability to the existing system.
>
> However, we can make this idea work for migrations started via the
> post-copy mechanism, because the entire migration already carries that
> known risk of completion failure.
>
> It seems like the likely outcome though is that migrations will be able
> to be completed with either mechanism in the future: either up-front
> migration or post-copy migration. In that light, it seems we won't be
> able to fully rid ourselves of the meta_bitmap idea, making the
> post-copy idea here not too useful in culling our complexity, since
> we'll have to support the current standard live migration anyway.
>
> So I have reviewed the current set of patches under the assumption that
> it seems like the right way to go for 2.4 and beyond.
>
> Thank you!
> --js
ok! seems reasonable.


Sorry for late answer.

Regards,
     Den

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[John Snow]

Ping for maintainer reviews -- I've given it a first pass and it should
hopefully be close.

Thanks,
--js

On 05/13/2015 11:29 AM, Vladimir Sementsov-Ogievskiy wrote:
> These patches provide dirty bitmap migration feature. Only named dirty
> bitmaps are to be migrated. Migration may be enabled using migration
> capabilities.
> 
> v5:
>     - rebase on master
>     - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
>     - remove rfc, as incremental backup series by John Snow are in
>       upstream
>     
> 
> v4 significant changes:
>  0001: tiny bugfix: out[i] -> out[i-start], same for 'in'
>  0007: set chunk size to 1kb, disable live iteration for
>        migrating data < 1mb size.
> 
>  tests: only one with md5 sum is here. used function event_wait
>         by John Snow. (I hope, you don't mind me just adding this
>         function with your 'Signed-off-by')
> 
>  rfc: This patch set is based on v13 of
>       "block: incremental backup series" by John Snow, which are
>       not pushed yet.
> 
> v3:
>  based on v13 of "block: incremental backup series" by John Snow.
> 
>  changes from v2:
>  removed patch for adding dirty parameter (migration capablities used
>  instead).
>  
>  0001: printf's dropped, qapi used
>  0002: part0 -> zeroes
>  0003: part0 -> zeroes
>  0005: dirty_dirty -> meta
>        add comments about meta bitmap
>        
>  0006: the format is changed, nodes used instead of devices.
> 
>  other patches are new.
> 
>  rfc: there are two tests. They are the same but using different
>  interfaces: md5 checksum of the bitmap last layer in query-block or
>  separate query-block-dirty-bitmap with dirty bitmap regions.
>  The second form is more appropriate for debugging, the first is more
>  appropriate for simple regression control. Which should go to
>  upstream?
> 
> v2:
>  1. bug-fixes, that are already in upstream, and renaming of function
>  bdrv_reset_dirty_bitmap (which is already in Snow's series) are
>  dropped
>  2. bitmap store/restore: the concept renamed to serialization, added
>  function hbitmap_deserialize_part0, to not transfer zero blocks
>  3. migration dirty parameter: added description comment
>  4. Other patches are new.
> 
> v2.rfc:
> Actually, in this version of the series I'm trying not use
> migration/block.c at all. Instead a separate migration unit is added
> in the new file migration/dirty-bitmap.c. Now bitmaps are migrated
> like blocks in block migration, they have their "dirty-dirty" bitmaps,
> for tracking set/unset changes during migration.
> 
> The advantages are:
>   - no complications of migration/block.c
>   - separate dirty-dirty bitmaps provide handling of "unset's"
>   - more effective meta-data/data ratio - no tiny bitmap-blocks.
> 
> 
> 
> v1:
> These patches provide dirty bitmap migration feature. Only named dirty
> bitmaps are to be migrated. Migration is made as a part of block
> migration in block-migration.c.
> 
> Dirty bitmap migration may be enabled by "dirty" parameter for qmp migrate
> command. If "blk" and "inc" parameters are false when "dirty" is true
> block migration is actually skipped: no allocatoions, no bdrv_read's,
> no bdrv_write's, only bitmaps are migrated.
> 
> The patch set includes two my previous bug fixes, which are necessary
> for it. The patch set is based on Incremental backup series by John
> Snow.
> Vladimir Sementsov-Ogievskiy (12):
>   hbitmap: serialization
>   block: BdrvDirtyBitmap serialization interface
>   block: tiny refactoring: minimize hbitmap_(set/reset) usage
>   block: add meta bitmaps
>   block: add bdrv_next_dirty_bitmap()
>   qapi: add dirty-bitmaps migration capability
>   migration: add migration/block-dirty-bitmap.c
>   iotests: maintain several vms in test
>   iotests: add add_incoming_migration to VM class
>   qapi: add md5 checksum of last dirty bitmap level to query-block
>   iotests: add dirty bitmap migration test
>   migration/qemu-file: make functions qemu_(get/put)_string public
> 
>  block.c                        |  98 +++++-
>  include/block/block.h          |  22 ++
>  include/migration/block.h      |   1 +
>  include/migration/migration.h  |   1 +
>  include/migration/qemu-file.h  |  17 +
>  include/qemu/hbitmap.h         |  67 ++++
>  migration/Makefile.objs        |   2 +-
>  migration/block-dirty-bitmap.c | 693 +++++++++++++++++++++++++++++++++++++++++
>  migration/migration.c          |   9 +
>  migration/qemu-file.c          |  18 ++
>  qapi-schema.json               |   4 +-
>  qapi/block-core.json           |   4 +-
>  tests/qemu-iotests/117         |  84 +++++
>  tests/qemu-iotests/117.out     |   5 +
>  tests/qemu-iotests/group       |   1 +
>  tests/qemu-iotests/iotests.py  |  14 +-
>  util/hbitmap.c                 | 106 +++++++
>  vl.c                           |   1 +
>  18 files changed, 1137 insertions(+), 10 deletions(-)
>  create mode 100644 migration/block-dirty-bitmap.c
>  create mode 100755 tests/qemu-iotests/117
>  create mode 100644 tests/qemu-iotests/117.out
> 

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Denis V. Lunev]

On 13/06/15 01:16, John Snow wrote:
> Ping for maintainer reviews -- I've given it a first pass and it should
> hopefully be close.
>
> Thanks,
> --js
do we need to respin? I do not see much attention from maintainers here.

Den


> On 05/13/2015 11:29 AM, Vladimir Sementsov-Ogievskiy wrote:
>> These patches provide dirty bitmap migration feature. Only named dirty
>> bitmaps are to be migrated. Migration may be enabled using migration
>> capabilities.
>>
>> v5:
>>      - rebase on master
>>      - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
>>      - remove rfc, as incremental backup series by John Snow are in
>>        upstream
>>      
>>
>> v4 significant changes:
>>   0001: tiny bugfix: out[i] -> out[i-start], same for 'in'
>>   0007: set chunk size to 1kb, disable live iteration for
>>         migrating data < 1mb size.
>>
>>   tests: only one with md5 sum is here. used function event_wait
>>          by John Snow. (I hope, you don't mind me just adding this
>>          function with your 'Signed-off-by')
>>
>>   rfc: This patch set is based on v13 of
>>        "block: incremental backup series" by John Snow, which are
>>        not pushed yet.
>>
>> v3:
>>   based on v13 of "block: incremental backup series" by John Snow.
>>
>>   changes from v2:
>>   removed patch for adding dirty parameter (migration capablities used
>>   instead).
>>   
>>   0001: printf's dropped, qapi used
>>   0002: part0 -> zeroes
>>   0003: part0 -> zeroes
>>   0005: dirty_dirty -> meta
>>         add comments about meta bitmap
>>         
>>   0006: the format is changed, nodes used instead of devices.
>>
>>   other patches are new.
>>
>>   rfc: there are two tests. They are the same but using different
>>   interfaces: md5 checksum of the bitmap last layer in query-block or
>>   separate query-block-dirty-bitmap with dirty bitmap regions.
>>   The second form is more appropriate for debugging, the first is more
>>   appropriate for simple regression control. Which should go to
>>   upstream?
>>
>> v2:
>>   1. bug-fixes, that are already in upstream, and renaming of function
>>   bdrv_reset_dirty_bitmap (which is already in Snow's series) are
>>   dropped
>>   2. bitmap store/restore: the concept renamed to serialization, added
>>   function hbitmap_deserialize_part0, to not transfer zero blocks
>>   3. migration dirty parameter: added description comment
>>   4. Other patches are new.
>>
>> v2.rfc:
>> Actually, in this version of the series I'm trying not use
>> migration/block.c at all. Instead a separate migration unit is added
>> in the new file migration/dirty-bitmap.c. Now bitmaps are migrated
>> like blocks in block migration, they have their "dirty-dirty" bitmaps,
>> for tracking set/unset changes during migration.
>>
>> The advantages are:
>>    - no complications of migration/block.c
>>    - separate dirty-dirty bitmaps provide handling of "unset's"
>>    - more effective meta-data/data ratio - no tiny bitmap-blocks.
>>
>>
>>
>> v1:
>> These patches provide dirty bitmap migration feature. Only named dirty
>> bitmaps are to be migrated. Migration is made as a part of block
>> migration in block-migration.c.
>>
>> Dirty bitmap migration may be enabled by "dirty" parameter for qmp migrate
>> command. If "blk" and "inc" parameters are false when "dirty" is true
>> block migration is actually skipped: no allocatoions, no bdrv_read's,
>> no bdrv_write's, only bitmaps are migrated.
>>
>> The patch set includes two my previous bug fixes, which are necessary
>> for it. The patch set is based on Incremental backup series by John
>> Snow.
>> Vladimir Sementsov-Ogievskiy (12):
>>    hbitmap: serialization
>>    block: BdrvDirtyBitmap serialization interface
>>    block: tiny refactoring: minimize hbitmap_(set/reset) usage
>>    block: add meta bitmaps
>>    block: add bdrv_next_dirty_bitmap()
>>    qapi: add dirty-bitmaps migration capability
>>    migration: add migration/block-dirty-bitmap.c
>>    iotests: maintain several vms in test
>>    iotests: add add_incoming_migration to VM class
>>    qapi: add md5 checksum of last dirty bitmap level to query-block
>>    iotests: add dirty bitmap migration test
>>    migration/qemu-file: make functions qemu_(get/put)_string public
>>
>>   block.c                        |  98 +++++-
>>   include/block/block.h          |  22 ++
>>   include/migration/block.h      |   1 +
>>   include/migration/migration.h  |   1 +
>>   include/migration/qemu-file.h  |  17 +
>>   include/qemu/hbitmap.h         |  67 ++++
>>   migration/Makefile.objs        |   2 +-
>>   migration/block-dirty-bitmap.c | 693 +++++++++++++++++++++++++++++++++++++++++
>>   migration/migration.c          |   9 +
>>   migration/qemu-file.c          |  18 ++
>>   qapi-schema.json               |   4 +-
>>   qapi/block-core.json           |   4 +-
>>   tests/qemu-iotests/117         |  84 +++++
>>   tests/qemu-iotests/117.out     |   5 +
>>   tests/qemu-iotests/group       |   1 +
>>   tests/qemu-iotests/iotests.py  |  14 +-
>>   util/hbitmap.c                 | 106 +++++++
>>   vl.c                           |   1 +
>>   18 files changed, 1137 insertions(+), 10 deletions(-)
>>   create mode 100644 migration/block-dirty-bitmap.c
>>   create mode 100755 tests/qemu-iotests/117
>>   create mode 100644 tests/qemu-iotests/117.out
>>

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[John Snow]

On 06/19/2015 12:57 PM, Denis V. Lunev wrote:
> On 13/06/15 01:16, John Snow wrote:
>> Ping for maintainer reviews -- I've given it a first pass and it should
>> hopefully be close.
>>
>> Thanks,
>> --js
> do we need to respin? I do not see much attention from maintainers here.
> 
> Den
> 

Not sure. Either everyone else wants to see the persistence series
refined before they merge/review this, or just haven't gotten to this
one yet.


> 
>> On 05/13/2015 11:29 AM, Vladimir Sementsov-Ogievskiy wrote:
>>> These patches provide dirty bitmap migration feature. Only named dirty
>>> bitmaps are to be migrated. Migration may be enabled using migration
>>> capabilities.
>>>
>>> v5:
>>>      - rebase on master
>>>      - drop [PATCH RFC v4 10/13] iotests: add event_wait to VM class
>>>      - remove rfc, as incremental backup series by John Snow are in
>>>        upstream
>>>     
>>> v4 significant changes:
>>>   0001: tiny bugfix: out[i] -> out[i-start], same for 'in'
>>>   0007: set chunk size to 1kb, disable live iteration for
>>>         migrating data < 1mb size.
>>>
>>>   tests: only one with md5 sum is here. used function event_wait
>>>          by John Snow. (I hope, you don't mind me just adding this
>>>          function with your 'Signed-off-by')
>>>
>>>   rfc: This patch set is based on v13 of
>>>        "block: incremental backup series" by John Snow, which are
>>>        not pushed yet.
>>>
>>> v3:
>>>   based on v13 of "block: incremental backup series" by John Snow.
>>>
>>>   changes from v2:
>>>   removed patch for adding dirty parameter (migration capablities used
>>>   instead).
>>>     0001: printf's dropped, qapi used
>>>   0002: part0 -> zeroes
>>>   0003: part0 -> zeroes
>>>   0005: dirty_dirty -> meta
>>>         add comments about meta bitmap
>>>           0006: the format is changed, nodes used instead of devices.
>>>
>>>   other patches are new.
>>>
>>>   rfc: there are two tests. They are the same but using different
>>>   interfaces: md5 checksum of the bitmap last layer in query-block or
>>>   separate query-block-dirty-bitmap with dirty bitmap regions.
>>>   The second form is more appropriate for debugging, the first is more
>>>   appropriate for simple regression control. Which should go to
>>>   upstream?
>>>
>>> v2:
>>>   1. bug-fixes, that are already in upstream, and renaming of function
>>>   bdrv_reset_dirty_bitmap (which is already in Snow's series) are
>>>   dropped
>>>   2. bitmap store/restore: the concept renamed to serialization, added
>>>   function hbitmap_deserialize_part0, to not transfer zero blocks
>>>   3. migration dirty parameter: added description comment
>>>   4. Other patches are new.
>>>
>>> v2.rfc:
>>> Actually, in this version of the series I'm trying not use
>>> migration/block.c at all. Instead a separate migration unit is added
>>> in the new file migration/dirty-bitmap.c. Now bitmaps are migrated
>>> like blocks in block migration, they have their "dirty-dirty" bitmaps,
>>> for tracking set/unset changes during migration.
>>>
>>> The advantages are:
>>>    - no complications of migration/block.c
>>>    - separate dirty-dirty bitmaps provide handling of "unset's"
>>>    - more effective meta-data/data ratio - no tiny bitmap-blocks.
>>>
>>>
>>>
>>> v1:
>>> These patches provide dirty bitmap migration feature. Only named dirty
>>> bitmaps are to be migrated. Migration is made as a part of block
>>> migration in block-migration.c.
>>>
>>> Dirty bitmap migration may be enabled by "dirty" parameter for qmp
>>> migrate
>>> command. If "blk" and "inc" parameters are false when "dirty" is true
>>> block migration is actually skipped: no allocatoions, no bdrv_read's,
>>> no bdrv_write's, only bitmaps are migrated.
>>>
>>> The patch set includes two my previous bug fixes, which are necessary
>>> for it. The patch set is based on Incremental backup series by John
>>> Snow.
>>> Vladimir Sementsov-Ogievskiy (12):
>>>    hbitmap: serialization
>>>    block: BdrvDirtyBitmap serialization interface
>>>    block: tiny refactoring: minimize hbitmap_(set/reset) usage
>>>    block: add meta bitmaps
>>>    block: add bdrv_next_dirty_bitmap()
>>>    qapi: add dirty-bitmaps migration capability
>>>    migration: add migration/block-dirty-bitmap.c
>>>    iotests: maintain several vms in test
>>>    iotests: add add_incoming_migration to VM class
>>>    qapi: add md5 checksum of last dirty bitmap level to query-block
>>>    iotests: add dirty bitmap migration test
>>>    migration/qemu-file: make functions qemu_(get/put)_string public
>>>
>>>   block.c                        |  98 +++++-
>>>   include/block/block.h          |  22 ++
>>>   include/migration/block.h      |   1 +
>>>   include/migration/migration.h  |   1 +
>>>   include/migration/qemu-file.h  |  17 +
>>>   include/qemu/hbitmap.h         |  67 ++++
>>>   migration/Makefile.objs        |   2 +-
>>>   migration/block-dirty-bitmap.c | 693
>>> +++++++++++++++++++++++++++++++++++++++++
>>>   migration/migration.c          |   9 +
>>>   migration/qemu-file.c          |  18 ++
>>>   qapi-schema.json               |   4 +-
>>>   qapi/block-core.json           |   4 +-
>>>   tests/qemu-iotests/117         |  84 +++++
>>>   tests/qemu-iotests/117.out     |   5 +
>>>   tests/qemu-iotests/group       |   1 +
>>>   tests/qemu-iotests/iotests.py  |  14 +-
>>>   util/hbitmap.c                 | 106 +++++++
>>>   vl.c                           |   1 +
>>>   18 files changed, 1137 insertions(+), 10 deletions(-)
>>>   create mode 100644 migration/block-dirty-bitmap.c
>>>   create mode 100755 tests/qemu-iotests/117
>>>   create mode 100644 tests/qemu-iotests/117.out
>>>
> 

-- 
—js

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Vladimir Sementsov-Ogievskiy]

On 03.06.2015 01:17, John Snow wrote:
>
> On 05/28/2015 04:56 PM, Denis V. Lunev wrote:
>> On 28/05/15 23:09, John Snow wrote:
>>> On 05/26/2015 10:51 AM, Denis V. Lunev wrote:
>>>> On 26/05/15 17:48, Denis V. Lunev wrote:
>>>>> On 21/05/15 19:44, John Snow wrote:
>>>>>> On 05/21/2015 09:57 AM, Denis V. Lunev wrote:
>>>>>>> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>>> Hi all.
>>>>>>>>
>>>>>>>> Hmm. There is an interesting suggestion from Denis Lunev (in CC)
>>>>>>>> about
>>>>>>>> how to drop meta bitmaps and make things easer.
>>>>>>>>
>>>>>>>> method:
>>>>>>>>
>>>>>>>>> start migration
>>>>>>>> disk and memory are migrated, but not dirty bitmaps.
>>>>>>>>> stop vm
>>>>>>>> create all necessary bitmaps in destination vm (empty, but with same
>>>>>>>> names and granularities and enabled flag)
>>>>>>>>> start destination vm
>>>>>>>> empty bitmaps are tracking now
>>>>>>>>> start migrating dirty bitmaps. merge them to corresponding bitmaps
>>>>>>>> in destination
>>>>>>>> while bitmaps are migrating, they should be in some kind of
>>>>>>>> 'inconsistent' state.
>>>>>>>> so, we can't start backup or other migration while bitmaps are
>>>>>>>> migrating, but vm is already _running_ on destination.
>>>>>>>>
>>>>>>>> what do you think about it?
>>>>>>>>
>>>>>>> the description is a bit incorrect
>>>>>>>
>>>>>>> - start migration process, perform memory and disk migration
>>>>>>>       as usual. VM is still executed at source
>>>>>>> - start VM on target. VM on source should be on pause as usual,
>>>>>>>       do not finish migration process. Running VM on target "writes"
>>>>>>>       normally setting dirty bits as usual
>>>>>>> - copy active dirty bitmaps from source to target. This is safe
>>>>>>>       as VM on source is not running
>>>>>>> - "OR" copied bitmaps with ones running on target
>>>>>>> - finish migration process (stop source VM).
>>>>>>>
>>>>>>> Downtime will not be increased due to dirty bitmaps with this
>>>>>>> approach, migration process is very simple - plain data copy.
>>>>>>>
>>>>>>> Regards,
>>>>>>>        Den
>>>>>>>
>>>>>> I was actually just discussing the live migration approach a little
>>>>>> bit
>>>>>> ago with Stefan, trying to decide on the "right" packet format (The
>>>>>> only
>>>>>> two patches I haven't ACKed yet are ones in which we need to choose a
>>>>>> send size) and we decided that 1KiB chunk sends would be
>>>>>> appropriate for
>>>>>> live migration.
>>>>>>
>>>>>> I think I'm okay with that method, but obviously this approach
>>>>>> outlined
>>>>>> here would also work very well and would avoid meta bitmaps, chunk
>>>>>> sizes, migration tuning, convergence questions, etc etc etc.
>>>>>>
>>>>>> You'd need to add a new status to the bitmap on the target (maybe
>>>>>> "INCOMPLETE" or "MIGRATING") that prevents it from being used for a
>>>>>> backup operation without preventing it from recording new writes.
>>>>>>
>>>>>> My only concern is how easy it will be to work this into the migration
>>>>>> workflow.
>>>>>>
>>>>>> It would require some sort of "post-migration" ternary phase, I
>>>>>> suppose,
>>>>>> for devices/data that can be transferred after the VM starts -- and I
>>>>>> suspect we'll be the only use of that phase for now.
>>>>>>
>>>>>> David, what are your thoughts, here? Would you prefer Vladimir and I
>>>>>> push forward on the live migration approach, or add a new post-hoc
>>>>>> phase? This approach might be simpler on the block layer, but I
>>>>>> would be
>>>>>> rather upset if he scrapped his entire series for the second time for
>>>>>> another approach that also didn't get accepted.
>>>>>>
>>>>>> --js
>>>>> hmmm.... It looks like we should proceed with this to fit 2.4 dates.
>>>>> There is not much interest at the moment. I think that we could
>>>>> implement this later in 2.5 etc...
>>>>>
>>>>> Regards,
>>>>>       Den
>>>> oops. I have written something strange. Anyway, I think that for
>>>> now we should proceed with this patchset to fit QEMU 2.4 dates.
>>>> The implementation with additional stage (my proposal) could be
>>>> added later, f.e. in 2.5 as I do not see much interest from migration
>>>> gurus.
>>>>
>>>> In this case the review will take a ... lot of time.
>>>>
>>>> Regards,
>>>>       Den
>>>>
>>> That sounds good to me. I think this solution is workable for 2.4, and
>>> we can begin working on a post-migration phase for the future to help
>>> simplify our cases a lot.
>>>
>>> I have been out sick much of this week, so apologies in my lack of
>>> fervor getting this series upstream recently.
>>>
>>> --js
>> no prob :)
> Had a chat with Stefan about this approach and apparently that's what
> the postcopy migration patches on-list are all about.
>
> Stefan brought up the point of post-hoc reliability: It's possible to
> transfer control to the new VM and then lose your link, making migration
> completion impossible. Adding a post-copy phase to our existing live
> migration is a non-starter, because it introduces unfairly this
> unreliability to the existing system.
>
> However, we can make this idea work for migrations started via the
> post-copy mechanism, because the entire migration already carries that
> known risk of completion failure.
>
> It seems like the likely outcome though is that migrations will be able
> to be completed with either mechanism in the future: either up-front
> migration or post-copy migration. In that light, it seems we won't be
> able to fully rid ourselves of the meta_bitmap idea, making the
> post-copy idea here not too useful in culling our complexity, since
> we'll have to support the current standard live migration anyway.
>
> So I have reviewed the current set of patches under the assumption that
> it seems like the right way to go for 2.4 and beyond.
>
> Thank you!
> --js

For now, post-copy migration is merged as I know. Is something changed 
for its reliability? Do we still need meta-bitmap approach for bitmap 
migration?

-- 
Best regards,
Vladimir
* now, @virtuozzo.com instead of @parallels.com. Sorry for this inconvenience.

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[John Snow]

On 01/26/2016 03:45 AM, Vladimir Sementsov-Ogievskiy wrote:
> On 03.06.2015 01:17, John Snow wrote:
>>
>> On 05/28/2015 04:56 PM, Denis V. Lunev wrote:
>>> On 28/05/15 23:09, John Snow wrote:
>>>> On 05/26/2015 10:51 AM, Denis V. Lunev wrote:
>>>>> On 26/05/15 17:48, Denis V. Lunev wrote:
>>>>>> On 21/05/15 19:44, John Snow wrote:
>>>>>>> On 05/21/2015 09:57 AM, Denis V. Lunev wrote:
>>>>>>>> On 21/05/15 16:51, Vladimir Sementsov-Ogievskiy wrote:
>>>>>>>>> Hi all.
>>>>>>>>>
>>>>>>>>> Hmm. There is an interesting suggestion from Denis Lunev (in CC)
>>>>>>>>> about
>>>>>>>>> how to drop meta bitmaps and make things easer.
>>>>>>>>>
>>>>>>>>> method:
>>>>>>>>>
>>>>>>>>>> start migration
>>>>>>>>> disk and memory are migrated, but not dirty bitmaps.
>>>>>>>>>> stop vm
>>>>>>>>> create all necessary bitmaps in destination vm (empty, but with
>>>>>>>>> same
>>>>>>>>> names and granularities and enabled flag)
>>>>>>>>>> start destination vm
>>>>>>>>> empty bitmaps are tracking now
>>>>>>>>>> start migrating dirty bitmaps. merge them to corresponding
>>>>>>>>>> bitmaps
>>>>>>>>> in destination
>>>>>>>>> while bitmaps are migrating, they should be in some kind of
>>>>>>>>> 'inconsistent' state.
>>>>>>>>> so, we can't start backup or other migration while bitmaps are
>>>>>>>>> migrating, but vm is already _running_ on destination.
>>>>>>>>>
>>>>>>>>> what do you think about it?
>>>>>>>>>
>>>>>>>> the description is a bit incorrect
>>>>>>>>
>>>>>>>> - start migration process, perform memory and disk migration
>>>>>>>>       as usual. VM is still executed at source
>>>>>>>> - start VM on target. VM on source should be on pause as usual,
>>>>>>>>       do not finish migration process. Running VM on target
>>>>>>>> "writes"
>>>>>>>>       normally setting dirty bits as usual
>>>>>>>> - copy active dirty bitmaps from source to target. This is safe
>>>>>>>>       as VM on source is not running
>>>>>>>> - "OR" copied bitmaps with ones running on target
>>>>>>>> - finish migration process (stop source VM).
>>>>>>>>
>>>>>>>> Downtime will not be increased due to dirty bitmaps with this
>>>>>>>> approach, migration process is very simple - plain data copy.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>>        Den
>>>>>>>>
>>>>>>> I was actually just discussing the live migration approach a little
>>>>>>> bit
>>>>>>> ago with Stefan, trying to decide on the "right" packet format (The
>>>>>>> only
>>>>>>> two patches I haven't ACKed yet are ones in which we need to
>>>>>>> choose a
>>>>>>> send size) and we decided that 1KiB chunk sends would be
>>>>>>> appropriate for
>>>>>>> live migration.
>>>>>>>
>>>>>>> I think I'm okay with that method, but obviously this approach
>>>>>>> outlined
>>>>>>> here would also work very well and would avoid meta bitmaps, chunk
>>>>>>> sizes, migration tuning, convergence questions, etc etc etc.
>>>>>>>
>>>>>>> You'd need to add a new status to the bitmap on the target (maybe
>>>>>>> "INCOMPLETE" or "MIGRATING") that prevents it from being used for a
>>>>>>> backup operation without preventing it from recording new writes.
>>>>>>>
>>>>>>> My only concern is how easy it will be to work this into the
>>>>>>> migration
>>>>>>> workflow.
>>>>>>>
>>>>>>> It would require some sort of "post-migration" ternary phase, I
>>>>>>> suppose,
>>>>>>> for devices/data that can be transferred after the VM starts --
>>>>>>> and I
>>>>>>> suspect we'll be the only use of that phase for now.
>>>>>>>
>>>>>>> David, what are your thoughts, here? Would you prefer Vladimir and I
>>>>>>> push forward on the live migration approach, or add a new post-hoc
>>>>>>> phase? This approach might be simpler on the block layer, but I
>>>>>>> would be
>>>>>>> rather upset if he scrapped his entire series for the second time
>>>>>>> for
>>>>>>> another approach that also didn't get accepted.
>>>>>>>
>>>>>>> --js
>>>>>> hmmm.... It looks like we should proceed with this to fit 2.4 dates.
>>>>>> There is not much interest at the moment. I think that we could
>>>>>> implement this later in 2.5 etc...
>>>>>>
>>>>>> Regards,
>>>>>>       Den
>>>>> oops. I have written something strange. Anyway, I think that for
>>>>> now we should proceed with this patchset to fit QEMU 2.4 dates.
>>>>> The implementation with additional stage (my proposal) could be
>>>>> added later, f.e. in 2.5 as I do not see much interest from migration
>>>>> gurus.
>>>>>
>>>>> In this case the review will take a ... lot of time.
>>>>>
>>>>> Regards,
>>>>>       Den
>>>>>
>>>> That sounds good to me. I think this solution is workable for 2.4, and
>>>> we can begin working on a post-migration phase for the future to help
>>>> simplify our cases a lot.
>>>>
>>>> I have been out sick much of this week, so apologies in my lack of
>>>> fervor getting this series upstream recently.
>>>>
>>>> --js
>>> no prob :)
>> Had a chat with Stefan about this approach and apparently that's what
>> the postcopy migration patches on-list are all about.
>>
>> Stefan brought up the point of post-hoc reliability: It's possible to
>> transfer control to the new VM and then lose your link, making migration
>> completion impossible. Adding a post-copy phase to our existing live
>> migration is a non-starter, because it introduces unfairly this
>> unreliability to the existing system.
>>
>> However, we can make this idea work for migrations started via the
>> post-copy mechanism, because the entire migration already carries that
>> known risk of completion failure.
>>
>> It seems like the likely outcome though is that migrations will be able
>> to be completed with either mechanism in the future: either up-front
>> migration or post-copy migration. In that light, it seems we won't be
>> able to fully rid ourselves of the meta_bitmap idea, making the
>> post-copy idea here not too useful in culling our complexity, since
>> we'll have to support the current standard live migration anyway.
>>
>> So I have reviewed the current set of patches under the assumption that
>> it seems like the right way to go for 2.4 and beyond.
>>
>> Thank you!
>> --js
> 
> For now, post-copy migration is merged as I know. Is something changed
> for its reliability? Do we still need meta-bitmap approach for bitmap
> migration?
> 

[Dropping a few people from the CC list, adding qemu-block]

There will always be the issue of post-hoc reliability, and I believe
for now all migrations still default to the non-postcopy version.

Still, losing a bitmap is not as catastrophic as losing ram, so maybe
it'd be OK to introduce bitmap migration as postcopy-only, allowing you
to ditch the meta bitmaps.

I think whether or not you need the meta bitmap hinges on convincing
migration maintainers that there is no need to ever do a live migration
of bitmap data, but that postcopying it is always preferred, enough so
that we never bother to merge the live migration version.

I imagine the post-copy bitmap migration looks something like this:

1) Migrate the metadata for the bitmap (name, size, granularity, etc)

2) Create this bitmap on the target, and immediately create an anonymous
child to move it into the frozen state. The anonymous child is marked
read-only for now to prevent block migration writes from corrupting the
bitmap. Neither bitmaps can be written to or used for operations.

3) At the time of migration pivot, the source bitmap is marked as
read-only, and the destination bitmap's anonymous child is marked as
read-write. Any disk IO that happens in this period is recorded in the
anonymous child.

4) The source bitmap is migrated using a simple for loop at about ~1KiB
at a time (to fit in Ethernet frames. Perhaps this can be configured if
we really, really want to.)

5) The destination stores the migrated bitmap in the frozen parent object.

6) When migration is fully complete, the parent and child bitmap object
can be merged into one and moved back into a normal operative state.


as for the reliability issue, we have a lot of mitigation options...

If the source machine loses connection, we can easily just re-start the
data transfer, since the bitmap on the target is still recording new
information.

Perhaps the source VM could attempt to write out the bitmap data to disk
automatically once it loses connection, and delete that data if it
manages to re-connect and transmit successfully. If that's not desired,
we can always add HMP/QMP commands to do a state dump.

If we do lose the network entirely, we'll need a matching QMP/HMP
command to load state from disk transferred via other means, then the
target bitmaps can resume normal operation once they get that missing state.

Obviously nothing that can be done for if the source just flat-out
crashes. We could attempt to dump all bitmaps to disk in the event of a
failure, but if things are that unstable that we crashed *after* pivot,
recovery doesn't sound likely.

The target will just simply have to be re-synced with a new full backup.

... actually, wait...

Can we implement a function that, in the event of a disaster, compares
the current state of the drive with the last known good incremental and
populates a bitmap based on the difference?

Actually, we probably really want this feature around regardless. It'll
allow us to start incremental backup chains from full backups made
before we even knew we wanted to start making them.

Something like:

block-dirty-bitmap-diff node=drive0 name=bitmap0 target=/path/to/file

I like this idea, I think I'll prototype just this little piece, since
it's useful even without postcopy bitmap migration.

Thanks!
--js

Re: [Qemu-devel] [PATCH v5 00/12] Dirty bitmaps migration

[Fam Zheng]

On Tue, 01/26 17:57, John Snow wrote:
> Can we implement a function that, in the event of a disaster, compares
> the current state of the drive with the last known good incremental and
> populates a bitmap based on the difference?
> 
> Actually, we probably really want this feature around regardless. It'll
> allow us to start incremental backup chains from full backups made
> before we even knew we wanted to start making them.
> 
> Something like:
> 
> block-dirty-bitmap-diff node=drive0 name=bitmap0 target=/path/to/file
> 
> I like this idea, I think I'll prototype just this little piece, since
> it's useful even without postcopy bitmap migration.

Good idea!  It looks like another type of block job.

Fam

[Qemu-devel] [PATCH 03/12] block: tiny refactoring: minimize hbitmap_(set/reset) usage

[Vladimir Sementsov-Ogievskiy]

From: Vladimir Sementsov-Ogievskiy <vsementsov@parallels.com>

Reviewed-by: John Snow <jsnow@redhat.com>
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
---
 block.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/block.c b/block.c
index 8937296..7a7307c 100644
--- a/block.c
+++ b/block.c
@@ -3616,7 +3616,7 @@ void bdrv_set_dirty(BlockDriverState *bs, int64_t cur_sector,
         if (!bdrv_dirty_bitmap_enabled(bitmap)) {
             continue;
         }
-        hbitmap_set(bitmap->bitmap, cur_sector, nr_sectors);
+        bdrv_set_dirty_bitmap(bitmap, cur_sector, nr_sectors);
     }
 }
 
-- 
2.1.4