* [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter
@ 2020-07-22 8:05 Max Reitz
2020-07-22 8:05 ` [PATCH for-5.2 v3 1/3] " Max Reitz
` (3 more replies)
0 siblings, 4 replies; 9+ messages in thread
From: Max Reitz @ 2020-07-22 8:05 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy, Juan Quintela,
John Snow, qemu-devel, Dr . David Alan Gilbert, Peter Krempa,
Max Reitz
RFC v1: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00912.html
RFC v2: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00915.html
v1: https://lists.nongnu.org/archive/html/qemu-devel/2020-06/msg09792.html
v2: https://lists.nongnu.org/archive/html/qemu-block/2020-07/msg01179.html
Branch: https://github.com/XanClic/qemu.git migration-bitmap-mapping-v3
Branch: https://git.xanclic.moe/XanClic/qemu.git migration-bitmap-mapping-v3
Hi,
This new migration parameter allows mapping block node names and bitmap
names to aliases for the purpose of block dirty bitmap migration.
This way, management tools can use different node names on the source
and destination and pass the mapping of how bitmaps are to be
transferred to qemu (on the source, the destination, or even both with
arbitrary aliases in the migration stream).
v3:
- Patch 1:
- Add notes on the fact that the destination won’t ever know about any
bitmaps that aren’t sent, so you can freely drop what you want, and
you’re completely free in renaming bitmaps and putting them on
“other” nodes (whatever “other” node means in the context of
migration, because that’s kind of one of the problems this series is
trying to solve: The fact that you can’t trivially match nodes
between source and destination)
- Fix an assertion
- Patch 2: s/pass/time.sleep(0.2)/
- Patch 3:
- Add copyright line
- Use format string instead of %
- s/pass/time.sleep(0.1)/
- s/wait_for_runstate/wait_migration/ on the destination to wait for
the migration to actually complete
- Replace the “info migrate_parameters” parsing code by a multiline
regex
- Test what happens when the destination has a mapping that isn’t used
because there are not bitmaps to be transferred (which breaks the
assertion in patch 1 as it was in v2)
- Let verify_dest_has_all_bitmaps() actually verify the bitmaps on the
destination instead of the source
git-backport-diff against v2:
Key:
[----] : patches are identical
[####] : number of functional differences between upstream/downstream patch
[down] : patch is downstream-only
The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
001/3:[0014] [FC] 'migration: Add block-bitmap-mapping parameter'
002/3:[0003] [FC] 'iotests.py: Add wait_for_runstate()'
003/3:[0046] [FC] 'iotests: Test node/bitmap aliases during migration'
Max Reitz (3):
migration: Add block-bitmap-mapping parameter
iotests.py: Add wait_for_runstate()
iotests: Test node/bitmap aliases during migration
qapi/migration.json | 104 ++++++-
migration/migration.h | 3 +
migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++----
migration/migration.c | 30 ++
monitor/hmp-cmds.c | 30 ++
tests/qemu-iotests/300 | 515 +++++++++++++++++++++++++++++++++
tests/qemu-iotests/300.out | 5 +
tests/qemu-iotests/group | 1 +
tests/qemu-iotests/iotests.py | 5 +
9 files changed, 1011 insertions(+), 55 deletions(-)
create mode 100755 tests/qemu-iotests/300
create mode 100644 tests/qemu-iotests/300.out
--
2.26.2
^ permalink raw reply [flat|nested] 9+ messages in thread
* [PATCH for-5.2 v3 1/3] migration: Add block-bitmap-mapping parameter
2020-07-22 8:05 [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter Max Reitz
@ 2020-07-22 8:05 ` Max Reitz
2020-08-12 14:32 ` Eric Blake
2020-07-22 8:05 ` [PATCH for-5.2 v3 2/3] iotests.py: Add wait_for_runstate() Max Reitz
` (2 subsequent siblings)
3 siblings, 1 reply; 9+ messages in thread
From: Max Reitz @ 2020-07-22 8:05 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy, Juan Quintela,
John Snow, qemu-devel, Dr . David Alan Gilbert, Peter Krempa,
Max Reitz
This migration parameter allows mapping block node names and bitmap
names to aliases for the purpose of block dirty bitmap migration.
This way, management tools can use different node and bitmap names on
the source and destination and pass the mapping of how bitmaps are to be
transferred to qemu (on the source, the destination, or even both with
arbitrary aliases in the migration stream).
Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
qapi/migration.json | 104 ++++++++-
migration/migration.h | 3 +
migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++++++++++-----
migration/migration.c | 30 +++
monitor/hmp-cmds.c | 30 +++
5 files changed, 485 insertions(+), 55 deletions(-)
diff --git a/qapi/migration.json b/qapi/migration.json
index d5000558c6..d7e953cd73 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -507,6 +507,44 @@
'data': [ 'none', 'zlib',
{ 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
+##
+# @BitmapMigrationBitmapAlias:
+#
+# @name: The name of the bitmap.
+#
+# @alias: An alias name for migration (for example the bitmap name on
+# the opposite site).
+#
+# Since: 5.1
+##
+{ 'struct': 'BitmapMigrationBitmapAlias',
+ 'data': {
+ 'name': 'str',
+ 'alias': 'str'
+ } }
+
+##
+# @BitmapMigrationNodeAlias:
+#
+# Maps a block node name and the bitmaps it has to aliases for dirty
+# bitmap migration.
+#
+# @node-name: A block node name.
+#
+# @alias: An alias block node name for migration (for example the
+# node name on the opposite site).
+#
+# @bitmaps: Mappings for the bitmaps on this node.
+#
+# Since: 5.1
+##
+{ 'struct': 'BitmapMigrationNodeAlias',
+ 'data': {
+ 'node-name': 'str',
+ 'alias': 'str',
+ 'bitmaps': [ 'BitmapMigrationBitmapAlias' ]
+ } }
+
##
# @MigrationParameter:
#
@@ -641,6 +679,25 @@
# will consume more CPU.
# Defaults to 1. (Since 5.0)
#
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+# aliases for the purpose of dirty bitmap migration. Such
+# aliases may for example be the corresponding names on the
+# opposite site.
+# The mapping must be one-to-one, and on the destination also
+# complete: On the source, bitmaps on nodes where either the
+# bitmap or the node is not mapped will be ignored. In
+# contrast, on the destination, receiving a bitmap (by alias)
+# from a node (by alias) when either alias is not mapped will
+# result in an error.
+# Note that the destination does not know about bitmaps it
+# does not receive, so there is no limitation or requirement
+# regarding the number of bitmaps received, or how they are
+# named, or on which nodes they are placed.
+# By default (when this parameter has never been set), bitmap
+# names are mapped to themselves. Nodes are mapped to their
+# block device name if there is one, and to their node name
+# otherwise. (Since 5.2)
+#
# Since: 2.4
##
{ 'enum': 'MigrationParameter',
@@ -655,7 +712,8 @@
'multifd-channels',
'xbzrle-cache-size', 'max-postcopy-bandwidth',
'max-cpu-throttle', 'multifd-compression',
- 'multifd-zlib-level' ,'multifd-zstd-level' ] }
+ 'multifd-zlib-level' ,'multifd-zstd-level',
+ 'block-bitmap-mapping' ] }
##
# @MigrateSetParameters:
@@ -781,6 +839,25 @@
# will consume more CPU.
# Defaults to 1. (Since 5.0)
#
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+# aliases for the purpose of dirty bitmap migration. Such
+# aliases may for example be the corresponding names on the
+# opposite site.
+# The mapping must be one-to-one, and on the destination also
+# complete: On the source, bitmaps on nodes where either the
+# bitmap or the node is not mapped will be ignored. In
+# contrast, on the destination, receiving a bitmap (by alias)
+# from a node (by alias) when either alias is not mapped will
+# result in an error.
+# Note that the destination does not know about bitmaps it
+# does not receive, so there is no limitation or requirement
+# regarding the number of bitmaps received, or how they are
+# named, or on which nodes they are placed.
+# By default (when this parameter has never been set), bitmap
+# names are mapped to themselves. Nodes are mapped to their
+# block device name if there is one, and to their node name
+# otherwise. (Since 5.2)
+#
# Since: 2.4
##
# TODO either fuse back into MigrationParameters, or make
@@ -811,7 +888,8 @@
'*max-cpu-throttle': 'int',
'*multifd-compression': 'MultiFDCompression',
'*multifd-zlib-level': 'int',
- '*multifd-zstd-level': 'int' } }
+ '*multifd-zstd-level': 'int',
+ '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } }
##
# @migrate-set-parameters:
@@ -957,6 +1035,25 @@
# will consume more CPU.
# Defaults to 1. (Since 5.0)
#
+# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
+# aliases for the purpose of dirty bitmap migration. Such
+# aliases may for example be the corresponding names on the
+# opposite site.
+# The mapping must be one-to-one, and on the destination also
+# complete: On the source, bitmaps on nodes where either the
+# bitmap or the node is not mapped will be ignored. In
+# contrast, on the destination, receiving a bitmap (by alias)
+# from a node (by alias) when either alias is not mapped will
+# result in an error.
+# Note that the destination does not know about bitmaps it
+# does not receive, so there is no limitation or requirement
+# regarding the number of bitmaps received, or how they are
+# named, or on which nodes they are placed.
+# By default (when this parameter has never been set), bitmap
+# names are mapped to themselves. Nodes are mapped to their
+# block device name if there is one, and to their node name
+# otherwise. (Since 5.2)
+#
# Since: 2.4
##
{ 'struct': 'MigrationParameters',
@@ -985,7 +1082,8 @@
'*max-cpu-throttle': 'uint8',
'*multifd-compression': 'MultiFDCompression',
'*multifd-zlib-level': 'uint8',
- '*multifd-zstd-level': 'uint8' } }
+ '*multifd-zstd-level': 'uint8',
+ '*block-bitmap-mapping': [ 'BitmapMigrationNodeAlias' ] } }
##
# @query-migrate-parameters:
diff --git a/migration/migration.h b/migration/migration.h
index f617960522..038c318b92 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -336,6 +336,9 @@ void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value);
void dirty_bitmap_mig_before_vm_start(void);
void init_dirty_bitmap_incoming_migration(void);
+bool check_dirty_bitmap_mig_alias_map(const BitmapMigrationNodeAliasList *bbm,
+ Error **errp);
+
void migrate_add_address(SocketAddress *address);
int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque);
diff --git a/migration/block-dirty-bitmap.c b/migration/block-dirty-bitmap.c
index b0dbf9eeed..ad48f1a863 100644
--- a/migration/block-dirty-bitmap.c
+++ b/migration/block-dirty-bitmap.c
@@ -29,10 +29,10 @@
*
* # Header (shared for different chunk types)
* 1, 2 or 4 bytes: flags (see qemu_{put,put}_flags)
- * [ 1 byte: node name size ] \ flags & DEVICE_NAME
- * [ n bytes: node name ] /
- * [ 1 byte: bitmap name size ] \ flags & BITMAP_NAME
- * [ n bytes: bitmap name ] /
+ * [ 1 byte: node alias size ] \ flags & DEVICE_NAME
+ * [ n bytes: node alias ] /
+ * [ 1 byte: bitmap alias size ] \ flags & BITMAP_NAME
+ * [ n bytes: bitmap alias ] /
*
* # Start of bitmap migration (flags & START)
* header
@@ -72,7 +72,9 @@
#include "migration/register.h"
#include "qemu/hbitmap.h"
#include "qemu/cutils.h"
+#include "qemu/id.h"
#include "qapi/error.h"
+#include "qapi/qapi-commands-migration.h"
#include "trace.h"
#define CHUNK_SIZE (1 << 10)
@@ -103,7 +105,8 @@
typedef struct DirtyBitmapMigBitmapState {
/* Written during setup phase. */
BlockDriverState *bs;
- const char *node_name;
+ char *node_alias;
+ char *bitmap_alias;
BdrvDirtyBitmap *bitmap;
uint64_t total_sectors;
uint64_t sectors_per_chunk;
@@ -128,7 +131,8 @@ typedef struct DirtyBitmapMigState {
typedef struct DirtyBitmapLoadState {
uint32_t flags;
- char node_name[256];
+ char node_alias[256];
+ char bitmap_alias[256];
char bitmap_name[256];
BlockDriverState *bs;
BdrvDirtyBitmap *bitmap;
@@ -144,6 +148,162 @@ typedef struct DirtyBitmapLoadBitmapState {
static GSList *enabled_bitmaps;
QemuMutex finish_lock;
+/* For hash tables that map node/bitmap names to aliases */
+typedef struct AliasMapInnerNode {
+ char *string;
+ GHashTable *subtree;
+} AliasMapInnerNode;
+
+static void free_alias_map_inner_node(void *amin_ptr)
+{
+ AliasMapInnerNode *amin = amin_ptr;
+
+ g_free(amin->string);
+ g_hash_table_unref(amin->subtree);
+ g_free(amin);
+}
+
+/**
+ * Construct an alias map based on the given QMP structure.
+ *
+ * (Note that we cannot store such maps in the MigrationParameters
+ * object, because that struct is defined by the QAPI schema, which
+ * makes it basically impossible to have dicts with arbitrary keys.
+ * Therefore, we instead have to construct these maps when migration
+ * starts.)
+ *
+ * @bbm is the block_bitmap_mapping from the migration parameters.
+ *
+ * If @name_to_alias is true, the returned hash table will map node
+ * and bitmap names to their respective aliases (for outgoing
+ * migration).
+ *
+ * If @name_to_alias is false, the returned hash table will map node
+ * and bitmap aliases to their respective names (for incoming
+ * migration).
+ *
+ * The hash table maps node names/aliases to AliasMapInnerNode
+ * objects, whose .string is the respective node alias/name, and whose
+ * .subtree table maps bitmap names/aliases to the respective bitmap
+ * alias/name.
+ */
+static GHashTable *construct_alias_map(const BitmapMigrationNodeAliasList *bbm,
+ bool name_to_alias,
+ Error **errp)
+{
+ GHashTable *alias_map;
+
+ alias_map = g_hash_table_new_full(g_str_hash, g_str_equal,
+ g_free, free_alias_map_inner_node);
+
+ for (; bbm; bbm = bbm->next) {
+ const BitmapMigrationNodeAlias *bmna = bbm->value;
+ const BitmapMigrationBitmapAliasList *bmbal;
+ AliasMapInnerNode *amin;
+ GHashTable *bitmaps_map;
+ const char *node_map_from, *node_map_to;
+
+ if (!id_wellformed(bmna->alias)) {
+ error_setg(errp, "The node alias %s is not well-formed",
+ bmna->alias);
+ goto fail;
+ }
+
+ if (name_to_alias) {
+ if (g_hash_table_contains(alias_map, bmna->node_name)) {
+ error_setg(errp, "The node name %s is mapped twice",
+ bmna->node_name);
+ goto fail;
+ }
+
+ node_map_from = bmna->node_name;
+ node_map_to = bmna->alias;
+ } else {
+ if (g_hash_table_contains(alias_map, bmna->alias)) {
+ error_setg(errp, "The node alias %s is used twice",
+ bmna->alias);
+ goto fail;
+ }
+
+ node_map_from = bmna->alias;
+ node_map_to = bmna->node_name;
+ }
+
+ bitmaps_map = g_hash_table_new_full(g_str_hash, g_str_equal,
+ g_free, g_free);
+
+ amin = g_new(AliasMapInnerNode, 1);
+ *amin = (AliasMapInnerNode){
+ .string = g_strdup(node_map_to),
+ .subtree = bitmaps_map,
+ };
+
+ g_hash_table_insert(alias_map, g_strdup(node_map_from), amin);
+
+ for (bmbal = bmna->bitmaps; bmbal; bmbal = bmbal->next) {
+ const BitmapMigrationBitmapAlias *bmba = bmbal->value;
+ const char *bmap_map_from, *bmap_map_to;
+
+ if (name_to_alias) {
+ bmap_map_from = bmba->name;
+ bmap_map_to = bmba->alias;
+
+ if (g_hash_table_contains(bitmaps_map, bmba->name)) {
+ error_setg(errp, "The bitmap %s/%s is mapped twice",
+ bmna->node_name, bmba->name);
+ goto fail;
+ }
+ } else {
+ bmap_map_from = bmba->alias;
+ bmap_map_to = bmba->name;
+
+ if (g_hash_table_contains(bitmaps_map, bmba->alias)) {
+ error_setg(errp, "The bitmap alias %s/%s is used twice",
+ bmna->alias, bmba->alias);
+ goto fail;
+ }
+ }
+
+ g_hash_table_insert(bitmaps_map,
+ g_strdup(bmap_map_from), g_strdup(bmap_map_to));
+ }
+ }
+
+ return alias_map;
+
+fail:
+ g_hash_table_destroy(alias_map);
+ return NULL;
+}
+
+/**
+ * Run construct_alias_map() in both directions to check whether @bbm
+ * is valid.
+ * (This function is to be used by migration/migration.c to validate
+ * the user-specified block-bitmap-mapping migration parameter.)
+ *
+ * Returns true if and only if the mapping is valid.
+ */
+bool check_dirty_bitmap_mig_alias_map(const BitmapMigrationNodeAliasList *bbm,
+ Error **errp)
+{
+ GHashTable *alias_map;
+
+ alias_map = construct_alias_map(bbm, true, errp);
+ if (!alias_map) {
+ return false;
+ }
+ g_hash_table_destroy(alias_map);
+
+ alias_map = construct_alias_map(bbm, false, errp);
+ if (!alias_map) {
+ return false;
+ }
+ g_hash_table_destroy(alias_map);
+
+ return true;
+}
+
void init_dirty_bitmap_incoming_migration(void)
{
qemu_mutex_init(&finish_lock);
@@ -191,11 +351,11 @@ static void send_bitmap_header(QEMUFile *f, DirtyBitmapMigBitmapState *dbms,
qemu_put_bitmap_flags(f, flags);
if (flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
- qemu_put_counted_string(f, dbms->node_name);
+ qemu_put_counted_string(f, dbms->node_alias);
}
if (flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
- qemu_put_counted_string(f, bdrv_dirty_bitmap_name(bitmap));
+ qemu_put_counted_string(f, dbms->bitmap_alias);
}
}
@@ -263,17 +423,25 @@ static void dirty_bitmap_mig_cleanup(void)
QSIMPLEQ_REMOVE_HEAD(&dirty_bitmap_mig_state.dbms_list, entry);
bdrv_dirty_bitmap_set_busy(dbms->bitmap, false);
bdrv_unref(dbms->bs);
+ g_free(dbms->node_alias);
+ g_free(dbms->bitmap_alias);
g_free(dbms);
}
}
/* Called with iothread lock taken. */
-static int add_bitmaps_to_list(BlockDriverState *bs, const char *bs_name)
+static int add_bitmaps_to_list(BlockDriverState *bs, const char *bs_name,
+ GHashTable *alias_map)
{
BdrvDirtyBitmap *bitmap;
DirtyBitmapMigBitmapState *dbms;
+ GHashTable *bitmap_aliases;
+ const char *node_alias, *bitmap_name, *bitmap_alias;
Error *local_err = NULL;
+ /* When an alias map is given, @bs_name must be @bs's node name */
+ assert(!alias_map || !strcmp(bs_name, bdrv_get_node_name(bs)));
+
FOR_EACH_DIRTY_BITMAP(bs, bitmap) {
if (bdrv_dirty_bitmap_name(bitmap)) {
break;
@@ -283,21 +451,39 @@ static int add_bitmaps_to_list(BlockDriverState *bs, const char *bs_name)
return 0;
}
+ bitmap_name = bdrv_dirty_bitmap_name(bitmap);
+
if (!bs_name || strcmp(bs_name, "") == 0) {
error_report("Bitmap '%s' in unnamed node can't be migrated",
- bdrv_dirty_bitmap_name(bitmap));
+ bitmap_name);
return -1;
}
- if (bs_name[0] == '#') {
+ if (alias_map) {
+ const AliasMapInnerNode *amin = g_hash_table_lookup(alias_map, bs_name);
+
+ if (!amin) {
+ /* Skip bitmaps on nodes with no alias */
+ return 0;
+ }
+
+ node_alias = amin->string;
+ bitmap_aliases = amin->subtree;
+ } else {
+ node_alias = bs_name;
+ bitmap_aliases = NULL;
+ }
+
+ if (node_alias[0] == '#') {
error_report("Bitmap '%s' in a node with auto-generated "
"name '%s' can't be migrated",
- bdrv_dirty_bitmap_name(bitmap), bs_name);
+ bitmap_name, node_alias);
return -1;
}
FOR_EACH_DIRTY_BITMAP(bs, bitmap) {
- if (!bdrv_dirty_bitmap_name(bitmap)) {
+ bitmap_name = bdrv_dirty_bitmap_name(bitmap);
+ if (!bitmap_name) {
continue;
}
@@ -306,12 +492,23 @@ static int add_bitmaps_to_list(BlockDriverState *bs, const char *bs_name)
return -1;
}
+ if (bitmap_aliases) {
+ bitmap_alias = g_hash_table_lookup(bitmap_aliases, bitmap_name);
+ if (!bitmap_alias) {
+ /* Skip bitmaps with no alias */
+ continue;
+ }
+ } else {
+ bitmap_alias = bitmap_name;
+ }
+
bdrv_ref(bs);
bdrv_dirty_bitmap_set_busy(bitmap, true);
dbms = g_new0(DirtyBitmapMigBitmapState, 1);
dbms->bs = bs;
- dbms->node_name = bs_name;
+ dbms->node_alias = g_strdup(node_alias);
+ dbms->bitmap_alias = g_strdup(bitmap_alias);
dbms->bitmap = bitmap;
dbms->total_sectors = bdrv_nb_sectors(bs);
dbms->sectors_per_chunk = CHUNK_SIZE * 8 *
@@ -337,43 +534,52 @@ static int init_dirty_bitmap_migration(void)
DirtyBitmapMigBitmapState *dbms;
GHashTable *handled_by_blk = g_hash_table_new(NULL, NULL);
BlockBackend *blk;
+ const MigrationParameters *mig_params = &migrate_get_current()->parameters;
+ GHashTable *alias_map = NULL;
+
+ if (mig_params->has_block_bitmap_mapping) {
+ alias_map = construct_alias_map(mig_params->block_bitmap_mapping, true,
+ &error_abort);
+ }
dirty_bitmap_mig_state.bulk_completed = false;
dirty_bitmap_mig_state.prev_bs = NULL;
dirty_bitmap_mig_state.prev_bitmap = NULL;
dirty_bitmap_mig_state.no_bitmaps = false;
- /*
- * Use blockdevice name for direct (or filtered) children of named block
- * backends.
- */
- for (blk = blk_next(NULL); blk; blk = blk_next(blk)) {
- const char *name = blk_name(blk);
-
- if (!name || strcmp(name, "") == 0) {
- continue;
- }
+ if (!alias_map) {
+ /*
+ * Use blockdevice name for direct (or filtered) children of named block
+ * backends.
+ */
+ for (blk = blk_next(NULL); blk; blk = blk_next(blk)) {
+ const char *name = blk_name(blk);
- bs = blk_bs(blk);
+ if (!name || strcmp(name, "") == 0) {
+ continue;
+ }
- /* Skip filters without bitmaps */
- while (bs && bs->drv && bs->drv->is_filter &&
- !bdrv_has_named_bitmaps(bs))
- {
- if (bs->backing) {
- bs = bs->backing->bs;
- } else if (bs->file) {
- bs = bs->file->bs;
- } else {
- bs = NULL;
+ bs = blk_bs(blk);
+
+ /* Skip filters without bitmaps */
+ while (bs && bs->drv && bs->drv->is_filter &&
+ !bdrv_has_named_bitmaps(bs))
+ {
+ if (bs->backing) {
+ bs = bs->backing->bs;
+ } else if (bs->file) {
+ bs = bs->file->bs;
+ } else {
+ bs = NULL;
+ }
}
- }
- if (bs && bs->drv && !bs->drv->is_filter) {
- if (add_bitmaps_to_list(bs, name)) {
- goto fail;
+ if (bs && bs->drv && !bs->drv->is_filter) {
+ if (add_bitmaps_to_list(bs, name, NULL)) {
+ goto fail;
+ }
+ g_hash_table_add(handled_by_blk, bs);
}
- g_hash_table_add(handled_by_blk, bs);
}
}
@@ -382,7 +588,7 @@ static int init_dirty_bitmap_migration(void)
continue;
}
- if (add_bitmaps_to_list(bs, bdrv_get_node_name(bs))) {
+ if (add_bitmaps_to_list(bs, bdrv_get_node_name(bs), alias_map)) {
goto fail;
}
}
@@ -397,11 +603,17 @@ static int init_dirty_bitmap_migration(void)
}
g_hash_table_destroy(handled_by_blk);
+ if (alias_map) {
+ g_hash_table_destroy(alias_map);
+ }
return 0;
fail:
g_hash_table_destroy(handled_by_blk);
+ if (alias_map) {
+ g_hash_table_destroy(alias_map);
+ }
dirty_bitmap_mig_cleanup();
return -1;
@@ -666,8 +878,10 @@ static int dirty_bitmap_load_bits(QEMUFile *f, DirtyBitmapLoadState *s)
return 0;
}
-static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s)
+static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s,
+ GHashTable *alias_map)
{
+ GHashTable *bitmap_alias_map = NULL;
Error *local_err = NULL;
bool nothing;
s->flags = qemu_get_bitmap_flags(f);
@@ -676,25 +890,68 @@ static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s)
nothing = s->flags == (s->flags & DIRTY_BITMAP_MIG_FLAG_EOS);
if (s->flags & DIRTY_BITMAP_MIG_FLAG_DEVICE_NAME) {
- if (!qemu_get_counted_string(f, s->node_name)) {
- error_report("Unable to read node name string");
+ if (!qemu_get_counted_string(f, s->node_alias)) {
+ error_report("Unable to read node alias string");
return -EINVAL;
}
- s->bs = bdrv_lookup_bs(s->node_name, s->node_name, &local_err);
+
+ if (alias_map) {
+ const AliasMapInnerNode *amin;
+
+ amin = g_hash_table_lookup(alias_map, s->node_alias);
+ if (!amin) {
+ error_report("Error: Unknown node alias '%s'", s->node_alias);
+ return -EINVAL;
+ }
+
+ bitmap_alias_map = amin->subtree;
+ s->bs = bdrv_lookup_bs(NULL, amin->string, &local_err);
+ } else {
+ s->bs = bdrv_lookup_bs(s->node_alias, s->node_alias, &local_err);
+ }
if (!s->bs) {
error_report_err(local_err);
return -EINVAL;
}
- } else if (!s->bs && !nothing) {
+ } else if (s->bs) {
+ if (alias_map) {
+ const AliasMapInnerNode *amin;
+
+ /* Must be present in the map, or s->bs would not be set */
+ amin = g_hash_table_lookup(alias_map, s->node_alias);
+ assert(amin != NULL);
+
+ bitmap_alias_map = amin->subtree;
+ }
+ } else if (!nothing) {
error_report("Error: block device name is not set");
return -EINVAL;
}
+ assert(nothing || !!alias_map == !!bitmap_alias_map);
+
if (s->flags & DIRTY_BITMAP_MIG_FLAG_BITMAP_NAME) {
- if (!qemu_get_counted_string(f, s->bitmap_name)) {
+ const char *bitmap_name;
+
+ if (!qemu_get_counted_string(f, s->bitmap_alias)) {
error_report("Unable to read bitmap name string");
return -EINVAL;
}
+
+ if (bitmap_alias_map) {
+ bitmap_name = g_hash_table_lookup(bitmap_alias_map,
+ s->bitmap_alias);
+ if (!bitmap_name) {
+ error_report("Error: Unknown bitmap alias '%s' on node '%s' "
+ "(alias '%s')", s->bitmap_alias, s->bs->node_name,
+ s->node_alias);
+ return -EINVAL;
+ }
+ } else {
+ bitmap_name = s->bitmap_alias;
+ }
+
+ g_strlcpy(s->bitmap_name, bitmap_name, sizeof(s->bitmap_name));
s->bitmap = bdrv_find_dirty_bitmap(s->bs, s->bitmap_name);
/* bitmap may be NULL here, it wouldn't be an error if it is the
@@ -702,7 +959,7 @@ static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s)
if (!s->bitmap && !(s->flags & DIRTY_BITMAP_MIG_FLAG_START)) {
error_report("Error: unknown dirty bitmap "
"'%s' for block device '%s'",
- s->bitmap_name, s->node_name);
+ s->bitmap_name, s->bs->node_name);
return -EINVAL;
}
} else if (!s->bitmap && !nothing) {
@@ -715,6 +972,8 @@ static int dirty_bitmap_load_header(QEMUFile *f, DirtyBitmapLoadState *s)
static int dirty_bitmap_load(QEMUFile *f, void *opaque, int version_id)
{
+ GHashTable *alias_map = NULL;
+ const MigrationParameters *mig_params = &migrate_get_current()->parameters;
static DirtyBitmapLoadState s;
int ret = 0;
@@ -724,10 +983,15 @@ static int dirty_bitmap_load(QEMUFile *f, void *opaque, int version_id)
return -EINVAL;
}
+ if (mig_params->has_block_bitmap_mapping) {
+ alias_map = construct_alias_map(mig_params->block_bitmap_mapping,
+ false, &error_abort);
+ }
+
do {
- ret = dirty_bitmap_load_header(f, &s);
+ ret = dirty_bitmap_load_header(f, &s, alias_map);
if (ret < 0) {
- return ret;
+ goto fail;
}
if (s.flags & DIRTY_BITMAP_MIG_FLAG_START) {
@@ -743,12 +1007,17 @@ static int dirty_bitmap_load(QEMUFile *f, void *opaque, int version_id)
}
if (ret) {
- return ret;
+ goto fail;
}
} while (!(s.flags & DIRTY_BITMAP_MIG_FLAG_EOS));
trace_dirty_bitmap_load_success();
- return 0;
+ ret = 0;
+fail:
+ if (alias_map) {
+ g_hash_table_destroy(alias_map);
+ }
+ return ret;
}
static int dirty_bitmap_save_setup(QEMUFile *f, void *opaque)
diff --git a/migration/migration.c b/migration/migration.c
index 2ed9923227..bcf7c3bf76 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -36,6 +36,7 @@
#include "block/block.h"
#include "qapi/error.h"
#include "qapi/clone-visitor.h"
+#include "qapi/qapi-visit-migration.h"
#include "qapi/qapi-visit-sockets.h"
#include "qapi/qapi-commands-migration.h"
#include "qapi/qapi-events-migration.h"
@@ -832,6 +833,13 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
params->has_announce_step = true;
params->announce_step = s->parameters.announce_step;
+ if (s->parameters.has_block_bitmap_mapping) {
+ params->has_block_bitmap_mapping = true;
+ params->block_bitmap_mapping =
+ QAPI_CLONE(BitmapMigrationNodeAliasList,
+ s->parameters.block_bitmap_mapping);
+ }
+
return params;
}
@@ -1297,6 +1305,13 @@ static bool migrate_params_check(MigrationParameters *params, Error **errp)
"is invalid, it must be in the range of 1 to 10000 ms");
return false;
}
+
+ if (params->has_block_bitmap_mapping &&
+ !check_dirty_bitmap_mig_alias_map(params->block_bitmap_mapping, errp)) {
+ error_prepend(errp, "Invalid mapping given for block-bitmap-mapping: ");
+ return false;
+ }
+
return true;
}
@@ -1391,6 +1406,11 @@ static void migrate_params_test_apply(MigrateSetParameters *params,
if (params->has_announce_step) {
dest->announce_step = params->announce_step;
}
+
+ if (params->has_block_bitmap_mapping) {
+ dest->has_block_bitmap_mapping = true;
+ dest->block_bitmap_mapping = params->block_bitmap_mapping;
+ }
}
static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
@@ -1503,6 +1523,16 @@ static void migrate_params_apply(MigrateSetParameters *params, Error **errp)
if (params->has_announce_step) {
s->parameters.announce_step = params->announce_step;
}
+
+ if (params->has_block_bitmap_mapping) {
+ qapi_free_BitmapMigrationNodeAliasList(
+ s->parameters.block_bitmap_mapping);
+
+ s->parameters.has_block_bitmap_mapping = true;
+ s->parameters.block_bitmap_mapping =
+ QAPI_CLONE(BitmapMigrationNodeAliasList,
+ params->block_bitmap_mapping);
+ }
}
void qmp_migrate_set_parameters(MigrateSetParameters *params, Error **errp)
diff --git a/monitor/hmp-cmds.c b/monitor/hmp-cmds.c
index ae4b6a4246..7711726fd2 100644
--- a/monitor/hmp-cmds.c
+++ b/monitor/hmp-cmds.c
@@ -469,6 +469,32 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
monitor_printf(mon, "%s: '%s'\n",
MigrationParameter_str(MIGRATION_PARAMETER_TLS_AUTHZ),
params->tls_authz);
+
+ if (params->has_block_bitmap_mapping) {
+ const BitmapMigrationNodeAliasList *bmnal;
+
+ monitor_printf(mon, "%s:\n",
+ MigrationParameter_str(
+ MIGRATION_PARAMETER_BLOCK_BITMAP_MAPPING));
+
+ for (bmnal = params->block_bitmap_mapping;
+ bmnal;
+ bmnal = bmnal->next)
+ {
+ const BitmapMigrationNodeAlias *bmna = bmnal->value;
+ const BitmapMigrationBitmapAliasList *bmbal;
+
+ monitor_printf(mon, " '%s' -> '%s'\n",
+ bmna->node_name, bmna->alias);
+
+ for (bmbal = bmna->bitmaps; bmbal; bmbal = bmbal->next) {
+ const BitmapMigrationBitmapAlias *bmba = bmbal->value;
+
+ monitor_printf(mon, " '%s' -> '%s'\n",
+ bmba->name, bmba->alias);
+ }
+ }
+ }
}
qapi_free_MigrationParameters(params);
@@ -1384,6 +1410,10 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
p->has_announce_step = true;
visit_type_size(v, param, &p->announce_step, &err);
break;
+ case MIGRATION_PARAMETER_BLOCK_BITMAP_MAPPING:
+ error_setg(&err, "The block-bitmap-mapping parameter can only be set "
+ "through QMP");
+ break;
default:
assert(0);
}
--
2.26.2
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH for-5.2 v3 2/3] iotests.py: Add wait_for_runstate()
2020-07-22 8:05 [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter Max Reitz
2020-07-22 8:05 ` [PATCH for-5.2 v3 1/3] " Max Reitz
@ 2020-07-22 8:05 ` Max Reitz
2020-07-22 8:05 ` [PATCH for-5.2 v3 3/3] iotests: Test node/bitmap aliases during migration Max Reitz
2020-08-12 14:15 ` [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter Max Reitz
3 siblings, 0 replies; 9+ messages in thread
From: Max Reitz @ 2020-07-22 8:05 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy, Juan Quintela,
John Snow, qemu-devel, Dr . David Alan Gilbert, Peter Krempa,
Max Reitz
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/iotests.py | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/tests/qemu-iotests/iotests.py b/tests/qemu-iotests/iotests.py
index 3590ed78a0..20645a6e7d 100644
--- a/tests/qemu-iotests/iotests.py
+++ b/tests/qemu-iotests/iotests.py
@@ -28,6 +28,7 @@ import signal
import struct
import subprocess
import sys
+import time
from typing import (Any, Callable, Dict, Iterable,
List, Optional, Sequence, Tuple, TypeVar)
import unittest
@@ -803,6 +804,10 @@ class VM(qtest.QEMUQtestMachine):
'Found node %s under %s (but expected %s)' % \
(node['name'], path, expected_node)
+ def wait_for_runstate(self, runstate: str) -> None:
+ while self.qmp('query-status')['return']['status'] != runstate:
+ time.sleep(0.2)
+
index_re = re.compile(r'([^\[]+)\[([^\]]+)\]')
class QMPTestCase(unittest.TestCase):
--
2.26.2
^ permalink raw reply related [flat|nested] 9+ messages in thread
* [PATCH for-5.2 v3 3/3] iotests: Test node/bitmap aliases during migration
2020-07-22 8:05 [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter Max Reitz
2020-07-22 8:05 ` [PATCH for-5.2 v3 1/3] " Max Reitz
2020-07-22 8:05 ` [PATCH for-5.2 v3 2/3] iotests.py: Add wait_for_runstate() Max Reitz
@ 2020-07-22 8:05 ` Max Reitz
2020-08-12 14:15 ` [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter Max Reitz
3 siblings, 0 replies; 9+ messages in thread
From: Max Reitz @ 2020-07-22 8:05 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy, Juan Quintela,
John Snow, qemu-devel, Dr . David Alan Gilbert, Peter Krempa,
Max Reitz
Signed-off-by: Max Reitz <mreitz@redhat.com>
---
tests/qemu-iotests/300 | 515 +++++++++++++++++++++++++++++++++++++
tests/qemu-iotests/300.out | 5 +
tests/qemu-iotests/group | 1 +
3 files changed, 521 insertions(+)
create mode 100755 tests/qemu-iotests/300
create mode 100644 tests/qemu-iotests/300.out
diff --git a/tests/qemu-iotests/300 b/tests/qemu-iotests/300
new file mode 100755
index 0000000000..6d684e882a
--- /dev/null
+++ b/tests/qemu-iotests/300
@@ -0,0 +1,515 @@
+#!/usr/bin/env python3
+#
+# Copyright (C) 2020 Red Hat, Inc.
+#
+# Tests for dirty bitmaps migration with node aliases
+#
+# This program is free software; you can redistribute it and/or modify
+# it under the terms of the GNU General Public License as published by
+# the Free Software Foundation; either version 2 of the License, or
+# (at your option) any later version.
+#
+# This program is distributed in the hope that it will be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program. If not, see <http://www.gnu.org/licenses/>.
+#
+
+import os
+import random
+import re
+import time
+from typing import Dict, List, Optional, Union
+import iotests
+import qemu
+
+BlockBitmapMapping = List[Dict[str, Union[str, List[Dict[str, str]]]]]
+
+assert iotests.sock_dir is not None
+mig_sock = os.path.join(iotests.sock_dir, 'mig_sock')
+
+class TestDirtyBitmapMigration(iotests.QMPTestCase):
+ src_node_name: str = ''
+ dst_node_name: str = ''
+ src_bmap_name: str = ''
+ dst_bmap_name: str = ''
+
+ def setUp(self) -> None:
+ self.vm_a = iotests.VM(path_suffix='-a')
+ self.vm_a.add_blockdev(f'node-name={self.src_node_name},' \
+ 'driver=null-co')
+ self.vm_a.launch()
+
+ self.vm_b = iotests.VM(path_suffix='-b')
+ self.vm_b.add_blockdev(f'node-name={self.dst_node_name},' \
+ 'driver=null-co')
+ self.vm_b.add_incoming(f'unix:{mig_sock}')
+ self.vm_b.launch()
+
+ result = self.vm_a.qmp('block-dirty-bitmap-add',
+ node=self.src_node_name,
+ name=self.src_bmap_name)
+ self.assert_qmp(result, 'return', {})
+
+ # Dirty some random megabytes
+ for _ in range(9):
+ mb_ofs = random.randrange(1024)
+ self.vm_a.hmp_qemu_io(self.src_node_name, f'write {mb_ofs}M 1M')
+
+ result = self.vm_a.qmp('x-debug-block-dirty-bitmap-sha256',
+ node=self.src_node_name,
+ name=self.src_bmap_name)
+ self.bitmap_hash_reference = result['return']['sha256']
+
+ caps = [{'capability': name, 'state': True} \
+ for name in ('dirty-bitmaps', 'events')]
+
+ for vm in (self.vm_a, self.vm_b):
+ result = vm.qmp('migrate-set-capabilities', capabilities=caps)
+ self.assert_qmp(result, 'return', {})
+
+ def tearDown(self) -> None:
+ self.vm_a.shutdown()
+ self.vm_b.shutdown()
+ try:
+ os.remove(mig_sock)
+ except OSError:
+ pass
+
+ def check_bitmap(self, bitmap_name_valid: bool) -> None:
+ result = self.vm_b.qmp('x-debug-block-dirty-bitmap-sha256',
+ node=self.dst_node_name,
+ name=self.dst_bmap_name)
+ if bitmap_name_valid:
+ self.assert_qmp(result, 'return/sha256',
+ self.bitmap_hash_reference)
+ else:
+ self.assert_qmp(result, 'error/desc',
+ f"Dirty bitmap '{self.dst_bmap_name}' not found")
+
+ def migrate(self, migration_success: bool = True,
+ bitmap_name_valid: bool = True) -> None:
+ result = self.vm_a.qmp('migrate', uri=f'unix:{mig_sock}')
+ self.assert_qmp(result, 'return', {})
+
+ status = None
+ while status not in ('completed', 'failed'):
+ status = self.vm_a.event_wait('MIGRATION')['data']['status']
+
+ self.assertEqual(status == 'completed', migration_success)
+ if status == 'failed':
+ # Wait until the migration has been cleaned up
+ # (Otherwise, bdrv_close_all() will abort because the
+ # dirty bitmap migration code still holds a reference to
+ # the BDS)
+ # (Unfortunately, there does not appear to be a nicer way
+ # of waiting until a failed migration has been cleaned up)
+ timeout_msg = 'Timeout waiting for migration to be cleaned up'
+ with iotests.Timeout(30, timeout_msg):
+ while os.path.exists(mig_sock):
+ time.sleep(0.1)
+
+ # Dropping src_node_name will only work once the
+ # bitmap migration code has released it
+ while 'error' in self.vm_a.qmp('blockdev-del',
+ node_name=self.src_node_name):
+ time.sleep(0.1)
+
+ return
+
+ self.vm_a.wait_for_runstate('postmigrate')
+ self.vm_b.wait_migration('running')
+
+ self.check_bitmap(bitmap_name_valid)
+
+ @staticmethod
+ def mapping(node_name: str, node_alias: str,
+ bitmap_name: str, bitmap_alias: str) \
+ -> BlockBitmapMapping:
+ return [{
+ 'node-name': node_name,
+ 'alias': node_alias,
+ 'bitmaps': [{
+ 'name': bitmap_name,
+ 'alias': bitmap_alias
+ }]
+ }]
+
+ def set_mapping(self, vm: iotests.VM, mapping: BlockBitmapMapping,
+ error: Optional[str] = None) \
+ -> None:
+ """
+ Invoke migrate-set-parameters on @vm to set the given @mapping.
+ Check for success if @error is None, or verify the error message
+ if it is not.
+ On success, verify that "info migrate_parameters" on HMP returns
+ our mapping. (Just to check its formatting code.)
+ """
+ result = vm.qmp('migrate-set-parameters',
+ block_bitmap_mapping=mapping)
+
+ if error is None:
+ self.assert_qmp(result, 'return', {})
+
+ result = vm.qmp('human-monitor-command',
+ command_line='info migrate_parameters')
+
+ hmp_mapping = re.search(r'^block-bitmap-mapping:\r?(\n .*)*\n',
+ result['return'], flags=re.MULTILINE)
+
+ self.assertEqual(hmp_mapping.group(0).replace('\r', ''),
+ self.to_hmp_mapping(mapping))
+ else:
+ self.assert_qmp(result, 'error/desc', error)
+
+ @staticmethod
+ def to_hmp_mapping(mapping: BlockBitmapMapping) -> str:
+ result = 'block-bitmap-mapping:\n'
+
+ for node in mapping:
+ result += f" '{node['node-name']}' -> '{node['alias']}'\n"
+
+ assert isinstance(node['bitmaps'], list)
+ for bitmap in node['bitmaps']:
+ result += f" '{bitmap['name']}' -> '{bitmap['alias']}'\n"
+
+ return result
+
+
+class TestAliasMigration(TestDirtyBitmapMigration):
+ src_node_name = 'node0'
+ dst_node_name = 'node0'
+ src_bmap_name = 'bmap0'
+ dst_bmap_name = 'bmap0'
+
+ def test_migration_without_alias(self) -> None:
+ self.migrate(self.src_node_name == self.dst_node_name,
+ self.src_bmap_name == self.dst_bmap_name)
+
+ # Expect abnormal shutdown of the destination VM on migration failure
+ if self.src_node_name != self.dst_node_name:
+ try:
+ self.vm_b.shutdown()
+ except qemu.machine.AbnormalShutdown:
+ pass
+
+ def test_alias_on_src_migration(self) -> None:
+ mapping = self.mapping(self.src_node_name, self.dst_node_name,
+ self.src_bmap_name, self.dst_bmap_name)
+
+ self.set_mapping(self.vm_a, mapping)
+ self.migrate()
+
+ def test_alias_on_dst_migration(self) -> None:
+ mapping = self.mapping(self.dst_node_name, self.src_node_name,
+ self.dst_bmap_name, self.src_bmap_name)
+
+ self.set_mapping(self.vm_b, mapping)
+ self.migrate()
+
+ def test_alias_on_both_migration(self) -> None:
+ src_map = self.mapping(self.src_node_name, 'node-alias',
+ self.src_bmap_name, 'bmap-alias')
+
+ dst_map = self.mapping(self.dst_node_name, 'node-alias',
+ self.dst_bmap_name, 'bmap-alias')
+
+ self.set_mapping(self.vm_a, src_map)
+ self.set_mapping(self.vm_b, dst_map)
+ self.migrate()
+
+
+class TestNodeAliasMigration(TestAliasMigration):
+ src_node_name = 'node-src'
+ dst_node_name = 'node-dst'
+
+
+class TestBitmapAliasMigration(TestAliasMigration):
+ src_bmap_name = 'bmap-src'
+ dst_bmap_name = 'bmap-dst'
+
+
+class TestFullAliasMigration(TestAliasMigration):
+ src_node_name = 'node-src'
+ dst_node_name = 'node-dst'
+ src_bmap_name = 'bmap-src'
+ dst_bmap_name = 'bmap-dst'
+
+
+class TestBlockBitmapMappingErrors(TestDirtyBitmapMigration):
+ src_node_name = 'node0'
+ dst_node_name = 'node0'
+ src_bmap_name = 'bmap0'
+ dst_bmap_name = 'bmap0'
+
+ """
+ Note that mapping nodes or bitmaps that do not exist is not an error.
+ """
+
+ def test_non_injective_node_mapping(self) -> None:
+ mapping: BlockBitmapMapping = [
+ {
+ 'node-name': 'node0',
+ 'alias': 'common-alias',
+ 'bitmaps': [{
+ 'name': 'bmap0',
+ 'alias': 'bmap-alias0'
+ }]
+ },
+ {
+ 'node-name': 'node1',
+ 'alias': 'common-alias',
+ 'bitmaps': [{
+ 'name': 'bmap1',
+ 'alias': 'bmap-alias1'
+ }]
+ }
+ ]
+
+ self.set_mapping(self.vm_a, mapping,
+ 'Invalid mapping given for block-bitmap-mapping: ' \
+ 'The node alias common-alias is used twice')
+
+ def test_non_injective_bitmap_mapping(self) -> None:
+ mapping: BlockBitmapMapping = [{
+ 'node-name': 'node0',
+ 'alias': 'node-alias0',
+ 'bitmaps': [
+ {
+ 'name': 'bmap0',
+ 'alias': 'common-alias'
+ },
+ {
+ 'name': 'bmap1',
+ 'alias': 'common-alias'
+ }
+ ]
+ }]
+
+ self.set_mapping(self.vm_a, mapping,
+ 'Invalid mapping given for block-bitmap-mapping: ' \
+ 'The bitmap alias node-alias0/common-alias is used ' \
+ 'twice')
+
+ def test_ambiguous_node_mapping(self) -> None:
+ mapping: BlockBitmapMapping = [
+ {
+ 'node-name': 'node0',
+ 'alias': 'node-alias0',
+ 'bitmaps': [{
+ 'name': 'bmap0',
+ 'alias': 'bmap-alias0'
+ }]
+ },
+ {
+ 'node-name': 'node0',
+ 'alias': 'node-alias1',
+ 'bitmaps': [{
+ 'name': 'bmap0',
+ 'alias': 'bmap-alias0'
+ }]
+ }
+ ]
+
+ self.set_mapping(self.vm_a, mapping,
+ 'Invalid mapping given for block-bitmap-mapping: ' \
+ 'The node name node0 is mapped twice')
+
+ def test_ambiguous_bitmap_mapping(self) -> None:
+ mapping: BlockBitmapMapping = [{
+ 'node-name': 'node0',
+ 'alias': 'node-alias0',
+ 'bitmaps': [
+ {
+ 'name': 'bmap0',
+ 'alias': 'bmap-alias0'
+ },
+ {
+ 'name': 'bmap0',
+ 'alias': 'bmap-alias1'
+ }
+ ]
+ }]
+
+ self.set_mapping(self.vm_a, mapping,
+ 'Invalid mapping given for block-bitmap-mapping: ' \
+ 'The bitmap node0/bmap0 is mapped twice')
+
+ def test_migratee_node_is_not_mapped_on_src(self) -> None:
+ self.set_mapping(self.vm_a, [])
+ # Should just ignore all bitmaps on unmapped nodes
+ self.migrate(True, False)
+
+ def test_migratee_node_is_not_mapped_on_dst(self) -> None:
+ self.set_mapping(self.vm_b, [])
+ self.migrate(False, False)
+
+ # Expect abnormal shutdown of the destination VM on migration failure
+ try:
+ self.vm_b.shutdown()
+ except qemu.machine.AbnormalShutdown:
+ pass
+
+ self.assertIn(f"Unknown node alias '{self.src_node_name}'",
+ self.vm_b.get_log())
+
+ def test_migratee_bitmap_is_not_mapped_on_src(self) -> None:
+ mapping: BlockBitmapMapping = [{
+ 'node-name': self.src_node_name,
+ 'alias': self.dst_node_name,
+ 'bitmaps': []
+ }]
+
+ self.set_mapping(self.vm_a, mapping)
+ # Should just ignore all unmapped bitmaps
+ self.migrate(True, False)
+
+ def test_migratee_bitmap_is_not_mapped_on_dst(self) -> None:
+ mapping: BlockBitmapMapping = [{
+ 'node-name': self.dst_node_name,
+ 'alias': self.src_node_name,
+ 'bitmaps': []
+ }]
+
+ self.set_mapping(self.vm_b, mapping)
+ self.migrate(False, False)
+
+ # Expect abnormal shutdown of the destination VM on migration failure
+ try:
+ self.vm_b.shutdown()
+ except qemu.machine.AbnormalShutdown:
+ pass
+
+ self.assertIn(f"Unknown bitmap alias '{self.src_bmap_name}' on node " \
+ f"'{self.dst_node_name}' (alias '{self.src_node_name}')",
+ self.vm_b.get_log())
+
+ def test_unused_mapping_on_dst(self) -> None:
+ # Let the source not send any bitmaps
+ self.set_mapping(self.vm_a, [])
+
+ # Establish some mapping on the destination
+ self.set_mapping(self.vm_b, [])
+
+ # The fact that there is a mapping on B without any bitmaps
+ # being received should be fine, not fatal
+ self.migrate(True, False)
+
+ def test_non_wellformed_node_alias(self) -> None:
+ mapping: BlockBitmapMapping = [{
+ 'node-name': self.src_bmap_name,
+ 'alias': '123-foo',
+ 'bitmaps': []
+ }]
+
+ self.set_mapping(self.vm_a, mapping,
+ 'Invalid mapping given for block-bitmap-mapping: ' \
+ 'The node alias 123-foo is not well-formed')
+
+
+class TestCrossAliasMigration(TestDirtyBitmapMigration):
+ """
+ Swap aliases, both to see that qemu does not get confused, and
+ that we can migrate multiple things at once.
+
+ So we migrate this:
+ node-a.bmap-a -> node-b.bmap-b
+ node-a.bmap-b -> node-b.bmap-a
+ node-b.bmap-a -> node-a.bmap-b
+ node-b.bmap-b -> node-a.bmap-a
+ """
+
+ src_node_name = 'node-a'
+ dst_node_name = 'node-b'
+ src_bmap_name = 'bmap-a'
+ dst_bmap_name = 'bmap-b'
+
+ def setUp(self) -> None:
+ TestDirtyBitmapMigration.setUp(self)
+
+ # Now create another block device and let both have two bitmaps each
+ result = self.vm_a.qmp('blockdev-add',
+ node_name='node-b', driver='null-co')
+ self.assert_qmp(result, 'return', {})
+
+ result = self.vm_b.qmp('blockdev-add',
+ node_name='node-a', driver='null-co')
+ self.assert_qmp(result, 'return', {})
+
+ bmaps_to_add = (('node-a', 'bmap-b'),
+ ('node-b', 'bmap-a'),
+ ('node-b', 'bmap-b'))
+
+ for (node, bmap) in bmaps_to_add:
+ result = self.vm_a.qmp('block-dirty-bitmap-add',
+ node=node, name=bmap)
+ self.assert_qmp(result, 'return', {})
+
+ @staticmethod
+ def cross_mapping() -> BlockBitmapMapping:
+ return [
+ {
+ 'node-name': 'node-a',
+ 'alias': 'node-b',
+ 'bitmaps': [
+ {
+ 'name': 'bmap-a',
+ 'alias': 'bmap-b'
+ },
+ {
+ 'name': 'bmap-b',
+ 'alias': 'bmap-a'
+ }
+ ]
+ },
+ {
+ 'node-name': 'node-b',
+ 'alias': 'node-a',
+ 'bitmaps': [
+ {
+ 'name': 'bmap-b',
+ 'alias': 'bmap-a'
+ },
+ {
+ 'name': 'bmap-a',
+ 'alias': 'bmap-b'
+ }
+ ]
+ }
+ ]
+
+ def verify_dest_has_all_bitmaps(self) -> None:
+ bitmaps = self.vm_b.query_bitmaps()
+
+ # Extract and sort bitmap names
+ for node in bitmaps:
+ bitmaps[node] = sorted((bmap['name'] for bmap in bitmaps[node]))
+
+ self.assertEqual(bitmaps,
+ {'node-a': ['bmap-a', 'bmap-b'],
+ 'node-b': ['bmap-a', 'bmap-b']})
+
+ def test_alias_on_src(self) -> None:
+ self.set_mapping(self.vm_a, self.cross_mapping())
+
+ # Checks that node-a.bmap-a was migrated to node-b.bmap-b, and
+ # that is enough
+ self.migrate()
+
+ self.verify_dest_has_all_bitmaps()
+
+ def test_alias_on_dst(self) -> None:
+ self.set_mapping(self.vm_b, self.cross_mapping())
+
+ # Checks that node-a.bmap-a was migrated to node-b.bmap-b, and
+ # that is enough
+ self.migrate()
+
+ self.verify_dest_has_all_bitmaps()
+
+
+if __name__ == '__main__':
+ iotests.main(supported_protocols=['file'])
diff --git a/tests/qemu-iotests/300.out b/tests/qemu-iotests/300.out
new file mode 100644
index 0000000000..af8dac10f9
--- /dev/null
+++ b/tests/qemu-iotests/300.out
@@ -0,0 +1,5 @@
+............................
+----------------------------------------------------------------------
+Ran 28 tests
+
+OK
diff --git a/tests/qemu-iotests/group b/tests/qemu-iotests/group
index e8d8747887..0f3907e7ad 100644
--- a/tests/qemu-iotests/group
+++ b/tests/qemu-iotests/group
@@ -308,4 +308,5 @@
296 rw
297 meta
299 auto quick
+300 migration
301 backing quick
--
2.26.2
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter
2020-07-22 8:05 [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter Max Reitz
` (2 preceding siblings ...)
2020-07-22 8:05 ` [PATCH for-5.2 v3 3/3] iotests: Test node/bitmap aliases during migration Max Reitz
@ 2020-08-12 14:15 ` Max Reitz
2020-08-12 14:38 ` Vladimir Sementsov-Ogievskiy
3 siblings, 1 reply; 9+ messages in thread
From: Max Reitz @ 2020-08-12 14:15 UTC (permalink / raw)
To: qemu-block
Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy, Juan Quintela,
John Snow, qemu-devel, Dr . David Alan Gilbert, Peter Krempa
[-- Attachment #1.1: Type: text/plain, Size: 3614 bytes --]
Ping – seems like everyone found v2 more or less acceptable bar the
failing assertion in patch 1, and some aspects of the test. How about
v3, are there any objections?
On 22.07.20 10:05, Max Reitz wrote:
> RFC v1: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00912.html
> RFC v2: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00915.html
> v1: https://lists.nongnu.org/archive/html/qemu-devel/2020-06/msg09792.html
> v2: https://lists.nongnu.org/archive/html/qemu-block/2020-07/msg01179.html
>
> Branch: https://github.com/XanClic/qemu.git migration-bitmap-mapping-v3
> Branch: https://git.xanclic.moe/XanClic/qemu.git migration-bitmap-mapping-v3
>
> Hi,
>
> This new migration parameter allows mapping block node names and bitmap
> names to aliases for the purpose of block dirty bitmap migration.
>
> This way, management tools can use different node names on the source
> and destination and pass the mapping of how bitmaps are to be
> transferred to qemu (on the source, the destination, or even both with
> arbitrary aliases in the migration stream).
>
>
> v3:
> - Patch 1:
> - Add notes on the fact that the destination won’t ever know about any
> bitmaps that aren’t sent, so you can freely drop what you want, and
> you’re completely free in renaming bitmaps and putting them on
> “other” nodes (whatever “other” node means in the context of
> migration, because that’s kind of one of the problems this series is
> trying to solve: The fact that you can’t trivially match nodes
> between source and destination)
> - Fix an assertion
>
> - Patch 2: s/pass/time.sleep(0.2)/
>
> - Patch 3:
> - Add copyright line
> - Use format string instead of %
> - s/pass/time.sleep(0.1)/
> - s/wait_for_runstate/wait_migration/ on the destination to wait for
> the migration to actually complete
> - Replace the “info migrate_parameters” parsing code by a multiline
> regex
> - Test what happens when the destination has a mapping that isn’t used
> because there are not bitmaps to be transferred (which breaks the
> assertion in patch 1 as it was in v2)
> - Let verify_dest_has_all_bitmaps() actually verify the bitmaps on the
> destination instead of the source
>
>
> git-backport-diff against v2:
>
> Key:
> [----] : patches are identical
> [####] : number of functional differences between upstream/downstream patch
> [down] : patch is downstream-only
> The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
>
> 001/3:[0014] [FC] 'migration: Add block-bitmap-mapping parameter'
> 002/3:[0003] [FC] 'iotests.py: Add wait_for_runstate()'
> 003/3:[0046] [FC] 'iotests: Test node/bitmap aliases during migration'
>
>
> Max Reitz (3):
> migration: Add block-bitmap-mapping parameter
> iotests.py: Add wait_for_runstate()
> iotests: Test node/bitmap aliases during migration
>
> qapi/migration.json | 104 ++++++-
> migration/migration.h | 3 +
> migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++----
> migration/migration.c | 30 ++
> monitor/hmp-cmds.c | 30 ++
> tests/qemu-iotests/300 | 515 +++++++++++++++++++++++++++++++++
> tests/qemu-iotests/300.out | 5 +
> tests/qemu-iotests/group | 1 +
> tests/qemu-iotests/iotests.py | 5 +
> 9 files changed, 1011 insertions(+), 55 deletions(-)
> create mode 100755 tests/qemu-iotests/300
> create mode 100644 tests/qemu-iotests/300.out
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH for-5.2 v3 1/3] migration: Add block-bitmap-mapping parameter
2020-07-22 8:05 ` [PATCH for-5.2 v3 1/3] " Max Reitz
@ 2020-08-12 14:32 ` Eric Blake
2020-08-13 13:03 ` Max Reitz
0 siblings, 1 reply; 9+ messages in thread
From: Eric Blake @ 2020-08-12 14:32 UTC (permalink / raw)
To: Max Reitz, qemu-block
Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy, Juan Quintela,
qemu-devel, Dr . David Alan Gilbert, Peter Krempa, John Snow
On 7/22/20 3:05 AM, Max Reitz wrote:
> This migration parameter allows mapping block node names and bitmap
> names to aliases for the purpose of block dirty bitmap migration.
>
> This way, management tools can use different node and bitmap names on
> the source and destination and pass the mapping of how bitmaps are to be
> transferred to qemu (on the source, the destination, or even both with
> arbitrary aliases in the migration stream).
>
> Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
> Signed-off-by: Max Reitz <mreitz@redhat.com>
> ---
> qapi/migration.json | 104 ++++++++-
> migration/migration.h | 3 +
> migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++++++++++-----
> migration/migration.c | 30 +++
> monitor/hmp-cmds.c | 30 +++
> 5 files changed, 485 insertions(+), 55 deletions(-)
>
> diff --git a/qapi/migration.json b/qapi/migration.json
> index d5000558c6..d7e953cd73 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -507,6 +507,44 @@
> 'data': [ 'none', 'zlib',
> { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
>
> +##
> +# @BitmapMigrationBitmapAlias:
> +#
> +# @name: The name of the bitmap.
> +#
> +# @alias: An alias name for migration (for example the bitmap name on
> +# the opposite site).
> +#
> +# Since: 5.1
5.2, now, but I can touch that up if it is the only problem raised.
> +# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
> +# aliases for the purpose of dirty bitmap migration. Such
> +# aliases may for example be the corresponding names on the
> +# opposite site.
> +# The mapping must be one-to-one, and on the destination also
> +# complete: On the source, bitmaps on nodes where either the
> +# bitmap or the node is not mapped will be ignored. In
> +# contrast, on the destination, receiving a bitmap (by alias)
> +# from a node (by alias) when either alias is not mapped will
> +# result in an error.
Grammar is a bit odd and it feels repetitive. How about:
The mapping must not provide more than one alias for a bitmap, nor reuse
the same alias across multiple bitmaps in the same node. On the source,
an omitted node or bitmap within a node will ignore those bitmaps. In
contrast, on the destination, receiving a bitmap (by alias) from a node
(by alias) when either alias is not mapped will result in an error.
> +# Note that the destination does not know about bitmaps it
> +# does not receive, so there is no limitation or requirement
> +# regarding the number of bitmaps received, or how they are
> +# named, or on which nodes they are placed.
> +# By default (when this parameter has never been set), bitmap
> +# names are mapped to themselves. Nodes are mapped to their
> +# block device name if there is one, and to their node name
> +# otherwise. (Since 5.2)
Looks good.
> @@ -781,6 +839,25 @@
> # will consume more CPU.
> # Defaults to 1. (Since 5.0)
> #
> +# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
> +# aliases for the purpose of dirty bitmap migration. Such
> +# aliases may for example be the corresponding names on the
> +# opposite site.
Ah, the joys of duplicated text.
> +++ b/migration/block-dirty-bitmap.c
> @@ -128,7 +131,8 @@ typedef struct DirtyBitmapMigState {
>
> typedef struct DirtyBitmapLoadState {
> uint32_t flags;
> - char node_name[256];
> + char node_alias[256];
> + char bitmap_alias[256];
Do we properly check that alias names will never overflow?
> +static GHashTable *construct_alias_map(const BitmapMigrationNodeAliasList *bbm,
> + bool name_to_alias,
> + Error **errp)
> +{
> + GHashTable *alias_map;
> +
> + alias_map = g_hash_table_new_full(g_str_hash, g_str_equal,
> + g_free, free_alias_map_inner_node);
> +
> + for (; bbm; bbm = bbm->next) {
> + const BitmapMigrationNodeAlias *bmna = bbm->value;
> + const BitmapMigrationBitmapAliasList *bmbal;
> + AliasMapInnerNode *amin;
> + GHashTable *bitmaps_map;
> + const char *node_map_from, *node_map_to;
> +
> + if (!id_wellformed(bmna->alias)) {
> + error_setg(errp, "The node alias %s is not well-formed",
> + bmna->alias);
> + goto fail;
> + }
Maybe id_wellformed should enforce lengths? Otherwise, I'm not seeing a
length limit applied during the mapping process.
Modulo that, I think we're ready to go.
--
Eric Blake, Principal Software Engineer
Red Hat, Inc. +1-919-301-3226
Virtualization: qemu.org | libvirt.org
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter
2020-08-12 14:15 ` [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter Max Reitz
@ 2020-08-12 14:38 ` Vladimir Sementsov-Ogievskiy
2020-08-13 13:04 ` Max Reitz
0 siblings, 1 reply; 9+ messages in thread
From: Vladimir Sementsov-Ogievskiy @ 2020-08-12 14:38 UTC (permalink / raw)
To: Max Reitz, qemu-block
Cc: Kevin Wolf, Peter Krempa, Juan Quintela, John Snow, qemu-devel,
Dr . David Alan Gilbert
Now this doesn't apply, as code changed a lot after my series
"[PATCH v4 for-5.1 00/21] Fix error handling during bitmap postcopy"
merged first.
I feel my responsibility for the mess with these series, so if you want I can try to rebase and post v4 of this one myself :)
12.08.2020 17:15, Max Reitz wrote:
> Ping – seems like everyone found v2 more or less acceptable bar the
> failing assertion in patch 1, and some aspects of the test. How about
> v3, are there any objections?
>
> On 22.07.20 10:05, Max Reitz wrote:
>> RFC v1: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00912.html
>> RFC v2: https://lists.nongnu.org/archive/html/qemu-block/2020-05/msg00915.html
>> v1: https://lists.nongnu.org/archive/html/qemu-devel/2020-06/msg09792.html
>> v2: https://lists.nongnu.org/archive/html/qemu-block/2020-07/msg01179.html
>>
>> Branch: https://github.com/XanClic/qemu.git migration-bitmap-mapping-v3
>> Branch: https://git.xanclic.moe/XanClic/qemu.git migration-bitmap-mapping-v3
>>
>> Hi,
>>
>> This new migration parameter allows mapping block node names and bitmap
>> names to aliases for the purpose of block dirty bitmap migration.
>>
>> This way, management tools can use different node names on the source
>> and destination and pass the mapping of how bitmaps are to be
>> transferred to qemu (on the source, the destination, or even both with
>> arbitrary aliases in the migration stream).
>>
>>
>> v3:
>> - Patch 1:
>> - Add notes on the fact that the destination won’t ever know about any
>> bitmaps that aren’t sent, so you can freely drop what you want, and
>> you’re completely free in renaming bitmaps and putting them on
>> “other” nodes (whatever “other” node means in the context of
>> migration, because that’s kind of one of the problems this series is
>> trying to solve: The fact that you can’t trivially match nodes
>> between source and destination)
>> - Fix an assertion
>>
>> - Patch 2: s/pass/time.sleep(0.2)/
>>
>> - Patch 3:
>> - Add copyright line
>> - Use format string instead of %
>> - s/pass/time.sleep(0.1)/
>> - s/wait_for_runstate/wait_migration/ on the destination to wait for
>> the migration to actually complete
>> - Replace the “info migrate_parameters” parsing code by a multiline
>> regex
>> - Test what happens when the destination has a mapping that isn’t used
>> because there are not bitmaps to be transferred (which breaks the
>> assertion in patch 1 as it was in v2)
>> - Let verify_dest_has_all_bitmaps() actually verify the bitmaps on the
>> destination instead of the source
>>
>>
>> git-backport-diff against v2:
>>
>> Key:
>> [----] : patches are identical
>> [####] : number of functional differences between upstream/downstream patch
>> [down] : patch is downstream-only
>> The flags [FC] indicate (F)unctional and (C)ontextual differences, respectively
>>
>> 001/3:[0014] [FC] 'migration: Add block-bitmap-mapping parameter'
>> 002/3:[0003] [FC] 'iotests.py: Add wait_for_runstate()'
>> 003/3:[0046] [FC] 'iotests: Test node/bitmap aliases during migration'
>>
>>
>> Max Reitz (3):
>> migration: Add block-bitmap-mapping parameter
>> iotests.py: Add wait_for_runstate()
>> iotests: Test node/bitmap aliases during migration
>>
>> qapi/migration.json | 104 ++++++-
>> migration/migration.h | 3 +
>> migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++----
>> migration/migration.c | 30 ++
>> monitor/hmp-cmds.c | 30 ++
>> tests/qemu-iotests/300 | 515 +++++++++++++++++++++++++++++++++
>> tests/qemu-iotests/300.out | 5 +
>> tests/qemu-iotests/group | 1 +
>> tests/qemu-iotests/iotests.py | 5 +
>> 9 files changed, 1011 insertions(+), 55 deletions(-)
>> create mode 100755 tests/qemu-iotests/300
>> create mode 100644 tests/qemu-iotests/300.out
>>
>
>
--
Best regards,
Vladimir
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH for-5.2 v3 1/3] migration: Add block-bitmap-mapping parameter
2020-08-12 14:32 ` Eric Blake
@ 2020-08-13 13:03 ` Max Reitz
0 siblings, 0 replies; 9+ messages in thread
From: Max Reitz @ 2020-08-13 13:03 UTC (permalink / raw)
To: Eric Blake, qemu-block
Cc: Kevin Wolf, Vladimir Sementsov-Ogievskiy, Juan Quintela,
qemu-devel, Dr . David Alan Gilbert, Peter Krempa, John Snow
[-- Attachment #1.1: Type: text/plain, Size: 6697 bytes --]
On 12.08.20 16:32, Eric Blake wrote:
> On 7/22/20 3:05 AM, Max Reitz wrote:
>> This migration parameter allows mapping block node names and bitmap
>> names to aliases for the purpose of block dirty bitmap migration.
>>
>> This way, management tools can use different node and bitmap names on
>> the source and destination and pass the mapping of how bitmaps are to be
>> transferred to qemu (on the source, the destination, or even both with
>> arbitrary aliases in the migration stream).
>>
>> Suggested-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
>> Signed-off-by: Max Reitz <mreitz@redhat.com>
>> ---
>> qapi/migration.json | 104 ++++++++-
>> migration/migration.h | 3 +
>> migration/block-dirty-bitmap.c | 373 ++++++++++++++++++++++++++++-----
>> migration/migration.c | 30 +++
>> monitor/hmp-cmds.c | 30 +++
>> 5 files changed, 485 insertions(+), 55 deletions(-)
>>
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index d5000558c6..d7e953cd73 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -507,6 +507,44 @@
>> 'data': [ 'none', 'zlib',
>> { 'name': 'zstd', 'if': 'defined(CONFIG_ZSTD)' } ] }
>> +##
>> +# @BitmapMigrationBitmapAlias:
>> +#
>> +# @name: The name of the bitmap.
>> +#
>> +# @alias: An alias name for migration (for example the bitmap name on
>> +# the opposite site).
>> +#
>> +# Since: 5.1
>
> 5.2, now, but I can touch that up if it is the only problem raised.
Yes, somehow I missed this, and amended it to 5.2 elsewhere...
>> +# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
>> +# aliases for the purpose of dirty bitmap migration. Such
>> +# aliases may for example be the corresponding names on the
>> +# opposite site.
>> +# The mapping must be one-to-one, and on the destination also
>> +# complete: On the source, bitmaps on nodes where either the
>> +# bitmap or the node is not mapped will be ignored. In
>> +# contrast, on the destination, receiving a bitmap (by alias)
>> +# from a node (by alias) when either alias is not mapped will
>> +# result in an error.
>
> Grammar is a bit odd and it feels repetitive. How about:
>
> The mapping must not provide more than one alias for a bitmap, nor reuse
> the same alias across multiple bitmaps in the same node.
This describes an injective function, as in, one-to-one.
> On the source,
> an omitted node or bitmap within a node will ignore those bitmaps. In
> contrast, on the destination, receiving a bitmap (by alias) from a node
> (by alias) when either alias is not mapped will result in an error.
This will need a rewrite anyway. Because of Vladimir’s patches you
merged in rc2, it looks like it makes more sense now to ignore unknown
aliases on the destination. So I’ll have to think up something new anyway.
>> +# Note that the destination does not know about bitmaps it
>> +# does not receive, so there is no limitation or requirement
>> +# regarding the number of bitmaps received, or how they are
>> +# named, or on which nodes they are placed.
>> +# By default (when this parameter has never been set), bitmap
>> +# names are mapped to themselves. Nodes are mapped to their
>> +# block device name if there is one, and to their node name
>> +# otherwise. (Since 5.2)
>
> Looks good.
>
>
>> @@ -781,6 +839,25 @@
>> # will consume more CPU.
>> # Defaults to 1. (Since 5.0)
>> #
>> +# @block-bitmap-mapping: Maps block nodes and bitmaps on them to
>> +# aliases for the purpose of dirty bitmap migration. Such
>> +# aliases may for example be the corresponding names on the
>> +# opposite site.
>
> Ah, the joys of duplicated text.
>
>> +++ b/migration/block-dirty-bitmap.c
>
>> @@ -128,7 +131,8 @@ typedef struct DirtyBitmapMigState {
>> typedef struct DirtyBitmapLoadState {
>> uint32_t flags;
>> - char node_name[256];
>> + char node_alias[256];
>> + char bitmap_alias[256];
>
> Do we properly check that alias names will never overflow?
Hm, well. There are assertions guarding against that, but they’re
assertions.
That’s an existing problem, actually. If you give a bitmap a name
longer than 255 bytes, you run into the same failed assertion.
>> +static GHashTable *construct_alias_map(const
>> BitmapMigrationNodeAliasList *bbm,
>> + bool name_to_alias,
>> + Error **errp)
>> +{
>> + GHashTable *alias_map;
>> +
>> + alias_map = g_hash_table_new_full(g_str_hash, g_str_equal,
>> + g_free,
>> free_alias_map_inner_node);
>> +
>> + for (; bbm; bbm = bbm->next) {
>> + const BitmapMigrationNodeAlias *bmna = bbm->value;
>> + const BitmapMigrationBitmapAliasList *bmbal;
>> + AliasMapInnerNode *amin;
>> + GHashTable *bitmaps_map;
>> + const char *node_map_from, *node_map_to;
>> +
>> + if (!id_wellformed(bmna->alias)) {
>> + error_setg(errp, "The node alias %s is not well-formed",
>> + bmna->alias);
>> + goto fail;
>> + }
>
> Maybe id_wellformed should enforce lengths? Otherwise, I'm not seeing a
> length limit applied during the mapping process.
The limit is applied by qemu_put_counted_string(), but it’s enforced via
assertion, so, yeah. I’ll fix it.
(That’s going to be the easy part. The harder part is fixing the
pre-existing issue, because we only see bitmap names once migration
starts. Thanks to the new permissive error policy (as in, no
configuration problem should ever produce errors during bitmap
migration), we then need to drop such bitmaps.)
> Modulo that, I think we're ready to go.
Unfortunately, no. As Vladimir has pointed out, this will require a
hefty rebase now.
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter
2020-08-12 14:38 ` Vladimir Sementsov-Ogievskiy
@ 2020-08-13 13:04 ` Max Reitz
0 siblings, 0 replies; 9+ messages in thread
From: Max Reitz @ 2020-08-13 13:04 UTC (permalink / raw)
To: Vladimir Sementsov-Ogievskiy, qemu-block
Cc: Kevin Wolf, Peter Krempa, Juan Quintela, John Snow, qemu-devel,
Dr . David Alan Gilbert
[-- Attachment #1.1: Type: text/plain, Size: 484 bytes --]
On 12.08.20 16:38, Vladimir Sementsov-Ogievskiy wrote:
> Now this doesn't apply, as code changed a lot after my series
> "[PATCH v4 for-5.1 00/21] Fix error handling during bitmap postcopy"
> merged first.
Yeah, well. Yeah.
> I feel my responsibility for the mess with these series, so if you want
> I can try to rebase and post v4 of this one myself :)
I feel like rebasing is simpler than reviewing, so I’ll be egoistic and
choose the rebaser’s side.
Max
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2020-08-13 13:05 UTC | newest]
Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-07-22 8:05 [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter Max Reitz
2020-07-22 8:05 ` [PATCH for-5.2 v3 1/3] " Max Reitz
2020-08-12 14:32 ` Eric Blake
2020-08-13 13:03 ` Max Reitz
2020-07-22 8:05 ` [PATCH for-5.2 v3 2/3] iotests.py: Add wait_for_runstate() Max Reitz
2020-07-22 8:05 ` [PATCH for-5.2 v3 3/3] iotests: Test node/bitmap aliases during migration Max Reitz
2020-08-12 14:15 ` [PATCH for-5.2 v3 0/3] migration: Add block-bitmap-mapping parameter Max Reitz
2020-08-12 14:38 ` Vladimir Sementsov-Ogievskiy
2020-08-13 13:04 ` Max Reitz
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.