* [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability
@ 2019-02-04 13:09 Yury Kotov
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 1/4] exec: Change RAMBlockIterFunc definition Yury Kotov
` (5 more replies)
0 siblings, 6 replies; 16+ messages in thread
From: Yury Kotov @ 2019-02-04 13:09 UTC (permalink / raw)
To: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela,
Dr. David Alan Gilbert, Eric Blake, Markus Armbruster,
Thomas Huth, Laurent Vivier
Cc: wrfsh
Hi,
The series adds a migration capability, which allows to skip shared RAM blocks
during the migration. It's useful for fast local migration. E.g. to update QEMU
for the running guests.
Usage example:
1. Start source VM:
qemu-system-x86 \
-m 4G \
-object memory-backend-file,id=mem0,size=4G,share=on,mem-path=/dev/shm/mem0 \
-numa node,memdev=mem0 \
-qmp unix:/tmp/qemu-qmp-1.sock,server,nowait \
2. Start target VM:
qemu-system-x86 \
-m 4G \
-object memory-backend-file,id=mem0,size=4G,share=on,mem-path=/dev/shm/mem0 \
-numa node,memdev=mem0 \
-qmp unix:/tmp/qemu-qmp-2.sock,server,nowait \
-incoming defer
3. Enable ignore-external capability on both VMs:
{ "execute": "migrate-set-capabilities" , "arguments":
{ "capabilities": [ { "capability": "x-ignore-external", "state": true } ] } }
4. Start migration.
Another use case I keep in mind is to migrate to file. Usage is very similar.
V1 to V2:
* Keep migration stream compatibility
* Reuse the existing code to ignore unwanted RAMBlocks
* Add capability validation feature
* ignore-external -> ignore-shared
Regards,
Yury
Yury Kotov (5):
exec: Change RAMBlockIterFunc definition
migration: Move qemu_ram_foreach_migratable_block to migration code
migration: Introduce ignore-shared capability
tests/migration-test: Add a test for ignore-shared capability
migration: Add capabilities validation
exec.c | 38 ++++++-------
include/exec/cpu-common.h | 7 +--
migration/migration.c | 9 ++++
migration/migration.h | 11 +++-
migration/postcopy-ram.c | 48 +++++++++--------
migration/ram.c | 86 ++++++++++++++++++++++++++----
migration/rdma.c | 9 ++--
migration/savevm.c | 101 +++++++++++++++++++++++++++++++++++
qapi/migration.json | 5 +-
stubs/ram-block.c | 15 ++++++
tests/migration-test.c | 109 +++++++++++++++++++++++++++++++-------
util/vfio-helpers.c | 6 +--
12 files changed, 361 insertions(+), 83 deletions(-)
--
2.20.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH v2 1/4] exec: Change RAMBlockIterFunc definition
2019-02-04 13:09 [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability Yury Kotov
@ 2019-02-04 13:09 ` Yury Kotov
2019-02-11 11:40 ` Dr. David Alan Gilbert
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 2/4] migration: Introduce ignore-shared capability Yury Kotov
` (4 subsequent siblings)
5 siblings, 1 reply; 16+ messages in thread
From: Yury Kotov @ 2019-02-04 13:09 UTC (permalink / raw)
To: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela,
Dr. David Alan Gilbert, Eric Blake, Markus Armbruster,
Thomas Huth, Laurent Vivier
Cc: wrfsh
Currently, qemu_ram_foreach_* calls RAMBlockIterFunc with many
block-specific arguments. But often iter func needs RAMBlock*.
It's more effective to call RAMBlockIterFunc with RAMBlock* argument.
So, fix RAMBlockIterFunc definition and add some functions to read RAMBlock*
fields witch were passed.
Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
---
exec.c | 21 +++++++++++++++++----
include/exec/cpu-common.h | 6 ++++--
migration/postcopy-ram.c | 36 +++++++++++++++++++++---------------
migration/rdma.c | 7 +++++--
util/vfio-helpers.c | 6 +++---
5 files changed, 50 insertions(+), 26 deletions(-)
diff --git a/exec.c b/exec.c
index da3e635f91..a61d501568 100644
--- a/exec.c
+++ b/exec.c
@@ -1970,6 +1970,21 @@ const char *qemu_ram_get_idstr(RAMBlock *rb)
return rb->idstr;
}
+void *qemu_ram_get_host_addr(RAMBlock *rb)
+{
+ return rb->host;
+}
+
+ram_addr_t qemu_ram_get_offset(RAMBlock *rb)
+{
+ return rb->offset;
+}
+
+ram_addr_t qemu_ram_get_used_length(RAMBlock *rb)
+{
+ return rb->used_length;
+}
+
bool qemu_ram_is_shared(RAMBlock *rb)
{
return rb->flags & RAM_SHARED;
@@ -3960,8 +3975,7 @@ int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
rcu_read_lock();
RAMBLOCK_FOREACH(block) {
- ret = func(block->idstr, block->host, block->offset,
- block->used_length, opaque);
+ ret = func(block, opaque);
if (ret) {
break;
}
@@ -3980,8 +3994,7 @@ int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque)
if (!qemu_ram_is_migratable(block)) {
continue;
}
- ret = func(block->idstr, block->host, block->offset,
- block->used_length, opaque);
+ ret = func(block, opaque);
if (ret) {
break;
}
diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
index 2ad2d6d86b..bdae5446d7 100644
--- a/include/exec/cpu-common.h
+++ b/include/exec/cpu-common.h
@@ -72,6 +72,9 @@ ram_addr_t qemu_ram_block_host_offset(RAMBlock *rb, void *host);
void qemu_ram_set_idstr(RAMBlock *block, const char *name, DeviceState *dev);
void qemu_ram_unset_idstr(RAMBlock *block);
const char *qemu_ram_get_idstr(RAMBlock *rb);
+void *qemu_ram_get_host_addr(RAMBlock *rb);
+ram_addr_t qemu_ram_get_offset(RAMBlock *rb);
+ram_addr_t qemu_ram_get_used_length(RAMBlock *rb);
bool qemu_ram_is_shared(RAMBlock *rb);
bool qemu_ram_is_uf_zeroable(RAMBlock *rb);
void qemu_ram_set_uf_zeroable(RAMBlock *rb);
@@ -116,8 +119,7 @@ void cpu_flush_icache_range(hwaddr start, int len);
extern struct MemoryRegion io_mem_rom;
extern struct MemoryRegion io_mem_notdirty;
-typedef int (RAMBlockIterFunc)(const char *block_name, void *host_addr,
- ram_addr_t offset, ram_addr_t length, void *opaque);
+typedef int (RAMBlockIterFunc)(RAMBlock *rb, void *opaque);
int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque);
diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index fa09dba534..b098816221 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -319,10 +319,10 @@ static bool ufd_check_and_apply(int ufd, MigrationIncomingState *mis)
/* Callback from postcopy_ram_supported_by_host block iterator.
*/
-static int test_ramblock_postcopiable(const char *block_name, void *host_addr,
- ram_addr_t offset, ram_addr_t length, void *opaque)
+static int test_ramblock_postcopiable(RAMBlock *rb, void *opaque)
{
- RAMBlock *rb = qemu_ram_block_by_name(block_name);
+ const char *block_name = qemu_ram_get_idstr(rb);
+ ram_addr_t length = qemu_ram_get_used_length(rb);
size_t pagesize = qemu_ram_pagesize(rb);
if (length % pagesize) {
@@ -443,9 +443,12 @@ out:
* must be done right at the start prior to pre-copy.
* opaque should be the MIS.
*/
-static int init_range(const char *block_name, void *host_addr,
- ram_addr_t offset, ram_addr_t length, void *opaque)
+static int init_range(RAMBlock *rb, void *opaque)
{
+ const char *block_name = qemu_ram_get_idstr(rb);
+ void *host_addr = qemu_ram_get_host_addr(rb);
+ ram_addr_t offset = qemu_ram_get_offset(rb);
+ ram_addr_t length = qemu_ram_get_used_length(rb);
trace_postcopy_init_range(block_name, host_addr, offset, length);
/*
@@ -465,9 +468,12 @@ static int init_range(const char *block_name, void *host_addr,
* At the end of migration, undo the effects of init_range
* opaque should be the MIS.
*/
-static int cleanup_range(const char *block_name, void *host_addr,
- ram_addr_t offset, ram_addr_t length, void *opaque)
+static int cleanup_range(RAMBlock *rb, void *opaque)
{
+ const char *block_name = qemu_ram_get_idstr(rb);
+ void *host_addr = qemu_ram_get_host_addr(rb);
+ ram_addr_t offset = qemu_ram_get_offset(rb);
+ ram_addr_t length = qemu_ram_get_used_length(rb);
MigrationIncomingState *mis = opaque;
struct uffdio_range range_struct;
trace_postcopy_cleanup_range(block_name, host_addr, offset, length);
@@ -586,9 +592,12 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
/*
* Disable huge pages on an area
*/
-static int nhp_range(const char *block_name, void *host_addr,
- ram_addr_t offset, ram_addr_t length, void *opaque)
+static int nhp_range(RAMBlock *rb, void *opaque)
{
+ const char *block_name = qemu_ram_get_idstr(rb);
+ void *host_addr = qemu_ram_get_host_addr(rb);
+ ram_addr_t offset = qemu_ram_get_offset(rb);
+ ram_addr_t length = qemu_ram_get_used_length(rb);
trace_postcopy_nhp_range(block_name, host_addr, offset, length);
/*
@@ -626,15 +635,13 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
* opaque: MigrationIncomingState pointer
* Returns 0 on success
*/
-static int ram_block_enable_notify(const char *block_name, void *host_addr,
- ram_addr_t offset, ram_addr_t length,
- void *opaque)
+static int ram_block_enable_notify(RAMBlock *rb, void *opaque)
{
MigrationIncomingState *mis = opaque;
struct uffdio_register reg_struct;
- reg_struct.range.start = (uintptr_t)host_addr;
- reg_struct.range.len = length;
+ reg_struct.range.start = (uintptr_t)qemu_ram_get_host_addr(rb);
+ reg_struct.range.len = qemu_ram_get_used_length(rb);
reg_struct.mode = UFFDIO_REGISTER_MODE_MISSING;
/* Now tell our userfault_fd that it's responsible for this area */
@@ -647,7 +654,6 @@ static int ram_block_enable_notify(const char *block_name, void *host_addr,
return -1;
}
if (reg_struct.ioctls & ((__u64)1 << _UFFDIO_ZEROPAGE)) {
- RAMBlock *rb = qemu_ram_block_by_name(block_name);
qemu_ram_set_uf_zeroable(rb);
}
diff --git a/migration/rdma.c b/migration/rdma.c
index 54a3c11540..7eb38ee764 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -624,9 +624,12 @@ static int rdma_add_block(RDMAContext *rdma, const char *block_name,
* in advanced before the migration starts. This tells us where the RAM blocks
* are so that we can register them individually.
*/
-static int qemu_rdma_init_one_block(const char *block_name, void *host_addr,
- ram_addr_t block_offset, ram_addr_t length, void *opaque)
+static int qemu_rdma_init_one_block(RAMBlock *rb, void *opaque)
{
+ const char *block_name = qemu_ram_get_idstr(rb);
+ void *host_addr = qemu_ram_get_host_addr(rb);
+ ram_addr_t block_offset = qemu_ram_get_offset(rb);
+ ram_addr_t length = qemu_ram_get_used_length(rb);
return rdma_add_block(opaque, block_name, host_addr, block_offset, length);
}
diff --git a/util/vfio-helpers.c b/util/vfio-helpers.c
index 342d4a2285..2367fe8f7f 100644
--- a/util/vfio-helpers.c
+++ b/util/vfio-helpers.c
@@ -391,10 +391,10 @@ static void qemu_vfio_ram_block_removed(RAMBlockNotifier *n,
}
}
-static int qemu_vfio_init_ramblock(const char *block_name, void *host_addr,
- ram_addr_t offset, ram_addr_t length,
- void *opaque)
+static int qemu_vfio_init_ramblock(RAMBlock *rb, void *opaque)
{
+ void *host_addr = qemu_ram_get_host_addr(rb);
+ ram_addr_t length = qemu_ram_get_used_length(rb);
int ret;
QEMUVFIOState *s = opaque;
--
2.20.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH v2 2/4] migration: Introduce ignore-shared capability
2019-02-04 13:09 [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability Yury Kotov
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 1/4] exec: Change RAMBlockIterFunc definition Yury Kotov
@ 2019-02-04 13:09 ` Yury Kotov
2019-02-11 12:45 ` Dr. David Alan Gilbert
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 3/4] tests/migration-test: Add a test for " Yury Kotov
` (3 subsequent siblings)
5 siblings, 1 reply; 16+ messages in thread
From: Yury Kotov @ 2019-02-04 13:09 UTC (permalink / raw)
To: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela,
Dr. David Alan Gilbert, Eric Blake, Markus Armbruster,
Thomas Huth, Laurent Vivier
Cc: wrfsh
We want to use local migration to update QEMU for running guests.
In this case we don't need to migrate shared (file backed) RAM.
So, add a capability to ignore such blocks during live migration.
Also, move qemu_ram_foreach_migratable_block (and rename) to the
migration code, because it requires access to the migration capabilities.
Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
---
exec.c | 19 -------
include/exec/cpu-common.h | 1 -
migration/migration.c | 9 ++++
migration/migration.h | 5 +-
migration/postcopy-ram.c | 12 ++---
migration/ram.c | 110 +++++++++++++++++++++++++++++---------
migration/rdma.c | 2 +-
qapi/migration.json | 5 +-
stubs/ram-block.c | 15 ++++++
9 files changed, 123 insertions(+), 55 deletions(-)
diff --git a/exec.c b/exec.c
index a61d501568..91bfe5fb62 100644
--- a/exec.c
+++ b/exec.c
@@ -3984,25 +3984,6 @@ int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
return ret;
}
-int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque)
-{
- RAMBlock *block;
- int ret = 0;
-
- rcu_read_lock();
- RAMBLOCK_FOREACH(block) {
- if (!qemu_ram_is_migratable(block)) {
- continue;
- }
- ret = func(block, opaque);
- if (ret) {
- break;
- }
- }
- rcu_read_unlock();
- return ret;
-}
-
/*
* Unmap pages of memory from start to start+length such that
* they a) read as 0, b) Trigger whatever fault mechanism
diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
index bdae5446d7..403463d7bb 100644
--- a/include/exec/cpu-common.h
+++ b/include/exec/cpu-common.h
@@ -122,7 +122,6 @@ extern struct MemoryRegion io_mem_notdirty;
typedef int (RAMBlockIterFunc)(RAMBlock *rb, void *opaque);
int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
-int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque);
int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length);
#endif
diff --git a/migration/migration.c b/migration/migration.c
index 37e06b76dc..c40776a40c 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -1983,6 +1983,15 @@ bool migrate_dirty_bitmaps(void)
return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS];
}
+bool migrate_ignore_shared(void)
+{
+ MigrationState *s;
+
+ s = migrate_get_current();
+
+ return s->enabled_capabilities[MIGRATION_CAPABILITY_X_IGNORE_SHARED];
+}
+
bool migrate_use_events(void)
{
MigrationState *s;
diff --git a/migration/migration.h b/migration/migration.h
index dcd05d9f87..2c88f8a555 100644
--- a/migration/migration.h
+++ b/migration/migration.h
@@ -261,6 +261,7 @@ bool migrate_release_ram(void);
bool migrate_postcopy_ram(void);
bool migrate_zero_blocks(void);
bool migrate_dirty_bitmaps(void);
+bool migrate_ignore_shared(void);
bool migrate_auto_converge(void);
bool migrate_use_multifd(void);
@@ -301,8 +302,10 @@ void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value);
void dirty_bitmap_mig_before_vm_start(void);
void init_dirty_bitmap_incoming_migration(void);
+int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque);
+
#define qemu_ram_foreach_block \
- #warning "Use qemu_ram_foreach_block_migratable in migration code"
+ #warning "Use foreach_not_ignored_block in migration code"
void migration_make_urgent_request(void);
void migration_consume_urgent_request(void);
diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index b098816221..e2aa57a701 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -374,7 +374,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis)
}
/* We don't support postcopy with shared RAM yet */
- if (qemu_ram_foreach_migratable_block(test_ramblock_postcopiable, NULL)) {
+ if (foreach_not_ignored_block(test_ramblock_postcopiable, NULL)) {
goto out;
}
@@ -508,7 +508,7 @@ static int cleanup_range(RAMBlock *rb, void *opaque)
*/
int postcopy_ram_incoming_init(MigrationIncomingState *mis)
{
- if (qemu_ram_foreach_migratable_block(init_range, NULL)) {
+ if (foreach_not_ignored_block(init_range, NULL)) {
return -1;
}
@@ -550,7 +550,7 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
return -1;
}
- if (qemu_ram_foreach_migratable_block(cleanup_range, mis)) {
+ if (foreach_not_ignored_block(cleanup_range, mis)) {
return -1;
}
@@ -617,7 +617,7 @@ static int nhp_range(RAMBlock *rb, void *opaque)
*/
int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
{
- if (qemu_ram_foreach_migratable_block(nhp_range, mis)) {
+ if (foreach_not_ignored_block(nhp_range, mis)) {
return -1;
}
@@ -628,7 +628,7 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
/*
* Mark the given area of RAM as requiring notification to unwritten areas
- * Used as a callback on qemu_ram_foreach_migratable_block.
+ * Used as a callback on foreach_not_ignored_block.
* host_addr: Base of area to mark
* offset: Offset in the whole ram arena
* length: Length of the section
@@ -1122,7 +1122,7 @@ int postcopy_ram_enable_notify(MigrationIncomingState *mis)
mis->have_fault_thread = true;
/* Mark so that we get notified of accesses to unwritten areas */
- if (qemu_ram_foreach_migratable_block(ram_block_enable_notify, mis)) {
+ if (foreach_not_ignored_block(ram_block_enable_notify, mis)) {
error_report("ram_block_enable_notify failed");
return -1;
}
diff --git a/migration/ram.c b/migration/ram.c
index 59191c1ed2..01315edd66 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -159,18 +159,44 @@ out:
return ret;
}
+static bool ramblock_is_ignored(RAMBlock *block)
+{
+ return !qemu_ram_is_migratable(block) ||
+ (migrate_ignore_shared() && qemu_ram_is_shared(block));
+}
+
/* Should be holding either ram_list.mutex, or the RCU lock. */
+#define RAMBLOCK_FOREACH_NOT_IGNORED(block) \
+ INTERNAL_RAMBLOCK_FOREACH(block) \
+ if (ramblock_is_ignored(block)) {} else
+
#define RAMBLOCK_FOREACH_MIGRATABLE(block) \
INTERNAL_RAMBLOCK_FOREACH(block) \
if (!qemu_ram_is_migratable(block)) {} else
#undef RAMBLOCK_FOREACH
+int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque)
+{
+ RAMBlock *block;
+ int ret = 0;
+
+ rcu_read_lock();
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
+ ret = func(block, opaque);
+ if (ret) {
+ break;
+ }
+ }
+ rcu_read_unlock();
+ return ret;
+}
+
static void ramblock_recv_map_init(void)
{
RAMBlock *rb;
- RAMBLOCK_FOREACH_MIGRATABLE(rb) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
assert(!rb->receivedmap);
rb->receivedmap = bitmap_new(rb->max_length >> qemu_target_page_bits());
}
@@ -1545,7 +1571,7 @@ unsigned long migration_bitmap_find_dirty(RAMState *rs, RAMBlock *rb,
unsigned long *bitmap = rb->bmap;
unsigned long next;
- if (!qemu_ram_is_migratable(rb)) {
+ if (ramblock_is_ignored(rb)) {
return size;
}
@@ -1594,7 +1620,7 @@ uint64_t ram_pagesize_summary(void)
RAMBlock *block;
uint64_t summary = 0;
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
summary |= block->page_size;
}
@@ -1664,7 +1690,7 @@ static void migration_bitmap_sync(RAMState *rs)
qemu_mutex_lock(&rs->bitmap_mutex);
rcu_read_lock();
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
migration_bitmap_sync_range(rs, block, 0, block->used_length);
}
ram_counters.remaining = ram_bytes_remaining();
@@ -2388,7 +2414,7 @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss,
size_t pagesize_bits =
qemu_ram_pagesize(pss->block) >> TARGET_PAGE_BITS;
- if (!qemu_ram_is_migratable(pss->block)) {
+ if (ramblock_is_ignored(pss->block)) {
error_report("block %s should not be migrated !", pss->block->idstr);
return 0;
}
@@ -2486,19 +2512,30 @@ void acct_update_position(QEMUFile *f, size_t size, bool zero)
}
}
-uint64_t ram_bytes_total(void)
+static uint64_t ram_bytes_total_common(bool count_ignored)
{
RAMBlock *block;
uint64_t total = 0;
rcu_read_lock();
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
- total += block->used_length;
+ if (count_ignored) {
+ RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ total += block->used_length;
+ }
+ } else {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
+ total += block->used_length;
+ }
}
rcu_read_unlock();
return total;
}
+uint64_t ram_bytes_total(void)
+{
+ return ram_bytes_total_common(false);
+}
+
static void xbzrle_load_setup(void)
{
XBZRLE.decoded_buf = g_malloc(TARGET_PAGE_SIZE);
@@ -2547,7 +2584,7 @@ static void ram_save_cleanup(void *opaque)
*/
memory_global_dirty_log_stop();
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
g_free(block->bmap);
block->bmap = NULL;
g_free(block->unsentmap);
@@ -2610,7 +2647,7 @@ void ram_postcopy_migrated_memory_release(MigrationState *ms)
{
struct RAMBlock *block;
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
unsigned long *bitmap = block->bmap;
unsigned long range = block->used_length >> TARGET_PAGE_BITS;
unsigned long run_start = find_next_zero_bit(bitmap, range, 0);
@@ -2688,7 +2725,7 @@ static int postcopy_each_ram_send_discard(MigrationState *ms)
struct RAMBlock *block;
int ret;
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
PostcopyDiscardState *pds =
postcopy_discard_send_init(ms, block->idstr);
@@ -2896,7 +2933,7 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms)
rs->last_sent_block = NULL;
rs->last_page = 0;
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
unsigned long pages = block->used_length >> TARGET_PAGE_BITS;
unsigned long *bitmap = block->bmap;
unsigned long *unsentmap = block->unsentmap;
@@ -3062,7 +3099,7 @@ static void ram_list_init_bitmaps(void)
/* Skip setting bitmap if there is no RAM */
if (ram_bytes_total()) {
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
pages = block->max_length >> TARGET_PAGE_BITS;
block->bmap = bitmap_new(pages);
bitmap_set(block->bmap, 0, pages);
@@ -3117,7 +3154,7 @@ static void ram_state_resume_prepare(RAMState *rs, QEMUFile *out)
* about dirty page logging as well.
*/
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
pages += bitmap_count_one(block->bmap,
block->used_length >> TARGET_PAGE_BITS);
}
@@ -3176,7 +3213,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
rcu_read_lock();
- qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
+ qemu_put_be64(f, ram_bytes_total_common(true) | RAM_SAVE_FLAG_MEM_SIZE);
RAMBLOCK_FOREACH_MIGRATABLE(block) {
qemu_put_byte(f, strlen(block->idstr));
@@ -3185,6 +3222,10 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
if (migrate_postcopy_ram() && block->page_size != qemu_host_page_size) {
qemu_put_be64(f, block->page_size);
}
+ if (migrate_ignore_shared()) {
+ qemu_put_be64(f, block->mr->addr);
+ qemu_put_byte(f, ramblock_is_ignored(block) ? 1 : 0);
+ }
}
rcu_read_unlock();
@@ -3443,7 +3484,7 @@ static inline RAMBlock *ram_block_from_stream(QEMUFile *f, int flags)
return NULL;
}
- if (!qemu_ram_is_migratable(block)) {
+ if (ramblock_is_ignored(block)) {
error_report("block %s should not be migrated !", id);
return NULL;
}
@@ -3698,7 +3739,7 @@ int colo_init_ram_cache(void)
RAMBlock *block;
rcu_read_lock();
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
block->colo_cache = qemu_anon_ram_alloc(block->used_length,
NULL,
false);
@@ -3719,7 +3760,7 @@ int colo_init_ram_cache(void)
if (ram_bytes_total()) {
RAMBlock *block;
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
unsigned long pages = block->max_length >> TARGET_PAGE_BITS;
block->bmap = bitmap_new(pages);
@@ -3734,7 +3775,7 @@ int colo_init_ram_cache(void)
out_locked:
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
if (block->colo_cache) {
qemu_anon_ram_free(block->colo_cache, block->used_length);
block->colo_cache = NULL;
@@ -3751,14 +3792,14 @@ void colo_release_ram_cache(void)
RAMBlock *block;
memory_global_dirty_log_stop();
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
g_free(block->bmap);
block->bmap = NULL;
}
rcu_read_lock();
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
if (block->colo_cache) {
qemu_anon_ram_free(block->colo_cache, block->used_length);
block->colo_cache = NULL;
@@ -3794,7 +3835,7 @@ static int ram_load_cleanup(void *opaque)
{
RAMBlock *rb;
- RAMBLOCK_FOREACH_MIGRATABLE(rb) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
if (ramblock_is_pmem(rb)) {
pmem_persist(rb->host, rb->used_length);
}
@@ -3803,7 +3844,7 @@ static int ram_load_cleanup(void *opaque)
xbzrle_load_cleanup();
compress_threads_load_cleanup();
- RAMBLOCK_FOREACH_MIGRATABLE(rb) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
g_free(rb->receivedmap);
rb->receivedmap = NULL;
}
@@ -4003,7 +4044,7 @@ static void colo_flush_ram_cache(void)
memory_global_dirty_log_sync();
rcu_read_lock();
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
migration_bitmap_sync_range(ram_state, block, 0, block->used_length);
}
rcu_read_unlock();
@@ -4146,6 +4187,23 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
ret = -EINVAL;
}
}
+ if (migrate_ignore_shared()) {
+ hwaddr addr = qemu_get_be64(f);
+ bool ignored = qemu_get_byte(f);
+ if (ignored != ramblock_is_ignored(block)) {
+ error_report("RAM block %s should %s be migrated",
+ id, ignored ? "" : "not");
+ ret = -EINVAL;
+ }
+ if (ramblock_is_ignored(block) &&
+ block->mr->addr != addr) {
+ error_report("Mismatched GPAs for block %s "
+ "%" PRId64 "!= %" PRId64,
+ id, (uint64_t)addr,
+ (uint64_t)block->mr->addr);
+ ret = -EINVAL;
+ }
+ }
ram_control_load_hook(f, RAM_CONTROL_BLOCK_REG,
block->idstr);
} else {
@@ -4216,7 +4274,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
static bool ram_has_postcopy(void *opaque)
{
RAMBlock *rb;
- RAMBLOCK_FOREACH_MIGRATABLE(rb) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
if (ramblock_is_pmem(rb)) {
info_report("Block: %s, host: %p is a nvdimm memory, postcopy"
"is not supported now!", rb->idstr, rb->host);
@@ -4236,7 +4294,7 @@ static int ram_dirty_bitmap_sync_all(MigrationState *s, RAMState *rs)
trace_ram_dirty_bitmap_sync_start();
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
qemu_savevm_send_recv_bitmap(file, block->idstr);
trace_ram_dirty_bitmap_request(block->idstr);
ramblock_count++;
diff --git a/migration/rdma.c b/migration/rdma.c
index 7eb38ee764..3cb579cc99 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -644,7 +644,7 @@ static int qemu_rdma_init_ram_blocks(RDMAContext *rdma)
assert(rdma->blockmap == NULL);
memset(local, 0, sizeof *local);
- qemu_ram_foreach_migratable_block(qemu_rdma_init_one_block, rdma);
+ foreach_not_ignored_block(qemu_rdma_init_one_block, rdma);
trace_qemu_rdma_init_ram_blocks(local->nb_blocks);
rdma->dest_blocks = g_new0(RDMADestBlock,
rdma->local_ram_blocks.nb_blocks);
diff --git a/qapi/migration.json b/qapi/migration.json
index 7a795ecc16..7105570cd3 100644
--- a/qapi/migration.json
+++ b/qapi/migration.json
@@ -409,13 +409,16 @@
# devices (and thus take locks) immediately at the end of migration.
# (since 3.0)
#
+# @x-ignore-shared: If enabled, QEMU will not migrate shared memory (since 4.0)
+#
# Since: 1.2
##
{ 'enum': 'MigrationCapability',
'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
'compress', 'events', 'postcopy-ram', 'x-colo', 'release-ram',
'block', 'return-path', 'pause-before-switchover', 'x-multifd',
- 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate' ] }
+ 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate',
+ 'x-ignore-shared' ] }
##
# @MigrationCapabilityStatus:
diff --git a/stubs/ram-block.c b/stubs/ram-block.c
index cfa5d8678f..73c0a3ee08 100644
--- a/stubs/ram-block.c
+++ b/stubs/ram-block.c
@@ -2,6 +2,21 @@
#include "exec/ramlist.h"
#include "exec/cpu-common.h"
+void *qemu_ram_get_host_addr(RAMBlock *rb)
+{
+ return 0;
+}
+
+ram_addr_t qemu_ram_get_offset(RAMBlock *rb)
+{
+ return 0;
+}
+
+ram_addr_t qemu_ram_get_used_length(RAMBlock *rb)
+{
+ return 0;
+}
+
void ram_block_notifier_add(RAMBlockNotifier *n)
{
}
--
2.20.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH v2 3/4] tests/migration-test: Add a test for ignore-shared capability
2019-02-04 13:09 [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability Yury Kotov
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 1/4] exec: Change RAMBlockIterFunc definition Yury Kotov
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 2/4] migration: Introduce ignore-shared capability Yury Kotov
@ 2019-02-04 13:09 ` Yury Kotov
2019-02-11 13:17 ` Dr. David Alan Gilbert
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 4/4] migration: Add capabilities validation Yury Kotov
` (2 subsequent siblings)
5 siblings, 1 reply; 16+ messages in thread
From: Yury Kotov @ 2019-02-04 13:09 UTC (permalink / raw)
To: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela,
Dr. David Alan Gilbert, Eric Blake, Markus Armbruster,
Thomas Huth, Laurent Vivier
Cc: wrfsh
Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
---
tests/migration-test.c | 109 +++++++++++++++++++++++++++++++++--------
1 file changed, 89 insertions(+), 20 deletions(-)
diff --git a/tests/migration-test.c b/tests/migration-test.c
index 8352612364..485f42b2d2 100644
--- a/tests/migration-test.c
+++ b/tests/migration-test.c
@@ -332,6 +332,13 @@ static void cleanup(const char *filename)
g_free(path);
}
+static char *get_shmem_opts(const char *mem_size, const char *shmem_path)
+{
+ return g_strdup_printf("-object memory-backend-file,id=mem0,size=%s"
+ ",mem-path=%s,share=on -numa node,memdev=mem0",
+ mem_size, shmem_path);
+}
+
static void migrate_check_parameter(QTestState *who, const char *parameter,
long long value)
{
@@ -430,73 +437,91 @@ static void migrate_postcopy_start(QTestState *from, QTestState *to)
}
static int test_migrate_start(QTestState **from, QTestState **to,
- const char *uri, bool hide_stderr)
+ const char *uri, bool hide_stderr,
+ bool use_shmem)
{
gchar *cmd_src, *cmd_dst;
char *bootpath = g_strdup_printf("%s/bootsect", tmpfs);
+ char *extra_opts = NULL;
+ char *shmem_path = NULL;
const char *arch = qtest_get_arch();
const char *accel = "kvm:tcg";
got_stop = false;
+ if (use_shmem) {
+ shmem_path = g_strdup_printf("/dev/shm/qemu-%d", getpid());
+ }
+
if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
init_bootfile(bootpath, x86_bootsect);
+ extra_opts = use_shmem ? get_shmem_opts("150M", shmem_path) : NULL;
cmd_src = g_strdup_printf("-machine accel=%s -m 150M"
" -name source,debug-threads=on"
" -serial file:%s/src_serial"
- " -drive file=%s,format=raw",
- accel, tmpfs, bootpath);
+ " -drive file=%s,format=raw %s",
+ accel, tmpfs, bootpath,
+ extra_opts ? extra_opts : "");
cmd_dst = g_strdup_printf("-machine accel=%s -m 150M"
" -name target,debug-threads=on"
" -serial file:%s/dest_serial"
" -drive file=%s,format=raw"
- " -incoming %s",
- accel, tmpfs, bootpath, uri);
+ " -incoming %s %s",
+ accel, tmpfs, bootpath, uri,
+ extra_opts ? extra_opts : "");
start_address = X86_TEST_MEM_START;
end_address = X86_TEST_MEM_END;
} else if (g_str_equal(arch, "s390x")) {
init_bootfile_s390x(bootpath);
+ extra_opts = use_shmem ? get_shmem_opts("128M", shmem_path) : NULL;
cmd_src = g_strdup_printf("-machine accel=%s -m 128M"
" -name source,debug-threads=on"
- " -serial file:%s/src_serial -bios %s",
- accel, tmpfs, bootpath);
+ " -serial file:%s/src_serial -bios %s %s",
+ accel, tmpfs, bootpath,
+ extra_opts ? extra_opts : "");
cmd_dst = g_strdup_printf("-machine accel=%s -m 128M"
" -name target,debug-threads=on"
" -serial file:%s/dest_serial -bios %s"
- " -incoming %s",
- accel, tmpfs, bootpath, uri);
+ " -incoming %s %s",
+ accel, tmpfs, bootpath, uri,
+ extra_opts ? extra_opts : "");
start_address = S390_TEST_MEM_START;
end_address = S390_TEST_MEM_END;
} else if (strcmp(arch, "ppc64") == 0) {
+ extra_opts = use_shmem ? get_shmem_opts("256M", shmem_path) : NULL;
cmd_src = g_strdup_printf("-machine accel=%s -m 256M -nodefaults"
" -name source,debug-threads=on"
" -serial file:%s/src_serial"
" -prom-env 'use-nvramrc?=true' -prom-env "
"'nvramrc=hex .\" _\" begin %x %x "
"do i c@ 1 + i c! 1000 +loop .\" B\" 0 "
- "until'", accel, tmpfs, end_address,
- start_address);
+ "until' %s", accel, tmpfs, end_address,
+ start_address, extra_opts ? extra_opts : "");
cmd_dst = g_strdup_printf("-machine accel=%s -m 256M"
" -name target,debug-threads=on"
" -serial file:%s/dest_serial"
- " -incoming %s",
- accel, tmpfs, uri);
+ " -incoming %s %s",
+ accel, tmpfs, uri,
+ extra_opts ? extra_opts : "");
start_address = PPC_TEST_MEM_START;
end_address = PPC_TEST_MEM_END;
} else if (strcmp(arch, "aarch64") == 0) {
init_bootfile(bootpath, aarch64_kernel);
+ extra_opts = use_shmem ? get_shmem_opts("150M", shmem_path) : NULL;
cmd_src = g_strdup_printf("-machine virt,accel=%s,gic-version=max "
"-name vmsource,debug-threads=on -cpu max "
"-m 150M -serial file:%s/src_serial "
- "-kernel %s ",
- accel, tmpfs, bootpath);
+ "-kernel %s %s",
+ accel, tmpfs, bootpath,
+ extra_opts ? extra_opts : "");
cmd_dst = g_strdup_printf("-machine virt,accel=%s,gic-version=max "
"-name vmdest,debug-threads=on -cpu max "
"-m 150M -serial file:%s/dest_serial "
"-kernel %s "
- "-incoming %s ",
- accel, tmpfs, bootpath, uri);
+ "-incoming %s %s",
+ accel, tmpfs, bootpath, uri,
+ extra_opts ? extra_opts : "");
start_address = ARM_TEST_MEM_START;
end_address = ARM_TEST_MEM_END;
@@ -507,6 +532,7 @@ static int test_migrate_start(QTestState **from, QTestState **to,
}
g_free(bootpath);
+ g_free(extra_opts);
if (hide_stderr) {
gchar *tmp;
@@ -524,6 +550,16 @@ static int test_migrate_start(QTestState **from, QTestState **to,
*to = qtest_init(cmd_dst);
g_free(cmd_dst);
+
+ /*
+ * Remove shmem file immediately to avoid memory leak in test failed case.
+ * It's valid becase QEMU has already opened this file
+ */
+ if (use_shmem) {
+ unlink(shmem_path);
+ g_free(shmem_path);
+ }
+
return 0;
}
@@ -603,7 +639,7 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
QTestState *from, *to;
- if (test_migrate_start(&from, &to, uri, hide_error)) {
+ if (test_migrate_start(&from, &to, uri, hide_error, false)) {
return -1;
}
@@ -720,7 +756,7 @@ static void test_baddest(void)
char *status;
bool failed;
- if (test_migrate_start(&from, &to, "tcp:0:0", true)) {
+ if (test_migrate_start(&from, &to, "tcp:0:0", true, false)) {
return;
}
migrate(from, "tcp:0:0", "{}");
@@ -745,7 +781,7 @@ static void test_precopy_unix(void)
char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
QTestState *from, *to;
- if (test_migrate_start(&from, &to, uri, false)) {
+ if (test_migrate_start(&from, &to, uri, false, false)) {
return;
}
@@ -781,6 +817,38 @@ static void test_precopy_unix(void)
g_free(uri);
}
+static void test_ignore_shared(void)
+{
+ char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
+ QTestState *from, *to;
+
+ if (test_migrate_start(&from, &to, uri, false, true)) {
+ return;
+ }
+
+ migrate_set_capability(from, "x-ignore-shared", true);
+ migrate_set_capability(to, "x-ignore-shared", true);
+
+ /* Wait for the first serial output from the source */
+ wait_for_serial("src_serial");
+
+ migrate(from, uri, "{}");
+
+ wait_for_migration_pass(from);
+
+ if (!got_stop) {
+ qtest_qmp_eventwait(from, "STOP");
+ }
+
+ qtest_qmp_eventwait(to, "RESUME");
+
+ wait_for_serial("dest_serial");
+ wait_for_migration_complete(from);
+
+ test_migrate_end(from, to, true);
+ g_free(uri);
+}
+
int main(int argc, char **argv)
{
char template[] = "/tmp/migration-test-XXXXXX";
@@ -832,6 +900,7 @@ int main(int argc, char **argv)
qtest_add_func("/migration/deprecated", test_deprecated);
qtest_add_func("/migration/bad_dest", test_baddest);
qtest_add_func("/migration/precopy/unix", test_precopy_unix);
+ qtest_add_func("/migration/ignore_shared", test_ignore_shared);
ret = g_test_run();
--
2.20.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* [Qemu-devel] [PATCH v2 4/4] migration: Add capabilities validation
2019-02-04 13:09 [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability Yury Kotov
` (2 preceding siblings ...)
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 3/4] tests/migration-test: Add a test for " Yury Kotov
@ 2019-02-04 13:09 ` Yury Kotov
2019-02-11 13:30 ` Dr. David Alan Gilbert
2019-02-11 11:24 ` [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability Yury Kotov
2019-02-11 16:03 ` Dr. David Alan Gilbert
5 siblings, 1 reply; 16+ messages in thread
From: Yury Kotov @ 2019-02-04 13:09 UTC (permalink / raw)
To: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela,
Dr. David Alan Gilbert, Eric Blake, Markus Armbruster,
Thomas Huth, Laurent Vivier
Cc: wrfsh
Currently we don't check which capabilities set in the source QEMU.
We just expect that the target QEMU has the same enabled capabilities.
Add explicit validation for capabilities to make sure that the target VM
has them too. This is enabled for only new capabilities to keep compatibily.
Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
---
migration/savevm.c | 101 +++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 101 insertions(+)
diff --git a/migration/savevm.c b/migration/savevm.c
index 322660438d..9603a38bca 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -57,6 +57,7 @@
#include "sysemu/replay.h"
#include "qjson.h"
#include "migration/colo.h"
+#include "qemu/bitmap.h"
#ifndef ETH_P_RARP
#define ETH_P_RARP 0x8035
@@ -316,6 +317,8 @@ typedef struct SaveState {
uint32_t len;
const char *name;
uint32_t target_page_bits;
+ uint32_t caps_count;
+ uint8_t *capabilities;
} SaveState;
static SaveState savevm_state = {
@@ -323,15 +326,51 @@ static SaveState savevm_state = {
.global_section_id = 0,
};
+static bool should_validate_capability(int capability)
+{
+ assert(capability >= 0 && capability < MIGRATION_CAPABILITY__MAX);
+ /* Validate only new capabilities to keep compatibility. */
+ switch (capability) {
+ case MIGRATION_CAPABILITY_X_IGNORE_SHARED:
+ return true;
+ default:
+ return false;
+ }
+}
+
+static uint32_t get_validatable_capabilities_count(void)
+{
+ MigrationState *s = migrate_get_current();
+ uint32_t result = 0;
+ int i;
+ for (i = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
+ if (should_validate_capability(i) && s->enabled_capabilities[i]) {
+ result++;
+ }
+ }
+ return result;
+}
+
static int configuration_pre_save(void *opaque)
{
SaveState *state = opaque;
const char *current_name = MACHINE_GET_CLASS(current_machine)->name;
+ MigrationState *s = migrate_get_current();
+ int i, j;
state->len = strlen(current_name);
state->name = current_name;
state->target_page_bits = qemu_target_page_bits();
+ state->caps_count = get_validatable_capabilities_count();
+ state->capabilities = g_renew(uint8_t, state->capabilities,
+ state->caps_count);
+ for (i = j = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
+ if (should_validate_capability(i) && s->enabled_capabilities[i]) {
+ state->capabilities[j++] = i;
+ }
+ }
+
return 0;
}
@@ -347,6 +386,45 @@ static int configuration_pre_load(void *opaque)
return 0;
}
+static bool configuration_validate_capabilities(SaveState *state)
+{
+ bool ret = true;
+ MigrationState *s = migrate_get_current();
+ unsigned long *source_caps_bm;
+ int i;
+
+ source_caps_bm = bitmap_new(MIGRATION_CAPABILITY__MAX);
+ for (i = 0; i < state->caps_count; i++) {
+ int capability = state->capabilities[i];
+ if (capability >= MIGRATION_CAPABILITY__MAX) {
+ error_report("Received unknown capability %d", capability);
+ ret = false;
+ } else {
+ set_bit(capability, source_caps_bm);
+ }
+ }
+
+ for (i = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
+ bool source_state, target_state;
+ if (!should_validate_capability(i)) {
+ continue;
+ }
+ source_state = test_bit(i, source_caps_bm);
+ target_state = s->enabled_capabilities[i];
+ if (source_state != target_state) {
+ error_report("Capability %s is %s, but received capability is %s",
+ MigrationCapability_str(i),
+ target_state ? "on" : "off",
+ source_state ? "on" : "off");
+ ret = false;
+ /* Don't break here to report all failed capabilities */
+ }
+ }
+
+ g_free(source_caps_bm);
+ return ret;
+}
+
static int configuration_post_load(void *opaque, int version_id)
{
SaveState *state = opaque;
@@ -364,6 +442,10 @@ static int configuration_post_load(void *opaque, int version_id)
return -EINVAL;
}
+ if (!configuration_validate_capabilities(state)) {
+ return -EINVAL;
+ }
+
return 0;
}
@@ -380,6 +462,11 @@ static bool vmstate_target_page_bits_needed(void *opaque)
> qemu_target_page_bits_min();
}
+static bool vmstate_capabilites_needed(void *opaque)
+{
+ return get_validatable_capabilities_count() > 0;
+}
+
static const VMStateDescription vmstate_target_page_bits = {
.name = "configuration/target-page-bits",
.version_id = 1,
@@ -391,6 +478,19 @@ static const VMStateDescription vmstate_target_page_bits = {
}
};
+static const VMStateDescription vmstate_capabilites = {
+ .name = "configuration/capabilities",
+ .version_id = 1,
+ .minimum_version_id = 1,
+ .needed = vmstate_capabilites_needed,
+ .fields = (VMStateField[]) {
+ VMSTATE_UINT32_V(caps_count, SaveState, 1),
+ VMSTATE_VARRAY_UINT32_ALLOC(capabilities, SaveState, caps_count, 1,
+ vmstate_info_uint8, uint8_t),
+ VMSTATE_END_OF_LIST()
+ }
+};
+
static const VMStateDescription vmstate_configuration = {
.name = "configuration",
.version_id = 1,
@@ -404,6 +504,7 @@ static const VMStateDescription vmstate_configuration = {
},
.subsections = (const VMStateDescription*[]) {
&vmstate_target_page_bits,
+ &vmstate_capabilites,
NULL
}
};
--
2.20.1
^ permalink raw reply related [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability
2019-02-04 13:09 [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability Yury Kotov
` (3 preceding siblings ...)
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 4/4] migration: Add capabilities validation Yury Kotov
@ 2019-02-11 11:24 ` Yury Kotov
2019-02-11 16:03 ` Dr. David Alan Gilbert
5 siblings, 0 replies; 16+ messages in thread
From: Yury Kotov @ 2019-02-11 11:24 UTC (permalink / raw)
To: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela,
Dr. David Alan Gilbert, Eric Blake, Markus Armbruster,
Thomas Huth, Laurent Vivier
Cc: wrfsh
Ping
04.02.2019, 16:27, "Yury Kotov" <yury-kotov@yandex-team.ru>:
> Hi,
>
> The series adds a migration capability, which allows to skip shared RAM blocks
> during the migration. It's useful for fast local migration. E.g. to update QEMU
> for the running guests.
>
> Usage example:
> 1. Start source VM:
> qemu-system-x86 \
> -m 4G \
> -object memory-backend-file,id=mem0,size=4G,share=on,mem-path=/dev/shm/mem0 \
> -numa node,memdev=mem0 \
> -qmp unix:/tmp/qemu-qmp-1.sock,server,nowait \
>
> 2. Start target VM:
> qemu-system-x86 \
> -m 4G \
> -object memory-backend-file,id=mem0,size=4G,share=on,mem-path=/dev/shm/mem0 \
> -numa node,memdev=mem0 \
> -qmp unix:/tmp/qemu-qmp-2.sock,server,nowait \
> -incoming defer
>
> 3. Enable ignore-external capability on both VMs:
> { "execute": "migrate-set-capabilities" , "arguments":
> { "capabilities": [ { "capability": "x-ignore-external", "state": true } ] } }
>
> 4. Start migration.
>
> Another use case I keep in mind is to migrate to file. Usage is very similar.
>
> V1 to V2:
> * Keep migration stream compatibility
> * Reuse the existing code to ignore unwanted RAMBlocks
> * Add capability validation feature
> * ignore-external -> ignore-shared
>
> Regards,
> Yury
>
> Yury Kotov (5):
> exec: Change RAMBlockIterFunc definition
> migration: Move qemu_ram_foreach_migratable_block to migration code
> migration: Introduce ignore-shared capability
> tests/migration-test: Add a test for ignore-shared capability
> migration: Add capabilities validation
>
> exec.c | 38 ++++++-------
> include/exec/cpu-common.h | 7 +--
> migration/migration.c | 9 ++++
> migration/migration.h | 11 +++-
> migration/postcopy-ram.c | 48 +++++++++--------
> migration/ram.c | 86 ++++++++++++++++++++++++++----
> migration/rdma.c | 9 ++--
> migration/savevm.c | 101 +++++++++++++++++++++++++++++++++++
> qapi/migration.json | 5 +-
> stubs/ram-block.c | 15 ++++++
> tests/migration-test.c | 109 +++++++++++++++++++++++++++++++-------
> util/vfio-helpers.c | 6 +--
> 12 files changed, 361 insertions(+), 83 deletions(-)
>
> --
> 2.20.1
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 1/4] exec: Change RAMBlockIterFunc definition
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 1/4] exec: Change RAMBlockIterFunc definition Yury Kotov
@ 2019-02-11 11:40 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2019-02-11 11:40 UTC (permalink / raw)
To: Yury Kotov
Cc: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela, Eric Blake,
Markus Armbruster, Thomas Huth, Laurent Vivier, wrfsh
* Yury Kotov (yury-kotov@yandex-team.ru) wrote:
> Currently, qemu_ram_foreach_* calls RAMBlockIterFunc with many
> block-specific arguments. But often iter func needs RAMBlock*.
> It's more effective to call RAMBlockIterFunc with RAMBlock* argument.
> So, fix RAMBlockIterFunc definition and add some functions to read RAMBlock*
> fields witch were passed.
>
> Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> exec.c | 21 +++++++++++++++++----
> include/exec/cpu-common.h | 6 ++++--
> migration/postcopy-ram.c | 36 +++++++++++++++++++++---------------
> migration/rdma.c | 7 +++++--
> util/vfio-helpers.c | 6 +++---
> 5 files changed, 50 insertions(+), 26 deletions(-)
>
> diff --git a/exec.c b/exec.c
> index da3e635f91..a61d501568 100644
> --- a/exec.c
> +++ b/exec.c
> @@ -1970,6 +1970,21 @@ const char *qemu_ram_get_idstr(RAMBlock *rb)
> return rb->idstr;
> }
>
> +void *qemu_ram_get_host_addr(RAMBlock *rb)
> +{
> + return rb->host;
> +}
> +
> +ram_addr_t qemu_ram_get_offset(RAMBlock *rb)
> +{
> + return rb->offset;
> +}
> +
> +ram_addr_t qemu_ram_get_used_length(RAMBlock *rb)
> +{
> + return rb->used_length;
> +}
> +
> bool qemu_ram_is_shared(RAMBlock *rb)
> {
> return rb->flags & RAM_SHARED;
> @@ -3960,8 +3975,7 @@ int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
>
> rcu_read_lock();
> RAMBLOCK_FOREACH(block) {
> - ret = func(block->idstr, block->host, block->offset,
> - block->used_length, opaque);
> + ret = func(block, opaque);
> if (ret) {
> break;
> }
> @@ -3980,8 +3994,7 @@ int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque)
> if (!qemu_ram_is_migratable(block)) {
> continue;
> }
> - ret = func(block->idstr, block->host, block->offset,
> - block->used_length, opaque);
> + ret = func(block, opaque);
> if (ret) {
> break;
> }
> diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
> index 2ad2d6d86b..bdae5446d7 100644
> --- a/include/exec/cpu-common.h
> +++ b/include/exec/cpu-common.h
> @@ -72,6 +72,9 @@ ram_addr_t qemu_ram_block_host_offset(RAMBlock *rb, void *host);
> void qemu_ram_set_idstr(RAMBlock *block, const char *name, DeviceState *dev);
> void qemu_ram_unset_idstr(RAMBlock *block);
> const char *qemu_ram_get_idstr(RAMBlock *rb);
> +void *qemu_ram_get_host_addr(RAMBlock *rb);
> +ram_addr_t qemu_ram_get_offset(RAMBlock *rb);
> +ram_addr_t qemu_ram_get_used_length(RAMBlock *rb);
> bool qemu_ram_is_shared(RAMBlock *rb);
> bool qemu_ram_is_uf_zeroable(RAMBlock *rb);
> void qemu_ram_set_uf_zeroable(RAMBlock *rb);
> @@ -116,8 +119,7 @@ void cpu_flush_icache_range(hwaddr start, int len);
> extern struct MemoryRegion io_mem_rom;
> extern struct MemoryRegion io_mem_notdirty;
>
> -typedef int (RAMBlockIterFunc)(const char *block_name, void *host_addr,
> - ram_addr_t offset, ram_addr_t length, void *opaque);
> +typedef int (RAMBlockIterFunc)(RAMBlock *rb, void *opaque);
>
> int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
> int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque);
> diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
> index fa09dba534..b098816221 100644
> --- a/migration/postcopy-ram.c
> +++ b/migration/postcopy-ram.c
> @@ -319,10 +319,10 @@ static bool ufd_check_and_apply(int ufd, MigrationIncomingState *mis)
>
> /* Callback from postcopy_ram_supported_by_host block iterator.
> */
> -static int test_ramblock_postcopiable(const char *block_name, void *host_addr,
> - ram_addr_t offset, ram_addr_t length, void *opaque)
> +static int test_ramblock_postcopiable(RAMBlock *rb, void *opaque)
> {
> - RAMBlock *rb = qemu_ram_block_by_name(block_name);
> + const char *block_name = qemu_ram_get_idstr(rb);
> + ram_addr_t length = qemu_ram_get_used_length(rb);
> size_t pagesize = qemu_ram_pagesize(rb);
>
> if (length % pagesize) {
> @@ -443,9 +443,12 @@ out:
> * must be done right at the start prior to pre-copy.
> * opaque should be the MIS.
> */
> -static int init_range(const char *block_name, void *host_addr,
> - ram_addr_t offset, ram_addr_t length, void *opaque)
> +static int init_range(RAMBlock *rb, void *opaque)
> {
> + const char *block_name = qemu_ram_get_idstr(rb);
> + void *host_addr = qemu_ram_get_host_addr(rb);
> + ram_addr_t offset = qemu_ram_get_offset(rb);
> + ram_addr_t length = qemu_ram_get_used_length(rb);
> trace_postcopy_init_range(block_name, host_addr, offset, length);
>
> /*
> @@ -465,9 +468,12 @@ static int init_range(const char *block_name, void *host_addr,
> * At the end of migration, undo the effects of init_range
> * opaque should be the MIS.
> */
> -static int cleanup_range(const char *block_name, void *host_addr,
> - ram_addr_t offset, ram_addr_t length, void *opaque)
> +static int cleanup_range(RAMBlock *rb, void *opaque)
> {
> + const char *block_name = qemu_ram_get_idstr(rb);
> + void *host_addr = qemu_ram_get_host_addr(rb);
> + ram_addr_t offset = qemu_ram_get_offset(rb);
> + ram_addr_t length = qemu_ram_get_used_length(rb);
> MigrationIncomingState *mis = opaque;
> struct uffdio_range range_struct;
> trace_postcopy_cleanup_range(block_name, host_addr, offset, length);
> @@ -586,9 +592,12 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
> /*
> * Disable huge pages on an area
> */
> -static int nhp_range(const char *block_name, void *host_addr,
> - ram_addr_t offset, ram_addr_t length, void *opaque)
> +static int nhp_range(RAMBlock *rb, void *opaque)
> {
> + const char *block_name = qemu_ram_get_idstr(rb);
> + void *host_addr = qemu_ram_get_host_addr(rb);
> + ram_addr_t offset = qemu_ram_get_offset(rb);
> + ram_addr_t length = qemu_ram_get_used_length(rb);
> trace_postcopy_nhp_range(block_name, host_addr, offset, length);
>
> /*
> @@ -626,15 +635,13 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
> * opaque: MigrationIncomingState pointer
> * Returns 0 on success
> */
> -static int ram_block_enable_notify(const char *block_name, void *host_addr,
> - ram_addr_t offset, ram_addr_t length,
> - void *opaque)
> +static int ram_block_enable_notify(RAMBlock *rb, void *opaque)
> {
> MigrationIncomingState *mis = opaque;
> struct uffdio_register reg_struct;
>
> - reg_struct.range.start = (uintptr_t)host_addr;
> - reg_struct.range.len = length;
> + reg_struct.range.start = (uintptr_t)qemu_ram_get_host_addr(rb);
> + reg_struct.range.len = qemu_ram_get_used_length(rb);
> reg_struct.mode = UFFDIO_REGISTER_MODE_MISSING;
>
> /* Now tell our userfault_fd that it's responsible for this area */
> @@ -647,7 +654,6 @@ static int ram_block_enable_notify(const char *block_name, void *host_addr,
> return -1;
> }
> if (reg_struct.ioctls & ((__u64)1 << _UFFDIO_ZEROPAGE)) {
> - RAMBlock *rb = qemu_ram_block_by_name(block_name);
> qemu_ram_set_uf_zeroable(rb);
> }
>
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 54a3c11540..7eb38ee764 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -624,9 +624,12 @@ static int rdma_add_block(RDMAContext *rdma, const char *block_name,
> * in advanced before the migration starts. This tells us where the RAM blocks
> * are so that we can register them individually.
> */
> -static int qemu_rdma_init_one_block(const char *block_name, void *host_addr,
> - ram_addr_t block_offset, ram_addr_t length, void *opaque)
> +static int qemu_rdma_init_one_block(RAMBlock *rb, void *opaque)
> {
> + const char *block_name = qemu_ram_get_idstr(rb);
> + void *host_addr = qemu_ram_get_host_addr(rb);
> + ram_addr_t block_offset = qemu_ram_get_offset(rb);
> + ram_addr_t length = qemu_ram_get_used_length(rb);
> return rdma_add_block(opaque, block_name, host_addr, block_offset, length);
> }
>
> diff --git a/util/vfio-helpers.c b/util/vfio-helpers.c
> index 342d4a2285..2367fe8f7f 100644
> --- a/util/vfio-helpers.c
> +++ b/util/vfio-helpers.c
> @@ -391,10 +391,10 @@ static void qemu_vfio_ram_block_removed(RAMBlockNotifier *n,
> }
> }
>
> -static int qemu_vfio_init_ramblock(const char *block_name, void *host_addr,
> - ram_addr_t offset, ram_addr_t length,
> - void *opaque)
> +static int qemu_vfio_init_ramblock(RAMBlock *rb, void *opaque)
> {
> + void *host_addr = qemu_ram_get_host_addr(rb);
> + ram_addr_t length = qemu_ram_get_used_length(rb);
> int ret;
> QEMUVFIOState *s = opaque;
>
> --
> 2.20.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 2/4] migration: Introduce ignore-shared capability
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 2/4] migration: Introduce ignore-shared capability Yury Kotov
@ 2019-02-11 12:45 ` Dr. David Alan Gilbert
2019-02-11 13:36 ` Yury Kotov
0 siblings, 1 reply; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2019-02-11 12:45 UTC (permalink / raw)
To: Yury Kotov
Cc: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela, Eric Blake,
Markus Armbruster, Thomas Huth, Laurent Vivier, wrfsh
* Yury Kotov (yury-kotov@yandex-team.ru) wrote:
> We want to use local migration to update QEMU for running guests.
> In this case we don't need to migrate shared (file backed) RAM.
> So, add a capability to ignore such blocks during live migration.
>
> Also, move qemu_ram_foreach_migratable_block (and rename) to the
> migration code, because it requires access to the migration capabilities.
>
> Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
You could split this patch into the one that introduces the capability
and then the one that wires it up. We could also remove the x- at some
point.
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> exec.c | 19 -------
> include/exec/cpu-common.h | 1 -
> migration/migration.c | 9 ++++
> migration/migration.h | 5 +-
> migration/postcopy-ram.c | 12 ++---
> migration/ram.c | 110 +++++++++++++++++++++++++++++---------
> migration/rdma.c | 2 +-
> qapi/migration.json | 5 +-
> stubs/ram-block.c | 15 ++++++
> 9 files changed, 123 insertions(+), 55 deletions(-)
>
> diff --git a/exec.c b/exec.c
> index a61d501568..91bfe5fb62 100644
> --- a/exec.c
> +++ b/exec.c
> @@ -3984,25 +3984,6 @@ int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
> return ret;
> }
>
> -int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque)
> -{
> - RAMBlock *block;
> - int ret = 0;
> -
> - rcu_read_lock();
> - RAMBLOCK_FOREACH(block) {
> - if (!qemu_ram_is_migratable(block)) {
> - continue;
> - }
> - ret = func(block, opaque);
> - if (ret) {
> - break;
> - }
> - }
> - rcu_read_unlock();
> - return ret;
> -}
> -
> /*
> * Unmap pages of memory from start to start+length such that
> * they a) read as 0, b) Trigger whatever fault mechanism
> diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
> index bdae5446d7..403463d7bb 100644
> --- a/include/exec/cpu-common.h
> +++ b/include/exec/cpu-common.h
> @@ -122,7 +122,6 @@ extern struct MemoryRegion io_mem_notdirty;
> typedef int (RAMBlockIterFunc)(RAMBlock *rb, void *opaque);
>
> int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
> -int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque);
> int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length);
>
> #endif
> diff --git a/migration/migration.c b/migration/migration.c
> index 37e06b76dc..c40776a40c 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -1983,6 +1983,15 @@ bool migrate_dirty_bitmaps(void)
> return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS];
> }
>
> +bool migrate_ignore_shared(void)
> +{
> + MigrationState *s;
> +
> + s = migrate_get_current();
> +
> + return s->enabled_capabilities[MIGRATION_CAPABILITY_X_IGNORE_SHARED];
> +}
> +
> bool migrate_use_events(void)
> {
> MigrationState *s;
> diff --git a/migration/migration.h b/migration/migration.h
> index dcd05d9f87..2c88f8a555 100644
> --- a/migration/migration.h
> +++ b/migration/migration.h
> @@ -261,6 +261,7 @@ bool migrate_release_ram(void);
> bool migrate_postcopy_ram(void);
> bool migrate_zero_blocks(void);
> bool migrate_dirty_bitmaps(void);
> +bool migrate_ignore_shared(void);
>
> bool migrate_auto_converge(void);
> bool migrate_use_multifd(void);
> @@ -301,8 +302,10 @@ void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value);
> void dirty_bitmap_mig_before_vm_start(void);
> void init_dirty_bitmap_incoming_migration(void);
>
> +int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque);
> +
> #define qemu_ram_foreach_block \
> - #warning "Use qemu_ram_foreach_block_migratable in migration code"
> + #warning "Use foreach_not_ignored_block in migration code"
>
> void migration_make_urgent_request(void);
> void migration_consume_urgent_request(void);
> diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
> index b098816221..e2aa57a701 100644
> --- a/migration/postcopy-ram.c
> +++ b/migration/postcopy-ram.c
> @@ -374,7 +374,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis)
> }
>
> /* We don't support postcopy with shared RAM yet */
> - if (qemu_ram_foreach_migratable_block(test_ramblock_postcopiable, NULL)) {
> + if (foreach_not_ignored_block(test_ramblock_postcopiable, NULL)) {
> goto out;
> }
>
> @@ -508,7 +508,7 @@ static int cleanup_range(RAMBlock *rb, void *opaque)
> */
> int postcopy_ram_incoming_init(MigrationIncomingState *mis)
> {
> - if (qemu_ram_foreach_migratable_block(init_range, NULL)) {
> + if (foreach_not_ignored_block(init_range, NULL)) {
> return -1;
> }
>
> @@ -550,7 +550,7 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
> return -1;
> }
>
> - if (qemu_ram_foreach_migratable_block(cleanup_range, mis)) {
> + if (foreach_not_ignored_block(cleanup_range, mis)) {
> return -1;
> }
>
> @@ -617,7 +617,7 @@ static int nhp_range(RAMBlock *rb, void *opaque)
> */
> int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
> {
> - if (qemu_ram_foreach_migratable_block(nhp_range, mis)) {
> + if (foreach_not_ignored_block(nhp_range, mis)) {
> return -1;
> }
>
> @@ -628,7 +628,7 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
>
> /*
> * Mark the given area of RAM as requiring notification to unwritten areas
> - * Used as a callback on qemu_ram_foreach_migratable_block.
> + * Used as a callback on foreach_not_ignored_block.
> * host_addr: Base of area to mark
> * offset: Offset in the whole ram arena
> * length: Length of the section
> @@ -1122,7 +1122,7 @@ int postcopy_ram_enable_notify(MigrationIncomingState *mis)
> mis->have_fault_thread = true;
>
> /* Mark so that we get notified of accesses to unwritten areas */
> - if (qemu_ram_foreach_migratable_block(ram_block_enable_notify, mis)) {
> + if (foreach_not_ignored_block(ram_block_enable_notify, mis)) {
> error_report("ram_block_enable_notify failed");
> return -1;
> }
> diff --git a/migration/ram.c b/migration/ram.c
> index 59191c1ed2..01315edd66 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -159,18 +159,44 @@ out:
> return ret;
> }
>
> +static bool ramblock_is_ignored(RAMBlock *block)
> +{
> + return !qemu_ram_is_migratable(block) ||
> + (migrate_ignore_shared() && qemu_ram_is_shared(block));
> +}
> +
> /* Should be holding either ram_list.mutex, or the RCU lock. */
> +#define RAMBLOCK_FOREACH_NOT_IGNORED(block) \
> + INTERNAL_RAMBLOCK_FOREACH(block) \
> + if (ramblock_is_ignored(block)) {} else
> +
> #define RAMBLOCK_FOREACH_MIGRATABLE(block) \
> INTERNAL_RAMBLOCK_FOREACH(block) \
> if (!qemu_ram_is_migratable(block)) {} else
>
> #undef RAMBLOCK_FOREACH
>
> +int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque)
> +{
> + RAMBlock *block;
> + int ret = 0;
> +
> + rcu_read_lock();
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> + ret = func(block, opaque);
> + if (ret) {
> + break;
> + }
> + }
> + rcu_read_unlock();
> + return ret;
> +}
> +
> static void ramblock_recv_map_init(void)
> {
> RAMBlock *rb;
>
> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
> assert(!rb->receivedmap);
> rb->receivedmap = bitmap_new(rb->max_length >> qemu_target_page_bits());
> }
> @@ -1545,7 +1571,7 @@ unsigned long migration_bitmap_find_dirty(RAMState *rs, RAMBlock *rb,
> unsigned long *bitmap = rb->bmap;
> unsigned long next;
>
> - if (!qemu_ram_is_migratable(rb)) {
> + if (ramblock_is_ignored(rb)) {
> return size;
> }
>
> @@ -1594,7 +1620,7 @@ uint64_t ram_pagesize_summary(void)
> RAMBlock *block;
> uint64_t summary = 0;
>
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> summary |= block->page_size;
> }
>
> @@ -1664,7 +1690,7 @@ static void migration_bitmap_sync(RAMState *rs)
>
> qemu_mutex_lock(&rs->bitmap_mutex);
> rcu_read_lock();
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> migration_bitmap_sync_range(rs, block, 0, block->used_length);
> }
> ram_counters.remaining = ram_bytes_remaining();
> @@ -2388,7 +2414,7 @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss,
> size_t pagesize_bits =
> qemu_ram_pagesize(pss->block) >> TARGET_PAGE_BITS;
>
> - if (!qemu_ram_is_migratable(pss->block)) {
> + if (ramblock_is_ignored(pss->block)) {
> error_report("block %s should not be migrated !", pss->block->idstr);
> return 0;
> }
> @@ -2486,19 +2512,30 @@ void acct_update_position(QEMUFile *f, size_t size, bool zero)
> }
> }
>
> -uint64_t ram_bytes_total(void)
> +static uint64_t ram_bytes_total_common(bool count_ignored)
> {
> RAMBlock *block;
> uint64_t total = 0;
>
> rcu_read_lock();
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> - total += block->used_length;
> + if (count_ignored) {
> + RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + total += block->used_length;
> + }
> + } else {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> + total += block->used_length;
> + }
> }
> rcu_read_unlock();
> return total;
> }
>
> +uint64_t ram_bytes_total(void)
> +{
> + return ram_bytes_total_common(false);
> +}
> +
> static void xbzrle_load_setup(void)
> {
> XBZRLE.decoded_buf = g_malloc(TARGET_PAGE_SIZE);
> @@ -2547,7 +2584,7 @@ static void ram_save_cleanup(void *opaque)
> */
> memory_global_dirty_log_stop();
>
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> g_free(block->bmap);
> block->bmap = NULL;
> g_free(block->unsentmap);
> @@ -2610,7 +2647,7 @@ void ram_postcopy_migrated_memory_release(MigrationState *ms)
> {
> struct RAMBlock *block;
>
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> unsigned long *bitmap = block->bmap;
> unsigned long range = block->used_length >> TARGET_PAGE_BITS;
> unsigned long run_start = find_next_zero_bit(bitmap, range, 0);
> @@ -2688,7 +2725,7 @@ static int postcopy_each_ram_send_discard(MigrationState *ms)
> struct RAMBlock *block;
> int ret;
>
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> PostcopyDiscardState *pds =
> postcopy_discard_send_init(ms, block->idstr);
>
> @@ -2896,7 +2933,7 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms)
> rs->last_sent_block = NULL;
> rs->last_page = 0;
>
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> unsigned long pages = block->used_length >> TARGET_PAGE_BITS;
> unsigned long *bitmap = block->bmap;
> unsigned long *unsentmap = block->unsentmap;
> @@ -3062,7 +3099,7 @@ static void ram_list_init_bitmaps(void)
>
> /* Skip setting bitmap if there is no RAM */
> if (ram_bytes_total()) {
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> pages = block->max_length >> TARGET_PAGE_BITS;
> block->bmap = bitmap_new(pages);
> bitmap_set(block->bmap, 0, pages);
> @@ -3117,7 +3154,7 @@ static void ram_state_resume_prepare(RAMState *rs, QEMUFile *out)
> * about dirty page logging as well.
> */
>
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> pages += bitmap_count_one(block->bmap,
> block->used_length >> TARGET_PAGE_BITS);
> }
> @@ -3176,7 +3213,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
>
> rcu_read_lock();
>
> - qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
> + qemu_put_be64(f, ram_bytes_total_common(true) | RAM_SAVE_FLAG_MEM_SIZE);
>
> RAMBLOCK_FOREACH_MIGRATABLE(block) {
> qemu_put_byte(f, strlen(block->idstr));
> @@ -3185,6 +3222,10 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
> if (migrate_postcopy_ram() && block->page_size != qemu_host_page_size) {
> qemu_put_be64(f, block->page_size);
> }
> + if (migrate_ignore_shared()) {
> + qemu_put_be64(f, block->mr->addr);
> + qemu_put_byte(f, ramblock_is_ignored(block) ? 1 : 0);
> + }
> }
>
> rcu_read_unlock();
> @@ -3443,7 +3484,7 @@ static inline RAMBlock *ram_block_from_stream(QEMUFile *f, int flags)
> return NULL;
> }
>
> - if (!qemu_ram_is_migratable(block)) {
> + if (ramblock_is_ignored(block)) {
> error_report("block %s should not be migrated !", id);
> return NULL;
> }
> @@ -3698,7 +3739,7 @@ int colo_init_ram_cache(void)
> RAMBlock *block;
>
> rcu_read_lock();
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> block->colo_cache = qemu_anon_ram_alloc(block->used_length,
> NULL,
> false);
> @@ -3719,7 +3760,7 @@ int colo_init_ram_cache(void)
> if (ram_bytes_total()) {
> RAMBlock *block;
>
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> unsigned long pages = block->max_length >> TARGET_PAGE_BITS;
>
> block->bmap = bitmap_new(pages);
> @@ -3734,7 +3775,7 @@ int colo_init_ram_cache(void)
>
> out_locked:
>
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> if (block->colo_cache) {
> qemu_anon_ram_free(block->colo_cache, block->used_length);
> block->colo_cache = NULL;
> @@ -3751,14 +3792,14 @@ void colo_release_ram_cache(void)
> RAMBlock *block;
>
> memory_global_dirty_log_stop();
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> g_free(block->bmap);
> block->bmap = NULL;
> }
>
> rcu_read_lock();
>
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> if (block->colo_cache) {
> qemu_anon_ram_free(block->colo_cache, block->used_length);
> block->colo_cache = NULL;
> @@ -3794,7 +3835,7 @@ static int ram_load_cleanup(void *opaque)
> {
> RAMBlock *rb;
>
> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
> if (ramblock_is_pmem(rb)) {
> pmem_persist(rb->host, rb->used_length);
> }
> @@ -3803,7 +3844,7 @@ static int ram_load_cleanup(void *opaque)
> xbzrle_load_cleanup();
> compress_threads_load_cleanup();
>
> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
> g_free(rb->receivedmap);
> rb->receivedmap = NULL;
> }
> @@ -4003,7 +4044,7 @@ static void colo_flush_ram_cache(void)
>
> memory_global_dirty_log_sync();
> rcu_read_lock();
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> migration_bitmap_sync_range(ram_state, block, 0, block->used_length);
> }
> rcu_read_unlock();
> @@ -4146,6 +4187,23 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> ret = -EINVAL;
> }
> }
> + if (migrate_ignore_shared()) {
> + hwaddr addr = qemu_get_be64(f);
> + bool ignored = qemu_get_byte(f);
> + if (ignored != ramblock_is_ignored(block)) {
> + error_report("RAM block %s should %s be migrated",
> + id, ignored ? "" : "not");
> + ret = -EINVAL;
> + }
> + if (ramblock_is_ignored(block) &&
> + block->mr->addr != addr) {
> + error_report("Mismatched GPAs for block %s "
> + "%" PRId64 "!= %" PRId64,
> + id, (uint64_t)addr,
> + (uint64_t)block->mr->addr);
> + ret = -EINVAL;
> + }
> + }
> ram_control_load_hook(f, RAM_CONTROL_BLOCK_REG,
> block->idstr);
> } else {
> @@ -4216,7 +4274,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> static bool ram_has_postcopy(void *opaque)
> {
> RAMBlock *rb;
> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
> if (ramblock_is_pmem(rb)) {
> info_report("Block: %s, host: %p is a nvdimm memory, postcopy"
> "is not supported now!", rb->idstr, rb->host);
> @@ -4236,7 +4294,7 @@ static int ram_dirty_bitmap_sync_all(MigrationState *s, RAMState *rs)
>
> trace_ram_dirty_bitmap_sync_start();
>
> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> qemu_savevm_send_recv_bitmap(file, block->idstr);
> trace_ram_dirty_bitmap_request(block->idstr);
> ramblock_count++;
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 7eb38ee764..3cb579cc99 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -644,7 +644,7 @@ static int qemu_rdma_init_ram_blocks(RDMAContext *rdma)
>
> assert(rdma->blockmap == NULL);
> memset(local, 0, sizeof *local);
> - qemu_ram_foreach_migratable_block(qemu_rdma_init_one_block, rdma);
> + foreach_not_ignored_block(qemu_rdma_init_one_block, rdma);
> trace_qemu_rdma_init_ram_blocks(local->nb_blocks);
> rdma->dest_blocks = g_new0(RDMADestBlock,
> rdma->local_ram_blocks.nb_blocks);
> diff --git a/qapi/migration.json b/qapi/migration.json
> index 7a795ecc16..7105570cd3 100644
> --- a/qapi/migration.json
> +++ b/qapi/migration.json
> @@ -409,13 +409,16 @@
> # devices (and thus take locks) immediately at the end of migration.
> # (since 3.0)
> #
> +# @x-ignore-shared: If enabled, QEMU will not migrate shared memory (since 4.0)
> +#
> # Since: 1.2
> ##
> { 'enum': 'MigrationCapability',
> 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
> 'compress', 'events', 'postcopy-ram', 'x-colo', 'release-ram',
> 'block', 'return-path', 'pause-before-switchover', 'x-multifd',
> - 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate' ] }
> + 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate',
> + 'x-ignore-shared' ] }
>
> ##
> # @MigrationCapabilityStatus:
> diff --git a/stubs/ram-block.c b/stubs/ram-block.c
> index cfa5d8678f..73c0a3ee08 100644
> --- a/stubs/ram-block.c
> +++ b/stubs/ram-block.c
> @@ -2,6 +2,21 @@
> #include "exec/ramlist.h"
> #include "exec/cpu-common.h"
>
> +void *qemu_ram_get_host_addr(RAMBlock *rb)
> +{
> + return 0;
> +}
> +
> +ram_addr_t qemu_ram_get_offset(RAMBlock *rb)
> +{
> + return 0;
> +}
> +
> +ram_addr_t qemu_ram_get_used_length(RAMBlock *rb)
> +{
> + return 0;
> +}
> +
> void ram_block_notifier_add(RAMBlockNotifier *n)
> {
> }
> --
> 2.20.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 3/4] tests/migration-test: Add a test for ignore-shared capability
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 3/4] tests/migration-test: Add a test for " Yury Kotov
@ 2019-02-11 13:17 ` Dr. David Alan Gilbert
2019-02-12 12:49 ` Yury Kotov
0 siblings, 1 reply; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2019-02-11 13:17 UTC (permalink / raw)
To: Yury Kotov
Cc: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela, Eric Blake,
Markus Armbruster, Thomas Huth, Laurent Vivier, wrfsh
* Yury Kotov (yury-kotov@yandex-team.ru) wrote:
> Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
> ---
> tests/migration-test.c | 109 +++++++++++++++++++++++++++++++++--------
> 1 file changed, 89 insertions(+), 20 deletions(-)
>
> diff --git a/tests/migration-test.c b/tests/migration-test.c
> index 8352612364..485f42b2d2 100644
> --- a/tests/migration-test.c
> +++ b/tests/migration-test.c
> @@ -332,6 +332,13 @@ static void cleanup(const char *filename)
> g_free(path);
> }
>
> +static char *get_shmem_opts(const char *mem_size, const char *shmem_path)
> +{
> + return g_strdup_printf("-object memory-backend-file,id=mem0,size=%s"
> + ",mem-path=%s,share=on -numa node,memdev=mem0",
> + mem_size, shmem_path);
> +}
> +
> static void migrate_check_parameter(QTestState *who, const char *parameter,
> long long value)
> {
> @@ -430,73 +437,91 @@ static void migrate_postcopy_start(QTestState *from, QTestState *to)
> }
>
> static int test_migrate_start(QTestState **from, QTestState **to,
> - const char *uri, bool hide_stderr)
> + const char *uri, bool hide_stderr,
> + bool use_shmem)
> {
> gchar *cmd_src, *cmd_dst;
> char *bootpath = g_strdup_printf("%s/bootsect", tmpfs);
> + char *extra_opts = NULL;
> + char *shmem_path = NULL;
> const char *arch = qtest_get_arch();
> const char *accel = "kvm:tcg";
>
> got_stop = false;
>
> + if (use_shmem) {
> + shmem_path = g_strdup_printf("/dev/shm/qemu-%d", getpid());
> + }
I think /dev/shm is non-portable; so I think you'll need to have a way
to skip the test on OSs that don't have it.
> if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
> init_bootfile(bootpath, x86_bootsect);
> + extra_opts = use_shmem ? get_shmem_opts("150M", shmem_path) : NULL;
> cmd_src = g_strdup_printf("-machine accel=%s -m 150M"
> " -name source,debug-threads=on"
> " -serial file:%s/src_serial"
> - " -drive file=%s,format=raw",
> - accel, tmpfs, bootpath);
> + " -drive file=%s,format=raw %s",
> + accel, tmpfs, bootpath,
> + extra_opts ? extra_opts : "");
It's painful to have to use the ?: here as well as above where you set
extra_opts, but I guess you need to, to allow you to free extra_opts
below.
> cmd_dst = g_strdup_printf("-machine accel=%s -m 150M"
> " -name target,debug-threads=on"
> " -serial file:%s/dest_serial"
> " -drive file=%s,format=raw"
> - " -incoming %s",
> - accel, tmpfs, bootpath, uri);
> + " -incoming %s %s",
> + accel, tmpfs, bootpath, uri,
> + extra_opts ? extra_opts : "");
> start_address = X86_TEST_MEM_START;
> end_address = X86_TEST_MEM_END;
> } else if (g_str_equal(arch, "s390x")) {
> init_bootfile_s390x(bootpath);
> + extra_opts = use_shmem ? get_shmem_opts("128M", shmem_path) : NULL;
> cmd_src = g_strdup_printf("-machine accel=%s -m 128M"
> " -name source,debug-threads=on"
> - " -serial file:%s/src_serial -bios %s",
> - accel, tmpfs, bootpath);
> + " -serial file:%s/src_serial -bios %s %s",
> + accel, tmpfs, bootpath,
> + extra_opts ? extra_opts : "");
> cmd_dst = g_strdup_printf("-machine accel=%s -m 128M"
> " -name target,debug-threads=on"
> " -serial file:%s/dest_serial -bios %s"
> - " -incoming %s",
> - accel, tmpfs, bootpath, uri);
> + " -incoming %s %s",
> + accel, tmpfs, bootpath, uri,
> + extra_opts ? extra_opts : "");
> start_address = S390_TEST_MEM_START;
> end_address = S390_TEST_MEM_END;
> } else if (strcmp(arch, "ppc64") == 0) {
> + extra_opts = use_shmem ? get_shmem_opts("256M", shmem_path) : NULL;
> cmd_src = g_strdup_printf("-machine accel=%s -m 256M -nodefaults"
> " -name source,debug-threads=on"
> " -serial file:%s/src_serial"
> " -prom-env 'use-nvramrc?=true' -prom-env "
> "'nvramrc=hex .\" _\" begin %x %x "
> "do i c@ 1 + i c! 1000 +loop .\" B\" 0 "
> - "until'", accel, tmpfs, end_address,
> - start_address);
> + "until' %s", accel, tmpfs, end_address,
> + start_address, extra_opts ? extra_opts : "");
> cmd_dst = g_strdup_printf("-machine accel=%s -m 256M"
> " -name target,debug-threads=on"
> " -serial file:%s/dest_serial"
> - " -incoming %s",
> - accel, tmpfs, uri);
> + " -incoming %s %s",
> + accel, tmpfs, uri,
> + extra_opts ? extra_opts : "");
>
> start_address = PPC_TEST_MEM_START;
> end_address = PPC_TEST_MEM_END;
> } else if (strcmp(arch, "aarch64") == 0) {
> init_bootfile(bootpath, aarch64_kernel);
> + extra_opts = use_shmem ? get_shmem_opts("150M", shmem_path) : NULL;
> cmd_src = g_strdup_printf("-machine virt,accel=%s,gic-version=max "
> "-name vmsource,debug-threads=on -cpu max "
> "-m 150M -serial file:%s/src_serial "
> - "-kernel %s ",
> - accel, tmpfs, bootpath);
> + "-kernel %s %s",
> + accel, tmpfs, bootpath,
> + extra_opts ? extra_opts : "");
> cmd_dst = g_strdup_printf("-machine virt,accel=%s,gic-version=max "
> "-name vmdest,debug-threads=on -cpu max "
> "-m 150M -serial file:%s/dest_serial "
> "-kernel %s "
> - "-incoming %s ",
> - accel, tmpfs, bootpath, uri);
> + "-incoming %s %s",
> + accel, tmpfs, bootpath, uri,
> + extra_opts ? extra_opts : "");
>
> start_address = ARM_TEST_MEM_START;
> end_address = ARM_TEST_MEM_END;
> @@ -507,6 +532,7 @@ static int test_migrate_start(QTestState **from, QTestState **to,
> }
>
> g_free(bootpath);
> + g_free(extra_opts);
>
> if (hide_stderr) {
> gchar *tmp;
> @@ -524,6 +550,16 @@ static int test_migrate_start(QTestState **from, QTestState **to,
>
> *to = qtest_init(cmd_dst);
> g_free(cmd_dst);
> +
> + /*
> + * Remove shmem file immediately to avoid memory leak in test failed case.
> + * It's valid becase QEMU has already opened this file
> + */
> + if (use_shmem) {
> + unlink(shmem_path);
> + g_free(shmem_path);
> + }
> +
> return 0;
> }
>
> @@ -603,7 +639,7 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
> char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
> QTestState *from, *to;
>
> - if (test_migrate_start(&from, &to, uri, hide_error)) {
> + if (test_migrate_start(&from, &to, uri, hide_error, false)) {
> return -1;
> }
>
> @@ -720,7 +756,7 @@ static void test_baddest(void)
> char *status;
> bool failed;
>
> - if (test_migrate_start(&from, &to, "tcp:0:0", true)) {
> + if (test_migrate_start(&from, &to, "tcp:0:0", true, false)) {
> return;
> }
> migrate(from, "tcp:0:0", "{}");
> @@ -745,7 +781,7 @@ static void test_precopy_unix(void)
> char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
> QTestState *from, *to;
>
> - if (test_migrate_start(&from, &to, uri, false)) {
> + if (test_migrate_start(&from, &to, uri, false, false)) {
> return;
> }
>
> @@ -781,6 +817,38 @@ static void test_precopy_unix(void)
> g_free(uri);
> }
>
> +static void test_ignore_shared(void)
> +{
> + char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
> + QTestState *from, *to;
> +
> + if (test_migrate_start(&from, &to, uri, false, true)) {
> + return;
> + }
> +
> + migrate_set_capability(from, "x-ignore-shared", true);
> + migrate_set_capability(to, "x-ignore-shared", true);
> +
> + /* Wait for the first serial output from the source */
> + wait_for_serial("src_serial");
> +
> + migrate(from, uri, "{}");
> +
> + wait_for_migration_pass(from);
> +
> + if (!got_stop) {
> + qtest_qmp_eventwait(from, "STOP");
> + }
> +
> + qtest_qmp_eventwait(to, "RESUME");
> +
> + wait_for_serial("dest_serial");
> + wait_for_migration_complete(from);
> +
> + test_migrate_end(from, to, true);
> + g_free(uri);
Can we reliably look at the migration stats and see if we've
not transferred the shared data to make sure?
Dave
> +}
> +
> int main(int argc, char **argv)
> {
> char template[] = "/tmp/migration-test-XXXXXX";
> @@ -832,6 +900,7 @@ int main(int argc, char **argv)
> qtest_add_func("/migration/deprecated", test_deprecated);
> qtest_add_func("/migration/bad_dest", test_baddest);
> qtest_add_func("/migration/precopy/unix", test_precopy_unix);
> + qtest_add_func("/migration/ignore_shared", test_ignore_shared);
>
> ret = g_test_run();
>
> --
> 2.20.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 4/4] migration: Add capabilities validation
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 4/4] migration: Add capabilities validation Yury Kotov
@ 2019-02-11 13:30 ` Dr. David Alan Gilbert
2019-02-12 13:58 ` Yury Kotov
0 siblings, 1 reply; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2019-02-11 13:30 UTC (permalink / raw)
To: Yury Kotov
Cc: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela, Eric Blake,
Markus Armbruster, Thomas Huth, Laurent Vivier, wrfsh
* Yury Kotov (yury-kotov@yandex-team.ru) wrote:
> Currently we don't check which capabilities set in the source QEMU.
> We just expect that the target QEMU has the same enabled capabilities.
>
> Add explicit validation for capabilities to make sure that the target VM
> has them too. This is enabled for only new capabilities to keep compatibily.
I'd rather keep the capaiblities on the wire as strings, rather than
indexes; indexes are too delicate.
I guess we also have to think about what happens when new capabilities
are added; what happens when we migrate to an older qemu that doesn't
know about that capability?
What happens when we migrate to a newer capability which thinks you
forgot to send that capability across?
While we're on capabilities, I think you also need to check if the
skip-shared is enabled with postcopy; I don't think they'll work
together.
Dave
> Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
> ---
> migration/savevm.c | 101 +++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 101 insertions(+)
>
> diff --git a/migration/savevm.c b/migration/savevm.c
> index 322660438d..9603a38bca 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -57,6 +57,7 @@
> #include "sysemu/replay.h"
> #include "qjson.h"
> #include "migration/colo.h"
> +#include "qemu/bitmap.h"
>
> #ifndef ETH_P_RARP
> #define ETH_P_RARP 0x8035
> @@ -316,6 +317,8 @@ typedef struct SaveState {
> uint32_t len;
> const char *name;
> uint32_t target_page_bits;
> + uint32_t caps_count;
> + uint8_t *capabilities;
> } SaveState;
>
> static SaveState savevm_state = {
> @@ -323,15 +326,51 @@ static SaveState savevm_state = {
> .global_section_id = 0,
> };
>
> +static bool should_validate_capability(int capability)
> +{
> + assert(capability >= 0 && capability < MIGRATION_CAPABILITY__MAX);
> + /* Validate only new capabilities to keep compatibility. */
> + switch (capability) {
> + case MIGRATION_CAPABILITY_X_IGNORE_SHARED:
> + return true;
> + default:
> + return false;
> + }
> +}
> +
> +static uint32_t get_validatable_capabilities_count(void)
> +{
> + MigrationState *s = migrate_get_current();
> + uint32_t result = 0;
> + int i;
> + for (i = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
> + if (should_validate_capability(i) && s->enabled_capabilities[i]) {
> + result++;
> + }
> + }
> + return result;
> +}
> +
> static int configuration_pre_save(void *opaque)
> {
> SaveState *state = opaque;
> const char *current_name = MACHINE_GET_CLASS(current_machine)->name;
> + MigrationState *s = migrate_get_current();
> + int i, j;
>
> state->len = strlen(current_name);
> state->name = current_name;
> state->target_page_bits = qemu_target_page_bits();
>
> + state->caps_count = get_validatable_capabilities_count();
> + state->capabilities = g_renew(uint8_t, state->capabilities,
> + state->caps_count);
> + for (i = j = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
> + if (should_validate_capability(i) && s->enabled_capabilities[i]) {
> + state->capabilities[j++] = i;
> + }
> + }
> +
> return 0;
> }
>
> @@ -347,6 +386,45 @@ static int configuration_pre_load(void *opaque)
> return 0;
> }
>
> +static bool configuration_validate_capabilities(SaveState *state)
> +{
> + bool ret = true;
> + MigrationState *s = migrate_get_current();
> + unsigned long *source_caps_bm;
> + int i;
> +
> + source_caps_bm = bitmap_new(MIGRATION_CAPABILITY__MAX);
> + for (i = 0; i < state->caps_count; i++) {
> + int capability = state->capabilities[i];
> + if (capability >= MIGRATION_CAPABILITY__MAX) {
> + error_report("Received unknown capability %d", capability);
> + ret = false;
> + } else {
> + set_bit(capability, source_caps_bm);
> + }
> + }
> +
> + for (i = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
> + bool source_state, target_state;
> + if (!should_validate_capability(i)) {
> + continue;
> + }
> + source_state = test_bit(i, source_caps_bm);
> + target_state = s->enabled_capabilities[i];
> + if (source_state != target_state) {
> + error_report("Capability %s is %s, but received capability is %s",
> + MigrationCapability_str(i),
> + target_state ? "on" : "off",
> + source_state ? "on" : "off");
> + ret = false;
> + /* Don't break here to report all failed capabilities */
> + }
> + }
> +
> + g_free(source_caps_bm);
> + return ret;
> +}
> +
> static int configuration_post_load(void *opaque, int version_id)
> {
> SaveState *state = opaque;
> @@ -364,6 +442,10 @@ static int configuration_post_load(void *opaque, int version_id)
> return -EINVAL;
> }
>
> + if (!configuration_validate_capabilities(state)) {
> + return -EINVAL;
> + }
> +
> return 0;
> }
>
> @@ -380,6 +462,11 @@ static bool vmstate_target_page_bits_needed(void *opaque)
> > qemu_target_page_bits_min();
> }
>
> +static bool vmstate_capabilites_needed(void *opaque)
> +{
> + return get_validatable_capabilities_count() > 0;
> +}
> +
> static const VMStateDescription vmstate_target_page_bits = {
> .name = "configuration/target-page-bits",
> .version_id = 1,
> @@ -391,6 +478,19 @@ static const VMStateDescription vmstate_target_page_bits = {
> }
> };
>
> +static const VMStateDescription vmstate_capabilites = {
> + .name = "configuration/capabilities",
> + .version_id = 1,
> + .minimum_version_id = 1,
> + .needed = vmstate_capabilites_needed,
> + .fields = (VMStateField[]) {
> + VMSTATE_UINT32_V(caps_count, SaveState, 1),
> + VMSTATE_VARRAY_UINT32_ALLOC(capabilities, SaveState, caps_count, 1,
> + vmstate_info_uint8, uint8_t),
> + VMSTATE_END_OF_LIST()
> + }
> +};
> +
> static const VMStateDescription vmstate_configuration = {
> .name = "configuration",
> .version_id = 1,
> @@ -404,6 +504,7 @@ static const VMStateDescription vmstate_configuration = {
> },
> .subsections = (const VMStateDescription*[]) {
> &vmstate_target_page_bits,
> + &vmstate_capabilites,
> NULL
> }
> };
> --
> 2.20.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 2/4] migration: Introduce ignore-shared capability
2019-02-11 12:45 ` Dr. David Alan Gilbert
@ 2019-02-11 13:36 ` Yury Kotov
2019-02-11 15:55 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 16+ messages in thread
From: Yury Kotov @ 2019-02-11 13:36 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela, Eric Blake,
Markus Armbruster, Thomas Huth, Laurent Vivier, wrfsh
11.02.2019, 15:45, "Dr. David Alan Gilbert" <dgilbert@redhat.com>:
> * Yury Kotov (yury-kotov@yandex-team.ru) wrote:
>> We want to use local migration to update QEMU for running guests.
>> In this case we don't need to migrate shared (file backed) RAM.
>> So, add a capability to ignore such blocks during live migration.
>>
>> Also, move qemu_ram_foreach_migratable_block (and rename) to the
>> migration code, because it requires access to the migration capabilities.
>>
>> Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
>
> You could split this patch into the one that introduces the capability
> and then the one that wires it up. We could also remove the x- at some
> point.
I.e. the patch that just adds the capability to json (and migrate_use_*), but
nothing more, and the second one which actually realize the capability?
Like this:
2a4c42f18c migration: add postcopy blocktime ctx into MigrationIncomingState
f22f928ec9 migration: introduce postcopy-blocktime capability
?
>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
>> ---
>> exec.c | 19 -------
>> include/exec/cpu-common.h | 1 -
>> migration/migration.c | 9 ++++
>> migration/migration.h | 5 +-
>> migration/postcopy-ram.c | 12 ++---
>> migration/ram.c | 110 +++++++++++++++++++++++++++++---------
>> migration/rdma.c | 2 +-
>> qapi/migration.json | 5 +-
>> stubs/ram-block.c | 15 ++++++
>> 9 files changed, 123 insertions(+), 55 deletions(-)
>>
>> diff --git a/exec.c b/exec.c
>> index a61d501568..91bfe5fb62 100644
>> --- a/exec.c
>> +++ b/exec.c
>> @@ -3984,25 +3984,6 @@ int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
>> return ret;
>> }
>>
>> -int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque)
>> -{
>> - RAMBlock *block;
>> - int ret = 0;
>> -
>> - rcu_read_lock();
>> - RAMBLOCK_FOREACH(block) {
>> - if (!qemu_ram_is_migratable(block)) {
>> - continue;
>> - }
>> - ret = func(block, opaque);
>> - if (ret) {
>> - break;
>> - }
>> - }
>> - rcu_read_unlock();
>> - return ret;
>> -}
>> -
>> /*
>> * Unmap pages of memory from start to start+length such that
>> * they a) read as 0, b) Trigger whatever fault mechanism
>> diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
>> index bdae5446d7..403463d7bb 100644
>> --- a/include/exec/cpu-common.h
>> +++ b/include/exec/cpu-common.h
>> @@ -122,7 +122,6 @@ extern struct MemoryRegion io_mem_notdirty;
>> typedef int (RAMBlockIterFunc)(RAMBlock *rb, void *opaque);
>>
>> int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
>> -int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque);
>> int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length);
>>
>> #endif
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 37e06b76dc..c40776a40c 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -1983,6 +1983,15 @@ bool migrate_dirty_bitmaps(void)
>> return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS];
>> }
>>
>> +bool migrate_ignore_shared(void)
>> +{
>> + MigrationState *s;
>> +
>> + s = migrate_get_current();
>> +
>> + return s->enabled_capabilities[MIGRATION_CAPABILITY_X_IGNORE_SHARED];
>> +}
>> +
>> bool migrate_use_events(void)
>> {
>> MigrationState *s;
>> diff --git a/migration/migration.h b/migration/migration.h
>> index dcd05d9f87..2c88f8a555 100644
>> --- a/migration/migration.h
>> +++ b/migration/migration.h
>> @@ -261,6 +261,7 @@ bool migrate_release_ram(void);
>> bool migrate_postcopy_ram(void);
>> bool migrate_zero_blocks(void);
>> bool migrate_dirty_bitmaps(void);
>> +bool migrate_ignore_shared(void);
>>
>> bool migrate_auto_converge(void);
>> bool migrate_use_multifd(void);
>> @@ -301,8 +302,10 @@ void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value);
>> void dirty_bitmap_mig_before_vm_start(void);
>> void init_dirty_bitmap_incoming_migration(void);
>>
>> +int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque);
>> +
>> #define qemu_ram_foreach_block \
>> - #warning "Use qemu_ram_foreach_block_migratable in migration code"
>> + #warning "Use foreach_not_ignored_block in migration code"
>>
>> void migration_make_urgent_request(void);
>> void migration_consume_urgent_request(void);
>> diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
>> index b098816221..e2aa57a701 100644
>> --- a/migration/postcopy-ram.c
>> +++ b/migration/postcopy-ram.c
>> @@ -374,7 +374,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis)
>> }
>>
>> /* We don't support postcopy with shared RAM yet */
>> - if (qemu_ram_foreach_migratable_block(test_ramblock_postcopiable, NULL)) {
>> + if (foreach_not_ignored_block(test_ramblock_postcopiable, NULL)) {
>> goto out;
>> }
>>
>> @@ -508,7 +508,7 @@ static int cleanup_range(RAMBlock *rb, void *opaque)
>> */
>> int postcopy_ram_incoming_init(MigrationIncomingState *mis)
>> {
>> - if (qemu_ram_foreach_migratable_block(init_range, NULL)) {
>> + if (foreach_not_ignored_block(init_range, NULL)) {
>> return -1;
>> }
>>
>> @@ -550,7 +550,7 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
>> return -1;
>> }
>>
>> - if (qemu_ram_foreach_migratable_block(cleanup_range, mis)) {
>> + if (foreach_not_ignored_block(cleanup_range, mis)) {
>> return -1;
>> }
>>
>> @@ -617,7 +617,7 @@ static int nhp_range(RAMBlock *rb, void *opaque)
>> */
>> int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
>> {
>> - if (qemu_ram_foreach_migratable_block(nhp_range, mis)) {
>> + if (foreach_not_ignored_block(nhp_range, mis)) {
>> return -1;
>> }
>>
>> @@ -628,7 +628,7 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
>>
>> /*
>> * Mark the given area of RAM as requiring notification to unwritten areas
>> - * Used as a callback on qemu_ram_foreach_migratable_block.
>> + * Used as a callback on foreach_not_ignored_block.
>> * host_addr: Base of area to mark
>> * offset: Offset in the whole ram arena
>> * length: Length of the section
>> @@ -1122,7 +1122,7 @@ int postcopy_ram_enable_notify(MigrationIncomingState *mis)
>> mis->have_fault_thread = true;
>>
>> /* Mark so that we get notified of accesses to unwritten areas */
>> - if (qemu_ram_foreach_migratable_block(ram_block_enable_notify, mis)) {
>> + if (foreach_not_ignored_block(ram_block_enable_notify, mis)) {
>> error_report("ram_block_enable_notify failed");
>> return -1;
>> }
>> diff --git a/migration/ram.c b/migration/ram.c
>> index 59191c1ed2..01315edd66 100644
>> --- a/migration/ram.c
>> +++ b/migration/ram.c
>> @@ -159,18 +159,44 @@ out:
>> return ret;
>> }
>>
>> +static bool ramblock_is_ignored(RAMBlock *block)
>> +{
>> + return !qemu_ram_is_migratable(block) ||
>> + (migrate_ignore_shared() && qemu_ram_is_shared(block));
>> +}
>> +
>> /* Should be holding either ram_list.mutex, or the RCU lock. */
>> +#define RAMBLOCK_FOREACH_NOT_IGNORED(block) \
>> + INTERNAL_RAMBLOCK_FOREACH(block) \
>> + if (ramblock_is_ignored(block)) {} else
>> +
>> #define RAMBLOCK_FOREACH_MIGRATABLE(block) \
>> INTERNAL_RAMBLOCK_FOREACH(block) \
>> if (!qemu_ram_is_migratable(block)) {} else
>>
>> #undef RAMBLOCK_FOREACH
>>
>> +int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque)
>> +{
>> + RAMBlock *block;
>> + int ret = 0;
>> +
>> + rcu_read_lock();
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> + ret = func(block, opaque);
>> + if (ret) {
>> + break;
>> + }
>> + }
>> + rcu_read_unlock();
>> + return ret;
>> +}
>> +
>> static void ramblock_recv_map_init(void)
>> {
>> RAMBlock *rb;
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
>> assert(!rb->receivedmap);
>> rb->receivedmap = bitmap_new(rb->max_length >> qemu_target_page_bits());
>> }
>> @@ -1545,7 +1571,7 @@ unsigned long migration_bitmap_find_dirty(RAMState *rs, RAMBlock *rb,
>> unsigned long *bitmap = rb->bmap;
>> unsigned long next;
>>
>> - if (!qemu_ram_is_migratable(rb)) {
>> + if (ramblock_is_ignored(rb)) {
>> return size;
>> }
>>
>> @@ -1594,7 +1620,7 @@ uint64_t ram_pagesize_summary(void)
>> RAMBlock *block;
>> uint64_t summary = 0;
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> summary |= block->page_size;
>> }
>>
>> @@ -1664,7 +1690,7 @@ static void migration_bitmap_sync(RAMState *rs)
>>
>> qemu_mutex_lock(&rs->bitmap_mutex);
>> rcu_read_lock();
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> migration_bitmap_sync_range(rs, block, 0, block->used_length);
>> }
>> ram_counters.remaining = ram_bytes_remaining();
>> @@ -2388,7 +2414,7 @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss,
>> size_t pagesize_bits =
>> qemu_ram_pagesize(pss->block) >> TARGET_PAGE_BITS;
>>
>> - if (!qemu_ram_is_migratable(pss->block)) {
>> + if (ramblock_is_ignored(pss->block)) {
>> error_report("block %s should not be migrated !", pss->block->idstr);
>> return 0;
>> }
>> @@ -2486,19 +2512,30 @@ void acct_update_position(QEMUFile *f, size_t size, bool zero)
>> }
>> }
>>
>> -uint64_t ram_bytes_total(void)
>> +static uint64_t ram_bytes_total_common(bool count_ignored)
>> {
>> RAMBlock *block;
>> uint64_t total = 0;
>>
>> rcu_read_lock();
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> - total += block->used_length;
>> + if (count_ignored) {
>> + RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + total += block->used_length;
>> + }
>> + } else {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> + total += block->used_length;
>> + }
>> }
>> rcu_read_unlock();
>> return total;
>> }
>>
>> +uint64_t ram_bytes_total(void)
>> +{
>> + return ram_bytes_total_common(false);
>> +}
>> +
>> static void xbzrle_load_setup(void)
>> {
>> XBZRLE.decoded_buf = g_malloc(TARGET_PAGE_SIZE);
>> @@ -2547,7 +2584,7 @@ static void ram_save_cleanup(void *opaque)
>> */
>> memory_global_dirty_log_stop();
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> g_free(block->bmap);
>> block->bmap = NULL;
>> g_free(block->unsentmap);
>> @@ -2610,7 +2647,7 @@ void ram_postcopy_migrated_memory_release(MigrationState *ms)
>> {
>> struct RAMBlock *block;
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> unsigned long *bitmap = block->bmap;
>> unsigned long range = block->used_length >> TARGET_PAGE_BITS;
>> unsigned long run_start = find_next_zero_bit(bitmap, range, 0);
>> @@ -2688,7 +2725,7 @@ static int postcopy_each_ram_send_discard(MigrationState *ms)
>> struct RAMBlock *block;
>> int ret;
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> PostcopyDiscardState *pds =
>> postcopy_discard_send_init(ms, block->idstr);
>>
>> @@ -2896,7 +2933,7 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms)
>> rs->last_sent_block = NULL;
>> rs->last_page = 0;
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> unsigned long pages = block->used_length >> TARGET_PAGE_BITS;
>> unsigned long *bitmap = block->bmap;
>> unsigned long *unsentmap = block->unsentmap;
>> @@ -3062,7 +3099,7 @@ static void ram_list_init_bitmaps(void)
>>
>> /* Skip setting bitmap if there is no RAM */
>> if (ram_bytes_total()) {
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> pages = block->max_length >> TARGET_PAGE_BITS;
>> block->bmap = bitmap_new(pages);
>> bitmap_set(block->bmap, 0, pages);
>> @@ -3117,7 +3154,7 @@ static void ram_state_resume_prepare(RAMState *rs, QEMUFile *out)
>> * about dirty page logging as well.
>> */
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> pages += bitmap_count_one(block->bmap,
>> block->used_length >> TARGET_PAGE_BITS);
>> }
>> @@ -3176,7 +3213,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
>>
>> rcu_read_lock();
>>
>> - qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
>> + qemu_put_be64(f, ram_bytes_total_common(true) | RAM_SAVE_FLAG_MEM_SIZE);
>>
>> RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> qemu_put_byte(f, strlen(block->idstr));
>> @@ -3185,6 +3222,10 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
>> if (migrate_postcopy_ram() && block->page_size != qemu_host_page_size) {
>> qemu_put_be64(f, block->page_size);
>> }
>> + if (migrate_ignore_shared()) {
>> + qemu_put_be64(f, block->mr->addr);
>> + qemu_put_byte(f, ramblock_is_ignored(block) ? 1 : 0);
>> + }
>> }
>>
>> rcu_read_unlock();
>> @@ -3443,7 +3484,7 @@ static inline RAMBlock *ram_block_from_stream(QEMUFile *f, int flags)
>> return NULL;
>> }
>>
>> - if (!qemu_ram_is_migratable(block)) {
>> + if (ramblock_is_ignored(block)) {
>> error_report("block %s should not be migrated !", id);
>> return NULL;
>> }
>> @@ -3698,7 +3739,7 @@ int colo_init_ram_cache(void)
>> RAMBlock *block;
>>
>> rcu_read_lock();
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> block->colo_cache = qemu_anon_ram_alloc(block->used_length,
>> NULL,
>> false);
>> @@ -3719,7 +3760,7 @@ int colo_init_ram_cache(void)
>> if (ram_bytes_total()) {
>> RAMBlock *block;
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> unsigned long pages = block->max_length >> TARGET_PAGE_BITS;
>>
>> block->bmap = bitmap_new(pages);
>> @@ -3734,7 +3775,7 @@ int colo_init_ram_cache(void)
>>
>> out_locked:
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> if (block->colo_cache) {
>> qemu_anon_ram_free(block->colo_cache, block->used_length);
>> block->colo_cache = NULL;
>> @@ -3751,14 +3792,14 @@ void colo_release_ram_cache(void)
>> RAMBlock *block;
>>
>> memory_global_dirty_log_stop();
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> g_free(block->bmap);
>> block->bmap = NULL;
>> }
>>
>> rcu_read_lock();
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> if (block->colo_cache) {
>> qemu_anon_ram_free(block->colo_cache, block->used_length);
>> block->colo_cache = NULL;
>> @@ -3794,7 +3835,7 @@ static int ram_load_cleanup(void *opaque)
>> {
>> RAMBlock *rb;
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
>> if (ramblock_is_pmem(rb)) {
>> pmem_persist(rb->host, rb->used_length);
>> }
>> @@ -3803,7 +3844,7 @@ static int ram_load_cleanup(void *opaque)
>> xbzrle_load_cleanup();
>> compress_threads_load_cleanup();
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
>> g_free(rb->receivedmap);
>> rb->receivedmap = NULL;
>> }
>> @@ -4003,7 +4044,7 @@ static void colo_flush_ram_cache(void)
>>
>> memory_global_dirty_log_sync();
>> rcu_read_lock();
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> migration_bitmap_sync_range(ram_state, block, 0, block->used_length);
>> }
>> rcu_read_unlock();
>> @@ -4146,6 +4187,23 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> ret = -EINVAL;
>> }
>> }
>> + if (migrate_ignore_shared()) {
>> + hwaddr addr = qemu_get_be64(f);
>> + bool ignored = qemu_get_byte(f);
>> + if (ignored != ramblock_is_ignored(block)) {
>> + error_report("RAM block %s should %s be migrated",
>> + id, ignored ? "" : "not");
>> + ret = -EINVAL;
>> + }
>> + if (ramblock_is_ignored(block) &&
>> + block->mr->addr != addr) {
>> + error_report("Mismatched GPAs for block %s "
>> + "%" PRId64 "!= %" PRId64,
>> + id, (uint64_t)addr,
>> + (uint64_t)block->mr->addr);
>> + ret = -EINVAL;
>> + }
>> + }
>> ram_control_load_hook(f, RAM_CONTROL_BLOCK_REG,
>> block->idstr);
>> } else {
>> @@ -4216,7 +4274,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> static bool ram_has_postcopy(void *opaque)
>> {
>> RAMBlock *rb;
>> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
>> if (ramblock_is_pmem(rb)) {
>> info_report("Block: %s, host: %p is a nvdimm memory, postcopy"
>> "is not supported now!", rb->idstr, rb->host);
>> @@ -4236,7 +4294,7 @@ static int ram_dirty_bitmap_sync_all(MigrationState *s, RAMState *rs)
>>
>> trace_ram_dirty_bitmap_sync_start();
>>
>> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
>> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>> qemu_savevm_send_recv_bitmap(file, block->idstr);
>> trace_ram_dirty_bitmap_request(block->idstr);
>> ramblock_count++;
>> diff --git a/migration/rdma.c b/migration/rdma.c
>> index 7eb38ee764..3cb579cc99 100644
>> --- a/migration/rdma.c
>> +++ b/migration/rdma.c
>> @@ -644,7 +644,7 @@ static int qemu_rdma_init_ram_blocks(RDMAContext *rdma)
>>
>> assert(rdma->blockmap == NULL);
>> memset(local, 0, sizeof *local);
>> - qemu_ram_foreach_migratable_block(qemu_rdma_init_one_block, rdma);
>> + foreach_not_ignored_block(qemu_rdma_init_one_block, rdma);
>> trace_qemu_rdma_init_ram_blocks(local->nb_blocks);
>> rdma->dest_blocks = g_new0(RDMADestBlock,
>> rdma->local_ram_blocks.nb_blocks);
>> diff --git a/qapi/migration.json b/qapi/migration.json
>> index 7a795ecc16..7105570cd3 100644
>> --- a/qapi/migration.json
>> +++ b/qapi/migration.json
>> @@ -409,13 +409,16 @@
>> # devices (and thus take locks) immediately at the end of migration.
>> # (since 3.0)
>> #
>> +# @x-ignore-shared: If enabled, QEMU will not migrate shared memory (since 4.0)
>> +#
>> # Since: 1.2
>> ##
>> { 'enum': 'MigrationCapability',
>> 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
>> 'compress', 'events', 'postcopy-ram', 'x-colo', 'release-ram',
>> 'block', 'return-path', 'pause-before-switchover', 'x-multifd',
>> - 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate' ] }
>> + 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate',
>> + 'x-ignore-shared' ] }
>>
>> ##
>> # @MigrationCapabilityStatus:
>> diff --git a/stubs/ram-block.c b/stubs/ram-block.c
>> index cfa5d8678f..73c0a3ee08 100644
>> --- a/stubs/ram-block.c
>> +++ b/stubs/ram-block.c
>> @@ -2,6 +2,21 @@
>> #include "exec/ramlist.h"
>> #include "exec/cpu-common.h"
>>
>> +void *qemu_ram_get_host_addr(RAMBlock *rb)
>> +{
>> + return 0;
>> +}
>> +
>> +ram_addr_t qemu_ram_get_offset(RAMBlock *rb)
>> +{
>> + return 0;
>> +}
>> +
>> +ram_addr_t qemu_ram_get_used_length(RAMBlock *rb)
>> +{
>> + return 0;
>> +}
>> +
>> void ram_block_notifier_add(RAMBlockNotifier *n)
>> {
>> }
>> --
>> 2.20.1
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
Regards,
Yury
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 2/4] migration: Introduce ignore-shared capability
2019-02-11 13:36 ` Yury Kotov
@ 2019-02-11 15:55 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2019-02-11 15:55 UTC (permalink / raw)
To: Yury Kotov
Cc: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela, Eric Blake,
Markus Armbruster, Thomas Huth, Laurent Vivier, wrfsh
* Yury Kotov (yury-kotov@yandex-team.ru) wrote:
> 11.02.2019, 15:45, "Dr. David Alan Gilbert" <dgilbert@redhat.com>:
> > * Yury Kotov (yury-kotov@yandex-team.ru) wrote:
> >> We want to use local migration to update QEMU for running guests.
> >> In this case we don't need to migrate shared (file backed) RAM.
> >> So, add a capability to ignore such blocks during live migration.
> >>
> >> Also, move qemu_ram_foreach_migratable_block (and rename) to the
> >> migration code, because it requires access to the migration capabilities.
> >>
> >> Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
> >
> > You could split this patch into the one that introduces the capability
> > and then the one that wires it up. We could also remove the x- at some
> > point.
>
> I.e. the patch that just adds the capability to json (and migrate_use_*), but
> nothing more, and the second one which actually realize the capability?
Right.
Dave
> Like this:
> 2a4c42f18c migration: add postcopy blocktime ctx into MigrationIncomingState
> f22f928ec9 migration: introduce postcopy-blocktime capability
> ?
>
> >
> > Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> >
> >> ---
> >> exec.c | 19 -------
> >> include/exec/cpu-common.h | 1 -
> >> migration/migration.c | 9 ++++
> >> migration/migration.h | 5 +-
> >> migration/postcopy-ram.c | 12 ++---
> >> migration/ram.c | 110 +++++++++++++++++++++++++++++---------
> >> migration/rdma.c | 2 +-
> >> qapi/migration.json | 5 +-
> >> stubs/ram-block.c | 15 ++++++
> >> 9 files changed, 123 insertions(+), 55 deletions(-)
> >>
> >> diff --git a/exec.c b/exec.c
> >> index a61d501568..91bfe5fb62 100644
> >> --- a/exec.c
> >> +++ b/exec.c
> >> @@ -3984,25 +3984,6 @@ int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque)
> >> return ret;
> >> }
> >>
> >> -int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque)
> >> -{
> >> - RAMBlock *block;
> >> - int ret = 0;
> >> -
> >> - rcu_read_lock();
> >> - RAMBLOCK_FOREACH(block) {
> >> - if (!qemu_ram_is_migratable(block)) {
> >> - continue;
> >> - }
> >> - ret = func(block, opaque);
> >> - if (ret) {
> >> - break;
> >> - }
> >> - }
> >> - rcu_read_unlock();
> >> - return ret;
> >> -}
> >> -
> >> /*
> >> * Unmap pages of memory from start to start+length such that
> >> * they a) read as 0, b) Trigger whatever fault mechanism
> >> diff --git a/include/exec/cpu-common.h b/include/exec/cpu-common.h
> >> index bdae5446d7..403463d7bb 100644
> >> --- a/include/exec/cpu-common.h
> >> +++ b/include/exec/cpu-common.h
> >> @@ -122,7 +122,6 @@ extern struct MemoryRegion io_mem_notdirty;
> >> typedef int (RAMBlockIterFunc)(RAMBlock *rb, void *opaque);
> >>
> >> int qemu_ram_foreach_block(RAMBlockIterFunc func, void *opaque);
> >> -int qemu_ram_foreach_migratable_block(RAMBlockIterFunc func, void *opaque);
> >> int ram_block_discard_range(RAMBlock *rb, uint64_t start, size_t length);
> >>
> >> #endif
> >> diff --git a/migration/migration.c b/migration/migration.c
> >> index 37e06b76dc..c40776a40c 100644
> >> --- a/migration/migration.c
> >> +++ b/migration/migration.c
> >> @@ -1983,6 +1983,15 @@ bool migrate_dirty_bitmaps(void)
> >> return s->enabled_capabilities[MIGRATION_CAPABILITY_DIRTY_BITMAPS];
> >> }
> >>
> >> +bool migrate_ignore_shared(void)
> >> +{
> >> + MigrationState *s;
> >> +
> >> + s = migrate_get_current();
> >> +
> >> + return s->enabled_capabilities[MIGRATION_CAPABILITY_X_IGNORE_SHARED];
> >> +}
> >> +
> >> bool migrate_use_events(void)
> >> {
> >> MigrationState *s;
> >> diff --git a/migration/migration.h b/migration/migration.h
> >> index dcd05d9f87..2c88f8a555 100644
> >> --- a/migration/migration.h
> >> +++ b/migration/migration.h
> >> @@ -261,6 +261,7 @@ bool migrate_release_ram(void);
> >> bool migrate_postcopy_ram(void);
> >> bool migrate_zero_blocks(void);
> >> bool migrate_dirty_bitmaps(void);
> >> +bool migrate_ignore_shared(void);
> >>
> >> bool migrate_auto_converge(void);
> >> bool migrate_use_multifd(void);
> >> @@ -301,8 +302,10 @@ void migrate_send_rp_resume_ack(MigrationIncomingState *mis, uint32_t value);
> >> void dirty_bitmap_mig_before_vm_start(void);
> >> void init_dirty_bitmap_incoming_migration(void);
> >>
> >> +int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque);
> >> +
> >> #define qemu_ram_foreach_block \
> >> - #warning "Use qemu_ram_foreach_block_migratable in migration code"
> >> + #warning "Use foreach_not_ignored_block in migration code"
> >>
> >> void migration_make_urgent_request(void);
> >> void migration_consume_urgent_request(void);
> >> diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
> >> index b098816221..e2aa57a701 100644
> >> --- a/migration/postcopy-ram.c
> >> +++ b/migration/postcopy-ram.c
> >> @@ -374,7 +374,7 @@ bool postcopy_ram_supported_by_host(MigrationIncomingState *mis)
> >> }
> >>
> >> /* We don't support postcopy with shared RAM yet */
> >> - if (qemu_ram_foreach_migratable_block(test_ramblock_postcopiable, NULL)) {
> >> + if (foreach_not_ignored_block(test_ramblock_postcopiable, NULL)) {
> >> goto out;
> >> }
> >>
> >> @@ -508,7 +508,7 @@ static int cleanup_range(RAMBlock *rb, void *opaque)
> >> */
> >> int postcopy_ram_incoming_init(MigrationIncomingState *mis)
> >> {
> >> - if (qemu_ram_foreach_migratable_block(init_range, NULL)) {
> >> + if (foreach_not_ignored_block(init_range, NULL)) {
> >> return -1;
> >> }
> >>
> >> @@ -550,7 +550,7 @@ int postcopy_ram_incoming_cleanup(MigrationIncomingState *mis)
> >> return -1;
> >> }
> >>
> >> - if (qemu_ram_foreach_migratable_block(cleanup_range, mis)) {
> >> + if (foreach_not_ignored_block(cleanup_range, mis)) {
> >> return -1;
> >> }
> >>
> >> @@ -617,7 +617,7 @@ static int nhp_range(RAMBlock *rb, void *opaque)
> >> */
> >> int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
> >> {
> >> - if (qemu_ram_foreach_migratable_block(nhp_range, mis)) {
> >> + if (foreach_not_ignored_block(nhp_range, mis)) {
> >> return -1;
> >> }
> >>
> >> @@ -628,7 +628,7 @@ int postcopy_ram_prepare_discard(MigrationIncomingState *mis)
> >>
> >> /*
> >> * Mark the given area of RAM as requiring notification to unwritten areas
> >> - * Used as a callback on qemu_ram_foreach_migratable_block.
> >> + * Used as a callback on foreach_not_ignored_block.
> >> * host_addr: Base of area to mark
> >> * offset: Offset in the whole ram arena
> >> * length: Length of the section
> >> @@ -1122,7 +1122,7 @@ int postcopy_ram_enable_notify(MigrationIncomingState *mis)
> >> mis->have_fault_thread = true;
> >>
> >> /* Mark so that we get notified of accesses to unwritten areas */
> >> - if (qemu_ram_foreach_migratable_block(ram_block_enable_notify, mis)) {
> >> + if (foreach_not_ignored_block(ram_block_enable_notify, mis)) {
> >> error_report("ram_block_enable_notify failed");
> >> return -1;
> >> }
> >> diff --git a/migration/ram.c b/migration/ram.c
> >> index 59191c1ed2..01315edd66 100644
> >> --- a/migration/ram.c
> >> +++ b/migration/ram.c
> >> @@ -159,18 +159,44 @@ out:
> >> return ret;
> >> }
> >>
> >> +static bool ramblock_is_ignored(RAMBlock *block)
> >> +{
> >> + return !qemu_ram_is_migratable(block) ||
> >> + (migrate_ignore_shared() && qemu_ram_is_shared(block));
> >> +}
> >> +
> >> /* Should be holding either ram_list.mutex, or the RCU lock. */
> >> +#define RAMBLOCK_FOREACH_NOT_IGNORED(block) \
> >> + INTERNAL_RAMBLOCK_FOREACH(block) \
> >> + if (ramblock_is_ignored(block)) {} else
> >> +
> >> #define RAMBLOCK_FOREACH_MIGRATABLE(block) \
> >> INTERNAL_RAMBLOCK_FOREACH(block) \
> >> if (!qemu_ram_is_migratable(block)) {} else
> >>
> >> #undef RAMBLOCK_FOREACH
> >>
> >> +int foreach_not_ignored_block(RAMBlockIterFunc func, void *opaque)
> >> +{
> >> + RAMBlock *block;
> >> + int ret = 0;
> >> +
> >> + rcu_read_lock();
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> + ret = func(block, opaque);
> >> + if (ret) {
> >> + break;
> >> + }
> >> + }
> >> + rcu_read_unlock();
> >> + return ret;
> >> +}
> >> +
> >> static void ramblock_recv_map_init(void)
> >> {
> >> RAMBlock *rb;
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
> >> assert(!rb->receivedmap);
> >> rb->receivedmap = bitmap_new(rb->max_length >> qemu_target_page_bits());
> >> }
> >> @@ -1545,7 +1571,7 @@ unsigned long migration_bitmap_find_dirty(RAMState *rs, RAMBlock *rb,
> >> unsigned long *bitmap = rb->bmap;
> >> unsigned long next;
> >>
> >> - if (!qemu_ram_is_migratable(rb)) {
> >> + if (ramblock_is_ignored(rb)) {
> >> return size;
> >> }
> >>
> >> @@ -1594,7 +1620,7 @@ uint64_t ram_pagesize_summary(void)
> >> RAMBlock *block;
> >> uint64_t summary = 0;
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> summary |= block->page_size;
> >> }
> >>
> >> @@ -1664,7 +1690,7 @@ static void migration_bitmap_sync(RAMState *rs)
> >>
> >> qemu_mutex_lock(&rs->bitmap_mutex);
> >> rcu_read_lock();
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> migration_bitmap_sync_range(rs, block, 0, block->used_length);
> >> }
> >> ram_counters.remaining = ram_bytes_remaining();
> >> @@ -2388,7 +2414,7 @@ static int ram_save_host_page(RAMState *rs, PageSearchStatus *pss,
> >> size_t pagesize_bits =
> >> qemu_ram_pagesize(pss->block) >> TARGET_PAGE_BITS;
> >>
> >> - if (!qemu_ram_is_migratable(pss->block)) {
> >> + if (ramblock_is_ignored(pss->block)) {
> >> error_report("block %s should not be migrated !", pss->block->idstr);
> >> return 0;
> >> }
> >> @@ -2486,19 +2512,30 @@ void acct_update_position(QEMUFile *f, size_t size, bool zero)
> >> }
> >> }
> >>
> >> -uint64_t ram_bytes_total(void)
> >> +static uint64_t ram_bytes_total_common(bool count_ignored)
> >> {
> >> RAMBlock *block;
> >> uint64_t total = 0;
> >>
> >> rcu_read_lock();
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> - total += block->used_length;
> >> + if (count_ignored) {
> >> + RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + total += block->used_length;
> >> + }
> >> + } else {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> + total += block->used_length;
> >> + }
> >> }
> >> rcu_read_unlock();
> >> return total;
> >> }
> >>
> >> +uint64_t ram_bytes_total(void)
> >> +{
> >> + return ram_bytes_total_common(false);
> >> +}
> >> +
> >> static void xbzrle_load_setup(void)
> >> {
> >> XBZRLE.decoded_buf = g_malloc(TARGET_PAGE_SIZE);
> >> @@ -2547,7 +2584,7 @@ static void ram_save_cleanup(void *opaque)
> >> */
> >> memory_global_dirty_log_stop();
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> g_free(block->bmap);
> >> block->bmap = NULL;
> >> g_free(block->unsentmap);
> >> @@ -2610,7 +2647,7 @@ void ram_postcopy_migrated_memory_release(MigrationState *ms)
> >> {
> >> struct RAMBlock *block;
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> unsigned long *bitmap = block->bmap;
> >> unsigned long range = block->used_length >> TARGET_PAGE_BITS;
> >> unsigned long run_start = find_next_zero_bit(bitmap, range, 0);
> >> @@ -2688,7 +2725,7 @@ static int postcopy_each_ram_send_discard(MigrationState *ms)
> >> struct RAMBlock *block;
> >> int ret;
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> PostcopyDiscardState *pds =
> >> postcopy_discard_send_init(ms, block->idstr);
> >>
> >> @@ -2896,7 +2933,7 @@ int ram_postcopy_send_discard_bitmap(MigrationState *ms)
> >> rs->last_sent_block = NULL;
> >> rs->last_page = 0;
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> unsigned long pages = block->used_length >> TARGET_PAGE_BITS;
> >> unsigned long *bitmap = block->bmap;
> >> unsigned long *unsentmap = block->unsentmap;
> >> @@ -3062,7 +3099,7 @@ static void ram_list_init_bitmaps(void)
> >>
> >> /* Skip setting bitmap if there is no RAM */
> >> if (ram_bytes_total()) {
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> pages = block->max_length >> TARGET_PAGE_BITS;
> >> block->bmap = bitmap_new(pages);
> >> bitmap_set(block->bmap, 0, pages);
> >> @@ -3117,7 +3154,7 @@ static void ram_state_resume_prepare(RAMState *rs, QEMUFile *out)
> >> * about dirty page logging as well.
> >> */
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> pages += bitmap_count_one(block->bmap,
> >> block->used_length >> TARGET_PAGE_BITS);
> >> }
> >> @@ -3176,7 +3213,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
> >>
> >> rcu_read_lock();
> >>
> >> - qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
> >> + qemu_put_be64(f, ram_bytes_total_common(true) | RAM_SAVE_FLAG_MEM_SIZE);
> >>
> >> RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> qemu_put_byte(f, strlen(block->idstr));
> >> @@ -3185,6 +3222,10 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
> >> if (migrate_postcopy_ram() && block->page_size != qemu_host_page_size) {
> >> qemu_put_be64(f, block->page_size);
> >> }
> >> + if (migrate_ignore_shared()) {
> >> + qemu_put_be64(f, block->mr->addr);
> >> + qemu_put_byte(f, ramblock_is_ignored(block) ? 1 : 0);
> >> + }
> >> }
> >>
> >> rcu_read_unlock();
> >> @@ -3443,7 +3484,7 @@ static inline RAMBlock *ram_block_from_stream(QEMUFile *f, int flags)
> >> return NULL;
> >> }
> >>
> >> - if (!qemu_ram_is_migratable(block)) {
> >> + if (ramblock_is_ignored(block)) {
> >> error_report("block %s should not be migrated !", id);
> >> return NULL;
> >> }
> >> @@ -3698,7 +3739,7 @@ int colo_init_ram_cache(void)
> >> RAMBlock *block;
> >>
> >> rcu_read_lock();
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> block->colo_cache = qemu_anon_ram_alloc(block->used_length,
> >> NULL,
> >> false);
> >> @@ -3719,7 +3760,7 @@ int colo_init_ram_cache(void)
> >> if (ram_bytes_total()) {
> >> RAMBlock *block;
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> unsigned long pages = block->max_length >> TARGET_PAGE_BITS;
> >>
> >> block->bmap = bitmap_new(pages);
> >> @@ -3734,7 +3775,7 @@ int colo_init_ram_cache(void)
> >>
> >> out_locked:
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> if (block->colo_cache) {
> >> qemu_anon_ram_free(block->colo_cache, block->used_length);
> >> block->colo_cache = NULL;
> >> @@ -3751,14 +3792,14 @@ void colo_release_ram_cache(void)
> >> RAMBlock *block;
> >>
> >> memory_global_dirty_log_stop();
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> g_free(block->bmap);
> >> block->bmap = NULL;
> >> }
> >>
> >> rcu_read_lock();
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> if (block->colo_cache) {
> >> qemu_anon_ram_free(block->colo_cache, block->used_length);
> >> block->colo_cache = NULL;
> >> @@ -3794,7 +3835,7 @@ static int ram_load_cleanup(void *opaque)
> >> {
> >> RAMBlock *rb;
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
> >> if (ramblock_is_pmem(rb)) {
> >> pmem_persist(rb->host, rb->used_length);
> >> }
> >> @@ -3803,7 +3844,7 @@ static int ram_load_cleanup(void *opaque)
> >> xbzrle_load_cleanup();
> >> compress_threads_load_cleanup();
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
> >> g_free(rb->receivedmap);
> >> rb->receivedmap = NULL;
> >> }
> >> @@ -4003,7 +4044,7 @@ static void colo_flush_ram_cache(void)
> >>
> >> memory_global_dirty_log_sync();
> >> rcu_read_lock();
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> migration_bitmap_sync_range(ram_state, block, 0, block->used_length);
> >> }
> >> rcu_read_unlock();
> >> @@ -4146,6 +4187,23 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> >> ret = -EINVAL;
> >> }
> >> }
> >> + if (migrate_ignore_shared()) {
> >> + hwaddr addr = qemu_get_be64(f);
> >> + bool ignored = qemu_get_byte(f);
> >> + if (ignored != ramblock_is_ignored(block)) {
> >> + error_report("RAM block %s should %s be migrated",
> >> + id, ignored ? "" : "not");
> >> + ret = -EINVAL;
> >> + }
> >> + if (ramblock_is_ignored(block) &&
> >> + block->mr->addr != addr) {
> >> + error_report("Mismatched GPAs for block %s "
> >> + "%" PRId64 "!= %" PRId64,
> >> + id, (uint64_t)addr,
> >> + (uint64_t)block->mr->addr);
> >> + ret = -EINVAL;
> >> + }
> >> + }
> >> ram_control_load_hook(f, RAM_CONTROL_BLOCK_REG,
> >> block->idstr);
> >> } else {
> >> @@ -4216,7 +4274,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> >> static bool ram_has_postcopy(void *opaque)
> >> {
> >> RAMBlock *rb;
> >> - RAMBLOCK_FOREACH_MIGRATABLE(rb) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(rb) {
> >> if (ramblock_is_pmem(rb)) {
> >> info_report("Block: %s, host: %p is a nvdimm memory, postcopy"
> >> "is not supported now!", rb->idstr, rb->host);
> >> @@ -4236,7 +4294,7 @@ static int ram_dirty_bitmap_sync_all(MigrationState *s, RAMState *rs)
> >>
> >> trace_ram_dirty_bitmap_sync_start();
> >>
> >> - RAMBLOCK_FOREACH_MIGRATABLE(block) {
> >> + RAMBLOCK_FOREACH_NOT_IGNORED(block) {
> >> qemu_savevm_send_recv_bitmap(file, block->idstr);
> >> trace_ram_dirty_bitmap_request(block->idstr);
> >> ramblock_count++;
> >> diff --git a/migration/rdma.c b/migration/rdma.c
> >> index 7eb38ee764..3cb579cc99 100644
> >> --- a/migration/rdma.c
> >> +++ b/migration/rdma.c
> >> @@ -644,7 +644,7 @@ static int qemu_rdma_init_ram_blocks(RDMAContext *rdma)
> >>
> >> assert(rdma->blockmap == NULL);
> >> memset(local, 0, sizeof *local);
> >> - qemu_ram_foreach_migratable_block(qemu_rdma_init_one_block, rdma);
> >> + foreach_not_ignored_block(qemu_rdma_init_one_block, rdma);
> >> trace_qemu_rdma_init_ram_blocks(local->nb_blocks);
> >> rdma->dest_blocks = g_new0(RDMADestBlock,
> >> rdma->local_ram_blocks.nb_blocks);
> >> diff --git a/qapi/migration.json b/qapi/migration.json
> >> index 7a795ecc16..7105570cd3 100644
> >> --- a/qapi/migration.json
> >> +++ b/qapi/migration.json
> >> @@ -409,13 +409,16 @@
> >> # devices (and thus take locks) immediately at the end of migration.
> >> # (since 3.0)
> >> #
> >> +# @x-ignore-shared: If enabled, QEMU will not migrate shared memory (since 4.0)
> >> +#
> >> # Since: 1.2
> >> ##
> >> { 'enum': 'MigrationCapability',
> >> 'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
> >> 'compress', 'events', 'postcopy-ram', 'x-colo', 'release-ram',
> >> 'block', 'return-path', 'pause-before-switchover', 'x-multifd',
> >> - 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate' ] }
> >> + 'dirty-bitmaps', 'postcopy-blocktime', 'late-block-activate',
> >> + 'x-ignore-shared' ] }
> >>
> >> ##
> >> # @MigrationCapabilityStatus:
> >> diff --git a/stubs/ram-block.c b/stubs/ram-block.c
> >> index cfa5d8678f..73c0a3ee08 100644
> >> --- a/stubs/ram-block.c
> >> +++ b/stubs/ram-block.c
> >> @@ -2,6 +2,21 @@
> >> #include "exec/ramlist.h"
> >> #include "exec/cpu-common.h"
> >>
> >> +void *qemu_ram_get_host_addr(RAMBlock *rb)
> >> +{
> >> + return 0;
> >> +}
> >> +
> >> +ram_addr_t qemu_ram_get_offset(RAMBlock *rb)
> >> +{
> >> + return 0;
> >> +}
> >> +
> >> +ram_addr_t qemu_ram_get_used_length(RAMBlock *rb)
> >> +{
> >> + return 0;
> >> +}
> >> +
> >> void ram_block_notifier_add(RAMBlockNotifier *n)
> >> {
> >> }
> >> --
> >> 2.20.1
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> Regards,
> Yury
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability
2019-02-04 13:09 [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability Yury Kotov
` (4 preceding siblings ...)
2019-02-11 11:24 ` [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability Yury Kotov
@ 2019-02-11 16:03 ` Dr. David Alan Gilbert
2019-02-11 16:13 ` Daniel P. Berrangé
5 siblings, 1 reply; 16+ messages in thread
From: Dr. David Alan Gilbert @ 2019-02-11 16:03 UTC (permalink / raw)
To: Yury Kotov, qemu-devel, jiangshanlai, peter.maydell
Cc: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela, Eric Blake,
Markus Armbruster, Thomas Huth, Laurent Vivier, wrfsh
* Yury Kotov (yury-kotov@yandex-team.ru) wrote:
> Hi,
>
> The series adds a migration capability, which allows to skip shared RAM blocks
> during the migration. It's useful for fast local migration. E.g. to update QEMU
> for the running guests.
>
> Usage example:
> 1. Start source VM:
> qemu-system-x86 \
> -m 4G \
> -object memory-backend-file,id=mem0,size=4G,share=on,mem-path=/dev/shm/mem0 \
> -numa node,memdev=mem0 \
> -qmp unix:/tmp/qemu-qmp-1.sock,server,nowait \
>
> 2. Start target VM:
> qemu-system-x86 \
> -m 4G \
> -object memory-backend-file,id=mem0,size=4G,share=on,mem-path=/dev/shm/mem0 \
> -numa node,memdev=mem0 \
> -qmp unix:/tmp/qemu-qmp-2.sock,server,nowait \
> -incoming defer
>
> 3. Enable ignore-external capability on both VMs:
> { "execute": "migrate-set-capabilities" , "arguments":
> { "capabilities": [ { "capability": "x-ignore-external", "state": true } ] } }
>
> 4. Start migration.
>
> Another use case I keep in mind is to migrate to file. Usage is very similar.
Hi,
I've cc'd in Eric, Lai and Peter, all who were asking for something
similar last year; can you all confirm this patch does what you need or
can work with what you needed to do?
Dave
> V1 to V2:
> * Keep migration stream compatibility
> * Reuse the existing code to ignore unwanted RAMBlocks
> * Add capability validation feature
> * ignore-external -> ignore-shared
>
> Regards,
> Yury
>
> Yury Kotov (5):
> exec: Change RAMBlockIterFunc definition
> migration: Move qemu_ram_foreach_migratable_block to migration code
> migration: Introduce ignore-shared capability
> tests/migration-test: Add a test for ignore-shared capability
> migration: Add capabilities validation
>
> exec.c | 38 ++++++-------
> include/exec/cpu-common.h | 7 +--
> migration/migration.c | 9 ++++
> migration/migration.h | 11 +++-
> migration/postcopy-ram.c | 48 +++++++++--------
> migration/ram.c | 86 ++++++++++++++++++++++++++----
> migration/rdma.c | 9 ++--
> migration/savevm.c | 101 +++++++++++++++++++++++++++++++++++
> qapi/migration.json | 5 +-
> stubs/ram-block.c | 15 ++++++
> tests/migration-test.c | 109 +++++++++++++++++++++++++++++++-------
> util/vfio-helpers.c | 6 +--
> 12 files changed, 361 insertions(+), 83 deletions(-)
>
> --
> 2.20.1
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability
2019-02-11 16:03 ` Dr. David Alan Gilbert
@ 2019-02-11 16:13 ` Daniel P. Berrangé
0 siblings, 0 replies; 16+ messages in thread
From: Daniel P. Berrangé @ 2019-02-11 16:13 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: Yury Kotov, qemu-devel, jiangshanlai, peter.maydell,
Laurent Vivier, Thomas Huth, Eduardo Habkost, Peter Crosthwaite,
Juan Quintela, qemu-devel, Markus Armbruster, Paolo Bonzini,
Igor Mammedov, wrfsh, Richard Henderson
On Mon, Feb 11, 2019 at 04:03:57PM +0000, Dr. David Alan Gilbert wrote:
> * Yury Kotov (yury-kotov@yandex-team.ru) wrote:
> > Hi,
> >
> > The series adds a migration capability, which allows to skip shared RAM blocks
> > during the migration. It's useful for fast local migration. E.g. to update QEMU
> > for the running guests.
> >
> > Usage example:
> > 1. Start source VM:
> > qemu-system-x86 \
> > -m 4G \
> > -object memory-backend-file,id=mem0,size=4G,share=on,mem-path=/dev/shm/mem0 \
> > -numa node,memdev=mem0 \
> > -qmp unix:/tmp/qemu-qmp-1.sock,server,nowait \
> >
> > 2. Start target VM:
> > qemu-system-x86 \
> > -m 4G \
> > -object memory-backend-file,id=mem0,size=4G,share=on,mem-path=/dev/shm/mem0 \
> > -numa node,memdev=mem0 \
> > -qmp unix:/tmp/qemu-qmp-2.sock,server,nowait \
> > -incoming defer
> >
> > 3. Enable ignore-external capability on both VMs:
> > { "execute": "migrate-set-capabilities" , "arguments":
> > { "capabilities": [ { "capability": "x-ignore-external", "state": true } ] } }
> >
> > 4. Start migration.
> >
> > Another use case I keep in mind is to migrate to file. Usage is very similar.
>
> Hi,
> I've cc'd in Eric, Lai and Peter, all who were asking for something
> similar last year; can you all confirm this patch does what you need or
> can work with what you needed to do?
x-ignore-external is a global setting so affects all memory-backend-file
instances. The obvious question is where there is any reasonable/conceivable
scenario in which QEMU would have multiple "-object memory-backend-file"
intsances and it be neccessary/desirable to migrate some, but skip migrate
of others ?
Could there ever be a device backend which is using a memory region
to communicate with an external process, where you would need to have
a new instance for the migration target QEMU, and explicitly not reuse
the source QEMU's memory-backend-file storage ?
Regards,
Daniel
--
|: https://berrange.com -o- https://www.flickr.com/photos/dberrange :|
|: https://libvirt.org -o- https://fstop138.berrange.com :|
|: https://entangle-photo.org -o- https://www.instagram.com/dberrange :|
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 3/4] tests/migration-test: Add a test for ignore-shared capability
2019-02-11 13:17 ` Dr. David Alan Gilbert
@ 2019-02-12 12:49 ` Yury Kotov
0 siblings, 0 replies; 16+ messages in thread
From: Yury Kotov @ 2019-02-12 12:49 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela, Eric Blake,
Markus Armbruster, Thomas Huth, Laurent Vivier, wrfsh
11.02.2019, 16:17, "Dr. David Alan Gilbert" <dgilbert@redhat.com>:
> * Yury Kotov (yury-kotov@yandex-team.ru) wrote:
>> Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
>> ---
>> tests/migration-test.c | 109 +++++++++++++++++++++++++++++++++--------
>> 1 file changed, 89 insertions(+), 20 deletions(-)
>>
>> diff --git a/tests/migration-test.c b/tests/migration-test.c
>> index 8352612364..485f42b2d2 100644
>> --- a/tests/migration-test.c
>> +++ b/tests/migration-test.c
>> @@ -332,6 +332,13 @@ static void cleanup(const char *filename)
>> g_free(path);
>> }
>>
>> +static char *get_shmem_opts(const char *mem_size, const char *shmem_path)
>> +{
>> + return g_strdup_printf("-object memory-backend-file,id=mem0,size=%s"
>> + ",mem-path=%s,share=on -numa node,memdev=mem0",
>> + mem_size, shmem_path);
>> +}
>> +
>> static void migrate_check_parameter(QTestState *who, const char *parameter,
>> long long value)
>> {
>> @@ -430,73 +437,91 @@ static void migrate_postcopy_start(QTestState *from, QTestState *to)
>> }
>>
>> static int test_migrate_start(QTestState **from, QTestState **to,
>> - const char *uri, bool hide_stderr)
>> + const char *uri, bool hide_stderr,
>> + bool use_shmem)
>> {
>> gchar *cmd_src, *cmd_dst;
>> char *bootpath = g_strdup_printf("%s/bootsect", tmpfs);
>> + char *extra_opts = NULL;
>> + char *shmem_path = NULL;
>> const char *arch = qtest_get_arch();
>> const char *accel = "kvm:tcg";
>>
>> got_stop = false;
>>
>> + if (use_shmem) {
>> + shmem_path = g_strdup_printf("/dev/shm/qemu-%d", getpid());
>> + }
>
> I think /dev/shm is non-portable; so I think you'll need to have a way
> to skip the test on OSs that don't have it.
>
Ok, will fix in v3.
>> if (strcmp(arch, "i386") == 0 || strcmp(arch, "x86_64") == 0) {
>> init_bootfile(bootpath, x86_bootsect);
>> + extra_opts = use_shmem ? get_shmem_opts("150M", shmem_path) : NULL;
>> cmd_src = g_strdup_printf("-machine accel=%s -m 150M"
>> " -name source,debug-threads=on"
>> " -serial file:%s/src_serial"
>> - " -drive file=%s,format=raw",
>> - accel, tmpfs, bootpath);
>> + " -drive file=%s,format=raw %s",
>> + accel, tmpfs, bootpath,
>> + extra_opts ? extra_opts : "");
>
> It's painful to have to use the ?: here as well as above where you set
> extra_opts, but I guess you need to, to allow you to free extra_opts
> below.
>
I'll think how to write it more clear. May be it's better to g_strdup("")
instead of NULL to eliminate some of ?:.
>> cmd_dst = g_strdup_printf("-machine accel=%s -m 150M"
>> " -name target,debug-threads=on"
>> " -serial file:%s/dest_serial"
>> " -drive file=%s,format=raw"
>> - " -incoming %s",
>> - accel, tmpfs, bootpath, uri);
>> + " -incoming %s %s",
>> + accel, tmpfs, bootpath, uri,
>> + extra_opts ? extra_opts : "");
>> start_address = X86_TEST_MEM_START;
>> end_address = X86_TEST_MEM_END;
>> } else if (g_str_equal(arch, "s390x")) {
>> init_bootfile_s390x(bootpath);
>> + extra_opts = use_shmem ? get_shmem_opts("128M", shmem_path) : NULL;
>> cmd_src = g_strdup_printf("-machine accel=%s -m 128M"
>> " -name source,debug-threads=on"
>> - " -serial file:%s/src_serial -bios %s",
>> - accel, tmpfs, bootpath);
>> + " -serial file:%s/src_serial -bios %s %s",
>> + accel, tmpfs, bootpath,
>> + extra_opts ? extra_opts : "");
>> cmd_dst = g_strdup_printf("-machine accel=%s -m 128M"
>> " -name target,debug-threads=on"
>> " -serial file:%s/dest_serial -bios %s"
>> - " -incoming %s",
>> - accel, tmpfs, bootpath, uri);
>> + " -incoming %s %s",
>> + accel, tmpfs, bootpath, uri,
>> + extra_opts ? extra_opts : "");
>> start_address = S390_TEST_MEM_START;
>> end_address = S390_TEST_MEM_END;
>> } else if (strcmp(arch, "ppc64") == 0) {
>> + extra_opts = use_shmem ? get_shmem_opts("256M", shmem_path) : NULL;
>> cmd_src = g_strdup_printf("-machine accel=%s -m 256M -nodefaults"
>> " -name source,debug-threads=on"
>> " -serial file:%s/src_serial"
>> " -prom-env 'use-nvramrc?=true' -prom-env "
>> "'nvramrc=hex .\" _\" begin %x %x "
>> "do i c@ 1 + i c! 1000 +loop .\" B\" 0 "
>> - "until'", accel, tmpfs, end_address,
>> - start_address);
>> + "until' %s", accel, tmpfs, end_address,
>> + start_address, extra_opts ? extra_opts : "");
>> cmd_dst = g_strdup_printf("-machine accel=%s -m 256M"
>> " -name target,debug-threads=on"
>> " -serial file:%s/dest_serial"
>> - " -incoming %s",
>> - accel, tmpfs, uri);
>> + " -incoming %s %s",
>> + accel, tmpfs, uri,
>> + extra_opts ? extra_opts : "");
>>
>> start_address = PPC_TEST_MEM_START;
>> end_address = PPC_TEST_MEM_END;
>> } else if (strcmp(arch, "aarch64") == 0) {
>> init_bootfile(bootpath, aarch64_kernel);
>> + extra_opts = use_shmem ? get_shmem_opts("150M", shmem_path) : NULL;
>> cmd_src = g_strdup_printf("-machine virt,accel=%s,gic-version=max "
>> "-name vmsource,debug-threads=on -cpu max "
>> "-m 150M -serial file:%s/src_serial "
>> - "-kernel %s ",
>> - accel, tmpfs, bootpath);
>> + "-kernel %s %s",
>> + accel, tmpfs, bootpath,
>> + extra_opts ? extra_opts : "");
>> cmd_dst = g_strdup_printf("-machine virt,accel=%s,gic-version=max "
>> "-name vmdest,debug-threads=on -cpu max "
>> "-m 150M -serial file:%s/dest_serial "
>> "-kernel %s "
>> - "-incoming %s ",
>> - accel, tmpfs, bootpath, uri);
>> + "-incoming %s %s",
>> + accel, tmpfs, bootpath, uri,
>> + extra_opts ? extra_opts : "");
>>
>> start_address = ARM_TEST_MEM_START;
>> end_address = ARM_TEST_MEM_END;
>> @@ -507,6 +532,7 @@ static int test_migrate_start(QTestState **from, QTestState **to,
>> }
>>
>> g_free(bootpath);
>> + g_free(extra_opts);
>>
>> if (hide_stderr) {
>> gchar *tmp;
>> @@ -524,6 +550,16 @@ static int test_migrate_start(QTestState **from, QTestState **to,
>>
>> *to = qtest_init(cmd_dst);
>> g_free(cmd_dst);
>> +
>> + /*
>> + * Remove shmem file immediately to avoid memory leak in test failed case.
>> + * It's valid becase QEMU has already opened this file
>> + */
>> + if (use_shmem) {
>> + unlink(shmem_path);
>> + g_free(shmem_path);
>> + }
>> +
>> return 0;
>> }
>>
>> @@ -603,7 +639,7 @@ static int migrate_postcopy_prepare(QTestState **from_ptr,
>> char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
>> QTestState *from, *to;
>>
>> - if (test_migrate_start(&from, &to, uri, hide_error)) {
>> + if (test_migrate_start(&from, &to, uri, hide_error, false)) {
>> return -1;
>> }
>>
>> @@ -720,7 +756,7 @@ static void test_baddest(void)
>> char *status;
>> bool failed;
>>
>> - if (test_migrate_start(&from, &to, "tcp:0:0", true)) {
>> + if (test_migrate_start(&from, &to, "tcp:0:0", true, false)) {
>> return;
>> }
>> migrate(from, "tcp:0:0", "{}");
>> @@ -745,7 +781,7 @@ static void test_precopy_unix(void)
>> char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
>> QTestState *from, *to;
>>
>> - if (test_migrate_start(&from, &to, uri, false)) {
>> + if (test_migrate_start(&from, &to, uri, false, false)) {
>> return;
>> }
>>
>> @@ -781,6 +817,38 @@ static void test_precopy_unix(void)
>> g_free(uri);
>> }
>>
>> +static void test_ignore_shared(void)
>> +{
>> + char *uri = g_strdup_printf("unix:%s/migsocket", tmpfs);
>> + QTestState *from, *to;
>> +
>> + if (test_migrate_start(&from, &to, uri, false, true)) {
>> + return;
>> + }
>> +
>> + migrate_set_capability(from, "x-ignore-shared", true);
>> + migrate_set_capability(to, "x-ignore-shared", true);
>> +
>> + /* Wait for the first serial output from the source */
>> + wait_for_serial("src_serial");
>> +
>> + migrate(from, uri, "{}");
>> +
>> + wait_for_migration_pass(from);
>> +
>> + if (!got_stop) {
>> + qtest_qmp_eventwait(from, "STOP");
>> + }
>> +
>> + qtest_qmp_eventwait(to, "RESUME");
>> +
>> + wait_for_serial("dest_serial");
>> + wait_for_migration_complete(from);
>> +
>> + test_migrate_end(from, to, true);
>> + g_free(uri);
>
> Can we reliably look at the migration stats and see if we've
> not transferred the shared data to make sure?
>
Agree, it makes sense, I'll add this check in v3.
> Dave
>
>> +}
>> +
>> int main(int argc, char **argv)
>> {
>> char template[] = "/tmp/migration-test-XXXXXX";
>> @@ -832,6 +900,7 @@ int main(int argc, char **argv)
>> qtest_add_func("/migration/deprecated", test_deprecated);
>> qtest_add_func("/migration/bad_dest", test_baddest);
>> qtest_add_func("/migration/precopy/unix", test_precopy_unix);
>> + qtest_add_func("/migration/ignore_shared", test_ignore_shared);
>>
>> ret = g_test_run();
>>
>> --
>> 2.20.1
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
Regards,
Yury
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [Qemu-devel] [PATCH v2 4/4] migration: Add capabilities validation
2019-02-11 13:30 ` Dr. David Alan Gilbert
@ 2019-02-12 13:58 ` Yury Kotov
0 siblings, 0 replies; 16+ messages in thread
From: Yury Kotov @ 2019-02-12 13:58 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: qemu-devel, Eduardo Habkost, Igor Mammedov, Paolo Bonzini,
Peter Crosthwaite, Richard Henderson, Juan Quintela, Eric Blake,
Markus Armbruster, Thomas Huth, Laurent Vivier, wrfsh
11.02.2019, 16:30, "Dr. David Alan Gilbert" <dgilbert@redhat.com>:
> * Yury Kotov (yury-kotov@yandex-team.ru) wrote:
>> Currently we don't check which capabilities set in the source QEMU.
>> We just expect that the target QEMU has the same enabled capabilities.
>>
>> Add explicit validation for capabilities to make sure that the target VM
>> has them too. This is enabled for only new capabilities to keep compatibily.
>
> I'd rather keep the capaiblities on the wire as strings, rather than
> indexes; indexes are too delicate.
>
It seems that strings also have a problem. E.g. when we remove 'x-' prefix from
x-some-capability. But I don't insist.
> I guess we also have to think about what happens when new capabilities
> are added; what happens when we migrate to an older qemu that doesn't
> know about that capability?
> What happens when we migrate to a newer capability which thinks you
> forgot to send that capability across?
>
I thought about such cases:
> what happens when new capabilities are added?
If new capability should be validated, then source will send it, target will
expect it. Otherwise, nothing will happen.
> what happens when we migrate to an older qemu that doesn't
> know about that capability?
Incoming migration will be failed as expected. If old qemu doesn't know the
capability, therefore qemu doesn't support it. In this case, user should disable
the capability on source before migration.
> What happens when we migrate to a newer capability which thinks you
> forgot to send that capability across?
Sorry, I'm not sure that I understood correctly. It seems that it's the same
case as the previous one.
> While we're on capabilities, I think you also need to check if the
> skip-shared is enabled with postcopy; I don't think they'll work
> together.
>
I agree, postcopy and skip-shared shouldn't be enabled together.
Will add a check in v3.
> Dave
>
>> Signed-off-by: Yury Kotov <yury-kotov@yandex-team.ru>
>> ---
>> migration/savevm.c | 101 +++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 101 insertions(+)
>>
>> diff --git a/migration/savevm.c b/migration/savevm.c
>> index 322660438d..9603a38bca 100644
>> --- a/migration/savevm.c
>> +++ b/migration/savevm.c
>> @@ -57,6 +57,7 @@
>> #include "sysemu/replay.h"
>> #include "qjson.h"
>> #include "migration/colo.h"
>> +#include "qemu/bitmap.h"
>>
>> #ifndef ETH_P_RARP
>> #define ETH_P_RARP 0x8035
>> @@ -316,6 +317,8 @@ typedef struct SaveState {
>> uint32_t len;
>> const char *name;
>> uint32_t target_page_bits;
>> + uint32_t caps_count;
>> + uint8_t *capabilities;
>> } SaveState;
>>
>> static SaveState savevm_state = {
>> @@ -323,15 +326,51 @@ static SaveState savevm_state = {
>> .global_section_id = 0,
>> };
>>
>> +static bool should_validate_capability(int capability)
>> +{
>> + assert(capability >= 0 && capability < MIGRATION_CAPABILITY__MAX);
>> + /* Validate only new capabilities to keep compatibility. */
>> + switch (capability) {
>> + case MIGRATION_CAPABILITY_X_IGNORE_SHARED:
>> + return true;
>> + default:
>> + return false;
>> + }
>> +}
>> +
>> +static uint32_t get_validatable_capabilities_count(void)
>> +{
>> + MigrationState *s = migrate_get_current();
>> + uint32_t result = 0;
>> + int i;
>> + for (i = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
>> + if (should_validate_capability(i) && s->enabled_capabilities[i]) {
>> + result++;
>> + }
>> + }
>> + return result;
>> +}
>> +
>> static int configuration_pre_save(void *opaque)
>> {
>> SaveState *state = opaque;
>> const char *current_name = MACHINE_GET_CLASS(current_machine)->name;
>> + MigrationState *s = migrate_get_current();
>> + int i, j;
>>
>> state->len = strlen(current_name);
>> state->name = current_name;
>> state->target_page_bits = qemu_target_page_bits();
>>
>> + state->caps_count = get_validatable_capabilities_count();
>> + state->capabilities = g_renew(uint8_t, state->capabilities,
>> + state->caps_count);
>> + for (i = j = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
>> + if (should_validate_capability(i) && s->enabled_capabilities[i]) {
>> + state->capabilities[j++] = i;
>> + }
>> + }
>> +
>> return 0;
>> }
>>
>> @@ -347,6 +386,45 @@ static int configuration_pre_load(void *opaque)
>> return 0;
>> }
>>
>> +static bool configuration_validate_capabilities(SaveState *state)
>> +{
>> + bool ret = true;
>> + MigrationState *s = migrate_get_current();
>> + unsigned long *source_caps_bm;
>> + int i;
>> +
>> + source_caps_bm = bitmap_new(MIGRATION_CAPABILITY__MAX);
>> + for (i = 0; i < state->caps_count; i++) {
>> + int capability = state->capabilities[i];
>> + if (capability >= MIGRATION_CAPABILITY__MAX) {
>> + error_report("Received unknown capability %d", capability);
>> + ret = false;
>> + } else {
>> + set_bit(capability, source_caps_bm);
>> + }
>> + }
>> +
>> + for (i = 0; i < MIGRATION_CAPABILITY__MAX; i++) {
>> + bool source_state, target_state;
>> + if (!should_validate_capability(i)) {
>> + continue;
>> + }
>> + source_state = test_bit(i, source_caps_bm);
>> + target_state = s->enabled_capabilities[i];
>> + if (source_state != target_state) {
>> + error_report("Capability %s is %s, but received capability is %s",
>> + MigrationCapability_str(i),
>> + target_state ? "on" : "off",
>> + source_state ? "on" : "off");
>> + ret = false;
>> + /* Don't break here to report all failed capabilities */
>> + }
>> + }
>> +
>> + g_free(source_caps_bm);
>> + return ret;
>> +}
>> +
>> static int configuration_post_load(void *opaque, int version_id)
>> {
>> SaveState *state = opaque;
>> @@ -364,6 +442,10 @@ static int configuration_post_load(void *opaque, int version_id)
>> return -EINVAL;
>> }
>>
>> + if (!configuration_validate_capabilities(state)) {
>> + return -EINVAL;
>> + }
>> +
>> return 0;
>> }
>>
>> @@ -380,6 +462,11 @@ static bool vmstate_target_page_bits_needed(void *opaque)
>> > qemu_target_page_bits_min();
>> }
>>
>> +static bool vmstate_capabilites_needed(void *opaque)
>> +{
>> + return get_validatable_capabilities_count() > 0;
>> +}
>> +
>> static const VMStateDescription vmstate_target_page_bits = {
>> .name = "configuration/target-page-bits",
>> .version_id = 1,
>> @@ -391,6 +478,19 @@ static const VMStateDescription vmstate_target_page_bits = {
>> }
>> };
>>
>> +static const VMStateDescription vmstate_capabilites = {
>> + .name = "configuration/capabilities",
>> + .version_id = 1,
>> + .minimum_version_id = 1,
>> + .needed = vmstate_capabilites_needed,
>> + .fields = (VMStateField[]) {
>> + VMSTATE_UINT32_V(caps_count, SaveState, 1),
>> + VMSTATE_VARRAY_UINT32_ALLOC(capabilities, SaveState, caps_count, 1,
>> + vmstate_info_uint8, uint8_t),
>> + VMSTATE_END_OF_LIST()
>> + }
>> +};
>> +
>> static const VMStateDescription vmstate_configuration = {
>> .name = "configuration",
>> .version_id = 1,
>> @@ -404,6 +504,7 @@ static const VMStateDescription vmstate_configuration = {
>> },
>> .subsections = (const VMStateDescription*[]) {
>> &vmstate_target_page_bits,
>> + &vmstate_capabilites,
>> NULL
>> }
>> };
>> --
>> 2.20.1
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
Regards,
Yury
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2019-02-12 13:58 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-04 13:09 [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability Yury Kotov
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 1/4] exec: Change RAMBlockIterFunc definition Yury Kotov
2019-02-11 11:40 ` Dr. David Alan Gilbert
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 2/4] migration: Introduce ignore-shared capability Yury Kotov
2019-02-11 12:45 ` Dr. David Alan Gilbert
2019-02-11 13:36 ` Yury Kotov
2019-02-11 15:55 ` Dr. David Alan Gilbert
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 3/4] tests/migration-test: Add a test for " Yury Kotov
2019-02-11 13:17 ` Dr. David Alan Gilbert
2019-02-12 12:49 ` Yury Kotov
2019-02-04 13:09 ` [Qemu-devel] [PATCH v2 4/4] migration: Add capabilities validation Yury Kotov
2019-02-11 13:30 ` Dr. David Alan Gilbert
2019-02-12 13:58 ` Yury Kotov
2019-02-11 11:24 ` [Qemu-devel] [PATCH v2 0/5] Add ignore-external migration capability Yury Kotov
2019-02-11 16:03 ` Dr. David Alan Gilbert
2019-02-11 16:13 ` Daniel P. Berrangé
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.