* [PATCH v2 0/2] migration: Store ram size value
@ 2022-07-28 12:07 Juan Quintela
2022-07-28 12:07 ` [PATCH v2 1/2] migration: Split ram_bytes_total_common() in two functions Juan Quintela
2022-07-28 12:07 ` [PATCH v2 2/2] migration: Calculate ram size once Juan Quintela
0 siblings, 2 replies; 3+ messages in thread
From: Juan Quintela @ 2022-07-28 12:07 UTC (permalink / raw)
To: qemu-devel; +Cc: Juan Quintela, Dr. David Alan Gilbert
Hi
I just resized this patch for latest upstream, we still have the same
trouble for huge guests, we are doing lots of RCU operations that are
not needed at all. As David explained on the previous submission,
ram_mig_ram_block_resized() aborts migration when size changes.
Please review.
[v1]
Current migration code recalculates the amount of RAM each time that
is needed. This calculation requires RCU and other operations.
During migration we disable hot/unplug of memory, so we can store it.
Notice the times difference, and specially that ram_bytes_total()
don't appears anymore in the perf output.
total time: 75852 ms
downtime: 264 ms
setup: 273 ms
transferred ram: 19671939 kbytes
throughput: 2132.28 mbps
remaining ram: 0 kbytes
total ram: 1077936904 kbytes
duplicate: 265170289 pages
skipped: 0 pages
normal: 4316628 pages
normal bytes: 17266512 kbytes
dirty sync count: 4
page size: 4 kbytes
multifd bytes: 17341329 kbytes
pages-per-second: 1236658
precopy ram: 2330608 kbytes
downtime ram: 1 kbytes
37.97% live_migration qemu-system-x86_64 [.] buffer_zero_avx512
10.42% live_migration qemu-system-x86_64 [.] ram_find_and_save_block.part.0
6.67% live_migration qemu-system-x86_64 [.] add_to_iovec
3.71% live_migration qemu-system-x86_64 [.] ram_bytes_total_common
2.79% live_migration qemu-system-x86_64 [.] qemu_ram_is_migratable
2.69% live_migration qemu-system-x86_64 [.] qemu_put_byte.part.0
2.41% live_migration qemu-system-x86_64 [.] bitmap_test_and_clear_atomic
1.55% live_migration qemu-system-x86_64 [.] qemu_put_be32
1.26% live_migration qemu-system-x86_64 [.] find_next_bit
1.07% multifdsend_0 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.07% multifdsend_13 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.06% multifdsend_6 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.05% multifdsend_2 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.04% multifdsend_15 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.03% multifdsend_12 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.02% live_migration qemu-system-x86_64 [.] migrate_ignore_shared
1.01% multifdsend_7 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.01% multifdsend_3 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
1.01% multifdsend_10 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.98% live_migration qemu-system-x86_64 [.] ram_save_iterate
0.96% multifdsend_4 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.93% multifdsend_8 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.92% multifdsend_5 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.90% multifdsend_14 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.88% multifdsend_9 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.85% multifdsend_1 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.83% multifdsend_11 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.61% live_migration qemu-system-x86_64 [.] save_zero_page_to_file.part.0
0.48% live_migration [kernel.kallsyms] [k] copy_user_enhanced_fast_string
Migration status: completed
total time: 70033 ms
downtime: 279 ms
setup: 280 ms
transferred ram: 19692747 kbytes
throughput: 2312.82 mbps
remaining ram: 0 kbytes
total ram: 1077936904 kbytes
duplicate: 265164421 pages
skipped: 0 pages
normal: 4322415 pages
normal bytes: 17289660 kbytes
dirty sync count: 3
page size: 4 kbytes
multifd bytes: 17362190 kbytes
pages-per-second: 2523447
precopy ram: 2330555 kbytes
downtime ram: 1 kbytes
43.64% live_migration qemu-system-x86_64 [.] buffer_zero_avx512
11.32% live_migration qemu-system-x86_64 [.] ram_find_and_save_block.part.0
7.60% live_migration qemu-system-x86_64 [.] add_to_iovec
2.95% live_migration qemu-system-x86_64 [.] qemu_put_byte.part.0
2.73% live_migration qemu-system-x86_64 [.] bitmap_test_and_clear_atomic
1.76% live_migration qemu-system-x86_64 [.] qemu_put_be32
1.44% live_migration qemu-system-x86_64 [.] find_next_bit
0.84% multifdsend_1 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.84% multifdsend_7 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.81% multifdsend_15 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.80% multifdsend_4 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.80% multifdsend_3 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.79% multifdsend_12 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.79% multifdsend_14 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.79% multifdsend_11 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.78% multifdsend_13 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.78% live_migration qemu-system-x86_64 [.] ram_save_iterate
0.77% multifdsend_9 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.77% multifdsend_5 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.77% multifdsend_10 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.77% multifdsend_2 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.77% multifdsend_6 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.76% multifdsend_8 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.71% multifdsend_0 [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.66% live_migration qemu-system-x86_64 [.] save_zero_page_to_file.part.0
0.62% live_migration qemu-system-x86_64 [.] qemu_ram_is_migratable
0.54% live_migration [kernel.kallsyms] [k] copy_user_enhanced_fast_string
0.51% live_migration qemu-system-x86_64 [.] qemu_put_byte
Please, review.
Thanks, Juan.
Juan Quintela (2):
migration: Split ram_bytes_total_common() in two functions
migration: Calculate ram size once
migration/ram.c | 31 ++++++++++++++++++-------------
1 file changed, 18 insertions(+), 13 deletions(-)
--
2.37.1
^ permalink raw reply [flat|nested] 3+ messages in thread
* [PATCH v2 1/2] migration: Split ram_bytes_total_common() in two functions
2022-07-28 12:07 [PATCH v2 0/2] migration: Store ram size value Juan Quintela
@ 2022-07-28 12:07 ` Juan Quintela
2022-07-28 12:07 ` [PATCH v2 2/2] migration: Calculate ram size once Juan Quintela
1 sibling, 0 replies; 3+ messages in thread
From: Juan Quintela @ 2022-07-28 12:07 UTC (permalink / raw)
To: qemu-devel; +Cc: Juan Quintela, Dr. David Alan Gilbert
It is just a big if in the middle of the function, and we need two
functions anways.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration/ram.c | 24 +++++++++++++-----------
1 file changed, 13 insertions(+), 11 deletions(-)
diff --git a/migration/ram.c b/migration/ram.c
index b94669ba5d..96a2b848da 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2581,28 +2581,30 @@ void acct_update_position(QEMUFile *f, size_t size, bool zero)
}
}
-static uint64_t ram_bytes_total_common(bool count_ignored)
+static uint64_t ram_bytes_total_with_ignored(void)
{
RAMBlock *block;
uint64_t total = 0;
RCU_READ_LOCK_GUARD();
- if (count_ignored) {
- RAMBLOCK_FOREACH_MIGRATABLE(block) {
- total += block->used_length;
- }
- } else {
- RAMBLOCK_FOREACH_NOT_IGNORED(block) {
- total += block->used_length;
- }
+ RAMBLOCK_FOREACH_MIGRATABLE(block) {
+ total += block->used_length;
}
return total;
}
uint64_t ram_bytes_total(void)
{
- return ram_bytes_total_common(false);
+ RAMBlock *block;
+ uint64_t total = 0;
+
+ RCU_READ_LOCK_GUARD();
+
+ RAMBLOCK_FOREACH_NOT_IGNORED(block) {
+ total += block->used_length;
+ }
+ return total;
}
static void xbzrle_load_setup(void)
@@ -3204,7 +3206,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
(*rsp)->f = f;
WITH_RCU_READ_LOCK_GUARD() {
- qemu_put_be64(f, ram_bytes_total_common(true) | RAM_SAVE_FLAG_MEM_SIZE);
+ qemu_put_be64(f, ram_bytes_total_with_ignored() | RAM_SAVE_FLAG_MEM_SIZE);
RAMBLOCK_FOREACH_MIGRATABLE(block) {
qemu_put_byte(f, strlen(block->idstr));
--
2.37.1
^ permalink raw reply related [flat|nested] 3+ messages in thread* [PATCH v2 2/2] migration: Calculate ram size once
2022-07-28 12:07 [PATCH v2 0/2] migration: Store ram size value Juan Quintela
2022-07-28 12:07 ` [PATCH v2 1/2] migration: Split ram_bytes_total_common() in two functions Juan Quintela
@ 2022-07-28 12:07 ` Juan Quintela
1 sibling, 0 replies; 3+ messages in thread
From: Juan Quintela @ 2022-07-28 12:07 UTC (permalink / raw)
To: qemu-devel; +Cc: Juan Quintela, Dr. David Alan Gilbert
We are recalculating ram size continously, when we know that it don't
change during migration. Create a field in RAMState to track it.
Signed-off-by: Juan Quintela <quintela@redhat.com>
---
migration/ram.c | 7 +++++--
1 file changed, 5 insertions(+), 2 deletions(-)
diff --git a/migration/ram.c b/migration/ram.c
index 96a2b848da..7bb4efd470 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -316,6 +316,8 @@ struct RAMState {
QEMUFile *f;
/* UFFD file descriptor, used in 'write-tracking' migration */
int uffdio_fd;
+ /* total ram size in bytes */
+ uint64_t ram_bytes_total;
/* Last block that we have visited searching for dirty pages */
RAMBlock *last_seen_block;
/* Last block from where we have sent data */
@@ -2527,7 +2529,7 @@ static int ram_find_and_save_block(RAMState *rs)
bool again, found;
/* No dirty page as there is zero RAM */
- if (!ram_bytes_total()) {
+ if (!rs->ram_bytes_total) {
return pages;
}
@@ -2986,13 +2988,14 @@ static int ram_state_init(RAMState **rsp)
qemu_mutex_init(&(*rsp)->bitmap_mutex);
qemu_mutex_init(&(*rsp)->src_page_req_mutex);
QSIMPLEQ_INIT(&(*rsp)->src_page_requests);
+ (*rsp)->ram_bytes_total = ram_bytes_total();
/*
* Count the total number of pages used by ram blocks not including any
* gaps due to alignment or unplugs.
* This must match with the initial values of dirty bitmap.
*/
- (*rsp)->migration_dirty_pages = ram_bytes_total() >> TARGET_PAGE_BITS;
+ (*rsp)->migration_dirty_pages = (*rsp)->ram_bytes_total >> TARGET_PAGE_BITS;
ram_state_reset(*rsp);
return 0;
--
2.37.1
^ permalink raw reply related [flat|nested] 3+ messages in thread
end of thread, other threads:[~2022-07-28 12:15 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2022-07-28 12:07 [PATCH v2 0/2] migration: Store ram size value Juan Quintela
2022-07-28 12:07 ` [PATCH v2 1/2] migration: Split ram_bytes_total_common() in two functions Juan Quintela
2022-07-28 12:07 ` [PATCH v2 2/2] migration: Calculate ram size once Juan Quintela
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.