[PATCH 00/19] migration: Remove QEMUFileHooks

[PATCH 00/19] migration: Remove QEMUFileHooks

[Juan Quintela]

Hi

In this series (v1):
- QEMUFileHooks only had a single user, RDMA migration.  Just remove the
  hooks and create stubs for when RDMA is not compiled in.

- This implies that we have to move all the operations from
  migration/qemu-file.c to migration/rdma.c.

- I now we can still simplify rdma_control_save_page(), but I don't
  have an easy setup for testing.

- Yes, the goal of the whole operations is to be able to move
  ram_file_limit from qemu-file to migration.c.

Please review.

Thanks, Juan.

Juan Quintela (19):
  multifd: We already account for this packet on the multifd thread
  migration: Move ram_stats to its own file migration-stats.[ch]
  migration: Rename ram_counters to mig_stats
  migration: Rename RAMStats to MigrationAtomicStats
  migration/rdma: Split the zero page case from acct_update_position
  migration/rdma: Unfold last user of acct_update_position()
  migration/rdma: Unflod ram_control_before_iterate()
  migration/rdma: Unflod ram_control_after_iterate()
  migration/rdma: simplify ram_control_load_hook()
  migration/rdma: Don't pass the QIOChannelRDMA as an opaque
  migration/rdma: We can calculate the rioc from the QEMUFile
  migration/rdma: It makes no sense to recive that flag without RDMA
  migration: Make RAM_SAVE_FLAG_HOOK a normal case entry
  migration/rdma: Remove all uses of RAM_CONTROL_HOOK
  migration/rdma: Unfold hook_ram_load()
  migration/rdma: Make ram_control_save_page() use exported interfaces
  migration/rdma: Create rdma_control_save_page()
  qemu-file: Remove QEMUFileHooks
  migration/rdma: Move rdma constants from qemu-file.h to rdma.h

 migration/meson.build       |   1 +
 migration/migration-stats.c |  17 +++++
 migration/migration-stats.h |  41 ++++++++++++
 migration/migration.c       |  33 +++++-----
 migration/multifd.c         |  10 ++-
 migration/qemu-file.c       |  77 +----------------------
 migration/qemu-file.h       |  51 ---------------
 migration/ram.c             | 120 +++++++++++++++++++-----------------
 migration/ram.h             |  24 --------
 migration/rdma.c            |  82 ++++++++++++------------
 migration/rdma.h            |  36 +++++++++++
 migration/savevm.c          |   3 +-
 12 files changed, 222 insertions(+), 273 deletions(-)
 create mode 100644 migration/migration-stats.c
 create mode 100644 migration/migration-stats.h

-- 
2.40.0

[PATCH 01/19] multifd: We already account for this packet on the multifd thread

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/multifd.c | 3 ---
 1 file changed, 3 deletions(-)

diff --git a/migration/multifd.c b/migration/multifd.c
index 6a59c03dd2..6053012ad9 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -626,10 +626,7 @@ int multifd_send_sync_main(QEMUFile *f)
         p->packet_num = multifd_send_state->packet_num++;
         p->flags |= MULTIFD_FLAG_SYNC;
         p->pending_job++;
-        qemu_file_acct_rate_limit(f, p->packet_len);
         qemu_mutex_unlock(&p->mutex);
-        stat64_add(&ram_counters.transferred, p->packet_len);
-        stat64_add(&ram_counters.multifd_bytes, p->packet_len);
         qemu_sem_post(&p->sem);
     }
     for (i = 0; i < migrate_multifd_channels(); i++) {
-- 
2.40.0

[PATCH 02/19] migration: Move ram_stats to its own file migration-stats.[ch]

[Juan Quintela]

There is already include/qemu/stats.h, so stats.h was a bad idea.
We want this file to not depend on anything else, we will move all the
migration counters/stats to this struct.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/meson.build       |  1 +
 migration/migration-stats.c | 17 +++++++++++++++
 migration/migration-stats.h | 41 +++++++++++++++++++++++++++++++++++++
 migration/migration.c       |  1 +
 migration/multifd.c         |  1 +
 migration/ram.c             |  3 +--
 migration/ram.h             | 23 ---------------------
 migration/savevm.c          |  1 +
 8 files changed, 63 insertions(+), 25 deletions(-)
 create mode 100644 migration/migration-stats.c
 create mode 100644 migration/migration-stats.h

diff --git a/migration/meson.build b/migration/meson.build
index 480ff6854a..da1897fadf 100644
--- a/migration/meson.build
+++ b/migration/meson.build
@@ -19,6 +19,7 @@ softmmu_ss.add(files(
   'fd.c',
   'global_state.c',
   'migration-hmp-cmds.c',
+  'migration-stats.c',
   'migration.c',
   'multifd.c',
   'multifd-zlib.c',
diff --git a/migration/migration-stats.c b/migration/migration-stats.c
new file mode 100644
index 0000000000..b0eb5ae73c
--- /dev/null
+++ b/migration/migration-stats.c
@@ -0,0 +1,17 @@
+/*
+ * Migration stats
+ *
+ * Copyright (c) 2012-2023 Red Hat Inc
+ *
+ * Authors:
+ *  Juan Quintela <quintela@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#include "qemu/osdep.h"
+#include "qemu/stats64.h"
+#include "migration-stats.h"
+
+RAMStats ram_counters;
diff --git a/migration/migration-stats.h b/migration/migration-stats.h
new file mode 100644
index 0000000000..2edea0c779
--- /dev/null
+++ b/migration/migration-stats.h
@@ -0,0 +1,41 @@
+/*
+ * Migration stats
+ *
+ * Copyright (c) 2012-2023 Red Hat Inc
+ *
+ * Authors:
+ *  Juan Quintela <quintela@redhat.com>
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or later.
+ * See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_MIGRATION_STATS_H
+#define QEMU_MIGRATION_STATS_H
+
+#include "qemu/stats64.h"
+
+/*
+ * These are the ram migration statistic counters.  It is loosely
+ * based on MigrationStats.  We change to Stat64 any counter that
+ * needs to be updated using atomic ops (can be accessed by more than
+ * one thread).
+ */
+typedef struct {
+    Stat64 dirty_bytes_last_sync;
+    Stat64 dirty_pages_rate;
+    Stat64 dirty_sync_count;
+    Stat64 dirty_sync_missed_zero_copy;
+    Stat64 downtime_bytes;
+    Stat64 zero_pages;
+    Stat64 multifd_bytes;
+    Stat64 normal_pages;
+    Stat64 postcopy_bytes;
+    Stat64 postcopy_requests;
+    Stat64 precopy_bytes;
+    Stat64 transferred;
+} RAMStats;
+
+extern RAMStats ram_counters;
+
+#endif
diff --git a/migration/migration.c b/migration/migration.c
index abcadbb619..5ecf3dc381 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -29,6 +29,7 @@
 #include "migration/global_state.h"
 #include "migration/misc.h"
 #include "migration.h"
+#include "migration-stats.h"
 #include "savevm.h"
 #include "qemu-file.h"
 #include "channel.h"
diff --git a/migration/multifd.c b/migration/multifd.c
index 6053012ad9..347999f84a 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -19,6 +19,7 @@
 #include "qapi/error.h"
 #include "ram.h"
 #include "migration.h"
+#include "migration-stats.h"
 #include "socket.h"
 #include "tls.h"
 #include "qemu-file.h"
diff --git a/migration/ram.c b/migration/ram.c
index 89be3e3320..a6d5478ef8 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -36,6 +36,7 @@
 #include "xbzrle.h"
 #include "ram.h"
 #include "migration.h"
+#include "migration-stats.h"
 #include "migration/register.h"
 #include "migration/misc.h"
 #include "qemu-file.h"
@@ -460,8 +461,6 @@ uint64_t ram_bytes_remaining(void)
                        0;
 }
 
-RAMStats ram_counters;
-
 void ram_transferred_add(uint64_t bytes)
 {
     if (runstate_is_running()) {
diff --git a/migration/ram.h b/migration/ram.h
index 04b05e1b2c..8692de6ba0 100644
--- a/migration/ram.h
+++ b/migration/ram.h
@@ -32,30 +32,7 @@
 #include "qapi/qapi-types-migration.h"
 #include "exec/cpu-common.h"
 #include "io/channel.h"
-#include "qemu/stats64.h"
 
-/*
- * These are the ram migration statistic counters.  It is loosely
- * based on MigrationStats.  We change to Stat64 any counter that
- * needs to be updated using atomic ops (can be accessed by more than
- * one thread).
- */
-typedef struct {
-    Stat64 dirty_bytes_last_sync;
-    Stat64 dirty_pages_rate;
-    Stat64 dirty_sync_count;
-    Stat64 dirty_sync_missed_zero_copy;
-    Stat64 downtime_bytes;
-    Stat64 zero_pages;
-    Stat64 multifd_bytes;
-    Stat64 normal_pages;
-    Stat64 postcopy_bytes;
-    Stat64 postcopy_requests;
-    Stat64 precopy_bytes;
-    Stat64 transferred;
-} RAMStats;
-
-extern RAMStats ram_counters;
 extern XBZRLECacheStats xbzrle_counters;
 extern CompressionStats compression_counters;
 
diff --git a/migration/savevm.c b/migration/savevm.c
index a9181b444b..8e2efb1a19 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -31,6 +31,7 @@
 #include "net/net.h"
 #include "migration.h"
 #include "migration/snapshot.h"
+#include "migration-stats.h"
 #include "migration/vmstate.h"
 #include "migration/misc.h"
 #include "migration/register.h"
-- 
2.40.0

[PATCH 03/19] migration: Rename ram_counters to mig_stats

[Juan Quintela]

migration_stats is just too long, and it is going to have more than
ram counters in the near future.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/migration-stats.c |  2 +-
 migration/migration-stats.h |  2 +-
 migration/migration.c       | 32 ++++++++++++-------------
 migration/multifd.c         |  6 ++---
 migration/ram.c             | 48 ++++++++++++++++++-------------------
 migration/savevm.c          |  2 +-
 6 files changed, 46 insertions(+), 46 deletions(-)

diff --git a/migration/migration-stats.c b/migration/migration-stats.c
index b0eb5ae73c..8c0af9b80a 100644
--- a/migration/migration-stats.c
+++ b/migration/migration-stats.c
@@ -14,4 +14,4 @@
 #include "qemu/stats64.h"
 #include "migration-stats.h"
 
-RAMStats ram_counters;
+RAMStats mig_stats;
diff --git a/migration/migration-stats.h b/migration/migration-stats.h
index 2edea0c779..197374b4f6 100644
--- a/migration/migration-stats.h
+++ b/migration/migration-stats.h
@@ -36,6 +36,6 @@ typedef struct {
     Stat64 transferred;
 } RAMStats;
 
-extern RAMStats ram_counters;
+extern RAMStats mig_stats;
 
 #endif
diff --git a/migration/migration.c b/migration/migration.c
index 5ecf3dc381..feb5ab7493 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -909,26 +909,26 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
     size_t page_size = qemu_target_page_size();
 
     info->ram = g_malloc0(sizeof(*info->ram));
-    info->ram->transferred = stat64_get(&ram_counters.transferred);
+    info->ram->transferred = stat64_get(&mig_stats.transferred);
     info->ram->total = ram_bytes_total();
-    info->ram->duplicate = stat64_get(&ram_counters.zero_pages);
+    info->ram->duplicate = stat64_get(&mig_stats.zero_pages);
     /* legacy value.  It is not used anymore */
     info->ram->skipped = 0;
-    info->ram->normal = stat64_get(&ram_counters.normal_pages);
+    info->ram->normal = stat64_get(&mig_stats.normal_pages);
     info->ram->normal_bytes = info->ram->normal * page_size;
     info->ram->mbps = s->mbps;
     info->ram->dirty_sync_count =
-        stat64_get(&ram_counters.dirty_sync_count);
+        stat64_get(&mig_stats.dirty_sync_count);
     info->ram->dirty_sync_missed_zero_copy =
-        stat64_get(&ram_counters.dirty_sync_missed_zero_copy);
+        stat64_get(&mig_stats.dirty_sync_missed_zero_copy);
     info->ram->postcopy_requests =
-        stat64_get(&ram_counters.postcopy_requests);
+        stat64_get(&mig_stats.postcopy_requests);
     info->ram->page_size = page_size;
-    info->ram->multifd_bytes = stat64_get(&ram_counters.multifd_bytes);
+    info->ram->multifd_bytes = stat64_get(&mig_stats.multifd_bytes);
     info->ram->pages_per_second = s->pages_per_second;
-    info->ram->precopy_bytes = stat64_get(&ram_counters.precopy_bytes);
-    info->ram->downtime_bytes = stat64_get(&ram_counters.downtime_bytes);
-    info->ram->postcopy_bytes = stat64_get(&ram_counters.postcopy_bytes);
+    info->ram->precopy_bytes = stat64_get(&mig_stats.precopy_bytes);
+    info->ram->downtime_bytes = stat64_get(&mig_stats.downtime_bytes);
+    info->ram->postcopy_bytes = stat64_get(&mig_stats.postcopy_bytes);
 
     if (migrate_xbzrle()) {
         info->xbzrle_cache = g_malloc0(sizeof(*info->xbzrle_cache));
@@ -960,7 +960,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
     if (s->state != MIGRATION_STATUS_COMPLETED) {
         info->ram->remaining = ram_bytes_remaining();
         info->ram->dirty_pages_rate =
-           stat64_get(&ram_counters.dirty_pages_rate);
+           stat64_get(&mig_stats.dirty_pages_rate);
     }
 }
 
@@ -1613,10 +1613,10 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
 
     migrate_init(s);
     /*
-     * set ram_counters compression_counters memory to zero for a
+     * set mig_stats compression_counters memory to zero for a
      * new migration
      */
-    memset(&ram_counters, 0, sizeof(ram_counters));
+    memset(&mig_stats, 0, sizeof(mig_stats));
     memset(&compression_counters, 0, sizeof(compression_counters));
 
     return true;
@@ -2627,7 +2627,7 @@ static MigThrError migration_detect_error(MigrationState *s)
 static uint64_t migration_total_bytes(MigrationState *s)
 {
     return qemu_file_total_transferred(s->to_dst_file) +
-        stat64_get(&ram_counters.multifd_bytes);
+        stat64_get(&mig_stats.multifd_bytes);
 }
 
 static void migration_calculate_complete(MigrationState *s)
@@ -2691,10 +2691,10 @@ static void migration_update_counters(MigrationState *s,
      * if we haven't sent anything, we don't want to
      * recalculate. 10000 is a small enough number for our purposes
      */
-    if (stat64_get(&ram_counters.dirty_pages_rate) &&
+    if (stat64_get(&mig_stats.dirty_pages_rate) &&
         transferred > 10000) {
         s->expected_downtime =
-            stat64_get(&ram_counters.dirty_bytes_last_sync) / bandwidth;
+            stat64_get(&mig_stats.dirty_bytes_last_sync) / bandwidth;
     }
 
     qemu_file_reset_rate_limit(s->to_dst_file);
diff --git a/migration/multifd.c b/migration/multifd.c
index 347999f84a..4a2e1a47ce 100644
--- a/migration/multifd.c
+++ b/migration/multifd.c
@@ -434,8 +434,8 @@ static int multifd_send_pages(QEMUFile *f)
     transferred = ((uint64_t) pages->num) * p->page_size + p->packet_len;
     qemu_file_acct_rate_limit(f, transferred);
     qemu_mutex_unlock(&p->mutex);
-    stat64_add(&ram_counters.transferred, transferred);
-    stat64_add(&ram_counters.multifd_bytes, transferred);
+    stat64_add(&mig_stats.transferred, transferred);
+    stat64_add(&mig_stats.multifd_bytes, transferred);
     qemu_sem_post(&p->sem);
 
     return 1;
@@ -577,7 +577,7 @@ static int multifd_zero_copy_flush(QIOChannel *c)
         return -1;
     }
     if (ret == 1) {
-        stat64_add(&ram_counters.dirty_sync_missed_zero_copy, 1);
+        stat64_add(&mig_stats.dirty_sync_missed_zero_copy, 1);
     }
 
     return ret;
diff --git a/migration/ram.c b/migration/ram.c
index a6d5478ef8..c3981f64e4 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -464,13 +464,13 @@ uint64_t ram_bytes_remaining(void)
 void ram_transferred_add(uint64_t bytes)
 {
     if (runstate_is_running()) {
-        stat64_add(&ram_counters.precopy_bytes, bytes);
+        stat64_add(&mig_stats.precopy_bytes, bytes);
     } else if (migration_in_postcopy()) {
-        stat64_add(&ram_counters.postcopy_bytes, bytes);
+        stat64_add(&mig_stats.postcopy_bytes, bytes);
     } else {
-        stat64_add(&ram_counters.downtime_bytes, bytes);
+        stat64_add(&mig_stats.downtime_bytes, bytes);
     }
-    stat64_add(&ram_counters.transferred, bytes);
+    stat64_add(&mig_stats.transferred, bytes);
 }
 
 struct MigrationOps {
@@ -744,7 +744,7 @@ void mig_throttle_counter_reset(void)
 
     rs->time_last_bitmap_sync = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
     rs->num_dirty_pages_period = 0;
-    rs->bytes_xfer_prev = stat64_get(&ram_counters.transferred);
+    rs->bytes_xfer_prev = stat64_get(&mig_stats.transferred);
 }
 
 /**
@@ -764,7 +764,7 @@ static void xbzrle_cache_zero_page(RAMState *rs, ram_addr_t current_addr)
     /* We don't care if this fails to allocate a new cache page
      * as long as it updated an old one */
     cache_insert(XBZRLE.cache, current_addr, XBZRLE.zero_target_page,
-                 stat64_get(&ram_counters.dirty_sync_count));
+                 stat64_get(&mig_stats.dirty_sync_count));
 }
 
 #define ENCODING_FLAG_XBZRLE 0x1
@@ -790,7 +790,7 @@ static int save_xbzrle_page(RAMState *rs, PageSearchStatus *pss,
     int encoded_len = 0, bytes_xbzrle;
     uint8_t *prev_cached_page;
     QEMUFile *file = pss->pss_channel;
-    uint64_t generation = stat64_get(&ram_counters.dirty_sync_count);
+    uint64_t generation = stat64_get(&mig_stats.dirty_sync_count);
 
     if (!cache_is_cached(XBZRLE.cache, current_addr, generation)) {
         xbzrle_counters.cache_miss++;
@@ -1118,8 +1118,8 @@ uint64_t ram_pagesize_summary(void)
 
 uint64_t ram_get_total_transferred_pages(void)
 {
-    return stat64_get(&ram_counters.normal_pages) +
-        stat64_get(&ram_counters.zero_pages) +
+    return stat64_get(&mig_stats.normal_pages) +
+        stat64_get(&mig_stats.zero_pages) +
         compression_counters.pages + xbzrle_counters.pages;
 }
 
@@ -1129,7 +1129,7 @@ static void migration_update_rates(RAMState *rs, int64_t end_time)
     double compressed_size;
 
     /* calculate period counters */
-    stat64_set(&ram_counters.dirty_pages_rate,
+    stat64_set(&mig_stats.dirty_pages_rate,
                rs->num_dirty_pages_period * 1000 /
                (end_time - rs->time_last_bitmap_sync));
 
@@ -1180,7 +1180,7 @@ static void migration_trigger_throttle(RAMState *rs)
 {
     uint64_t threshold = migrate_throttle_trigger_threshold();
     uint64_t bytes_xfer_period =
-        stat64_get(&ram_counters.transferred) - rs->bytes_xfer_prev;
+        stat64_get(&mig_stats.transferred) - rs->bytes_xfer_prev;
     uint64_t bytes_dirty_period = rs->num_dirty_pages_period * TARGET_PAGE_SIZE;
     uint64_t bytes_dirty_threshold = bytes_xfer_period * threshold / 100;
 
@@ -1209,7 +1209,7 @@ static void migration_bitmap_sync(RAMState *rs)
     RAMBlock *block;
     int64_t end_time;
 
-    stat64_add(&ram_counters.dirty_sync_count, 1);
+    stat64_add(&mig_stats.dirty_sync_count, 1);
 
     if (!rs->time_last_bitmap_sync) {
         rs->time_last_bitmap_sync = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
@@ -1223,7 +1223,7 @@ static void migration_bitmap_sync(RAMState *rs)
         RAMBLOCK_FOREACH_NOT_IGNORED(block) {
             ramblock_sync_dirty_bitmap(rs, block);
         }
-        stat64_set(&ram_counters.dirty_bytes_last_sync, ram_bytes_remaining());
+        stat64_set(&mig_stats.dirty_bytes_last_sync, ram_bytes_remaining());
     }
     qemu_mutex_unlock(&rs->bitmap_mutex);
 
@@ -1243,10 +1243,10 @@ static void migration_bitmap_sync(RAMState *rs)
         /* reset period counters */
         rs->time_last_bitmap_sync = end_time;
         rs->num_dirty_pages_period = 0;
-        rs->bytes_xfer_prev = stat64_get(&ram_counters.transferred);
+        rs->bytes_xfer_prev = stat64_get(&mig_stats.transferred);
     }
     if (migrate_events()) {
-        uint64_t generation = stat64_get(&ram_counters.dirty_sync_count);
+        uint64_t generation = stat64_get(&mig_stats.dirty_sync_count);
         qapi_event_send_migration_pass(generation);
     }
 }
@@ -1320,7 +1320,7 @@ static int save_zero_page(PageSearchStatus *pss, QEMUFile *f, RAMBlock *block,
     int len = save_zero_page_to_file(pss, f, block, offset);
 
     if (len) {
-        stat64_add(&ram_counters.zero_pages, 1);
+        stat64_add(&mig_stats.zero_pages, 1);
         ram_transferred_add(len);
         return 1;
     }
@@ -1357,9 +1357,9 @@ static bool control_save_page(PageSearchStatus *pss, RAMBlock *block,
     }
 
     if (bytes_xmit > 0) {
-        stat64_add(&ram_counters.normal_pages, 1);
+        stat64_add(&mig_stats.normal_pages, 1);
     } else if (bytes_xmit == 0) {
-        stat64_add(&ram_counters.zero_pages, 1);
+        stat64_add(&mig_stats.zero_pages, 1);
     }
 
     return true;
@@ -1391,7 +1391,7 @@ static int save_normal_page(PageSearchStatus *pss, RAMBlock *block,
         qemu_put_buffer(file, buf, TARGET_PAGE_SIZE);
     }
     ram_transferred_add(TARGET_PAGE_SIZE);
-    stat64_add(&ram_counters.normal_pages, 1);
+    stat64_add(&mig_stats.normal_pages, 1);
     return 1;
 }
 
@@ -1447,7 +1447,7 @@ static int ram_save_multifd_page(QEMUFile *file, RAMBlock *block,
     if (multifd_queue_page(file, block, offset) < 0) {
         return -1;
     }
-    stat64_add(&ram_counters.normal_pages, 1);
+    stat64_add(&mig_stats.normal_pages, 1);
 
     return 1;
 }
@@ -1486,7 +1486,7 @@ update_compress_thread_counts(const CompressParam *param, int bytes_xmit)
     ram_transferred_add(bytes_xmit);
 
     if (param->zero_page) {
-        stat64_add(&ram_counters.zero_pages, 1);
+        stat64_add(&mig_stats.zero_pages, 1);
         return;
     }
 
@@ -2179,7 +2179,7 @@ int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len)
     RAMBlock *ramblock;
     RAMState *rs = ram_state;
 
-    stat64_add(&ram_counters.postcopy_requests, 1);
+    stat64_add(&mig_stats.postcopy_requests, 1);
     RCU_READ_LOCK_GUARD();
 
     if (!rbname) {
@@ -2634,9 +2634,9 @@ void acct_update_position(QEMUFile *f, size_t size, bool zero)
     uint64_t pages = size / TARGET_PAGE_SIZE;
 
     if (zero) {
-        stat64_add(&ram_counters.zero_pages, pages);
+        stat64_add(&mig_stats.zero_pages, pages);
     } else {
-        stat64_add(&ram_counters.normal_pages, pages);
+        stat64_add(&mig_stats.normal_pages, pages);
         ram_transferred_add(size);
         qemu_file_credit_transfer(f, size);
     }
diff --git a/migration/savevm.c b/migration/savevm.c
index 8e2efb1a19..a9d0a88e62 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1622,7 +1622,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
     }
 
     migrate_init(ms);
-    memset(&ram_counters, 0, sizeof(ram_counters));
+    memset(&mig_stats, 0, sizeof(mig_stats));
     memset(&compression_counters, 0, sizeof(compression_counters));
     ms->to_dst_file = f;
 
-- 
2.40.0

[PATCH 04/19] migration: Rename RAMStats to MigrationAtomicStats

[Juan Quintela]

It is lousely based on MigrationStats, but that name is taken, so this
is the best one that I came with.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---

If you have any good suggestion for the name, I am all ears.
---
 migration/migration-stats.c | 2 +-
 migration/migration-stats.h | 4 ++--
 2 files changed, 3 insertions(+), 3 deletions(-)

diff --git a/migration/migration-stats.c b/migration/migration-stats.c
index 8c0af9b80a..2f2cea965c 100644
--- a/migration/migration-stats.c
+++ b/migration/migration-stats.c
@@ -14,4 +14,4 @@
 #include "qemu/stats64.h"
 #include "migration-stats.h"
 
-RAMStats mig_stats;
+MigrationAtomicStats mig_stats;
diff --git a/migration/migration-stats.h b/migration/migration-stats.h
index 197374b4f6..149af932d7 100644
--- a/migration/migration-stats.h
+++ b/migration/migration-stats.h
@@ -34,8 +34,8 @@ typedef struct {
     Stat64 postcopy_requests;
     Stat64 precopy_bytes;
     Stat64 transferred;
-} RAMStats;
+} MigrationAtomicStats;
 
-extern RAMStats mig_stats;
+extern MigrationAtomicStats mig_stats;
 
 #endif
-- 
2.40.0

[PATCH 05/19] migration/rdma: Split the zero page case from acct_update_position

[Juan Quintela]

Now that we have atomic counters, we can do it on the place that we
need it, no need to do it inside ram.c.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/ram.c  | 12 ++++--------
 migration/ram.h  |  2 +-
 migration/rdma.c |  7 +++++--
 3 files changed, 10 insertions(+), 11 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index c3981f64e4..c249a1f468 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2629,17 +2629,13 @@ static int ram_find_and_save_block(RAMState *rs)
     return pages;
 }
 
-void acct_update_position(QEMUFile *f, size_t size, bool zero)
+void acct_update_position(QEMUFile *f, size_t size)
 {
     uint64_t pages = size / TARGET_PAGE_SIZE;
 
-    if (zero) {
-        stat64_add(&mig_stats.zero_pages, pages);
-    } else {
-        stat64_add(&mig_stats.normal_pages, pages);
-        ram_transferred_add(size);
-        qemu_file_credit_transfer(f, size);
-    }
+    stat64_add(&mig_stats.normal_pages, pages);
+    ram_transferred_add(size);
+    qemu_file_credit_transfer(f, size);
 }
 
 static uint64_t ram_bytes_total_with_ignored(void)
diff --git a/migration/ram.h b/migration/ram.h
index 8692de6ba0..3804753ca3 100644
--- a/migration/ram.h
+++ b/migration/ram.h
@@ -53,7 +53,7 @@ void mig_throttle_counter_reset(void);
 
 uint64_t ram_pagesize_summary(void);
 int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len);
-void acct_update_position(QEMUFile *f, size_t size, bool zero);
+void acct_update_position(QEMUFile *f, size_t size);
 void ram_postcopy_migrated_memory_release(MigrationState *ms);
 /* For outgoing discard bitmap */
 void ram_postcopy_send_discard_bitmap(MigrationState *ms);
diff --git a/migration/rdma.c b/migration/rdma.c
index 0af5e944f0..7a9b284c3f 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -17,8 +17,10 @@
 #include "qemu/osdep.h"
 #include "qapi/error.h"
 #include "qemu/cutils.h"
+#include "exec/target_page.h"
 #include "rdma.h"
 #include "migration.h"
+#include "migration-stats.h"
 #include "qemu-file.h"
 #include "ram.h"
 #include "qemu/error-report.h"
@@ -2120,7 +2122,8 @@ retry:
                     return -EIO;
                 }
 
-                acct_update_position(f, sge.length, true);
+                stat64_add(&mig_stats.zero_pages,
+                           sge.length / qemu_target_page_size());
 
                 return 1;
             }
@@ -2228,7 +2231,7 @@ retry:
     }
 
     set_bit(chunk, block->transit_bitmap);
-    acct_update_position(f, sge.length, false);
+    acct_update_position(f, sge.length);
     rdma->total_writes++;
 
     return 0;
-- 
2.40.0

[PATCH 06/19] migration/rdma: Unfold last user of acct_update_position()

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/ram.c  | 9 ---------
 migration/ram.h  | 1 -
 migration/rdma.c | 4 +++-
 3 files changed, 3 insertions(+), 11 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index c249a1f468..7d81c4a39e 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2629,15 +2629,6 @@ static int ram_find_and_save_block(RAMState *rs)
     return pages;
 }
 
-void acct_update_position(QEMUFile *f, size_t size)
-{
-    uint64_t pages = size / TARGET_PAGE_SIZE;
-
-    stat64_add(&mig_stats.normal_pages, pages);
-    ram_transferred_add(size);
-    qemu_file_credit_transfer(f, size);
-}
-
 static uint64_t ram_bytes_total_with_ignored(void)
 {
     RAMBlock *block;
diff --git a/migration/ram.h b/migration/ram.h
index 3804753ca3..6fffbeb5f1 100644
--- a/migration/ram.h
+++ b/migration/ram.h
@@ -53,7 +53,6 @@ void mig_throttle_counter_reset(void);
 
 uint64_t ram_pagesize_summary(void);
 int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len);
-void acct_update_position(QEMUFile *f, size_t size);
 void ram_postcopy_migrated_memory_release(MigrationState *ms);
 /* For outgoing discard bitmap */
 void ram_postcopy_send_discard_bitmap(MigrationState *ms);
diff --git a/migration/rdma.c b/migration/rdma.c
index 7a9b284c3f..7e747b2595 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -2231,7 +2231,9 @@ retry:
     }
 
     set_bit(chunk, block->transit_bitmap);
-    acct_update_position(f, sge.length);
+    stat64_add(&mig_stats.normal_pages, sge.length / qemu_target_page_size());
+    ram_transferred_add(sge.length);
+    qemu_file_credit_transfer(f, sge.length);
     rdma->total_writes++;
 
     return 0;
-- 
2.40.0

[PATCH 07/19] migration/rdma: Unflod ram_control_before_iterate()

[Juan Quintela]

Once there:
- Remove unused data parameter
- unfold it in its callers.
- change all callers to call qemu_rdma_registration_start()

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/qemu-file.c | 13 +------------
 migration/qemu-file.h |  2 --
 migration/ram.c       | 16 +++++++++++++---
 migration/rdma.c      |  4 +---
 migration/rdma.h      |  6 ++++++
 5 files changed, 21 insertions(+), 20 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index ee04240a21..b6dca23706 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -30,6 +30,7 @@
 #include "qemu-file.h"
 #include "trace.h"
 #include "qapi/error.h"
+#include "rdma.h"
 
 #define IO_BUF_SIZE 32768
 #define MAX_IOV_SIZE MIN_CONST(IOV_MAX, 64)
@@ -314,18 +315,6 @@ void qemu_fflush(QEMUFile *f)
     f->iovcnt = 0;
 }
 
-void ram_control_before_iterate(QEMUFile *f, uint64_t flags)
-{
-    int ret = 0;
-
-    if (f->hooks && f->hooks->before_ram_iterate) {
-        ret = f->hooks->before_ram_iterate(f, flags, NULL);
-        if (ret < 0) {
-            qemu_file_set_error(f, ret);
-        }
-    }
-}
-
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags)
 {
     int ret = 0;
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index d16cd50448..c898c5c537 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -56,7 +56,6 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f,
                                  uint64_t *bytes_sent);
 
 typedef struct QEMUFileHooks {
-    QEMURamHookFunc *before_ram_iterate;
     QEMURamHookFunc *after_ram_iterate;
     QEMURamHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
@@ -150,7 +149,6 @@ void qemu_fflush(QEMUFile *f);
 void qemu_file_set_blocking(QEMUFile *f, bool block);
 int qemu_file_get_to_fd(QEMUFile *f, int fd, size_t size);
 
-void ram_control_before_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data);
 
diff --git a/migration/ram.c b/migration/ram.c
index 7d81c4a39e..ce5dfc3c86 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -58,6 +58,7 @@
 #include "qemu/iov.h"
 #include "multifd.h"
 #include "sysemu/runstate.h"
+#include "rdma.h"
 #include "options.h"
 
 #include "hw/boards.h" /* for machine_dump_guest_core() */
@@ -3277,7 +3278,10 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
         }
     }
 
-    ram_control_before_iterate(f, RAM_CONTROL_SETUP);
+    ret = qemu_rdma_registration_start(f, RAM_CONTROL_SETUP);
+    if (ret < 0) {
+        qemu_file_set_error(f, ret);
+    }
     ram_control_after_iterate(f, RAM_CONTROL_SETUP);
 
     migration_ops = g_malloc0(sizeof(MigrationOps));
@@ -3337,7 +3341,10 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
         /* Read version before ram_list.blocks */
         smp_rmb();
 
-        ram_control_before_iterate(f, RAM_CONTROL_ROUND);
+        ret = qemu_rdma_registration_start(f, RAM_CONTROL_ROUND);
+        if (ret < 0) {
+            qemu_file_set_error(f, ret);
+        }
 
         t0 = qemu_clock_get_ns(QEMU_CLOCK_REALTIME);
         i = 0;
@@ -3442,7 +3449,10 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
             migration_bitmap_sync_precopy(rs);
         }
 
-        ram_control_before_iterate(f, RAM_CONTROL_FINISH);
+        ret = qemu_rdma_registration_start(f, RAM_CONTROL_FINISH);
+        if (ret < 0) {
+            qemu_file_set_error(f, ret);
+        }
 
         /* try transferring iterative blocks of memory */
 
diff --git a/migration/rdma.c b/migration/rdma.c
index 7e747b2595..56b7f6901e 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3860,8 +3860,7 @@ static int rdma_load_hook(QEMUFile *f, uint64_t flags, void *data)
     }
 }
 
-static int qemu_rdma_registration_start(QEMUFile *f,
-                                        uint64_t flags, void *data)
+int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags)
 {
     QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(qemu_file_get_ioc(f));
     RDMAContext *rdma;
@@ -4004,7 +4003,6 @@ static const QEMUFileHooks rdma_read_hooks = {
 };
 
 static const QEMUFileHooks rdma_write_hooks = {
-    .before_ram_iterate = qemu_rdma_registration_start,
     .after_ram_iterate  = qemu_rdma_registration_stop,
     .save_page          = qemu_rdma_save_page,
 };
diff --git a/migration/rdma.h b/migration/rdma.h
index de2ba09dc5..901c829c8b 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -22,4 +22,10 @@ void rdma_start_outgoing_migration(void *opaque, const char *host_port,
 
 void rdma_start_incoming_migration(const char *host_port, Error **errp);
 
+
+#ifdef CONFIG_RDMA
+int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags);
+#else
+int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags) { return 0; }
+#endif
 #endif
-- 
2.40.0

[PATCH 08/19] migration/rdma: Unflod ram_control_after_iterate()

[Juan Quintela]

Once there:
- Remove unused data parameter
- unfold it in its callers
- change all callers to call qemu_rdma_registration_stop()

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/qemu-file.c | 12 ------------
 migration/qemu-file.h |  2 --
 migration/ram.c       | 17 ++++++++++++++---
 migration/rdma.c      |  4 +---
 migration/rdma.h      |  2 ++
 5 files changed, 17 insertions(+), 20 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index b6dca23706..22af45a5db 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -315,18 +315,6 @@ void qemu_fflush(QEMUFile *f)
     f->iovcnt = 0;
 }
 
-void ram_control_after_iterate(QEMUFile *f, uint64_t flags)
-{
-    int ret = 0;
-
-    if (f->hooks && f->hooks->after_ram_iterate) {
-        ret = f->hooks->after_ram_iterate(f, flags, NULL);
-        if (ret < 0) {
-            qemu_file_set_error(f, ret);
-        }
-    }
-}
-
 void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data)
 {
     int ret = -EINVAL;
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index c898c5c537..fac26d7869 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -56,7 +56,6 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f,
                                  uint64_t *bytes_sent);
 
 typedef struct QEMUFileHooks {
-    QEMURamHookFunc *after_ram_iterate;
     QEMURamHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
 } QEMUFileHooks;
@@ -149,7 +148,6 @@ void qemu_fflush(QEMUFile *f);
 void qemu_file_set_blocking(QEMUFile *f, bool block);
 int qemu_file_get_to_fd(QEMUFile *f, int fd, size_t size);
 
-void ram_control_after_iterate(QEMUFile *f, uint64_t flags);
 void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data);
 
 /* Whenever this is found in the data stream, the flags
diff --git a/migration/ram.c b/migration/ram.c
index ce5dfc3c86..a5109a0f77 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -3282,7 +3282,11 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
     if (ret < 0) {
         qemu_file_set_error(f, ret);
     }
-    ram_control_after_iterate(f, RAM_CONTROL_SETUP);
+
+    ret = qemu_rdma_registration_stop(f, RAM_CONTROL_SETUP);
+    if (ret < 0) {
+        qemu_file_set_error(f, ret);
+    }
 
     migration_ops = g_malloc0(sizeof(MigrationOps));
     migration_ops->ram_save_target_page = ram_save_target_page_legacy;
@@ -3401,7 +3405,10 @@ static int ram_save_iterate(QEMUFile *f, void *opaque)
      * Must occur before EOS (or any QEMUFile operation)
      * because of RDMA protocol.
      */
-    ram_control_after_iterate(f, RAM_CONTROL_ROUND);
+    ret = qemu_rdma_registration_stop(f, RAM_CONTROL_ROUND);
+    if (ret < 0) {
+        qemu_file_set_error(f, ret);
+    }
 
 out:
     if (ret >= 0
@@ -3474,7 +3481,11 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
         qemu_mutex_unlock(&rs->bitmap_mutex);
 
         flush_compressed_data(rs);
-        ram_control_after_iterate(f, RAM_CONTROL_FINISH);
+
+        int ret = qemu_rdma_registration_stop(f, RAM_CONTROL_FINISH);
+        if (ret < 0) {
+            qemu_file_set_error(f, ret);
+        }
     }
 
     if (ret < 0) {
diff --git a/migration/rdma.c b/migration/rdma.c
index 56b7f6901e..77bbe3da36 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3888,8 +3888,7 @@ int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags)
  * Inform dest that dynamic registrations are done for now.
  * First, flush writes, if any.
  */
-static int qemu_rdma_registration_stop(QEMUFile *f,
-                                       uint64_t flags, void *data)
+int qemu_rdma_registration_stop(QEMUFile *f, uint64_t flags)
 {
     QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(qemu_file_get_ioc(f));
     RDMAContext *rdma;
@@ -4003,7 +4002,6 @@ static const QEMUFileHooks rdma_read_hooks = {
 };
 
 static const QEMUFileHooks rdma_write_hooks = {
-    .after_ram_iterate  = qemu_rdma_registration_stop,
     .save_page          = qemu_rdma_save_page,
 };
 
diff --git a/migration/rdma.h b/migration/rdma.h
index 901c829c8b..a16a8d8bc6 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -25,7 +25,9 @@ void rdma_start_incoming_migration(const char *host_port, Error **errp);
 
 #ifdef CONFIG_RDMA
 int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags);
+int qemu_rdma_registration_stop(QEMUFile *f, uint64_t flags);
 #else
 int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags) { return 0; }
+int qemu_rdma_registration_stop(QEMUFile *f, uint64_t flags) { return 0; }
 #endif
 #endif
-- 
2.40.0

[PATCH 09/19] migration/rdma: simplify ram_control_load_hook()

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/qemu-file.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 22af45a5db..9b5e14a2ef 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -317,10 +317,8 @@ void qemu_fflush(QEMUFile *f)
 
 void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data)
 {
-    int ret = -EINVAL;
-
     if (f->hooks && f->hooks->hook_ram_load) {
-        ret = f->hooks->hook_ram_load(f, flags, data);
+        int ret = f->hooks->hook_ram_load(f, flags, data);
         if (ret < 0) {
             qemu_file_set_error(f, ret);
         }
@@ -330,7 +328,7 @@ void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data)
          * that expects there to be a hook on the destination.
          */
         if (flags == RAM_CONTROL_HOOK) {
-            qemu_file_set_error(f, ret);
+            qemu_file_set_error(f, -EINVAL);
         }
     }
 }
-- 
2.40.0

[PATCH 10/19] migration/rdma: Don't pass the QIOChannelRDMA as an opaque

[Juan Quintela]

We can calculate it from the QEMUFile like the caller.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/rdma.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 77bbe3da36..40b8e2da51 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3527,7 +3527,7 @@ static int dest_ram_sort_func(const void *a, const void *b)
  *
  * Keep doing this until the source tells us to stop.
  */
-static int qemu_rdma_registration_handle(QEMUFile *f, void *opaque)
+static int qemu_rdma_registration_handle(QEMUFile *f)
 {
     RDMAControlHeader reg_resp = { .len = sizeof(RDMARegisterResult),
                                .type = RDMA_CONTROL_REGISTER_RESULT,
@@ -3539,7 +3539,7 @@ static int qemu_rdma_registration_handle(QEMUFile *f, void *opaque)
                              };
     RDMAControlHeader blocks = { .type = RDMA_CONTROL_RAM_BLOCKS_RESULT,
                                  .repeat = 1 };
-    QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(opaque);
+    QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(qemu_file_get_ioc(f));
     RDMAContext *rdma;
     RDMALocalBlocks *local;
     RDMAControlHeader head;
@@ -3852,7 +3852,7 @@ static int rdma_load_hook(QEMUFile *f, uint64_t flags, void *data)
         return rdma_block_notification_handle(rioc, data);
 
     case RAM_CONTROL_HOOK:
-        return qemu_rdma_registration_handle(f, rioc);
+        return qemu_rdma_registration_handle(f);
 
     default:
         /* Shouldn't be called with any other values */
-- 
2.40.0

[PATCH 11/19] migration/rdma: We can calculate the rioc from the QEMUFile

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/rdma.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/migration/rdma.c b/migration/rdma.c
index 40b8e2da51..d7fdc963fd 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3811,9 +3811,10 @@ out:
  * the source.
  */
 static int
-rdma_block_notification_handle(QIOChannelRDMA *rioc, const char *name)
+rdma_block_notification_handle(QEMUFile *f, const char *name)
 {
     RDMAContext *rdma;
+    QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(qemu_file_get_ioc(f));
     int curr;
     int found = -1;
 
@@ -3846,10 +3847,9 @@ rdma_block_notification_handle(QIOChannelRDMA *rioc, const char *name)
 
 static int rdma_load_hook(QEMUFile *f, uint64_t flags, void *data)
 {
-    QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(qemu_file_get_ioc(f));
     switch (flags) {
     case RAM_CONTROL_BLOCK_REG:
-        return rdma_block_notification_handle(rioc, data);
+        return rdma_block_notification_handle(f, data);
 
     case RAM_CONTROL_HOOK:
         return qemu_rdma_registration_handle(f);
-- 
2.40.0

[PATCH 12/19] migration/rdma: It makes no sense to recive that flag without RDMA

[Juan Quintela]

This could only happen if the source send
RAM_SAVE_FLAG_HOOK (i.e. rdma) and destination don't have CONFIG_RDMA.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/qemu-file.c | 8 --------
 1 file changed, 8 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 9b5e14a2ef..014db96984 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -322,14 +322,6 @@ void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data)
         if (ret < 0) {
             qemu_file_set_error(f, ret);
         }
-    } else {
-        /*
-         * Hook is a hook specifically requested by the source sending a flag
-         * that expects there to be a hook on the destination.
-         */
-        if (flags == RAM_CONTROL_HOOK) {
-            qemu_file_set_error(f, -EINVAL);
-        }
     }
 }
 
-- 
2.40.0

[PATCH 13/19] migration: Make RAM_SAVE_FLAG_HOOK a normal case entry

[Juan Quintela]

Fixes this commit, clearly a bad merge after a rebase or similar, it
should have been its own case since that point.

commit 5b0e9dd46fbda5152566a4a26fd96bc0d0452bf7
Author: Peter Lieven <pl@kamp.de>
Date:   Tue Jun 24 11:32:36 2014 +0200

    migration: catch unknown flag combinations in ram_load

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/ram.c | 12 +++++-------
 1 file changed, 5 insertions(+), 7 deletions(-)

diff --git a/migration/ram.c b/migration/ram.c
index a5109a0f77..67ba2d7f7e 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -4466,14 +4466,12 @@ static int ram_load_precopy(QEMUFile *f)
                 multifd_recv_sync_main();
             }
             break;
+        case RAM_SAVE_FLAG_HOOK:
+            ram_control_load_hook(f, RAM_CONTROL_HOOK, NULL);
+            break;
         default:
-            if (flags & RAM_SAVE_FLAG_HOOK) {
-                ram_control_load_hook(f, RAM_CONTROL_HOOK, NULL);
-            } else {
-                error_report("Unknown combination of migration flags: 0x%x",
-                             flags);
-                ret = -EINVAL;
-            }
+            error_report("Unknown combination of migration flags: 0x%x", flags);
+            ret = -EINVAL;
         }
         if (!ret) {
             ret = qemu_file_get_error(f);
-- 
2.40.0

[PATCH 14/19] migration/rdma: Remove all uses of RAM_CONTROL_HOOK

[Juan Quintela]

Instead of going trhough ram_control_load_hook(), call
qemu_rdma_registration_handle() directly.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/qemu-file.h | 1 -
 migration/ram.c       | 5 ++++-
 migration/rdma.c      | 5 +----
 migration/rdma.h      | 2 ++
 4 files changed, 7 insertions(+), 6 deletions(-)

diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index fac26d7869..7623e3c475 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -41,7 +41,6 @@ typedef int (QEMURamHookFunc)(QEMUFile *f, uint64_t flags, void *data);
  */
 #define RAM_CONTROL_SETUP     0
 #define RAM_CONTROL_ROUND     1
-#define RAM_CONTROL_HOOK      2
 #define RAM_CONTROL_FINISH    3
 #define RAM_CONTROL_BLOCK_REG 4
 
diff --git a/migration/ram.c b/migration/ram.c
index 67ba2d7f7e..c0110ca8cb 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -4467,7 +4467,10 @@ static int ram_load_precopy(QEMUFile *f)
             }
             break;
         case RAM_SAVE_FLAG_HOOK:
-            ram_control_load_hook(f, RAM_CONTROL_HOOK, NULL);
+            ret = qemu_rdma_registration_handle(f);
+            if (ret < 0) {
+                qemu_file_set_error(f, ret);
+            }
             break;
         default:
             error_report("Unknown combination of migration flags: 0x%x", flags);
diff --git a/migration/rdma.c b/migration/rdma.c
index d7fdc963fd..e72584483a 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3527,7 +3527,7 @@ static int dest_ram_sort_func(const void *a, const void *b)
  *
  * Keep doing this until the source tells us to stop.
  */
-static int qemu_rdma_registration_handle(QEMUFile *f)
+int qemu_rdma_registration_handle(QEMUFile *f)
 {
     RDMAControlHeader reg_resp = { .len = sizeof(RDMARegisterResult),
                                .type = RDMA_CONTROL_REGISTER_RESULT,
@@ -3851,9 +3851,6 @@ static int rdma_load_hook(QEMUFile *f, uint64_t flags, void *data)
     case RAM_CONTROL_BLOCK_REG:
         return rdma_block_notification_handle(f, data);
 
-    case RAM_CONTROL_HOOK:
-        return qemu_rdma_registration_handle(f);
-
     default:
         /* Shouldn't be called with any other values */
         abort();
diff --git a/migration/rdma.h b/migration/rdma.h
index a16a8d8bc6..8d0253047c 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -24,9 +24,11 @@ void rdma_start_incoming_migration(const char *host_port, Error **errp);
 
 
 #ifdef CONFIG_RDMA
+int qemu_rdma_registration_handle(QEMUFile *f);
 int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags);
 int qemu_rdma_registration_stop(QEMUFile *f, uint64_t flags);
 #else
+int qemu_rdma_registration_handle(QEMUFile *f) { return 0; }
 int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags) { return 0; }
 int qemu_rdma_registration_stop(QEMUFile *f, uint64_t flags) { return 0; }
 #endif
-- 
2.40.0

[PATCH 15/19] migration/rdma: Unfold hook_ram_load()

[Juan Quintela]

There is only one flag called with: RAM_CONTROL_BLOCK_REG.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/qemu-file.c | 10 ----------
 migration/qemu-file.h | 11 -----------
 migration/ram.c       |  6 ++++--
 migration/rdma.c      | 25 +++++--------------------
 migration/rdma.h      |  2 ++
 5 files changed, 11 insertions(+), 43 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 014db96984..9d86900efe 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -315,16 +315,6 @@ void qemu_fflush(QEMUFile *f)
     f->iovcnt = 0;
 }
 
-void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data)
-{
-    if (f->hooks && f->hooks->hook_ram_load) {
-        int ret = f->hooks->hook_ram_load(f, flags, data);
-        if (ret < 0) {
-            qemu_file_set_error(f, ret);
-        }
-    }
-}
-
 size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
                              ram_addr_t offset, size_t size,
                              uint64_t *bytes_sent)
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index 7623e3c475..d69f5d65e8 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -29,20 +29,12 @@
 #include "exec/cpu-common.h"
 #include "io/channel.h"
 
-/*
- * This function provides hooks around different
- * stages of RAM migration.
- * 'data' is call specific data associated with the 'flags' value
- */
-typedef int (QEMURamHookFunc)(QEMUFile *f, uint64_t flags, void *data);
-
 /*
  * Constants used by ram_control_* hooks
  */
 #define RAM_CONTROL_SETUP     0
 #define RAM_CONTROL_ROUND     1
 #define RAM_CONTROL_FINISH    3
-#define RAM_CONTROL_BLOCK_REG 4
 
 /*
  * This function allows override of where the RAM page
@@ -55,7 +47,6 @@ typedef size_t (QEMURamSaveFunc)(QEMUFile *f,
                                  uint64_t *bytes_sent);
 
 typedef struct QEMUFileHooks {
-    QEMURamHookFunc *hook_ram_load;
     QEMURamSaveFunc *save_page;
 } QEMUFileHooks;
 
@@ -147,8 +138,6 @@ void qemu_fflush(QEMUFile *f);
 void qemu_file_set_blocking(QEMUFile *f, bool block);
 int qemu_file_get_to_fd(QEMUFile *f, int fd, size_t size);
 
-void ram_control_load_hook(QEMUFile *f, uint64_t flags, void *data);
-
 /* Whenever this is found in the data stream, the flags
  * will be passed to ram_control_load_hook in the incoming-migration
  * side. This lets before_ram_iterate/after_ram_iterate add
diff --git a/migration/ram.c b/migration/ram.c
index c0110ca8cb..d29dd67d5f 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -4418,8 +4418,10 @@ static int ram_load_precopy(QEMUFile *f)
                             ret = -EINVAL;
                         }
                     }
-                    ram_control_load_hook(f, RAM_CONTROL_BLOCK_REG,
-                                          block->idstr);
+                    ret = rdma_block_notification_handle(f, block->idstr);
+                    if (ret < 0) {
+                        qemu_file_set_error(f, ret);
+                    }
                 } else {
                     error_report("Unknown ramblock \"%s\", cannot "
                                  "accept migration", id);
diff --git a/migration/rdma.c b/migration/rdma.c
index e72584483a..72ea08ee95 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3804,14 +3804,12 @@ out:
 }
 
 /* Destination:
- * Called via a ram_control_load_hook during the initial RAM load section which
- * lists the RAMBlocks by name.  This lets us know the order of the RAMBlocks
- * on the source.
- * We've already built our local RAMBlock list, but not yet sent the list to
- * the source.
+ * Called during the initial RAM load section which lists the
+ * RAMBlocks by name.  This lets us know the order of the RAMBlocks on
+ * the source.  We've already built our local RAMBlock list, but not
+ * yet sent the list to the source.
  */
-static int
-rdma_block_notification_handle(QEMUFile *f, const char *name)
+int rdma_block_notification_handle(QEMUFile *f, const char *name)
 {
     RDMAContext *rdma;
     QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(qemu_file_get_ioc(f));
@@ -3845,18 +3843,6 @@ rdma_block_notification_handle(QEMUFile *f, const char *name)
     return 0;
 }
 
-static int rdma_load_hook(QEMUFile *f, uint64_t flags, void *data)
-{
-    switch (flags) {
-    case RAM_CONTROL_BLOCK_REG:
-        return rdma_block_notification_handle(f, data);
-
-    default:
-        /* Shouldn't be called with any other values */
-        abort();
-    }
-}
-
 int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags)
 {
     QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(qemu_file_get_ioc(f));
@@ -3995,7 +3981,6 @@ err:
 }
 
 static const QEMUFileHooks rdma_read_hooks = {
-    .hook_ram_load = rdma_load_hook,
 };
 
 static const QEMUFileHooks rdma_write_hooks = {
diff --git a/migration/rdma.h b/migration/rdma.h
index 8d0253047c..1266a90e07 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -27,9 +27,11 @@ void rdma_start_incoming_migration(const char *host_port, Error **errp);
 int qemu_rdma_registration_handle(QEMUFile *f);
 int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags);
 int qemu_rdma_registration_stop(QEMUFile *f, uint64_t flags);
+int rdma_block_notification_handle(QEMUFile *f, const char *name);
 #else
 int qemu_rdma_registration_handle(QEMUFile *f) { return 0; }
 int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags) { return 0; }
 int qemu_rdma_registration_stop(QEMUFile *f, uint64_t flags) { return 0; }
+int rdma_block_notification_handle(QEMUFile *f, const char *name) { return 0; }
 #endif
 #endif
-- 
2.40.0

[PATCH 16/19] migration/rdma: Make ram_control_save_page() use exported interfaces

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/qemu-file.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 9d86900efe..17b3c2ea21 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -323,7 +323,7 @@ size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
         int ret = f->hooks->save_page(f, block_offset,
                                       offset, size, bytes_sent);
         if (ret != RAM_SAVE_CONTROL_NOT_SUPP) {
-            f->rate_limit_used += size;
+            qemu_file_acct_rate_limit(f, size);
         }
 
         if (ret != RAM_SAVE_CONTROL_DELAYED &&
-- 
2.40.0

[PATCH 17/19] migration/rdma: Create rdma_control_save_page()

[Juan Quintela]

The only user of ram_control_save_page() and save_page() hook was
rdma. Just move the function to rdma.c, rename it to
rdma_control_save_page().

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/qemu-file.c | 26 --------------------------
 migration/qemu-file.h | 14 --------------
 migration/ram.c       |  4 ++--
 migration/rdma.c      | 22 +++++++++++++++++++++-
 migration/rdma.h      |  8 ++++++++
 5 files changed, 31 insertions(+), 43 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 17b3c2ea21..8d3f33fe41 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -315,32 +315,6 @@ void qemu_fflush(QEMUFile *f)
     f->iovcnt = 0;
 }
 
-size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
-                             ram_addr_t offset, size_t size,
-                             uint64_t *bytes_sent)
-{
-    if (f->hooks && f->hooks->save_page) {
-        int ret = f->hooks->save_page(f, block_offset,
-                                      offset, size, bytes_sent);
-        if (ret != RAM_SAVE_CONTROL_NOT_SUPP) {
-            qemu_file_acct_rate_limit(f, size);
-        }
-
-        if (ret != RAM_SAVE_CONTROL_DELAYED &&
-            ret != RAM_SAVE_CONTROL_NOT_SUPP) {
-            if (bytes_sent && *bytes_sent > 0) {
-                qemu_file_credit_transfer(f, *bytes_sent);
-            } else if (ret < 0) {
-                qemu_file_set_error(f, ret);
-            }
-        }
-
-        return ret;
-    }
-
-    return RAM_SAVE_CONTROL_NOT_SUPP;
-}
-
 /*
  * Attempt to fill the buffer from the underlying file
  * Returns the number of bytes read, or negative value for an error.
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index d69f5d65e8..ae3a704772 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -36,18 +36,7 @@
 #define RAM_CONTROL_ROUND     1
 #define RAM_CONTROL_FINISH    3
 
-/*
- * This function allows override of where the RAM page
- * is saved (such as RDMA, for example.)
- */
-typedef size_t (QEMURamSaveFunc)(QEMUFile *f,
-                                 ram_addr_t block_offset,
-                                 ram_addr_t offset,
-                                 size_t size,
-                                 uint64_t *bytes_sent);
-
 typedef struct QEMUFileHooks {
-    QEMURamSaveFunc *save_page;
 } QEMUFileHooks;
 
 QEMUFile *qemu_file_new_input(QIOChannel *ioc);
@@ -148,9 +137,6 @@ int qemu_file_get_to_fd(QEMUFile *f, int fd, size_t size);
 #define RAM_SAVE_CONTROL_NOT_SUPP -1000
 #define RAM_SAVE_CONTROL_DELAYED  -2000
 
-size_t ram_control_save_page(QEMUFile *f, ram_addr_t block_offset,
-                             ram_addr_t offset, size_t size,
-                             uint64_t *bytes_sent);
 QIOChannel *qemu_file_get_ioc(QEMUFile *file);
 
 #endif
diff --git a/migration/ram.c b/migration/ram.c
index d29dd67d5f..a085ce8cae 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -1342,8 +1342,8 @@ static bool control_save_page(PageSearchStatus *pss, RAMBlock *block,
     int ret;
 
     *pages = -1;
-    ret = ram_control_save_page(pss->pss_channel, block->offset, offset,
-                                TARGET_PAGE_SIZE, &bytes_xmit);
+    ret = rdma_control_save_page(pss->pss_channel, block->offset, offset,
+                                 TARGET_PAGE_SIZE, &bytes_xmit);
     if (ret == RAM_SAVE_CONTROL_NOT_SUPP) {
         return false;
     }
diff --git a/migration/rdma.c b/migration/rdma.c
index 72ea08ee95..42ac7fd1e2 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -3315,6 +3315,27 @@ err:
     return ret;
 }
 
+size_t rdma_control_save_page(QEMUFile *f, ram_addr_t block_offset,
+                              ram_addr_t offset, size_t size,
+                              uint64_t *bytes_sent)
+{
+    int ret = qemu_rdma_save_page(f, block_offset, offset, size, bytes_sent);
+    if (ret != RAM_SAVE_CONTROL_NOT_SUPP) {
+        qemu_file_acct_rate_limit(f, size);
+    }
+
+    if (ret != RAM_SAVE_CONTROL_DELAYED &&
+        ret != RAM_SAVE_CONTROL_NOT_SUPP) {
+        if (bytes_sent && *bytes_sent > 0) {
+            qemu_file_credit_transfer(f, *bytes_sent);
+        } else if (ret < 0) {
+            qemu_file_set_error(f, ret);
+        }
+    }
+    return ret;
+}
+
+
 static void rdma_accept_incoming_migration(void *opaque);
 
 static void rdma_cm_poll_handler(void *opaque)
@@ -3984,7 +4005,6 @@ static const QEMUFileHooks rdma_read_hooks = {
 };
 
 static const QEMUFileHooks rdma_write_hooks = {
-    .save_page          = qemu_rdma_save_page,
 };
 
 
diff --git a/migration/rdma.h b/migration/rdma.h
index 1266a90e07..ed3650ef67 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -17,6 +17,8 @@
 #ifndef QEMU_MIGRATION_RDMA_H
 #define QEMU_MIGRATION_RDMA_H
 
+#include "exec/memory.h"
+
 void rdma_start_outgoing_migration(void *opaque, const char *host_port,
                                    Error **errp);
 
@@ -28,10 +30,16 @@ int qemu_rdma_registration_handle(QEMUFile *f);
 int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags);
 int qemu_rdma_registration_stop(QEMUFile *f, uint64_t flags);
 int rdma_block_notification_handle(QEMUFile *f, const char *name);
+size_t rdma_control_save_page(QEMUFile *f, ram_addr_t block_offset,
+                              ram_addr_t offset, size_t size,
+                              uint64_t *bytes_sent);
 #else
 int qemu_rdma_registration_handle(QEMUFile *f) { return 0; }
 int qemu_rdma_registration_start(QEMUFile *f, uint64_t flags) { return 0; }
 int qemu_rdma_registration_stop(QEMUFile *f, uint64_t flags) { return 0; }
 int rdma_block_notification_handle(QEMUFile *f, const char *name) { return 0; }
+size_t rdma_control_save_page(QEMUFile *f, ram_addr_t block_offset,
+                              ram_addr_t offset, size_t size,
+                              uint64_t *bytes_sent) { return false; }
 #endif
 #endif
-- 
2.40.0

[PATCH 18/19] qemu-file: Remove QEMUFileHooks

[Juan Quintela]

The only user was rdma, and its use is gone.

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/qemu-file.c | 6 ------
 migration/qemu-file.h | 4 ----
 migration/rdma.c      | 9 ---------
 3 files changed, 19 deletions(-)

diff --git a/migration/qemu-file.c b/migration/qemu-file.c
index 8d3f33fe41..c0c2195a6e 100644
--- a/migration/qemu-file.c
+++ b/migration/qemu-file.c
@@ -36,7 +36,6 @@
 #define MAX_IOV_SIZE MIN_CONST(IOV_MAX, 64)
 
 struct QEMUFile {
-    const QEMUFileHooks *hooks;
     QIOChannel *ioc;
     bool is_writable;
 
@@ -160,11 +159,6 @@ QEMUFile *qemu_file_new_input(QIOChannel *ioc)
     return qemu_file_new_impl(ioc, false);
 }
 
-void qemu_file_set_hooks(QEMUFile *f, const QEMUFileHooks *hooks)
-{
-    f->hooks = hooks;
-}
-
 /*
  * Get last error for stream f with optional Error*
  *
diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index ae3a704772..9c99914b21 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -36,12 +36,8 @@
 #define RAM_CONTROL_ROUND     1
 #define RAM_CONTROL_FINISH    3
 
-typedef struct QEMUFileHooks {
-} QEMUFileHooks;
-
 QEMUFile *qemu_file_new_input(QIOChannel *ioc);
 QEMUFile *qemu_file_new_output(QIOChannel *ioc);
-void qemu_file_set_hooks(QEMUFile *f, const QEMUFileHooks *hooks);
 int qemu_fclose(QEMUFile *f);
 
 /*
diff --git a/migration/rdma.c b/migration/rdma.c
index 42ac7fd1e2..63492fd4ac 100644
--- a/migration/rdma.c
+++ b/migration/rdma.c
@@ -4001,13 +4001,6 @@ err:
     return ret;
 }
 
-static const QEMUFileHooks rdma_read_hooks = {
-};
-
-static const QEMUFileHooks rdma_write_hooks = {
-};
-
-
 static void qio_channel_rdma_finalize(Object *obj)
 {
     QIOChannelRDMA *rioc = QIO_CHANNEL_RDMA(obj);
@@ -4066,12 +4059,10 @@ static QEMUFile *qemu_fopen_rdma(RDMAContext *rdma, const char *mode)
         rioc->file = qemu_file_new_output(QIO_CHANNEL(rioc));
         rioc->rdmaout = rdma;
         rioc->rdmain = rdma->return_path;
-        qemu_file_set_hooks(rioc->file, &rdma_write_hooks);
     } else {
         rioc->file = qemu_file_new_input(QIO_CHANNEL(rioc));
         rioc->rdmain = rdma;
         rioc->rdmaout = rdma->return_path;
-        qemu_file_set_hooks(rioc->file, &rdma_read_hooks);
     }
 
     return rioc->file;
-- 
2.40.0

[PATCH 19/19] migration/rdma: Move rdma constants from qemu-file.h to rdma.h

[Juan Quintela]

Signed-off-by: Juan Quintela <quintela@redhat.com>
---
 migration/qemu-file.h | 17 -----------------
 migration/ram.c       |  2 +-
 migration/rdma.h      | 16 ++++++++++++++++
 3 files changed, 17 insertions(+), 18 deletions(-)

diff --git a/migration/qemu-file.h b/migration/qemu-file.h
index 9c99914b21..5129b6f196 100644
--- a/migration/qemu-file.h
+++ b/migration/qemu-file.h
@@ -29,13 +29,6 @@
 #include "exec/cpu-common.h"
 #include "io/channel.h"
 
-/*
- * Constants used by ram_control_* hooks
- */
-#define RAM_CONTROL_SETUP     0
-#define RAM_CONTROL_ROUND     1
-#define RAM_CONTROL_FINISH    3
-
 QEMUFile *qemu_file_new_input(QIOChannel *ioc);
 QEMUFile *qemu_file_new_output(QIOChannel *ioc);
 int qemu_fclose(QEMUFile *f);
@@ -123,16 +116,6 @@ void qemu_fflush(QEMUFile *f);
 void qemu_file_set_blocking(QEMUFile *f, bool block);
 int qemu_file_get_to_fd(QEMUFile *f, int fd, size_t size);
 
-/* Whenever this is found in the data stream, the flags
- * will be passed to ram_control_load_hook in the incoming-migration
- * side. This lets before_ram_iterate/after_ram_iterate add
- * transport-specific sections to the RAM migration data.
- */
-#define RAM_SAVE_FLAG_HOOK     0x80
-
-#define RAM_SAVE_CONTROL_NOT_SUPP -1000
-#define RAM_SAVE_CONTROL_DELAYED  -2000
-
 QIOChannel *qemu_file_get_ioc(QEMUFile *file);
 
 #endif
diff --git a/migration/ram.c b/migration/ram.c
index a085ce8cae..ac2296d740 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -86,7 +86,7 @@
 #define RAM_SAVE_FLAG_EOS      0x10
 #define RAM_SAVE_FLAG_CONTINUE 0x20
 #define RAM_SAVE_FLAG_XBZRLE   0x40
-/* 0x80 is reserved in qemu-file.h for RAM_SAVE_FLAG_HOOK */
+/* 0x80 is reserved in rdma.h for RAM_SAVE_FLAG_HOOK */
 #define RAM_SAVE_FLAG_COMPRESS_PAGE    0x100
 #define RAM_SAVE_FLAG_MULTIFD_FLUSH    0x200
 /* We can't use any flag that is bigger than 0x200 */
diff --git a/migration/rdma.h b/migration/rdma.h
index ed3650ef67..96ec2cc8f0 100644
--- a/migration/rdma.h
+++ b/migration/rdma.h
@@ -24,6 +24,22 @@ void rdma_start_outgoing_migration(void *opaque, const char *host_port,
 
 void rdma_start_incoming_migration(const char *host_port, Error **errp);
 
+/*
+ * Constants used by rdma return codes
+ */
+#define RAM_CONTROL_SETUP     0
+#define RAM_CONTROL_ROUND     1
+#define RAM_CONTROL_FINISH    3
+
+/*
+ * Whenever this is found in the data stream, the flags
+ * will be passed to rdma functions in the incoming-migration
+ * side.
+ */
+#define RAM_SAVE_FLAG_HOOK     0x80
+
+#define RAM_SAVE_CONTROL_NOT_SUPP -1000
+#define RAM_SAVE_CONTROL_DELAYED  -2000
 
 #ifdef CONFIG_RDMA
 int qemu_rdma_registration_handle(QEMUFile *f);
-- 
2.40.0

Re: [PATCH 01/19] multifd: We already account for this packet on the multifd thread

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 972 bytes --]

On Thu, 27 Apr 2023 18:34:31 +0200
Juan Quintela <quintela@redhat.com> wrote:

> Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Lukas Straub <lukasstraub2@web.de>

> ---
>  migration/multifd.c | 3 ---
>  1 file changed, 3 deletions(-)
> 
> diff --git a/migration/multifd.c b/migration/multifd.c
> index 6a59c03dd2..6053012ad9 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -626,10 +626,7 @@ int multifd_send_sync_main(QEMUFile *f)
>          p->packet_num = multifd_send_state->packet_num++;
>          p->flags |= MULTIFD_FLAG_SYNC;
>          p->pending_job++;
> -        qemu_file_acct_rate_limit(f, p->packet_len);
>          qemu_mutex_unlock(&p->mutex);
> -        stat64_add(&ram_counters.transferred, p->packet_len);
> -        stat64_add(&ram_counters.multifd_bytes, p->packet_len);
>          qemu_sem_post(&p->sem);
>      }
>      for (i = 0; i < migrate_multifd_channels(); i++) {



-- 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 02/19] migration: Move ram_stats to its own file migration-stats.[ch]

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 6125 bytes --]

On Thu, 27 Apr 2023 18:34:32 +0200
Juan Quintela <quintela@redhat.com> wrote:

> There is already include/qemu/stats.h, so stats.h was a bad idea.
> We want this file to not depend on anything else, we will move all the
> migration counters/stats to this struct.
> 
> Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Lukas Straub <lukasstraub2@web.de>

> ---
>  migration/meson.build       |  1 +
>  migration/migration-stats.c | 17 +++++++++++++++
>  migration/migration-stats.h | 41 +++++++++++++++++++++++++++++++++++++
>  migration/migration.c       |  1 +
>  migration/multifd.c         |  1 +
>  migration/ram.c             |  3 +--
>  migration/ram.h             | 23 ---------------------
>  migration/savevm.c          |  1 +
>  8 files changed, 63 insertions(+), 25 deletions(-)
>  create mode 100644 migration/migration-stats.c
>  create mode 100644 migration/migration-stats.h
> 
> diff --git a/migration/meson.build b/migration/meson.build
> index 480ff6854a..da1897fadf 100644
> --- a/migration/meson.build
> +++ b/migration/meson.build
> @@ -19,6 +19,7 @@ softmmu_ss.add(files(
>    'fd.c',
>    'global_state.c',
>    'migration-hmp-cmds.c',
> +  'migration-stats.c',
>    'migration.c',
>    'multifd.c',
>    'multifd-zlib.c',
> diff --git a/migration/migration-stats.c b/migration/migration-stats.c
> new file mode 100644
> index 0000000000..b0eb5ae73c
> --- /dev/null
> +++ b/migration/migration-stats.c
> @@ -0,0 +1,17 @@
> +/*
> + * Migration stats
> + *
> + * Copyright (c) 2012-2023 Red Hat Inc
> + *
> + * Authors:
> + *  Juan Quintela <quintela@redhat.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#include "qemu/osdep.h"
> +#include "qemu/stats64.h"
> +#include "migration-stats.h"
> +
> +RAMStats ram_counters;
> diff --git a/migration/migration-stats.h b/migration/migration-stats.h
> new file mode 100644
> index 0000000000..2edea0c779
> --- /dev/null
> +++ b/migration/migration-stats.h
> @@ -0,0 +1,41 @@
> +/*
> + * Migration stats
> + *
> + * Copyright (c) 2012-2023 Red Hat Inc
> + *
> + * Authors:
> + *  Juan Quintela <quintela@redhat.com>
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or later.
> + * See the COPYING file in the top-level directory.
> + */
> +
> +#ifndef QEMU_MIGRATION_STATS_H
> +#define QEMU_MIGRATION_STATS_H
> +
> +#include "qemu/stats64.h"
> +
> +/*
> + * These are the ram migration statistic counters.  It is loosely
> + * based on MigrationStats.  We change to Stat64 any counter that
> + * needs to be updated using atomic ops (can be accessed by more than
> + * one thread).
> + */
> +typedef struct {
> +    Stat64 dirty_bytes_last_sync;
> +    Stat64 dirty_pages_rate;
> +    Stat64 dirty_sync_count;
> +    Stat64 dirty_sync_missed_zero_copy;
> +    Stat64 downtime_bytes;
> +    Stat64 zero_pages;
> +    Stat64 multifd_bytes;
> +    Stat64 normal_pages;
> +    Stat64 postcopy_bytes;
> +    Stat64 postcopy_requests;
> +    Stat64 precopy_bytes;
> +    Stat64 transferred;
> +} RAMStats;
> +
> +extern RAMStats ram_counters;
> +
> +#endif
> diff --git a/migration/migration.c b/migration/migration.c
> index abcadbb619..5ecf3dc381 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -29,6 +29,7 @@
>  #include "migration/global_state.h"
>  #include "migration/misc.h"
>  #include "migration.h"
> +#include "migration-stats.h"
>  #include "savevm.h"
>  #include "qemu-file.h"
>  #include "channel.h"
> diff --git a/migration/multifd.c b/migration/multifd.c
> index 6053012ad9..347999f84a 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -19,6 +19,7 @@
>  #include "qapi/error.h"
>  #include "ram.h"
>  #include "migration.h"
> +#include "migration-stats.h"
>  #include "socket.h"
>  #include "tls.h"
>  #include "qemu-file.h"
> diff --git a/migration/ram.c b/migration/ram.c
> index 89be3e3320..a6d5478ef8 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -36,6 +36,7 @@
>  #include "xbzrle.h"
>  #include "ram.h"
>  #include "migration.h"
> +#include "migration-stats.h"
>  #include "migration/register.h"
>  #include "migration/misc.h"
>  #include "qemu-file.h"
> @@ -460,8 +461,6 @@ uint64_t ram_bytes_remaining(void)
>                         0;
>  }
>  
> -RAMStats ram_counters;
> -
>  void ram_transferred_add(uint64_t bytes)
>  {
>      if (runstate_is_running()) {
> diff --git a/migration/ram.h b/migration/ram.h
> index 04b05e1b2c..8692de6ba0 100644
> --- a/migration/ram.h
> +++ b/migration/ram.h
> @@ -32,30 +32,7 @@
>  #include "qapi/qapi-types-migration.h"
>  #include "exec/cpu-common.h"
>  #include "io/channel.h"
> -#include "qemu/stats64.h"
>  
> -/*
> - * These are the ram migration statistic counters.  It is loosely
> - * based on MigrationStats.  We change to Stat64 any counter that
> - * needs to be updated using atomic ops (can be accessed by more than
> - * one thread).
> - */
> -typedef struct {
> -    Stat64 dirty_bytes_last_sync;
> -    Stat64 dirty_pages_rate;
> -    Stat64 dirty_sync_count;
> -    Stat64 dirty_sync_missed_zero_copy;
> -    Stat64 downtime_bytes;
> -    Stat64 zero_pages;
> -    Stat64 multifd_bytes;
> -    Stat64 normal_pages;
> -    Stat64 postcopy_bytes;
> -    Stat64 postcopy_requests;
> -    Stat64 precopy_bytes;
> -    Stat64 transferred;
> -} RAMStats;
> -
> -extern RAMStats ram_counters;
>  extern XBZRLECacheStats xbzrle_counters;
>  extern CompressionStats compression_counters;
>  
> diff --git a/migration/savevm.c b/migration/savevm.c
> index a9181b444b..8e2efb1a19 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -31,6 +31,7 @@
>  #include "net/net.h"
>  #include "migration.h"
>  #include "migration/snapshot.h"
> +#include "migration-stats.h"
>  #include "migration/vmstate.h"
>  #include "migration/misc.h"
>  #include "migration/register.h"



-- 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 03/19] migration: Rename ram_counters to mig_stats

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 14461 bytes --]

On Thu, 27 Apr 2023 18:34:33 +0200
Juan Quintela <quintela@redhat.com> wrote:

> migration_stats is just too long, and it is going to have more than
> ram counters in the near future.
> 
> Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Lukas Straub <lukasstraub2@web.de>

> ---
>  migration/migration-stats.c |  2 +-
>  migration/migration-stats.h |  2 +-
>  migration/migration.c       | 32 ++++++++++++-------------
>  migration/multifd.c         |  6 ++---
>  migration/ram.c             | 48 ++++++++++++++++++-------------------
>  migration/savevm.c          |  2 +-
>  6 files changed, 46 insertions(+), 46 deletions(-)
> 
> diff --git a/migration/migration-stats.c b/migration/migration-stats.c
> index b0eb5ae73c..8c0af9b80a 100644
> --- a/migration/migration-stats.c
> +++ b/migration/migration-stats.c
> @@ -14,4 +14,4 @@
>  #include "qemu/stats64.h"
>  #include "migration-stats.h"
>  
> -RAMStats ram_counters;
> +RAMStats mig_stats;
> diff --git a/migration/migration-stats.h b/migration/migration-stats.h
> index 2edea0c779..197374b4f6 100644
> --- a/migration/migration-stats.h
> +++ b/migration/migration-stats.h
> @@ -36,6 +36,6 @@ typedef struct {
>      Stat64 transferred;
>  } RAMStats;
>  
> -extern RAMStats ram_counters;
> +extern RAMStats mig_stats;
>  
>  #endif
> diff --git a/migration/migration.c b/migration/migration.c
> index 5ecf3dc381..feb5ab7493 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -909,26 +909,26 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
>      size_t page_size = qemu_target_page_size();
>  
>      info->ram = g_malloc0(sizeof(*info->ram));
> -    info->ram->transferred = stat64_get(&ram_counters.transferred);
> +    info->ram->transferred = stat64_get(&mig_stats.transferred);
>      info->ram->total = ram_bytes_total();
> -    info->ram->duplicate = stat64_get(&ram_counters.zero_pages);
> +    info->ram->duplicate = stat64_get(&mig_stats.zero_pages);
>      /* legacy value.  It is not used anymore */
>      info->ram->skipped = 0;
> -    info->ram->normal = stat64_get(&ram_counters.normal_pages);
> +    info->ram->normal = stat64_get(&mig_stats.normal_pages);
>      info->ram->normal_bytes = info->ram->normal * page_size;
>      info->ram->mbps = s->mbps;
>      info->ram->dirty_sync_count =
> -        stat64_get(&ram_counters.dirty_sync_count);
> +        stat64_get(&mig_stats.dirty_sync_count);
>      info->ram->dirty_sync_missed_zero_copy =
> -        stat64_get(&ram_counters.dirty_sync_missed_zero_copy);
> +        stat64_get(&mig_stats.dirty_sync_missed_zero_copy);
>      info->ram->postcopy_requests =
> -        stat64_get(&ram_counters.postcopy_requests);
> +        stat64_get(&mig_stats.postcopy_requests);
>      info->ram->page_size = page_size;
> -    info->ram->multifd_bytes = stat64_get(&ram_counters.multifd_bytes);
> +    info->ram->multifd_bytes = stat64_get(&mig_stats.multifd_bytes);
>      info->ram->pages_per_second = s->pages_per_second;
> -    info->ram->precopy_bytes = stat64_get(&ram_counters.precopy_bytes);
> -    info->ram->downtime_bytes = stat64_get(&ram_counters.downtime_bytes);
> -    info->ram->postcopy_bytes = stat64_get(&ram_counters.postcopy_bytes);
> +    info->ram->precopy_bytes = stat64_get(&mig_stats.precopy_bytes);
> +    info->ram->downtime_bytes = stat64_get(&mig_stats.downtime_bytes);
> +    info->ram->postcopy_bytes = stat64_get(&mig_stats.postcopy_bytes);
>  
>      if (migrate_xbzrle()) {
>          info->xbzrle_cache = g_malloc0(sizeof(*info->xbzrle_cache));
> @@ -960,7 +960,7 @@ static void populate_ram_info(MigrationInfo *info, MigrationState *s)
>      if (s->state != MIGRATION_STATUS_COMPLETED) {
>          info->ram->remaining = ram_bytes_remaining();
>          info->ram->dirty_pages_rate =
> -           stat64_get(&ram_counters.dirty_pages_rate);
> +           stat64_get(&mig_stats.dirty_pages_rate);
>      }
>  }
>  
> @@ -1613,10 +1613,10 @@ static bool migrate_prepare(MigrationState *s, bool blk, bool blk_inc,
>  
>      migrate_init(s);
>      /*
> -     * set ram_counters compression_counters memory to zero for a
> +     * set mig_stats compression_counters memory to zero for a
>       * new migration
>       */
> -    memset(&ram_counters, 0, sizeof(ram_counters));
> +    memset(&mig_stats, 0, sizeof(mig_stats));
>      memset(&compression_counters, 0, sizeof(compression_counters));
>  
>      return true;
> @@ -2627,7 +2627,7 @@ static MigThrError migration_detect_error(MigrationState *s)
>  static uint64_t migration_total_bytes(MigrationState *s)
>  {
>      return qemu_file_total_transferred(s->to_dst_file) +
> -        stat64_get(&ram_counters.multifd_bytes);
> +        stat64_get(&mig_stats.multifd_bytes);
>  }
>  
>  static void migration_calculate_complete(MigrationState *s)
> @@ -2691,10 +2691,10 @@ static void migration_update_counters(MigrationState *s,
>       * if we haven't sent anything, we don't want to
>       * recalculate. 10000 is a small enough number for our purposes
>       */
> -    if (stat64_get(&ram_counters.dirty_pages_rate) &&
> +    if (stat64_get(&mig_stats.dirty_pages_rate) &&
>          transferred > 10000) {
>          s->expected_downtime =
> -            stat64_get(&ram_counters.dirty_bytes_last_sync) / bandwidth;
> +            stat64_get(&mig_stats.dirty_bytes_last_sync) / bandwidth;
>      }
>  
>      qemu_file_reset_rate_limit(s->to_dst_file);
> diff --git a/migration/multifd.c b/migration/multifd.c
> index 347999f84a..4a2e1a47ce 100644
> --- a/migration/multifd.c
> +++ b/migration/multifd.c
> @@ -434,8 +434,8 @@ static int multifd_send_pages(QEMUFile *f)
>      transferred = ((uint64_t) pages->num) * p->page_size + p->packet_len;
>      qemu_file_acct_rate_limit(f, transferred);
>      qemu_mutex_unlock(&p->mutex);
> -    stat64_add(&ram_counters.transferred, transferred);
> -    stat64_add(&ram_counters.multifd_bytes, transferred);
> +    stat64_add(&mig_stats.transferred, transferred);
> +    stat64_add(&mig_stats.multifd_bytes, transferred);
>      qemu_sem_post(&p->sem);
>  
>      return 1;
> @@ -577,7 +577,7 @@ static int multifd_zero_copy_flush(QIOChannel *c)
>          return -1;
>      }
>      if (ret == 1) {
> -        stat64_add(&ram_counters.dirty_sync_missed_zero_copy, 1);
> +        stat64_add(&mig_stats.dirty_sync_missed_zero_copy, 1);
>      }
>  
>      return ret;
> diff --git a/migration/ram.c b/migration/ram.c
> index a6d5478ef8..c3981f64e4 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -464,13 +464,13 @@ uint64_t ram_bytes_remaining(void)
>  void ram_transferred_add(uint64_t bytes)
>  {
>      if (runstate_is_running()) {
> -        stat64_add(&ram_counters.precopy_bytes, bytes);
> +        stat64_add(&mig_stats.precopy_bytes, bytes);
>      } else if (migration_in_postcopy()) {
> -        stat64_add(&ram_counters.postcopy_bytes, bytes);
> +        stat64_add(&mig_stats.postcopy_bytes, bytes);
>      } else {
> -        stat64_add(&ram_counters.downtime_bytes, bytes);
> +        stat64_add(&mig_stats.downtime_bytes, bytes);
>      }
> -    stat64_add(&ram_counters.transferred, bytes);
> +    stat64_add(&mig_stats.transferred, bytes);
>  }
>  
>  struct MigrationOps {
> @@ -744,7 +744,7 @@ void mig_throttle_counter_reset(void)
>  
>      rs->time_last_bitmap_sync = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
>      rs->num_dirty_pages_period = 0;
> -    rs->bytes_xfer_prev = stat64_get(&ram_counters.transferred);
> +    rs->bytes_xfer_prev = stat64_get(&mig_stats.transferred);
>  }
>  
>  /**
> @@ -764,7 +764,7 @@ static void xbzrle_cache_zero_page(RAMState *rs, ram_addr_t current_addr)
>      /* We don't care if this fails to allocate a new cache page
>       * as long as it updated an old one */
>      cache_insert(XBZRLE.cache, current_addr, XBZRLE.zero_target_page,
> -                 stat64_get(&ram_counters.dirty_sync_count));
> +                 stat64_get(&mig_stats.dirty_sync_count));
>  }
>  
>  #define ENCODING_FLAG_XBZRLE 0x1
> @@ -790,7 +790,7 @@ static int save_xbzrle_page(RAMState *rs, PageSearchStatus *pss,
>      int encoded_len = 0, bytes_xbzrle;
>      uint8_t *prev_cached_page;
>      QEMUFile *file = pss->pss_channel;
> -    uint64_t generation = stat64_get(&ram_counters.dirty_sync_count);
> +    uint64_t generation = stat64_get(&mig_stats.dirty_sync_count);
>  
>      if (!cache_is_cached(XBZRLE.cache, current_addr, generation)) {
>          xbzrle_counters.cache_miss++;
> @@ -1118,8 +1118,8 @@ uint64_t ram_pagesize_summary(void)
>  
>  uint64_t ram_get_total_transferred_pages(void)
>  {
> -    return stat64_get(&ram_counters.normal_pages) +
> -        stat64_get(&ram_counters.zero_pages) +
> +    return stat64_get(&mig_stats.normal_pages) +
> +        stat64_get(&mig_stats.zero_pages) +
>          compression_counters.pages + xbzrle_counters.pages;
>  }
>  
> @@ -1129,7 +1129,7 @@ static void migration_update_rates(RAMState *rs, int64_t end_time)
>      double compressed_size;
>  
>      /* calculate period counters */
> -    stat64_set(&ram_counters.dirty_pages_rate,
> +    stat64_set(&mig_stats.dirty_pages_rate,
>                 rs->num_dirty_pages_period * 1000 /
>                 (end_time - rs->time_last_bitmap_sync));
>  
> @@ -1180,7 +1180,7 @@ static void migration_trigger_throttle(RAMState *rs)
>  {
>      uint64_t threshold = migrate_throttle_trigger_threshold();
>      uint64_t bytes_xfer_period =
> -        stat64_get(&ram_counters.transferred) - rs->bytes_xfer_prev;
> +        stat64_get(&mig_stats.transferred) - rs->bytes_xfer_prev;
>      uint64_t bytes_dirty_period = rs->num_dirty_pages_period * TARGET_PAGE_SIZE;
>      uint64_t bytes_dirty_threshold = bytes_xfer_period * threshold / 100;
>  
> @@ -1209,7 +1209,7 @@ static void migration_bitmap_sync(RAMState *rs)
>      RAMBlock *block;
>      int64_t end_time;
>  
> -    stat64_add(&ram_counters.dirty_sync_count, 1);
> +    stat64_add(&mig_stats.dirty_sync_count, 1);
>  
>      if (!rs->time_last_bitmap_sync) {
>          rs->time_last_bitmap_sync = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
> @@ -1223,7 +1223,7 @@ static void migration_bitmap_sync(RAMState *rs)
>          RAMBLOCK_FOREACH_NOT_IGNORED(block) {
>              ramblock_sync_dirty_bitmap(rs, block);
>          }
> -        stat64_set(&ram_counters.dirty_bytes_last_sync, ram_bytes_remaining());
> +        stat64_set(&mig_stats.dirty_bytes_last_sync, ram_bytes_remaining());
>      }
>      qemu_mutex_unlock(&rs->bitmap_mutex);
>  
> @@ -1243,10 +1243,10 @@ static void migration_bitmap_sync(RAMState *rs)
>          /* reset period counters */
>          rs->time_last_bitmap_sync = end_time;
>          rs->num_dirty_pages_period = 0;
> -        rs->bytes_xfer_prev = stat64_get(&ram_counters.transferred);
> +        rs->bytes_xfer_prev = stat64_get(&mig_stats.transferred);
>      }
>      if (migrate_events()) {
> -        uint64_t generation = stat64_get(&ram_counters.dirty_sync_count);
> +        uint64_t generation = stat64_get(&mig_stats.dirty_sync_count);
>          qapi_event_send_migration_pass(generation);
>      }
>  }
> @@ -1320,7 +1320,7 @@ static int save_zero_page(PageSearchStatus *pss, QEMUFile *f, RAMBlock *block,
>      int len = save_zero_page_to_file(pss, f, block, offset);
>  
>      if (len) {
> -        stat64_add(&ram_counters.zero_pages, 1);
> +        stat64_add(&mig_stats.zero_pages, 1);
>          ram_transferred_add(len);
>          return 1;
>      }
> @@ -1357,9 +1357,9 @@ static bool control_save_page(PageSearchStatus *pss, RAMBlock *block,
>      }
>  
>      if (bytes_xmit > 0) {
> -        stat64_add(&ram_counters.normal_pages, 1);
> +        stat64_add(&mig_stats.normal_pages, 1);
>      } else if (bytes_xmit == 0) {
> -        stat64_add(&ram_counters.zero_pages, 1);
> +        stat64_add(&mig_stats.zero_pages, 1);
>      }
>  
>      return true;
> @@ -1391,7 +1391,7 @@ static int save_normal_page(PageSearchStatus *pss, RAMBlock *block,
>          qemu_put_buffer(file, buf, TARGET_PAGE_SIZE);
>      }
>      ram_transferred_add(TARGET_PAGE_SIZE);
> -    stat64_add(&ram_counters.normal_pages, 1);
> +    stat64_add(&mig_stats.normal_pages, 1);
>      return 1;
>  }
>  
> @@ -1447,7 +1447,7 @@ static int ram_save_multifd_page(QEMUFile *file, RAMBlock *block,
>      if (multifd_queue_page(file, block, offset) < 0) {
>          return -1;
>      }
> -    stat64_add(&ram_counters.normal_pages, 1);
> +    stat64_add(&mig_stats.normal_pages, 1);
>  
>      return 1;
>  }
> @@ -1486,7 +1486,7 @@ update_compress_thread_counts(const CompressParam *param, int bytes_xmit)
>      ram_transferred_add(bytes_xmit);
>  
>      if (param->zero_page) {
> -        stat64_add(&ram_counters.zero_pages, 1);
> +        stat64_add(&mig_stats.zero_pages, 1);
>          return;
>      }
>  
> @@ -2179,7 +2179,7 @@ int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len)
>      RAMBlock *ramblock;
>      RAMState *rs = ram_state;
>  
> -    stat64_add(&ram_counters.postcopy_requests, 1);
> +    stat64_add(&mig_stats.postcopy_requests, 1);
>      RCU_READ_LOCK_GUARD();
>  
>      if (!rbname) {
> @@ -2634,9 +2634,9 @@ void acct_update_position(QEMUFile *f, size_t size, bool zero)
>      uint64_t pages = size / TARGET_PAGE_SIZE;
>  
>      if (zero) {
> -        stat64_add(&ram_counters.zero_pages, pages);
> +        stat64_add(&mig_stats.zero_pages, pages);
>      } else {
> -        stat64_add(&ram_counters.normal_pages, pages);
> +        stat64_add(&mig_stats.normal_pages, pages);
>          ram_transferred_add(size);
>          qemu_file_credit_transfer(f, size);
>      }
> diff --git a/migration/savevm.c b/migration/savevm.c
> index 8e2efb1a19..a9d0a88e62 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -1622,7 +1622,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
>      }
>  
>      migrate_init(ms);
> -    memset(&ram_counters, 0, sizeof(ram_counters));
> +    memset(&mig_stats, 0, sizeof(mig_stats));
>      memset(&compression_counters, 0, sizeof(compression_counters));
>      ms->to_dst_file = f;
>  



-- 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 04/19] migration: Rename RAMStats to MigrationAtomicStats

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 1326 bytes --]

On Thu, 27 Apr 2023 18:34:34 +0200
Juan Quintela <quintela@redhat.com> wrote:

> It is lousely based on MigrationStats, but that name is taken, so this
> is the best one that I came with.
> 
> Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Lukas Straub <lukasstraub2@web.de>
> ---
> 
> If you have any good suggestion for the name, I am all ears.
> ---
>  migration/migration-stats.c | 2 +-
>  migration/migration-stats.h | 4 ++--
>  2 files changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/migration/migration-stats.c b/migration/migration-stats.c
> index 8c0af9b80a..2f2cea965c 100644
> --- a/migration/migration-stats.c
> +++ b/migration/migration-stats.c
> @@ -14,4 +14,4 @@
>  #include "qemu/stats64.h"
>  #include "migration-stats.h"
>  
> -RAMStats mig_stats;
> +MigrationAtomicStats mig_stats;
> diff --git a/migration/migration-stats.h b/migration/migration-stats.h
> index 197374b4f6..149af932d7 100644
> --- a/migration/migration-stats.h
> +++ b/migration/migration-stats.h
> @@ -34,8 +34,8 @@ typedef struct {
>      Stat64 postcopy_requests;
>      Stat64 precopy_bytes;
>      Stat64 transferred;
> -} RAMStats;
> +} MigrationAtomicStats;
>  
> -extern RAMStats mig_stats;
> +extern MigrationAtomicStats mig_stats;
>  
>  #endif



-- 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 05/19] migration/rdma: Split the zero page case from acct_update_position

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 3012 bytes --]

On Thu, 27 Apr 2023 18:34:35 +0200
Juan Quintela <quintela@redhat.com> wrote:

> Now that we have atomic counters, we can do it on the place that we
> need it, no need to do it inside ram.c.
> 
> Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Lukas Straub <lukasstraub2@web.de>

> ---
>  migration/ram.c  | 12 ++++--------
>  migration/ram.h  |  2 +-
>  migration/rdma.c |  7 +++++--
>  3 files changed, 10 insertions(+), 11 deletions(-)
> 
> diff --git a/migration/ram.c b/migration/ram.c
> index c3981f64e4..c249a1f468 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -2629,17 +2629,13 @@ static int ram_find_and_save_block(RAMState *rs)
>      return pages;
>  }
>  
> -void acct_update_position(QEMUFile *f, size_t size, bool zero)
> +void acct_update_position(QEMUFile *f, size_t size)
>  {
>      uint64_t pages = size / TARGET_PAGE_SIZE;
>  
> -    if (zero) {
> -        stat64_add(&mig_stats.zero_pages, pages);
> -    } else {
> -        stat64_add(&mig_stats.normal_pages, pages);
> -        ram_transferred_add(size);
> -        qemu_file_credit_transfer(f, size);
> -    }
> +    stat64_add(&mig_stats.normal_pages, pages);
> +    ram_transferred_add(size);
> +    qemu_file_credit_transfer(f, size);
>  }
>  
>  static uint64_t ram_bytes_total_with_ignored(void)
> diff --git a/migration/ram.h b/migration/ram.h
> index 8692de6ba0..3804753ca3 100644
> --- a/migration/ram.h
> +++ b/migration/ram.h
> @@ -53,7 +53,7 @@ void mig_throttle_counter_reset(void);
>  
>  uint64_t ram_pagesize_summary(void);
>  int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len);
> -void acct_update_position(QEMUFile *f, size_t size, bool zero);
> +void acct_update_position(QEMUFile *f, size_t size);
>  void ram_postcopy_migrated_memory_release(MigrationState *ms);
>  /* For outgoing discard bitmap */
>  void ram_postcopy_send_discard_bitmap(MigrationState *ms);
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 0af5e944f0..7a9b284c3f 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -17,8 +17,10 @@
>  #include "qemu/osdep.h"
>  #include "qapi/error.h"
>  #include "qemu/cutils.h"
> +#include "exec/target_page.h"
>  #include "rdma.h"
>  #include "migration.h"
> +#include "migration-stats.h"
>  #include "qemu-file.h"
>  #include "ram.h"
>  #include "qemu/error-report.h"
> @@ -2120,7 +2122,8 @@ retry:
>                      return -EIO;
>                  }
>  
> -                acct_update_position(f, sge.length, true);
> +                stat64_add(&mig_stats.zero_pages,
> +                           sge.length / qemu_target_page_size());
>  
>                  return 1;
>              }
> @@ -2228,7 +2231,7 @@ retry:
>      }
>  
>      set_bit(chunk, block->transit_bitmap);
> -    acct_update_position(f, sge.length, false);
> +    acct_update_position(f, sge.length);
>      rdma->total_writes++;
>  
>      return 0;



-- 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 06/19] migration/rdma: Unfold last user of acct_update_position()

[Lukas Straub]

[-- Attachment #1: Type: text/plain, Size: 2014 bytes --]

On Thu, 27 Apr 2023 18:34:36 +0200
Juan Quintela <quintela@redhat.com> wrote:

> Signed-off-by: Juan Quintela <quintela@redhat.com>

Reviewed-by: Lukas Straub <lukasstraub2@web.de>

> ---
>  migration/ram.c  | 9 ---------
>  migration/ram.h  | 1 -
>  migration/rdma.c | 4 +++-
>  3 files changed, 3 insertions(+), 11 deletions(-)
> 
> diff --git a/migration/ram.c b/migration/ram.c
> index c249a1f468..7d81c4a39e 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -2629,15 +2629,6 @@ static int ram_find_and_save_block(RAMState *rs)
>      return pages;
>  }
>  
> -void acct_update_position(QEMUFile *f, size_t size)
> -{
> -    uint64_t pages = size / TARGET_PAGE_SIZE;
> -
> -    stat64_add(&mig_stats.normal_pages, pages);
> -    ram_transferred_add(size);
> -    qemu_file_credit_transfer(f, size);
> -}
> -
>  static uint64_t ram_bytes_total_with_ignored(void)
>  {
>      RAMBlock *block;
> diff --git a/migration/ram.h b/migration/ram.h
> index 3804753ca3..6fffbeb5f1 100644
> --- a/migration/ram.h
> +++ b/migration/ram.h
> @@ -53,7 +53,6 @@ void mig_throttle_counter_reset(void);
>  
>  uint64_t ram_pagesize_summary(void);
>  int ram_save_queue_pages(const char *rbname, ram_addr_t start, ram_addr_t len);
> -void acct_update_position(QEMUFile *f, size_t size);
>  void ram_postcopy_migrated_memory_release(MigrationState *ms);
>  /* For outgoing discard bitmap */
>  void ram_postcopy_send_discard_bitmap(MigrationState *ms);
> diff --git a/migration/rdma.c b/migration/rdma.c
> index 7a9b284c3f..7e747b2595 100644
> --- a/migration/rdma.c
> +++ b/migration/rdma.c
> @@ -2231,7 +2231,9 @@ retry:
>      }
>  
>      set_bit(chunk, block->transit_bitmap);
> -    acct_update_position(f, sge.length);
> +    stat64_add(&mig_stats.normal_pages, sge.length / qemu_target_page_size());
> +    ram_transferred_add(sge.length);
> +    qemu_file_credit_transfer(f, sge.length);
>      rdma->total_writes++;
>  
>      return 0;



-- 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH 07/19] migration/rdma: Unflod ram_control_before_iterate()

[Juan Quintela]

Juan Quintela <quintela@redhat.com> wrote:
> Once there:
> - Remove unused data parameter
> - unfold it in its callers.
> - change all callers to call qemu_rdma_registration_start()
>
> Signed-off-by: Juan Quintela <quintela@redhat.com>

self-Nack from here.

I just break the case when there is CONFIG_RDMA but it is not being
used.

Later, Juan.