The issues about architecture of the COLO checkpoint

The issues about architecture of the COLO checkpoint

[Daniel Cho]

[-- Attachment #1: Type: text/plain, Size: 712 bytes --]

Hi everyone,
     We have some issues about setting COLO feature. Hope somebody could
give us some advice.

Issue 1:
     We dynamic to set COLO feature for PVM(2 core, 16G memory),  but the
Primary VM will pause a long time(based on memory size) for waiting SVM
start. Does it have any idea to reduce the pause time?


Issue 2:
     In https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
could we move start_vm() before Line 488? Because at first checkpoint PVM
will wait for SVM's reply, it cause PVM stop for a while.

     We set the COLO feature on running VM, so we hope the running VM could
continuous service for users.
Do you have any suggestions for those issues?

Best regards,
Daniel Cho

[-- Attachment #2: Type: text/html, Size: 1062 bytes --]

Re: The issues about architecture of the COLO checkpoint

[Dr. David Alan Gilbert]

cc'ing in COLO people:

* Daniel Cho (danielcho@qnap.com) wrote:
> Hi everyone,
>      We have some issues about setting COLO feature. Hope somebody could
> give us some advice.
> 
> Issue 1:
>      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but the
> Primary VM will pause a long time(based on memory size) for waiting SVM
> start. Does it have any idea to reduce the pause time?
> 
> 
> Issue 2:
>      In https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> could we move start_vm() before Line 488? Because at first checkpoint PVM
> will wait for SVM's reply, it cause PVM stop for a while.
> 
>      We set the COLO feature on running VM, so we hope the running VM could
> continuous service for users.
> Do you have any suggestions for those issues?
> 
> Best regards,
> Daniel Cho
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

RE: The issues about architecture of the COLO checkpoint

[Zhanghailiang]

[-- Attachment #1: Type: text/plain, Size: 3167 bytes --]

Hi,

Thank you Dave,

I'll reply here directly.

-----Original Message-----
From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com] 
Sent: Wednesday, February 12, 2020 1:48 AM
To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com; Zhanghailiang <zhang.zhanghailiang@huawei.com>
Cc: qemu-devel@nongnu.org
Subject: Re: The issues about architecture of the COLO checkpoint


cc'ing in COLO people:


* Daniel Cho (danielcho@qnap.com) wrote:
> Hi everyone,
>      We have some issues about setting COLO feature. Hope somebody 
> could give us some advice.
> 
> Issue 1:
>      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but 
> the Primary VM will pause a long time(based on memory size) for 
> waiting SVM start. Does it have any idea to reduce the pause time?
> 

Yes, we do have some ideas to optimize this downtime.

The main problem for current version is, for each checkpoint, we have to send the whole PVM's pages
To SVM, and then copy the whole VM's state into SVM from ram cache, in this process, we need both of them be paused. 
Just as you said, the downtime is based on memory size. 

So firstly, we need to reduce the sending data while do checkpoint, actually, we can migrate parts of PVM's dirty pages in background
While both of VMs are running. And then we load these pages into ram cache (backup memory) in SVM temporarily. While do checkpoint,
We just send the last dirty pages of PVM to slave side and then copy the ram cache into SVM. Further on, we don't have
To send the whole PVM's dirty pages, we can only send the pages that dirtied by PVM or SVM during two checkpoints. (Because
If one page is not dirtied by both PVM and SVM, the data of this pages will keep same in SVM, PVM, backup memory). This method can reduce
the time that consumed in sending data.

For the second problem, we can reduce the memory copy by two methods, first one, we don't have to copy the whole pages in ram cache,
We can only copy the pages that dirtied by PVM and SVM in last checkpoint. Second, we can use userfault missing function to reduce the
Time consumed in memory copy. (For the second time, in theory, we can reduce time consumed in memory into ms level).

You can find the first optimization in attachment, it is based on an old qemu version (qemu-2.6), it should not be difficult to rebase it
Into master or your version. And please feel free to send the new version if you want into community ;)


> 
> Issue 2:
>      In 
> https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> could we move start_vm() before Line 488? Because at first checkpoint 
> PVM will wait for SVM's reply, it cause PVM stop for a while.
> 

No, that makes no sense, because if PVM runs firstly, it still need to wait for
The network packets from SVM to compare before send it to client side.


Thanks,
Hailiang

>      We set the COLO feature on running VM, so we hope the running VM 
> could continuous service for users.
> Do you have any suggestions for those issues?
> 
> Best regards,
> Daniel Cho
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK


[-- Attachment #2: 0001-COLO-Migrate-dirty-pages-during-the-gap-of-checkpoin.patch --]
[-- Type: application/octet-stream, Size: 7479 bytes --]

From 02153f7b65f9228cd13041274990b6265ffca71f Mon Sep 17 00:00:00 2001
From: zhanghailiang <zhang.zhanghailiang@huawei.com>
Date: Thu, 30 Mar 2017 10:05:30 +0800
Subject: [PATCH] COLO: Migrate dirty pages during the gap of checkpointing
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

We can migrate some dirty pages during the gap of checkpointing,
by this way, we can reduce the amount of ram migrated during checkpointing.

Change-Id: Iad2d0e972f81f860fad30cda4b816db6c603d992
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
---
 include/migration/migration.h |  1 +
 migration/colo.c              | 72 ++++++++++++++++++++++++++++++++---
 qapi-schema.json              |  5 ++-
 trace-events                  |  1 +
 4 files changed, 72 insertions(+), 7 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 8d82db9e..826c9aa3 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -198,6 +198,7 @@ struct MigrationState
     int64_t colo_checkpoint_time;
     QEMUTimer *colo_delay_timer;
 
+    QEMUTimer *pending_ram_check_timer;
 };
 
 void migrate_set_state(int *state, int old_state, int new_state);
diff --git a/migration/colo.c b/migration/colo.c
index a841dfd0..02886d0a 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -37,6 +37,13 @@ static Notifier packets_compare_notifier;
 static int heartbeat_fd = -1;
 static int heartbeat_listen_fd = -1;
 
+#define DEFAULT_RAM_PENDING_CHECK 1000
+
+/* should be calculated by bandwidth and max downtime ? */
+#define THRESHOLD_PENDING_SIZE (100 * 1024 * 1024UL)
+
+static int checkpoint_request;
+
 bool colo_supported(void)
 {
     return true;
@@ -519,6 +526,21 @@ static void colo_heartbeat_init(Error **errp)
     }
 }
 
+static bool colo_need_migrate_ram_background(MigrationState *s)
+{
+    uint64_t pending_size;
+    uint64_t pend_post, pend_nonpost;
+    int64_t max_size = THRESHOLD_PENDING_SIZE;
+
+    migration_bitmap_sync_init();
+    qemu_savevm_state_pending(s->to_dst_file, max_size, &pend_nonpost,
+                                      &pend_post);
+    pending_size = pend_post + pend_nonpost;
+
+    trace_colo_need_migrate_ram_background(pending_size);
+    return (pending_size >= max_size);
+}
+
 static void colo_process_checkpoint(MigrationState *s)
 {
     QEMUSizedBuffer *buffer = NULL;
@@ -584,6 +606,8 @@ static void colo_process_checkpoint(MigrationState *s)
 
     timer_mod(s->colo_delay_timer,
         current_time + s->parameters[MIGRATION_PARAMETER_X_CHECKPOINT_DELAY]);
+    timer_mod(s->pending_ram_check_timer,
+        current_time + DEFAULT_RAM_PENDING_CHECK);
 
     while (s->state == MIGRATION_STATUS_COLO) {
         if (failover_request_is_active()) {
@@ -596,11 +620,26 @@ static void colo_process_checkpoint(MigrationState *s)
         if (s->state != MIGRATION_STATUS_COLO) {
             goto out;
         }
-        /* start a colo checkpoint */
-        ret = colo_do_checkpoint_transaction(s, buffer);
-        if (ret < 0) {
-            goto out;
+        if (atomic_xchg(&checkpoint_request, 0)) {
+            /* start a colo checkpoint */
+            ret = colo_do_checkpoint_transaction(s, buffer);
+            if (ret < 0) {
+                goto out;
+            }
+        } else {
+            if (colo_need_migrate_ram_background(s)) {
+                colo_send_message(s->to_dst_file,
+                                  COLO_MESSAGE_MIGRATE_RAM_BACKGROUND,
+                                  &local_err);
+                if (local_err) {
+                    goto out;
+                }
+
+                qemu_savevm_state_iterate(s->to_dst_file, false);
+                qemu_put_byte(s->to_dst_file, QEMU_VM_EOF);
+            }
         }
+
     }
 
 out:
@@ -626,7 +665,6 @@ out:
     }
 
     qsb_free(buffer);
-    buffer = NULL;
 
     /* Hope this not to be too long to wait here */
     qemu_sem_wait(&s->colo_exit_sem);
@@ -635,6 +673,8 @@ out:
     colo_compare_unregister_notifier(&packets_compare_notifier);
     timer_del(s->colo_delay_timer);
     timer_free(s->colo_delay_timer);
+    timer_del(s->pending_ram_check_timer);
+    timer_free(s->pending_ram_check_timer);
     /*
     * Must be called after failover BH is completed,
     * Or the failover BH may shutdown the wrong fd, that
@@ -651,6 +691,7 @@ void colo_checkpoint_notify(void *opaque)
     int64_t next_notify_time;
 
     if (migration_in_colo_state() || failover_request_is_active()) {
+        atomic_inc(&checkpoint_request);
         qemu_sem_post(&s->colo_checkpoint_sem);
         s->colo_checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
         next_notify_time = s->colo_checkpoint_time +
@@ -659,13 +700,27 @@ void colo_checkpoint_notify(void *opaque)
     }
 }
 
+static void colo_pending_ram_check_notify(void *opaque)
+{
+    int64_t next_notify_time;
+    MigrationState *s = opaque;
+
+    if (migration_in_colo_state()) {
+        next_notify_time = DEFAULT_RAM_PENDING_CHECK +
+                           qemu_clock_get_ms(QEMU_CLOCK_HOST);
+        timer_mod(s->pending_ram_check_timer, next_notify_time);
+        qemu_sem_post(&s->colo_checkpoint_sem);
+    }
+}
+
 void migrate_start_colo_process(MigrationState *s)
 {
     qemu_mutex_unlock_iothread();
     qemu_sem_init(&s->colo_checkpoint_sem, 0);
     s->colo_delay_timer =  timer_new_ms(QEMU_CLOCK_HOST,
                                 colo_checkpoint_notify, s);
-
+    s->pending_ram_check_timer =  timer_new_ms(QEMU_CLOCK_HOST,
+                                colo_pending_ram_check_notify, s);
     qemu_sem_init(&s->colo_exit_sem, 0);
     migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
                       MIGRATION_STATUS_COLO);
@@ -819,6 +874,11 @@ static void colo_wait_handle_message(MigrationIncomingState *mis, Error **errp)
         * process, do we need some cleanup?
         */
         qemu_thread_exit(0);
+    case COLO_MESSAGE_MIGRATE_RAM_BACKGROUND:
+        if (qemu_loadvm_state_main(mis->from_src_file, mis) < 0) {
+            error_setg(errp, "Load ram background failed");
+        }
+        break;
     default:
         error_setg(errp, "Got unknown COLO message: %d", msg);
         break;
diff --git a/qapi-schema.json b/qapi-schema.json
index a58965b0..55f4962e 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -756,12 +756,15 @@
 #
 # @colo-stop: Resume VM from COLO state.
 #
+# @migrate-ram-background: Send some dirty pages during the gap of COLO checkpoint
+#
 # Since: 2.7
 ##
 { 'enum': 'COLOMessage',
   'data': [ 'checkpoint-ready', 'checkpoint-request', 'checkpoint-reply',
             'vmstate-send', 'vmstate-size', 'vmstate-received',
-            'vmstate-loaded', 'guest-shutdown', 'colo-stop' ] }
+            'vmstate-loaded', 'guest-shutdown', 'colo-stop',
+            'migrate-ram-background' ] }
 
 ##
 # @COLOMode
diff --git a/trace-events b/trace-events
index d54e1f5d..296aa403 100644
--- a/trace-events
+++ b/trace-events
@@ -1604,6 +1604,7 @@ colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
 colo_send_message(const char *msg) "Send '%s' message"
 colo_receive_message(const char *msg) "Receive '%s' message"
 colo_failover_set_state(int new_state) "new state %d"
+colo_need_migrate_ram_background(uint64_t pending_size) "Pending %" PRIx64 " dirty ram"
 
 # kvm-all.c
 kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
-- 
2.21.0.windows.1


[-- Attachment #3: 0001-COLO-Optimize-memory-back-up-process.patch --]
[-- Type: application/octet-stream, Size: 8986 bytes --]

From 3345680364a3fce2b84c3038f41e3ed4bf60faca Mon Sep 17 00:00:00 2001
From: zhanghailiang <zhang.zhanghailiang@huawei.com>
Date: Wed, 5 Apr 2017 15:47:22 +0800
Subject: [PATCH] COLO: Optimize memory back-up process
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: 8bit

This patch will reduce the downtime of VM for the initial process,
Privously, we copied all these memory in preparing stage of COLO
while we need to stop VM, which is a time-consuming process.
Here we optimize it by a trick, back-up every page while in migration
process while COLO is enabled, though it affects the speed of the
migration, but it obviously reduce the downtime of back-up all SVM'S
memory in COLO preparing stage.

Change-Id: I44a1a6ed2969d5a89a6e0b3e8f16c34986d742c3
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
---
 include/migration/migration.h |  1 +
 migration/colo.c              | 11 +------
 migration/migration.c         |  3 ++
 migration/ram.c               | 61 ++++++++++++++++++++++-------------
 migration/savevm.c            |  6 +++-
 5 files changed, 48 insertions(+), 34 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index 76810bd8..8afb64bc 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -367,6 +367,7 @@ PostcopyState postcopy_state_set(PostcopyState new_state);
 int colo_init_ram_cache(void);
 void colo_release_ram_cache(void);
 void colo_flush_ram_cache(void);
+void colo_incoming_start_dirty_log(void);
 
 void set_check_bitmap_rate(uint64_t time);
 uint64_t get_check_bitmap_rate(void);
diff --git a/migration/colo.c b/migration/colo.c
index 35eb2e6c..9262c568 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -938,11 +938,7 @@ void *colo_process_incoming_thread(void *opaque)
     */
     qemu_file_set_blocking(mis->from_src_file, true);
 
-    ret = colo_init_ram_cache();
-    if (ret < 0) {
-        error_report("Failed to initialize ram cache");
-        goto out;
-    }
+    colo_incoming_start_dirty_log();
 
     buffer = qsb_create(NULL, COLO_BUFFER_BASE_SIZE);
     if (buffer == NULL) {
@@ -996,11 +992,6 @@ out:
     }
 
     qsb_free(buffer);
-    /* Here, we can ensure BH is hold the global lock, and will join colo
-    * incoming thread, so here it is not necessary to lock here again,
-    * or there will be a deadlock error.
-    */
-    colo_release_ram_cache();
 
     /* Hope this not to be too long to loop here */
     qemu_sem_wait(&mis->colo_incoming_sem);
diff --git a/migration/migration.c b/migration/migration.c
index e4ec7a26..eb9aecb0 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -560,6 +560,7 @@ static void process_incoming_migration_co(void *opaque)
             migrate_decompress_threads_join();
             exit(EXIT_FAILURE);
         }
+
         qemu_thread_create(&mis->colo_incoming_thread, "colo incoming",
              colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE);
         mis->have_colo_incoming_thread = true;
@@ -567,6 +568,8 @@ static void process_incoming_migration_co(void *opaque)
 
         /* Wait checkpoint incoming thread exit before free resource */
         qemu_thread_join(&mis->colo_incoming_thread);
+        /* We hold the global iothread lock, so it is safe here */
+        colo_release_ram_cache();
     }
 
     qemu_fclose(f);
diff --git a/migration/ram.c b/migration/ram.c
index fa1c5ab8..da55e566 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -228,7 +228,6 @@ static RAMBlock *last_sent_block;
 static ram_addr_t last_offset;
 static QemuMutex migration_bitmap_mutex;
 static uint64_t migration_dirty_pages;
-static bool ram_cache_enable;
 static uint32_t last_version;
 static bool ram_bulk_stage;
 
@@ -2546,7 +2545,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
 
     while (!postcopy_running && !ret && !(flags & RAM_SAVE_FLAG_EOS)) {
         ram_addr_t addr, total_ram_bytes;
-        void *host = NULL;
+        void *host = NULL, *host_bak = NULL;
         uint8_t ch;
 
         addr = qemu_get_be64(f);
@@ -2557,10 +2556,24 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
                      RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) {
             RAMBlock *block = ram_block_from_stream(f, flags);
 
-            /* After going into COLO, we should load the Page into colo_cache */
-            if (ram_cache_enable) {
+            /*
+             * After going into COLO, we should load the Page into colo_cache
+             * NOTE: We need to keep a copy of SVM's ram in colo_cache.
+             * Privously, we copied all these memory in preparing stage of COLO
+             * while we need to stop VM, which is a time-consuming process.
+             * Here we optimize it by a trick, back-up every page while in
+             * migration process while COLO is enabled, though it affects the
+             * speed of the migration, but it obviously reduce the downtime of
+             * back-up all SVM'S memory in COLO preparing stage.
+             */
+            if (migration_incoming_colo_enabled()) {
                 host = colo_cache_from_block_offset(block, addr);
-            } else {
+                /* After goes into COLO state, don't backup it any more */
+                if (!migration_incoming_in_colo_state()) {
+                    host_bak = host;
+                }
+            }
+            if (!migration_incoming_in_colo_state()) {
                 host = host_from_ram_block_offset(block, addr);
             }
             if (!host) {
@@ -2649,6 +2662,9 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
         if (!ret) {
             ret = qemu_file_get_error(f);
         }
+        if (!ret && host_bak && host) {
+            memcpy(host_bak, host, TARGET_PAGE_SIZE);
+        }
     }
 
     rcu_read_unlock();
@@ -2665,38 +2681,27 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
 int colo_init_ram_cache(void)
 {
     RAMBlock *block;
-    int64_t ram_cache_pages = last_ram_offset() >> TARGET_PAGE_BITS;
 
-    rcu_read_lock();
     QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
         block->colo_cache = qemu_anon_ram_alloc(block->used_length, NULL);
+
         if (!block->colo_cache) {
             error_report("%s: Can't alloc memory for COLO cache of block %s,"
                          "size 0x" RAM_ADDR_FMT, __func__, block->idstr,
                          block->used_length);
             goto out_locked;
         }
+
         memcpy(block->colo_cache, block->host, block->used_length);
     }
-    rcu_read_unlock();
-    ram_cache_enable = true;
     /*
     * Record the dirty pages that sent by PVM, we use this dirty bitmap together
     * with to decide which page in cache should be flushed into SVM's RAM. Here
     * we use the same name 'migration_bitmap_rcu' as for migration.
     */
     migration_bitmap_rcu = g_new0(struct BitmapRcu, 1);
-    migration_bitmap_rcu->bmap = bitmap_new(ram_cache_pages);
-    migration_dirty_pages = 0;
-
-    migration_dirty_pages = ram_bytes_total() >> TARGET_PAGE_BITS;
-    memory_global_dirty_log_start();
-    migration_bitmap_sync();
-    bitmap_zero(migration_bitmap_rcu->bmap,
-                last_ram_offset() >> TARGET_PAGE_BITS);
-    migration_dirty_pages = 0;
-
-    ram_cache_enable = true;
+    migration_bitmap_rcu->bmap =
+        bitmap_new(last_ram_offset() >> TARGET_PAGE_BITS);
 
     return 0;
 
@@ -2708,17 +2713,26 @@ out_locked:
         }
     }
 
-    rcu_read_unlock();
     return -errno;
 }
 
+void colo_incoming_start_dirty_log(void)
+{
+    qemu_mutex_lock_iothread();
+    migration_dirty_pages = ram_bytes_total() >> TARGET_PAGE_BITS;
+    memory_global_dirty_log_start();
+    migration_bitmap_sync();
+    bitmap_zero(migration_bitmap_rcu->bmap,
+            last_ram_offset() >> TARGET_PAGE_BITS);
+    migration_dirty_pages = 0;
+    qemu_mutex_unlock_iothread();
+}
+
 void colo_release_ram_cache(void)
 {
     RAMBlock *block;
     struct BitmapRcu *bitmap = migration_bitmap_rcu;
 
-    ram_cache_enable = false;
-
     atomic_rcu_set(&migration_bitmap_rcu, NULL);
     if (bitmap) {
         memory_global_dirty_log_stop();
@@ -2784,6 +2798,7 @@ void colo_flush_ram_cache(void)
     }
 
     rcu_read_unlock();
+
     if (trace_event_get_state(TRACE_COLO_FLUSH_RAM_CACHE)) {
         end_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
         trace_colo_flush_ram_cache(num_dirty, end_time - start_time);
diff --git a/migration/savevm.c b/migration/savevm.c
index 5f3c8a71..a66acf09 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1636,8 +1636,12 @@ static int loadvm_handle_cmd_packaged(MigrationIncomingState *mis)
 
 static int loadvm_process_enable_colo(MigrationIncomingState *mis)
 {
+    int ret;
+
     migration_incoming_enable_colo();
-    return 0;
+    ret = colo_init_ram_cache();
+
+    return ret;
 }
 
 /*
-- 
2.21.0.windows.1

RE: The issues about architecture of the COLO checkpoint

[Zhang, Chen]

> -----Original Message-----
> From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
> Sent: Wednesday, February 12, 2020 11:18 AM
> To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
> <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
> Cc: qemu-devel@nongnu.org
> Subject: RE: The issues about architecture of the COLO checkpoint
> 
> Hi,
> 
> Thank you Dave,
> 
> I'll reply here directly.
> 
> -----Original Message-----
> From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> Sent: Wednesday, February 12, 2020 1:48 AM
> To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
> Zhanghailiang <zhang.zhanghailiang@huawei.com>
> Cc: qemu-devel@nongnu.org
> Subject: Re: The issues about architecture of the COLO checkpoint
> 
> 
> cc'ing in COLO people:
> 
> 
> * Daniel Cho (danielcho@qnap.com) wrote:
> > Hi everyone,
> >      We have some issues about setting COLO feature. Hope somebody
> > could give us some advice.
> >
> > Issue 1:
> >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > the Primary VM will pause a long time(based on memory size) for
> > waiting SVM start. Does it have any idea to reduce the pause time?
> >
> 
> Yes, we do have some ideas to optimize this downtime.
> 
> The main problem for current version is, for each checkpoint, we have to
> send the whole PVM's pages
> To SVM, and then copy the whole VM's state into SVM from ram cache, in
> this process, we need both of them be paused.
> Just as you said, the downtime is based on memory size.
> 
> So firstly, we need to reduce the sending data while do checkpoint, actually,
> we can migrate parts of PVM's dirty pages in background
> While both of VMs are running. And then we load these pages into ram
> cache (backup memory) in SVM temporarily. While do checkpoint,
> We just send the last dirty pages of PVM to slave side and then copy the ram
> cache into SVM. Further on, we don't have
> To send the whole PVM's dirty pages, we can only send the pages that
> dirtied by PVM or SVM during two checkpoints. (Because
> If one page is not dirtied by both PVM and SVM, the data of this pages will
> keep same in SVM, PVM, backup memory). This method can reduce
> the time that consumed in sending data.
> 
> For the second problem, we can reduce the memory copy by two methods,
> first one, we don't have to copy the whole pages in ram cache,
> We can only copy the pages that dirtied by PVM and SVM in last checkpoint.
> Second, we can use userfault missing function to reduce the
> Time consumed in memory copy. (For the second time, in theory, we can
> reduce time consumed in memory into ms level).
> 
> You can find the first optimization in attachment, it is based on an old qemu
> version (qemu-2.6), it should not be difficult to rebase it
> Into master or your version. And please feel free to send the new version if
> you want into community ;)
> 
> 

Thanks Hailiang!
By the way, Do you have time to push the patches to upstream?
I think this is a better and faster option.

Thanks
Zhang Chen

> >
> > Issue 2:
> >      In
> > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > could we move start_vm() before Line 488? Because at first checkpoint
> > PVM will wait for SVM's reply, it cause PVM stop for a while.
> >
> 
> No, that makes no sense, because if PVM runs firstly, it still need to wait for
> The network packets from SVM to compare before send it to client side.
> 
> 
> Thanks,
> Hailiang
> 
> >      We set the COLO feature on running VM, so we hope the running VM
> > could continuous service for users.
> > Do you have any suggestions for those issues?
> >
> > Best regards,
> > Daniel Cho
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: The issues about architecture of the COLO checkpoint

[Daniel Cho]

[-- Attachment #1: Type: text/plain, Size: 5341 bytes --]

Hi Hailiang,

Thanks for your replaying and explain in detail.
We will try to use the attachments to enhance memory copy.

However, we have some questions for your replying.

1.  As you said, "for each checkpoint, we have to send the whole PVM's
pages To SVM", why the only first checkpoint will takes more pause time?
In our observing, the first checkpoint will take more time for pausing,
then other checkpoints will takes a few time for pausing. Does it means
only the first checkpoint will send the whole pages to SVM, and the other
checkpoints send the dirty pages to SVM for reloading?

2. We notice the COLO-COMPARE component will stuck the packet until
receive packets from PVM and SVM, as this rule, when we add the
COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an
other issue to make PVM stuck while setting COLO feature. With this issue,
could we let colo-compare to pass the PVM's packet when the SVM's packet
queue is empty? Then, the PVM's network won't stock, and "if PVM runs
firstly, it still need to wait for The network packets from SVM to compare
before send it to client side" won't happened either.

Best regard,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com> 於 2020年2月12日 週三 下午1:45寫道：

>
>
> > -----Original Message-----
> > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > Sent: Wednesday, February 12, 2020 11:18 AM
> > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
> > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
> > Cc: qemu-devel@nongnu.org
> > Subject: RE: The issues about architecture of the COLO checkpoint
> >
> > Hi,
> >
> > Thank you Dave,
> >
> > I'll reply here directly.
> >
> > -----Original Message-----
> > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > Sent: Wednesday, February 12, 2020 1:48 AM
> > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
> > Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > Cc: qemu-devel@nongnu.org
> > Subject: Re: The issues about architecture of the COLO checkpoint
> >
> >
> > cc'ing in COLO people:
> >
> >
> > * Daniel Cho (danielcho@qnap.com) wrote:
> > > Hi everyone,
> > >      We have some issues about setting COLO feature. Hope somebody
> > > could give us some advice.
> > >
> > > Issue 1:
> > >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > > the Primary VM will pause a long time(based on memory size) for
> > > waiting SVM start. Does it have any idea to reduce the pause time?
> > >
> >
> > Yes, we do have some ideas to optimize this downtime.
> >
> > The main problem for current version is, for each checkpoint, we have to
> > send the whole PVM's pages
> > To SVM, and then copy the whole VM's state into SVM from ram cache, in
> > this process, we need both of them be paused.
> > Just as you said, the downtime is based on memory size.
> >
> > So firstly, we need to reduce the sending data while do checkpoint,
> actually,
> > we can migrate parts of PVM's dirty pages in background
> > While both of VMs are running. And then we load these pages into ram
> > cache (backup memory) in SVM temporarily. While do checkpoint,
> > We just send the last dirty pages of PVM to slave side and then copy the
> ram
> > cache into SVM. Further on, we don't have
> > To send the whole PVM's dirty pages, we can only send the pages that
> > dirtied by PVM or SVM during two checkpoints. (Because
> > If one page is not dirtied by both PVM and SVM, the data of this pages
> will
> > keep same in SVM, PVM, backup memory). This method can reduce
> > the time that consumed in sending data.
> >
> > For the second problem, we can reduce the memory copy by two methods,
> > first one, we don't have to copy the whole pages in ram cache,
> > We can only copy the pages that dirtied by PVM and SVM in last
> checkpoint.
> > Second, we can use userfault missing function to reduce the
> > Time consumed in memory copy. (For the second time, in theory, we can
> > reduce time consumed in memory into ms level).
> >
> > You can find the first optimization in attachment, it is based on an old
> qemu
> > version (qemu-2.6), it should not be difficult to rebase it
> > Into master or your version. And please feel free to send the new
> version if
> > you want into community ;)
> >
> >
>
> Thanks Hailiang!
> By the way, Do you have time to push the patches to upstream?
> I think this is a better and faster option.
>
> Thanks
> Zhang Chen
>
> > >
> > > Issue 2:
> > >      In
> > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > > could we move start_vm() before Line 488? Because at first checkpoint
> > > PVM will wait for SVM's reply, it cause PVM stop for a while.
> > >
> >
> > No, that makes no sense, because if PVM runs firstly, it still need to
> wait for
> > The network packets from SVM to compare before send it to client side.
> >
> >
> > Thanks,
> > Hailiang
> >
> > >      We set the COLO feature on running VM, so we hope the running VM
> > > could continuous service for users.
> > > Do you have any suggestions for those issues?
> > >
> > > Best regards,
> > > Daniel Cho
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>

[-- Attachment #2: Type: text/html, Size: 7382 bytes --]

RE: The issues about architecture of the COLO checkpoint

[Zhanghailiang]

[-- Attachment #1: Type: text/plain, Size: 6890 bytes --]

Hi,


1.      After re-walked through the codes, yes, you are right, actually, after the first migration, we will keep dirty log on in primary side,

And only send the dirty pages in PVM to SVM. The ram cache in secondary side is always a backup of PVM, so we don’t have to

Re-send the none-dirtied pages.

The reason why the first checkpoint takes longer time is we have to backup the whole VM’s ram into ram cache, that is colo_init_ram_cache().

It is time consuming, but I have optimized in the second patch “0001-COLO-Optimize-memory-back-up-process.patch” which you can find in my previous reply.



Besides, I found that, In my previous reply “We can only copy the pages that dirtied by PVM and SVM in last checkpoint.”,

We have done this optimization in current upstream codes.

2．I don’t quite understand this question. For COLO, we always need both network packets of PVM’s and SVM’s to compare before send this packets to client.
It depends on this to decide whether or not PVM and SVM are in same state.

Thanks,
hailiang
From: Daniel Cho [mailto:danielcho@qnap.com]
Sent: Wednesday, February 12, 2020 4:37 PM
To: Zhang, Chen <chen.zhang@intel.com>
Cc: Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
Subject: Re: The issues about architecture of the COLO checkpoint

Hi Hailiang,

Thanks for your replaying and explain in detail.
We will try to use the attachments to enhance memory copy.

However, we have some questions for your replying.

1.  As you said, "for each checkpoint, we have to send the whole PVM's pages To SVM", why the only first checkpoint will takes more pause time?
In our observing, the first checkpoint will take more time for pausing, then other checkpoints will takes a few time for pausing. Does it means only the first checkpoint will send the whole pages to SVM, and the other checkpoints send the dirty pages to SVM for reloading?

2. We notice the COLO-COMPARE component will stuck the packet until receive packets from PVM and SVM, as this rule, when we add the COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an other issue to make PVM stuck while setting COLO feature. With this issue, could we let colo-compare to pass the PVM's packet when the SVM's packet queue is empty? Then, the PVM's network won't stock, and "if PVM runs firstly, it still need to wait for The network packets from SVM to compare before send it to client side" won't happened either.

Best regard,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月12日 週三 下午1:45寫道：


> -----Original Message-----
> From: Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> Sent: Wednesday, February 12, 2020 11:18 AM
> To: Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; Daniel Cho
> <danielcho@qnap.com<mailto:danielcho@qnap.com>>; Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> Subject: RE: The issues about architecture of the COLO checkpoint
>
> Hi,
>
> Thank you Dave,
>
> I'll reply here directly.
>
> -----Original Message-----
> From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com<mailto:dgilbert@redhat.com>]
> Sent: Wednesday, February 12, 2020 1:48 AM
> To: Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>; chen.zhang@intel.com<mailto:chen.zhang@intel.com>;
> Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> Subject: Re: The issues about architecture of the COLO checkpoint
>
>
> cc'ing in COLO people:
>
>
> * Daniel Cho (danielcho@qnap.com<mailto:danielcho@qnap.com>) wrote:
> > Hi everyone,
> >      We have some issues about setting COLO feature. Hope somebody
> > could give us some advice.
> >
> > Issue 1:
> >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > the Primary VM will pause a long time(based on memory size) for
> > waiting SVM start. Does it have any idea to reduce the pause time?
> >
>
> Yes, we do have some ideas to optimize this downtime.
>
> The main problem for current version is, for each checkpoint, we have to
> send the whole PVM's pages
> To SVM, and then copy the whole VM's state into SVM from ram cache, in
> this process, we need both of them be paused.
> Just as you said, the downtime is based on memory size.
>
> So firstly, we need to reduce the sending data while do checkpoint, actually,
> we can migrate parts of PVM's dirty pages in background
> While both of VMs are running. And then we load these pages into ram
> cache (backup memory) in SVM temporarily. While do checkpoint,
> We just send the last dirty pages of PVM to slave side and then copy the ram
> cache into SVM. Further on, we don't have
> To send the whole PVM's dirty pages, we can only send the pages that
> dirtied by PVM or SVM during two checkpoints. (Because
> If one page is not dirtied by both PVM and SVM, the data of this pages will
> keep same in SVM, PVM, backup memory). This method can reduce
> the time that consumed in sending data.
>
> For the second problem, we can reduce the memory copy by two methods,
> first one, we don't have to copy the whole pages in ram cache,
> We can only copy the pages that dirtied by PVM and SVM in last checkpoint.
> Second, we can use userfault missing function to reduce the
> Time consumed in memory copy. (For the second time, in theory, we can
> reduce time consumed in memory into ms level).
>
> You can find the first optimization in attachment, it is based on an old qemu
> version (qemu-2.6), it should not be difficult to rebase it
> Into master or your version. And please feel free to send the new version if
> you want into community ;)
>
>

Thanks Hailiang!
By the way, Do you have time to push the patches to upstream?
I think this is a better and faster option.

Thanks
Zhang Chen

> >
> > Issue 2:
> >      In
> > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > could we move start_vm() before Line 488? Because at first checkpoint
> > PVM will wait for SVM's reply, it cause PVM stop for a while.
> >
>
> No, that makes no sense, because if PVM runs firstly, it still need to wait for
> The network packets from SVM to compare before send it to client side.
>
>
> Thanks,
> Hailiang
>
> >      We set the COLO feature on running VM, so we hope the running VM
> > could continuous service for users.
> > Do you have any suggestions for those issues?
> >
> > Best regards,
> > Daniel Cho
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK

[-- Attachment #2: Type: text/html, Size: 17613 bytes --]

RE: The issues about architecture of the COLO checkpoint

[Zhang, Chen]

[-- Attachment #1: Type: text/plain, Size: 7637 bytes --]

For the issue 2:

COLO need use the network packets to confirm PVM and SVM in the same state,
Generally speaking, we can’t send PVM packets without compared with SVM packets.
But to prevent jamming, I think COLO can do force checkpoint and send the PVM packets in this case.

Thanks
Zhang Chen

From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
Sent: Thursday, February 13, 2020 9:45 AM
To: Daniel Cho <danielcho@qnap.com>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org; Zhang, Chen <chen.zhang@intel.com>
Subject: RE: The issues about architecture of the COLO checkpoint

Hi,


1.       After re-walked through the codes, yes, you are right, actually, after the first migration, we will keep dirty log on in primary side,

And only send the dirty pages in PVM to SVM. The ram cache in secondary side is always a backup of PVM, so we don’t have to

Re-send the none-dirtied pages.

The reason why the first checkpoint takes longer time is we have to backup the whole VM’s ram into ram cache, that is colo_init_ram_cache().

It is time consuming, but I have optimized in the second patch “0001-COLO-Optimize-memory-back-up-process.patch” which you can find in my previous reply.



Besides, I found that, In my previous reply “We can only copy the pages that dirtied by PVM and SVM in last checkpoint.”,

We have done this optimization in current upstream codes.

2．I don’t quite understand this question. For COLO, we always need both network packets of PVM’s and SVM’s to compare before send this packets to client.
It depends on this to decide whether or not PVM and SVM are in same state.

Thanks,
hailiang

From: Daniel Cho [mailto:danielcho@qnap.com]
Sent: Wednesday, February 12, 2020 4:37 PM
To: Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
Cc: Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>; Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
Subject: Re: The issues about architecture of the COLO checkpoint

Hi Hailiang,

Thanks for your replaying and explain in detail.
We will try to use the attachments to enhance memory copy.

However, we have some questions for your replying.

1.  As you said, "for each checkpoint, we have to send the whole PVM's pages To SVM", why the only first checkpoint will takes more pause time?
In our observing, the first checkpoint will take more time for pausing, then other checkpoints will takes a few time for pausing. Does it means only the first checkpoint will send the whole pages to SVM, and the other checkpoints send the dirty pages to SVM for reloading?

2. We notice the COLO-COMPARE component will stuck the packet until receive packets from PVM and SVM, as this rule, when we add the COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an other issue to make PVM stuck while setting COLO feature. With this issue, could we let colo-compare to pass the PVM's packet when the SVM's packet queue is empty? Then, the PVM's network won't stock, and "if PVM runs firstly, it still need to wait for The network packets from SVM to compare before send it to client side" won't happened either.

Best regard,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月12日 週三 下午1:45寫道：


> -----Original Message-----
> From: Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> Sent: Wednesday, February 12, 2020 11:18 AM
> To: Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; Daniel Cho
> <danielcho@qnap.com<mailto:danielcho@qnap.com>>; Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> Subject: RE: The issues about architecture of the COLO checkpoint
>
> Hi,
>
> Thank you Dave,
>
> I'll reply here directly.
>
> -----Original Message-----
> From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com<mailto:dgilbert@redhat.com>]
> Sent: Wednesday, February 12, 2020 1:48 AM
> To: Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>; chen.zhang@intel.com<mailto:chen.zhang@intel.com>;
> Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> Subject: Re: The issues about architecture of the COLO checkpoint
>
>
> cc'ing in COLO people:
>
>
> * Daniel Cho (danielcho@qnap.com<mailto:danielcho@qnap.com>) wrote:
> > Hi everyone,
> >      We have some issues about setting COLO feature. Hope somebody
> > could give us some advice.
> >
> > Issue 1:
> >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > the Primary VM will pause a long time(based on memory size) for
> > waiting SVM start. Does it have any idea to reduce the pause time?
> >
>
> Yes, we do have some ideas to optimize this downtime.
>
> The main problem for current version is, for each checkpoint, we have to
> send the whole PVM's pages
> To SVM, and then copy the whole VM's state into SVM from ram cache, in
> this process, we need both of them be paused.
> Just as you said, the downtime is based on memory size.
>
> So firstly, we need to reduce the sending data while do checkpoint, actually,
> we can migrate parts of PVM's dirty pages in background
> While both of VMs are running. And then we load these pages into ram
> cache (backup memory) in SVM temporarily. While do checkpoint,
> We just send the last dirty pages of PVM to slave side and then copy the ram
> cache into SVM. Further on, we don't have
> To send the whole PVM's dirty pages, we can only send the pages that
> dirtied by PVM or SVM during two checkpoints. (Because
> If one page is not dirtied by both PVM and SVM, the data of this pages will
> keep same in SVM, PVM, backup memory). This method can reduce
> the time that consumed in sending data.
>
> For the second problem, we can reduce the memory copy by two methods,
> first one, we don't have to copy the whole pages in ram cache,
> We can only copy the pages that dirtied by PVM and SVM in last checkpoint.
> Second, we can use userfault missing function to reduce the
> Time consumed in memory copy. (For the second time, in theory, we can
> reduce time consumed in memory into ms level).
>
> You can find the first optimization in attachment, it is based on an old qemu
> version (qemu-2.6), it should not be difficult to rebase it
> Into master or your version. And please feel free to send the new version if
> you want into community ;)
>
>

Thanks Hailiang!
By the way, Do you have time to push the patches to upstream?
I think this is a better and faster option.

Thanks
Zhang Chen

> >
> > Issue 2:
> >      In
> > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > could we move start_vm() before Line 488? Because at first checkpoint
> > PVM will wait for SVM's reply, it cause PVM stop for a while.
> >
>
> No, that makes no sense, because if PVM runs firstly, it still need to wait for
> The network packets from SVM to compare before send it to client side.
>
>
> Thanks,
> Hailiang
>
> >      We set the COLO feature on running VM, so we hope the running VM
> > could continuous service for users.
> > Do you have any suggestions for those issues?
> >
> > Best regards,
> > Daniel Cho
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK

[-- Attachment #2: Type: text/html, Size: 19797 bytes --]

RE: The issues about architecture of the COLO checkpoint

[Zhang, Chen]

[-- Attachment #1: Type: text/plain, Size: 8191 bytes --]

Add cc Jason Wang, he is a network expert.
In case some network things goes wrong.

Thanks
Zhang Chen

From: Zhang, Chen
Sent: Thursday, February 13, 2020 10:10 AM
To: 'Zhanghailiang' <zhang.zhanghailiang@huawei.com>; Daniel Cho <danielcho@qnap.com>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
Subject: RE: The issues about architecture of the COLO checkpoint

For the issue 2:

COLO need use the network packets to confirm PVM and SVM in the same state,
Generally speaking, we can’t send PVM packets without compared with SVM packets.
But to prevent jamming, I think COLO can do force checkpoint and send the PVM packets in this case.

Thanks
Zhang Chen

From: Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
Sent: Thursday, February 13, 2020 9:45 AM
To: Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>; Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
Subject: RE: The issues about architecture of the COLO checkpoint

Hi,


1.       After re-walked through the codes, yes, you are right, actually, after the first migration, we will keep dirty log on in primary side,

And only send the dirty pages in PVM to SVM. The ram cache in secondary side is always a backup of PVM, so we don’t have to

Re-send the none-dirtied pages.

The reason why the first checkpoint takes longer time is we have to backup the whole VM’s ram into ram cache, that is colo_init_ram_cache().

It is time consuming, but I have optimized in the second patch “0001-COLO-Optimize-memory-back-up-process.patch” which you can find in my previous reply.



Besides, I found that, In my previous reply “We can only copy the pages that dirtied by PVM and SVM in last checkpoint.”,

We have done this optimization in current upstream codes.

2．I don’t quite understand this question. For COLO, we always need both network packets of PVM’s and SVM’s to compare before send this packets to client.
It depends on this to decide whether or not PVM and SVM are in same state.

Thanks,
hailiang

From: Daniel Cho [mailto:danielcho@qnap.com]
Sent: Wednesday, February 12, 2020 4:37 PM
To: Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
Cc: Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>; Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
Subject: Re: The issues about architecture of the COLO checkpoint

Hi Hailiang,

Thanks for your replaying and explain in detail.
We will try to use the attachments to enhance memory copy.

However, we have some questions for your replying.

1.  As you said, "for each checkpoint, we have to send the whole PVM's pages To SVM", why the only first checkpoint will takes more pause time?
In our observing, the first checkpoint will take more time for pausing, then other checkpoints will takes a few time for pausing. Does it means only the first checkpoint will send the whole pages to SVM, and the other checkpoints send the dirty pages to SVM for reloading?

2. We notice the COLO-COMPARE component will stuck the packet until receive packets from PVM and SVM, as this rule, when we add the COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an other issue to make PVM stuck while setting COLO feature. With this issue, could we let colo-compare to pass the PVM's packet when the SVM's packet queue is empty? Then, the PVM's network won't stock, and "if PVM runs firstly, it still need to wait for The network packets from SVM to compare before send it to client side" won't happened either.

Best regard,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月12日 週三 下午1:45寫道：


> -----Original Message-----
> From: Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> Sent: Wednesday, February 12, 2020 11:18 AM
> To: Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; Daniel Cho
> <danielcho@qnap.com<mailto:danielcho@qnap.com>>; Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> Subject: RE: The issues about architecture of the COLO checkpoint
>
> Hi,
>
> Thank you Dave,
>
> I'll reply here directly.
>
> -----Original Message-----
> From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com<mailto:dgilbert@redhat.com>]
> Sent: Wednesday, February 12, 2020 1:48 AM
> To: Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>; chen.zhang@intel.com<mailto:chen.zhang@intel.com>;
> Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> Subject: Re: The issues about architecture of the COLO checkpoint
>
>
> cc'ing in COLO people:
>
>
> * Daniel Cho (danielcho@qnap.com<mailto:danielcho@qnap.com>) wrote:
> > Hi everyone,
> >      We have some issues about setting COLO feature. Hope somebody
> > could give us some advice.
> >
> > Issue 1:
> >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > the Primary VM will pause a long time(based on memory size) for
> > waiting SVM start. Does it have any idea to reduce the pause time?
> >
>
> Yes, we do have some ideas to optimize this downtime.
>
> The main problem for current version is, for each checkpoint, we have to
> send the whole PVM's pages
> To SVM, and then copy the whole VM's state into SVM from ram cache, in
> this process, we need both of them be paused.
> Just as you said, the downtime is based on memory size.
>
> So firstly, we need to reduce the sending data while do checkpoint, actually,
> we can migrate parts of PVM's dirty pages in background
> While both of VMs are running. And then we load these pages into ram
> cache (backup memory) in SVM temporarily. While do checkpoint,
> We just send the last dirty pages of PVM to slave side and then copy the ram
> cache into SVM. Further on, we don't have
> To send the whole PVM's dirty pages, we can only send the pages that
> dirtied by PVM or SVM during two checkpoints. (Because
> If one page is not dirtied by both PVM and SVM, the data of this pages will
> keep same in SVM, PVM, backup memory). This method can reduce
> the time that consumed in sending data.
>
> For the second problem, we can reduce the memory copy by two methods,
> first one, we don't have to copy the whole pages in ram cache,
> We can only copy the pages that dirtied by PVM and SVM in last checkpoint.
> Second, we can use userfault missing function to reduce the
> Time consumed in memory copy. (For the second time, in theory, we can
> reduce time consumed in memory into ms level).
>
> You can find the first optimization in attachment, it is based on an old qemu
> version (qemu-2.6), it should not be difficult to rebase it
> Into master or your version. And please feel free to send the new version if
> you want into community ;)
>
>

Thanks Hailiang!
By the way, Do you have time to push the patches to upstream?
I think this is a better and faster option.

Thanks
Zhang Chen

> >
> > Issue 2:
> >      In
> > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > could we move start_vm() before Line 488? Because at first checkpoint
> > PVM will wait for SVM's reply, it cause PVM stop for a while.
> >
>
> No, that makes no sense, because if PVM runs firstly, it still need to wait for
> The network packets from SVM to compare before send it to client side.
>
>
> Thanks,
> Hailiang
>
> >      We set the COLO feature on running VM, so we hope the running VM
> > could continuous service for users.
> > Do you have any suggestions for those issues?
> >
> > Best regards,
> > Daniel Cho
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK

[-- Attachment #2: Type: text/html, Size: 21743 bytes --]

Re: The issues about architecture of the COLO checkpoint

[Daniel Cho]

[-- Attachment #1: Type: text/plain, Size: 9047 bytes --]

Hi Hailiang,

1.
    OK, we will try the patch
“0001-COLO-Optimize-memory-back-up-process.patch”,
and thanks for your help.

2.
    We understand the reason to compare PVM and SVM's packet. However, the
empty of SVM's packet queue might happened on setting COLO feature and SVM
broken.

On situation 1 ( setting COLO feature ):
    We could force do checkpoint after setting COLO feature finish, then it
will protect the state of PVM and SVM . As the Zhang Chen said.

On situation 2 ( SVM broken ):
    COLO will do failover for PVM, so it might not cause any wrong on PVM.

However, those situations are our views, so there might be a big difference
between reality and our views.
If we have any wrong views and opinions, please let us know, and correct
us.
Thanks.

Best regards,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com> 於 2020年2月13日 週四 上午10:17寫道：

> Add cc Jason Wang, he is a network expert.
>
> In case some network things goes wrong.
>
>
>
> Thanks
>
> Zhang Chen
>
>
>
> *From:* Zhang, Chen
> *Sent:* Thursday, February 13, 2020 10:10 AM
> *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com>; Daniel Cho <
> danielcho@qnap.com>
> *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> *Subject:* RE: The issues about architecture of the COLO checkpoint
>
>
>
> For the issue 2:
>
>
>
> COLO need use the network packets to confirm PVM and SVM in the same state,
>
> Generally speaking, we can’t send PVM packets without compared with SVM
> packets.
>
> But to prevent jamming, I think COLO can do force checkpoint and send the
> PVM packets in this case.
>
>
>
> Thanks
>
> Zhang Chen
>
>
>
> *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com>
> *Sent:* Thursday, February 13, 2020 9:45 AM
> *To:* Daniel Cho <danielcho@qnap.com>
> *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org;
> Zhang, Chen <chen.zhang@intel.com>
> *Subject:* RE: The issues about architecture of the COLO checkpoint
>
>
>
> Hi,
>
>
>
> 1.       After re-walked through the codes, yes, you are right, actually,
> after the first migration, we will keep dirty log on in primary side,
>
> And only send the dirty pages in PVM to SVM. The ram cache in secondary
> side is always a backup of PVM, so we don’t have to
>
> Re-send the none-dirtied pages.
>
> The reason why the first checkpoint takes longer time is we have to backup
> the whole VM’s ram into ram cache, that is colo_init_ram_cache().
>
> It is time consuming, but I have optimized in the second patch
> “0001-COLO-Optimize-memory-back-up-process.patch” which you can find in my
> previous reply.
>
>
>
> Besides, I found that, In my previous reply “We can only copy the pages
> that dirtied by PVM and SVM in last checkpoint.”,
>
> We have done this optimization in current upstream codes.
>
>
>
> 2．I don’t quite understand this question. For COLO, we always need both
> network packets of PVM’s and SVM’s to compare before send this packets to
> client.
>
> It depends on this to decide whether or not PVM and SVM are in same state.
>
>
>
> Thanks,
>
> hailiang
>
>
>
> *From:* Daniel Cho [mailto:danielcho@qnap.com <danielcho@qnap.com>]
> *Sent:* Wednesday, February 12, 2020 4:37 PM
> *To:* Zhang, Chen <chen.zhang@intel.com>
> *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David Alan
> Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> *Subject:* Re: The issues about architecture of the COLO checkpoint
>
>
>
> Hi Hailiang,
>
>
>
> Thanks for your replaying and explain in detail.
>
> We will try to use the attachments to enhance memory copy.
>
>
>
> However, we have some questions for your replying.
>
>
>
> 1.  As you said, "for each checkpoint, we have to send the whole PVM's
> pages To SVM", why the only first checkpoint will takes more pause time?
>
> In our observing, the first checkpoint will take more time for pausing,
> then other checkpoints will takes a few time for pausing. Does it means
> only the first checkpoint will send the whole pages to SVM, and the other
> checkpoints send the dirty pages to SVM for reloading?
>
>
>
> 2. We notice the COLO-COMPARE component will stuck the packet until
> receive packets from PVM and SVM, as this rule, when we add the
> COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an
> other issue to make PVM stuck while setting COLO feature. With this issue,
> could we let colo-compare to pass the PVM's packet when the SVM's packet
> queue is empty? Then, the PVM's network won't stock, and "if PVM runs
> firstly, it still need to wait for The network packets from SVM to
> compare before send it to client side" won't happened either.
>
>
>
> Best regard,
>
> Daniel Cho
>
>
>
> Zhang, Chen <chen.zhang@intel.com> 於 2020年2月12日 週三 下午1:45寫道：
>
>
>
> > -----Original Message-----
> > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > Sent: Wednesday, February 12, 2020 11:18 AM
> > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
> > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
> > Cc: qemu-devel@nongnu.org
> > Subject: RE: The issues about architecture of the COLO checkpoint
> >
> > Hi,
> >
> > Thank you Dave,
> >
> > I'll reply here directly.
> >
> > -----Original Message-----
> > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > Sent: Wednesday, February 12, 2020 1:48 AM
> > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
> > Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > Cc: qemu-devel@nongnu.org
> > Subject: Re: The issues about architecture of the COLO checkpoint
> >
> >
> > cc'ing in COLO people:
> >
> >
> > * Daniel Cho (danielcho@qnap.com) wrote:
> > > Hi everyone,
> > >      We have some issues about setting COLO feature. Hope somebody
> > > could give us some advice.
> > >
> > > Issue 1:
> > >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > > the Primary VM will pause a long time(based on memory size) for
> > > waiting SVM start. Does it have any idea to reduce the pause time?
> > >
> >
> > Yes, we do have some ideas to optimize this downtime.
> >
> > The main problem for current version is, for each checkpoint, we have to
> > send the whole PVM's pages
> > To SVM, and then copy the whole VM's state into SVM from ram cache, in
> > this process, we need both of them be paused.
> > Just as you said, the downtime is based on memory size.
> >
> > So firstly, we need to reduce the sending data while do checkpoint,
> actually,
> > we can migrate parts of PVM's dirty pages in background
> > While both of VMs are running. And then we load these pages into ram
> > cache (backup memory) in SVM temporarily. While do checkpoint,
> > We just send the last dirty pages of PVM to slave side and then copy the
> ram
> > cache into SVM. Further on, we don't have
> > To send the whole PVM's dirty pages, we can only send the pages that
> > dirtied by PVM or SVM during two checkpoints. (Because
> > If one page is not dirtied by both PVM and SVM, the data of this pages
> will
> > keep same in SVM, PVM, backup memory). This method can reduce
> > the time that consumed in sending data.
> >
> > For the second problem, we can reduce the memory copy by two methods,
> > first one, we don't have to copy the whole pages in ram cache,
> > We can only copy the pages that dirtied by PVM and SVM in last
> checkpoint.
> > Second, we can use userfault missing function to reduce the
> > Time consumed in memory copy. (For the second time, in theory, we can
> > reduce time consumed in memory into ms level).
> >
> > You can find the first optimization in attachment, it is based on an old
> qemu
> > version (qemu-2.6), it should not be difficult to rebase it
> > Into master or your version. And please feel free to send the new
> version if
> > you want into community ;)
> >
> >
>
> Thanks Hailiang!
> By the way, Do you have time to push the patches to upstream?
> I think this is a better and faster option.
>
> Thanks
> Zhang Chen
>
> > >
> > > Issue 2:
> > >      In
> > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > > could we move start_vm() before Line 488? Because at first checkpoint
> > > PVM will wait for SVM's reply, it cause PVM stop for a while.
> > >
> >
> > No, that makes no sense, because if PVM runs firstly, it still need to
> wait for
> > The network packets from SVM to compare before send it to client side.
> >
> >
> > Thanks,
> > Hailiang
> >
> > >      We set the COLO feature on running VM, so we hope the running VM
> > > could continuous service for users.
> > > Do you have any suggestions for those issues?
> > >
> > > Best regards,
> > > Daniel Cho
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>

[-- Attachment #2: Type: text/html, Size: 19729 bytes --]

Re: The issues about architecture of the COLO checkpoint

[Dr. David Alan Gilbert]

* Daniel Cho (danielcho@qnap.com) wrote:
> Hi Hailiang,
> 
> 1.
>     OK, we will try the patch
> “0001-COLO-Optimize-memory-back-up-process.patch”,
> and thanks for your help.
> 
> 2.
>     We understand the reason to compare PVM and SVM's packet. However, the
> empty of SVM's packet queue might happened on setting COLO feature and SVM
> broken.
> 
> On situation 1 ( setting COLO feature ):
>     We could force do checkpoint after setting COLO feature finish, then it
> will protect the state of PVM and SVM . As the Zhang Chen said.
> 
> On situation 2 ( SVM broken ):
>     COLO will do failover for PVM, so it might not cause any wrong on PVM.
> 
> However, those situations are our views, so there might be a big difference
> between reality and our views.
> If we have any wrong views and opinions, please let us know, and correct
> us.

It does need a timeout; the SVM being broken or being in a state where
it never sends the corresponding packet (because of a state difference)
can happen and COLO needs to timeout when the packet hasn't arrived
after a while and trigger the checkpoint.

Dave

> Thanks.
> 
> Best regards,
> Daniel Cho
> 
> Zhang, Chen <chen.zhang@intel.com> 於 2020年2月13日 週四 上午10:17寫道：
> 
> > Add cc Jason Wang, he is a network expert.
> >
> > In case some network things goes wrong.
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> > *From:* Zhang, Chen
> > *Sent:* Thursday, February 13, 2020 10:10 AM
> > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com>; Daniel Cho <
> > danielcho@qnap.com>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > For the issue 2:
> >
> >
> >
> > COLO need use the network packets to confirm PVM and SVM in the same state,
> >
> > Generally speaking, we can’t send PVM packets without compared with SVM
> > packets.
> >
> > But to prevent jamming, I think COLO can do force checkpoint and send the
> > PVM packets in this case.
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > *Sent:* Thursday, February 13, 2020 9:45 AM
> > *To:* Daniel Cho <danielcho@qnap.com>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org;
> > Zhang, Chen <chen.zhang@intel.com>
> > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi,
> >
> >
> >
> > 1.       After re-walked through the codes, yes, you are right, actually,
> > after the first migration, we will keep dirty log on in primary side,
> >
> > And only send the dirty pages in PVM to SVM. The ram cache in secondary
> > side is always a backup of PVM, so we don’t have to
> >
> > Re-send the none-dirtied pages.
> >
> > The reason why the first checkpoint takes longer time is we have to backup
> > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
> >
> > It is time consuming, but I have optimized in the second patch
> > “0001-COLO-Optimize-memory-back-up-process.patch” which you can find in my
> > previous reply.
> >
> >
> >
> > Besides, I found that, In my previous reply “We can only copy the pages
> > that dirtied by PVM and SVM in last checkpoint.”,
> >
> > We have done this optimization in current upstream codes.
> >
> >
> >
> > 2．I don’t quite understand this question. For COLO, we always need both
> > network packets of PVM’s and SVM’s to compare before send this packets to
> > client.
> >
> > It depends on this to decide whether or not PVM and SVM are in same state.
> >
> >
> >
> > Thanks,
> >
> > hailiang
> >
> >
> >
> > *From:* Daniel Cho [mailto:danielcho@qnap.com <danielcho@qnap.com>]
> > *Sent:* Wednesday, February 12, 2020 4:37 PM
> > *To:* Zhang, Chen <chen.zhang@intel.com>
> > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David Alan
> > Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> > *Subject:* Re: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi Hailiang,
> >
> >
> >
> > Thanks for your replaying and explain in detail.
> >
> > We will try to use the attachments to enhance memory copy.
> >
> >
> >
> > However, we have some questions for your replying.
> >
> >
> >
> > 1.  As you said, "for each checkpoint, we have to send the whole PVM's
> > pages To SVM", why the only first checkpoint will takes more pause time?
> >
> > In our observing, the first checkpoint will take more time for pausing,
> > then other checkpoints will takes a few time for pausing. Does it means
> > only the first checkpoint will send the whole pages to SVM, and the other
> > checkpoints send the dirty pages to SVM for reloading?
> >
> >
> >
> > 2. We notice the COLO-COMPARE component will stuck the packet until
> > receive packets from PVM and SVM, as this rule, when we add the
> > COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an
> > other issue to make PVM stuck while setting COLO feature. With this issue,
> > could we let colo-compare to pass the PVM's packet when the SVM's packet
> > queue is empty? Then, the PVM's network won't stock, and "if PVM runs
> > firstly, it still need to wait for The network packets from SVM to
> > compare before send it to client side" won't happened either.
> >
> >
> >
> > Best regard,
> >
> > Daniel Cho
> >
> >
> >
> > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月12日 週三 下午1:45寫道：
> >
> >
> >
> > > -----Original Message-----
> > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > Sent: Wednesday, February 12, 2020 11:18 AM
> > > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
> > > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
> > > Cc: qemu-devel@nongnu.org
> > > Subject: RE: The issues about architecture of the COLO checkpoint
> > >
> > > Hi,
> > >
> > > Thank you Dave,
> > >
> > > I'll reply here directly.
> > >
> > > -----Original Message-----
> > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > > Sent: Wednesday, February 12, 2020 1:48 AM
> > > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
> > > Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > Cc: qemu-devel@nongnu.org
> > > Subject: Re: The issues about architecture of the COLO checkpoint
> > >
> > >
> > > cc'ing in COLO people:
> > >
> > >
> > > * Daniel Cho (danielcho@qnap.com) wrote:
> > > > Hi everyone,
> > > >      We have some issues about setting COLO feature. Hope somebody
> > > > could give us some advice.
> > > >
> > > > Issue 1:
> > > >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > > > the Primary VM will pause a long time(based on memory size) for
> > > > waiting SVM start. Does it have any idea to reduce the pause time?
> > > >
> > >
> > > Yes, we do have some ideas to optimize this downtime.
> > >
> > > The main problem for current version is, for each checkpoint, we have to
> > > send the whole PVM's pages
> > > To SVM, and then copy the whole VM's state into SVM from ram cache, in
> > > this process, we need both of them be paused.
> > > Just as you said, the downtime is based on memory size.
> > >
> > > So firstly, we need to reduce the sending data while do checkpoint,
> > actually,
> > > we can migrate parts of PVM's dirty pages in background
> > > While both of VMs are running. And then we load these pages into ram
> > > cache (backup memory) in SVM temporarily. While do checkpoint,
> > > We just send the last dirty pages of PVM to slave side and then copy the
> > ram
> > > cache into SVM. Further on, we don't have
> > > To send the whole PVM's dirty pages, we can only send the pages that
> > > dirtied by PVM or SVM during two checkpoints. (Because
> > > If one page is not dirtied by both PVM and SVM, the data of this pages
> > will
> > > keep same in SVM, PVM, backup memory). This method can reduce
> > > the time that consumed in sending data.
> > >
> > > For the second problem, we can reduce the memory copy by two methods,
> > > first one, we don't have to copy the whole pages in ram cache,
> > > We can only copy the pages that dirtied by PVM and SVM in last
> > checkpoint.
> > > Second, we can use userfault missing function to reduce the
> > > Time consumed in memory copy. (For the second time, in theory, we can
> > > reduce time consumed in memory into ms level).
> > >
> > > You can find the first optimization in attachment, it is based on an old
> > qemu
> > > version (qemu-2.6), it should not be difficult to rebase it
> > > Into master or your version. And please feel free to send the new
> > version if
> > > you want into community ;)
> > >
> > >
> >
> > Thanks Hailiang!
> > By the way, Do you have time to push the patches to upstream?
> > I think this is a better and faster option.
> >
> > Thanks
> > Zhang Chen
> >
> > > >
> > > > Issue 2:
> > > >      In
> > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > > > could we move start_vm() before Line 488? Because at first checkpoint
> > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
> > > >
> > >
> > > No, that makes no sense, because if PVM runs firstly, it still need to
> > wait for
> > > The network packets from SVM to compare before send it to client side.
> > >
> > >
> > > Thanks,
> > > Hailiang
> > >
> > > >      We set the COLO feature on running VM, so we hope the running VM
> > > > could continuous service for users.
> > > > Do you have any suggestions for those issues?
> > > >
> > > > Best regards,
> > > > Daniel Cho
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> >
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: The issues about architecture of the COLO checkpoint

[Daniel Cho]

[-- Attachment #1: Type: text/plain, Size: 11088 bytes --]

Hi Dave,

Yes, I agree with you, it does need a timeout.

Hi Hailiang,

We base on qemu-4.1.0 for using COLO feature, in your patch, we found a lot
of difference  between your version and ours.
Could you give us a latest release version which is close your developing
code?

Thanks.

Regards
Daniel Cho

Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2020年2月13日 週四 下午6:38寫道：

> * Daniel Cho (danielcho@qnap.com) wrote:
> > Hi Hailiang,
> >
> > 1.
> >     OK, we will try the patch
> > “0001-COLO-Optimize-memory-back-up-process.patch”,
> > and thanks for your help.
> >
> > 2.
> >     We understand the reason to compare PVM and SVM's packet. However,
> the
> > empty of SVM's packet queue might happened on setting COLO feature and
> SVM
> > broken.
> >
> > On situation 1 ( setting COLO feature ):
> >     We could force do checkpoint after setting COLO feature finish, then
> it
> > will protect the state of PVM and SVM . As the Zhang Chen said.
> >
> > On situation 2 ( SVM broken ):
> >     COLO will do failover for PVM, so it might not cause any wrong on
> PVM.
> >
> > However, those situations are our views, so there might be a big
> difference
> > between reality and our views.
> > If we have any wrong views and opinions, please let us know, and correct
> > us.
>
> It does need a timeout; the SVM being broken or being in a state where
> it never sends the corresponding packet (because of a state difference)
> can happen and COLO needs to timeout when the packet hasn't arrived
> after a while and trigger the checkpoint.
>
> Dave
>
> > Thanks.
> >
> > Best regards,
> > Daniel Cho
> >
> > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月13日 週四 上午10:17寫道：
> >
> > > Add cc Jason Wang, he is a network expert.
> > >
> > > In case some network things goes wrong.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Zhang Chen
> > >
> > >
> > >
> > > *From:* Zhang, Chen
> > > *Sent:* Thursday, February 13, 2020 10:10 AM
> > > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com>; Daniel Cho <
> > > danielcho@qnap.com>
> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
> qemu-devel@nongnu.org
> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
> > >
> > >
> > >
> > > For the issue 2:
> > >
> > >
> > >
> > > COLO need use the network packets to confirm PVM and SVM in the same
> state,
> > >
> > > Generally speaking, we can’t send PVM packets without compared with SVM
> > > packets.
> > >
> > > But to prevent jamming, I think COLO can do force checkpoint and send
> the
> > > PVM packets in this case.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Zhang Chen
> > >
> > >
> > >
> > > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > *Sent:* Thursday, February 13, 2020 9:45 AM
> > > *To:* Daniel Cho <danielcho@qnap.com>
> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
> qemu-devel@nongnu.org;
> > > Zhang, Chen <chen.zhang@intel.com>
> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
> > >
> > >
> > >
> > > Hi,
> > >
> > >
> > >
> > > 1.       After re-walked through the codes, yes, you are right,
> actually,
> > > after the first migration, we will keep dirty log on in primary side,
> > >
> > > And only send the dirty pages in PVM to SVM. The ram cache in secondary
> > > side is always a backup of PVM, so we don’t have to
> > >
> > > Re-send the none-dirtied pages.
> > >
> > > The reason why the first checkpoint takes longer time is we have to
> backup
> > > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
> > >
> > > It is time consuming, but I have optimized in the second patch
> > > “0001-COLO-Optimize-memory-back-up-process.patch” which you can find
> in my
> > > previous reply.
> > >
> > >
> > >
> > > Besides, I found that, In my previous reply “We can only copy the pages
> > > that dirtied by PVM and SVM in last checkpoint.”,
> > >
> > > We have done this optimization in current upstream codes.
> > >
> > >
> > >
> > > 2．I don’t quite understand this question. For COLO, we always need both
> > > network packets of PVM’s and SVM’s to compare before send this packets
> to
> > > client.
> > >
> > > It depends on this to decide whether or not PVM and SVM are in same
> state.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > hailiang
> > >
> > >
> > >
> > > *From:* Daniel Cho [mailto:danielcho@qnap.com <danielcho@qnap.com>]
> > > *Sent:* Wednesday, February 12, 2020 4:37 PM
> > > *To:* Zhang, Chen <chen.zhang@intel.com>
> > > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David Alan
> > > Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> > > *Subject:* Re: The issues about architecture of the COLO checkpoint
> > >
> > >
> > >
> > > Hi Hailiang,
> > >
> > >
> > >
> > > Thanks for your replaying and explain in detail.
> > >
> > > We will try to use the attachments to enhance memory copy.
> > >
> > >
> > >
> > > However, we have some questions for your replying.
> > >
> > >
> > >
> > > 1.  As you said, "for each checkpoint, we have to send the whole PVM's
> > > pages To SVM", why the only first checkpoint will takes more pause
> time?
> > >
> > > In our observing, the first checkpoint will take more time for pausing,
> > > then other checkpoints will takes a few time for pausing. Does it means
> > > only the first checkpoint will send the whole pages to SVM, and the
> other
> > > checkpoints send the dirty pages to SVM for reloading?
> > >
> > >
> > >
> > > 2. We notice the COLO-COMPARE component will stuck the packet until
> > > receive packets from PVM and SVM, as this rule, when we add the
> > > COLO-COMPARE to PVM, its network will stuck until SVM start. So it is
> an
> > > other issue to make PVM stuck while setting COLO feature. With this
> issue,
> > > could we let colo-compare to pass the PVM's packet when the SVM's
> packet
> > > queue is empty? Then, the PVM's network won't stock, and "if PVM runs
> > > firstly, it still need to wait for The network packets from SVM to
> > > compare before send it to client side" won't happened either.
> > >
> > >
> > >
> > > Best regard,
> > >
> > > Daniel Cho
> > >
> > >
> > >
> > > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月12日 週三 下午1:45寫道：
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > > Sent: Wednesday, February 12, 2020 11:18 AM
> > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
> > > > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
> > > > Cc: qemu-devel@nongnu.org
> > > > Subject: RE: The issues about architecture of the COLO checkpoint
> > > >
> > > > Hi,
> > > >
> > > > Thank you Dave,
> > > >
> > > > I'll reply here directly.
> > > >
> > > > -----Original Message-----
> > > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > > > Sent: Wednesday, February 12, 2020 1:48 AM
> > > > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
> > > > Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > > Cc: qemu-devel@nongnu.org
> > > > Subject: Re: The issues about architecture of the COLO checkpoint
> > > >
> > > >
> > > > cc'ing in COLO people:
> > > >
> > > >
> > > > * Daniel Cho (danielcho@qnap.com) wrote:
> > > > > Hi everyone,
> > > > >      We have some issues about setting COLO feature. Hope somebody
> > > > > could give us some advice.
> > > > >
> > > > > Issue 1:
> > > > >      We dynamic to set COLO feature for PVM(2 core, 16G memory),
> but
> > > > > the Primary VM will pause a long time(based on memory size) for
> > > > > waiting SVM start. Does it have any idea to reduce the pause time?
> > > > >
> > > >
> > > > Yes, we do have some ideas to optimize this downtime.
> > > >
> > > > The main problem for current version is, for each checkpoint, we
> have to
> > > > send the whole PVM's pages
> > > > To SVM, and then copy the whole VM's state into SVM from ram cache,
> in
> > > > this process, we need both of them be paused.
> > > > Just as you said, the downtime is based on memory size.
> > > >
> > > > So firstly, we need to reduce the sending data while do checkpoint,
> > > actually,
> > > > we can migrate parts of PVM's dirty pages in background
> > > > While both of VMs are running. And then we load these pages into ram
> > > > cache (backup memory) in SVM temporarily. While do checkpoint,
> > > > We just send the last dirty pages of PVM to slave side and then copy
> the
> > > ram
> > > > cache into SVM. Further on, we don't have
> > > > To send the whole PVM's dirty pages, we can only send the pages that
> > > > dirtied by PVM or SVM during two checkpoints. (Because
> > > > If one page is not dirtied by both PVM and SVM, the data of this
> pages
> > > will
> > > > keep same in SVM, PVM, backup memory). This method can reduce
> > > > the time that consumed in sending data.
> > > >
> > > > For the second problem, we can reduce the memory copy by two methods,
> > > > first one, we don't have to copy the whole pages in ram cache,
> > > > We can only copy the pages that dirtied by PVM and SVM in last
> > > checkpoint.
> > > > Second, we can use userfault missing function to reduce the
> > > > Time consumed in memory copy. (For the second time, in theory, we can
> > > > reduce time consumed in memory into ms level).
> > > >
> > > > You can find the first optimization in attachment, it is based on an
> old
> > > qemu
> > > > version (qemu-2.6), it should not be difficult to rebase it
> > > > Into master or your version. And please feel free to send the new
> > > version if
> > > > you want into community ;)
> > > >
> > > >
> > >
> > > Thanks Hailiang!
> > > By the way, Do you have time to push the patches to upstream?
> > > I think this is a better and faster option.
> > >
> > > Thanks
> > > Zhang Chen
> > >
> > > > >
> > > > > Issue 2:
> > > > >      In
> > > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > > > > could we move start_vm() before Line 488? Because at first
> checkpoint
> > > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
> > > > >
> > > >
> > > > No, that makes no sense, because if PVM runs firstly, it still need
> to
> > > wait for
> > > > The network packets from SVM to compare before send it to client
> side.
> > > >
> > > >
> > > > Thanks,
> > > > Hailiang
> > > >
> > > > >      We set the COLO feature on running VM, so we hope the running
> VM
> > > > > could continuous service for users.
> > > > > Do you have any suggestions for those issues?
> > > > >
> > > > > Best regards,
> > > > > Daniel Cho
> > > > --
> > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > >
> > >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
>

[-- Attachment #2: Type: text/html, Size: 16172 bytes --]

RE: The issues about architecture of the COLO checkpoint

[Zhanghailiang]

[-- Attachment #1: Type: text/plain, Size: 11952 bytes --]

Hi Daniel,

I have rebased these patches with newest upstream version, this series “Optimize VM's downtime while do checkpoint in COLO”,
It is not been tested, please let me known if there are any problems.

Thanks,
Hailiang

From: Daniel Cho [mailto:danielcho@qnap.com]
Sent: Saturday, February 15, 2020 11:36 AM
To: Dr. David Alan Gilbert <dgilbert@redhat.com>
Cc: Zhang, Chen <chen.zhang@intel.com>; Zhanghailiang <zhang.zhanghailiang@huawei.com>; qemu-devel@nongnu.org; Jason Wang <jasowang@redhat.com>
Subject: Re: The issues about architecture of the COLO checkpoint

Hi Dave,

Yes, I agree with you, it does need a timeout.

Hi Hailiang,

We base on qemu-4.1.0 for using COLO feature, in your patch, we found a lot of difference  between your version and ours.
Could you give us a latest release version which is close your developing code?

Thanks.

Regards
Daniel Cho

Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>> 於 2020年2月13日 週四 下午6:38寫道：
* Daniel Cho (danielcho@qnap.com<mailto:danielcho@qnap.com>) wrote:
> Hi Hailiang,
>
> 1.
>     OK, we will try the patch
> “0001-COLO-Optimize-memory-back-up-process.patch”,
> and thanks for your help.
>
> 2.
>     We understand the reason to compare PVM and SVM's packet. However, the
> empty of SVM's packet queue might happened on setting COLO feature and SVM
> broken.
>
> On situation 1 ( setting COLO feature ):
>     We could force do checkpoint after setting COLO feature finish, then it
> will protect the state of PVM and SVM . As the Zhang Chen said.
>
> On situation 2 ( SVM broken ):
>     COLO will do failover for PVM, so it might not cause any wrong on PVM.
>
> However, those situations are our views, so there might be a big difference
> between reality and our views.
> If we have any wrong views and opinions, please let us know, and correct
> us.

It does need a timeout; the SVM being broken or being in a state where
it never sends the corresponding packet (because of a state difference)
can happen and COLO needs to timeout when the packet hasn't arrived
after a while and trigger the checkpoint.

Dave

> Thanks.
>
> Best regards,
> Daniel Cho
>
> Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月13日 週四 上午10:17寫道：
>
> > Add cc Jason Wang, he is a network expert.
> >
> > In case some network things goes wrong.
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> > *From:* Zhang, Chen
> > *Sent:* Thursday, February 13, 2020 10:10 AM
> > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>; Daniel Cho <
> > danielcho@qnap.com<mailto:danielcho@qnap.com>>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > For the issue 2:
> >
> >
> >
> > COLO need use the network packets to confirm PVM and SVM in the same state,
> >
> > Generally speaking, we can’t send PVM packets without compared with SVM
> > packets.
> >
> > But to prevent jamming, I think COLO can do force checkpoint and send the
> > PVM packets in this case.
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> > *Sent:* Thursday, February 13, 2020 9:45 AM
> > *To:* Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>;
> > Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi,
> >
> >
> >
> > 1.       After re-walked through the codes, yes, you are right, actually,
> > after the first migration, we will keep dirty log on in primary side,
> >
> > And only send the dirty pages in PVM to SVM. The ram cache in secondary
> > side is always a backup of PVM, so we don’t have to
> >
> > Re-send the none-dirtied pages.
> >
> > The reason why the first checkpoint takes longer time is we have to backup
> > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
> >
> > It is time consuming, but I have optimized in the second patch
> > “0001-COLO-Optimize-memory-back-up-process.patch” which you can find in my
> > previous reply.
> >
> >
> >
> > Besides, I found that, In my previous reply “We can only copy the pages
> > that dirtied by PVM and SVM in last checkpoint.”,
> >
> > We have done this optimization in current upstream codes.
> >
> >
> >
> > 2．I don’t quite understand this question. For COLO, we always need both
> > network packets of PVM’s and SVM’s to compare before send this packets to
> > client.
> >
> > It depends on this to decide whether or not PVM and SVM are in same state.
> >
> >
> >
> > Thanks,
> >
> > hailiang
> >
> >
> >
> > *From:* Daniel Cho [mailto:danielcho@qnap.com<mailto:danielcho@qnap.com> <danielcho@qnap.com<mailto:danielcho@qnap.com>>]
> > *Sent:* Wednesday, February 12, 2020 4:37 PM
> > *To:* Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>; Dr. David Alan
> > Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > *Subject:* Re: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi Hailiang,
> >
> >
> >
> > Thanks for your replaying and explain in detail.
> >
> > We will try to use the attachments to enhance memory copy.
> >
> >
> >
> > However, we have some questions for your replying.
> >
> >
> >
> > 1.  As you said, "for each checkpoint, we have to send the whole PVM's
> > pages To SVM", why the only first checkpoint will takes more pause time?
> >
> > In our observing, the first checkpoint will take more time for pausing,
> > then other checkpoints will takes a few time for pausing. Does it means
> > only the first checkpoint will send the whole pages to SVM, and the other
> > checkpoints send the dirty pages to SVM for reloading?
> >
> >
> >
> > 2. We notice the COLO-COMPARE component will stuck the packet until
> > receive packets from PVM and SVM, as this rule, when we add the
> > COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an
> > other issue to make PVM stuck while setting COLO feature. With this issue,
> > could we let colo-compare to pass the PVM's packet when the SVM's packet
> > queue is empty? Then, the PVM's network won't stock, and "if PVM runs
> > firstly, it still need to wait for The network packets from SVM to
> > compare before send it to client side" won't happened either.
> >
> >
> >
> > Best regard,
> >
> > Daniel Cho
> >
> >
> >
> > Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月12日 週三 下午1:45寫道：
> >
> >
> >
> > > -----Original Message-----
> > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> > > Sent: Wednesday, February 12, 2020 11:18 AM
> > > To: Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; Daniel Cho
> > > <danielcho@qnap.com<mailto:danielcho@qnap.com>>; Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> > > Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > > Subject: RE: The issues about architecture of the COLO checkpoint
> > >
> > > Hi,
> > >
> > > Thank you Dave,
> > >
> > > I'll reply here directly.
> > >
> > > -----Original Message-----
> > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com<mailto:dgilbert@redhat.com>]
> > > Sent: Wednesday, February 12, 2020 1:48 AM
> > > To: Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>; chen.zhang@intel.com<mailto:chen.zhang@intel.com>;
> > > Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> > > Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > > Subject: Re: The issues about architecture of the COLO checkpoint
> > >
> > >
> > > cc'ing in COLO people:
> > >
> > >
> > > * Daniel Cho (danielcho@qnap.com<mailto:danielcho@qnap.com>) wrote:
> > > > Hi everyone,
> > > >      We have some issues about setting COLO feature. Hope somebody
> > > > could give us some advice.
> > > >
> > > > Issue 1:
> > > >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > > > the Primary VM will pause a long time(based on memory size) for
> > > > waiting SVM start. Does it have any idea to reduce the pause time?
> > > >
> > >
> > > Yes, we do have some ideas to optimize this downtime.
> > >
> > > The main problem for current version is, for each checkpoint, we have to
> > > send the whole PVM's pages
> > > To SVM, and then copy the whole VM's state into SVM from ram cache, in
> > > this process, we need both of them be paused.
> > > Just as you said, the downtime is based on memory size.
> > >
> > > So firstly, we need to reduce the sending data while do checkpoint,
> > actually,
> > > we can migrate parts of PVM's dirty pages in background
> > > While both of VMs are running. And then we load these pages into ram
> > > cache (backup memory) in SVM temporarily. While do checkpoint,
> > > We just send the last dirty pages of PVM to slave side and then copy the
> > ram
> > > cache into SVM. Further on, we don't have
> > > To send the whole PVM's dirty pages, we can only send the pages that
> > > dirtied by PVM or SVM during two checkpoints. (Because
> > > If one page is not dirtied by both PVM and SVM, the data of this pages
> > will
> > > keep same in SVM, PVM, backup memory). This method can reduce
> > > the time that consumed in sending data.
> > >
> > > For the second problem, we can reduce the memory copy by two methods,
> > > first one, we don't have to copy the whole pages in ram cache,
> > > We can only copy the pages that dirtied by PVM and SVM in last
> > checkpoint.
> > > Second, we can use userfault missing function to reduce the
> > > Time consumed in memory copy. (For the second time, in theory, we can
> > > reduce time consumed in memory into ms level).
> > >
> > > You can find the first optimization in attachment, it is based on an old
> > qemu
> > > version (qemu-2.6), it should not be difficult to rebase it
> > > Into master or your version. And please feel free to send the new
> > version if
> > > you want into community ;)
> > >
> > >
> >
> > Thanks Hailiang!
> > By the way, Do you have time to push the patches to upstream?
> > I think this is a better and faster option.
> >
> > Thanks
> > Zhang Chen
> >
> > > >
> > > > Issue 2:
> > > >      In
> > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > > > could we move start_vm() before Line 488? Because at first checkpoint
> > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
> > > >
> > >
> > > No, that makes no sense, because if PVM runs firstly, it still need to
> > wait for
> > > The network packets from SVM to compare before send it to client side.
> > >
> > >
> > > Thanks,
> > > Hailiang
> > >
> > > >      We set the COLO feature on running VM, so we hope the running VM
> > > > could continuous service for users.
> > > > Do you have any suggestions for those issues?
> > > >
> > > > Best regards,
> > > > Daniel Cho
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK
> >
> >
--
Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK

[-- Attachment #2: Type: text/html, Size: 21757 bytes --]

Re: The issues about architecture of the COLO checkpoint

[Zhang, Chen]

[-- Attachment #1: Type: text/plain, Size: 13842 bytes --]


On 2/15/2020 11:35 AM, Daniel Cho wrote:
> Hi Dave,
>
> Yes, I agree with you, it does need a timeout.


Hi Daniel and Dave,

Current colo-compare already have the timeout mechanism.

Named packet_check_timer,  It will scan primary packet queue to make 
sure all the primary packet not stay too long time.

If colo-compare got a primary packet without related secondary packet in 
a certain time , it will automatic trigger checkpoint.

https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847


Thanks

Zhang Chen


>
> Hi Hailiang,
>
> We base on qemu-4.1.0 for using COLO feature, in your patch, we found 
> a lot of difference  between your version and ours.
> Could you give us a latest release version which is close your 
> developing code?
>
> Thanks.
>
> Regards
> Daniel Cho
>
> Dr. David Alan Gilbert <dgilbert@redhat.com 
> <mailto:dgilbert@redhat.com>> 於 2020年2月13日 週四 下午6:38寫道：
>
>     * Daniel Cho (danielcho@qnap.com <mailto:danielcho@qnap.com>) wrote:
>     > Hi Hailiang,
>     >
>     > 1.
>     >     OK, we will try the patch
>     > “0001-COLO-Optimize-memory-back-up-process.patch”,
>     > and thanks for your help.
>     >
>     > 2.
>     >     We understand the reason to compare PVM and SVM's packet.
>     However, the
>     > empty of SVM's packet queue might happened on setting COLO
>     feature and SVM
>     > broken.
>     >
>     > On situation 1 ( setting COLO feature ):
>     >     We could force do checkpoint after setting COLO feature
>     finish, then it
>     > will protect the state of PVM and SVM . As the Zhang Chen said.
>     >
>     > On situation 2 ( SVM broken ):
>     >     COLO will do failover for PVM, so it might not cause any
>     wrong on PVM.
>     >
>     > However, those situations are our views, so there might be a big
>     difference
>     > between reality and our views.
>     > If we have any wrong views and opinions, please let us know, and
>     correct
>     > us.
>
>     It does need a timeout; the SVM being broken or being in a state where
>     it never sends the corresponding packet (because of a state
>     difference)
>     can happen and COLO needs to timeout when the packet hasn't arrived
>     after a while and trigger the checkpoint.
>
>     Dave
>
>     > Thanks.
>     >
>     > Best regards,
>     > Daniel Cho
>     >
>     > Zhang, Chen <chen.zhang@intel.com <mailto:chen.zhang@intel.com>>
>     於 2020年2月13日 週四 上午10:17寫道：
>     >
>     > > Add cc Jason Wang, he is a network expert.
>     > >
>     > > In case some network things goes wrong.
>     > >
>     > >
>     > >
>     > > Thanks
>     > >
>     > > Zhang Chen
>     > >
>     > >
>     > >
>     > > *From:* Zhang, Chen
>     > > *Sent:* Thursday, February 13, 2020 10:10 AM
>     > > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com
>     <mailto:zhang.zhanghailiang@huawei.com>>; Daniel Cho <
>     > > danielcho@qnap.com <mailto:danielcho@qnap.com>>
>     > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org
>     <mailto:qemu-devel@nongnu.org>
>     > > *Subject:* RE: The issues about architecture of the COLO
>     checkpoint
>     > >
>     > >
>     > >
>     > > For the issue 2:
>     > >
>     > >
>     > >
>     > > COLO need use the network packets to confirm PVM and SVM in
>     the same state,
>     > >
>     > > Generally speaking, we can’t send PVM packets without compared
>     with SVM
>     > > packets.
>     > >
>     > > But to prevent jamming, I think COLO can do force checkpoint
>     and send the
>     > > PVM packets in this case.
>     > >
>     > >
>     > >
>     > > Thanks
>     > >
>     > > Zhang Chen
>     > >
>     > >
>     > >
>     > > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com
>     <mailto:zhang.zhanghailiang@huawei.com>>
>     > > *Sent:* Thursday, February 13, 2020 9:45 AM
>     > > *To:* Daniel Cho <danielcho@qnap.com <mailto:danielcho@qnap.com>>
>     > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org
>     <mailto:qemu-devel@nongnu.org>;
>     > > Zhang, Chen <chen.zhang@intel.com <mailto:chen.zhang@intel.com>>
>     > > *Subject:* RE: The issues about architecture of the COLO
>     checkpoint
>     > >
>     > >
>     > >
>     > > Hi,
>     > >
>     > >
>     > >
>     > > 1.       After re-walked through the codes, yes, you are
>     right, actually,
>     > > after the first migration, we will keep dirty log on in
>     primary side,
>     > >
>     > > And only send the dirty pages in PVM to SVM. The ram cache in
>     secondary
>     > > side is always a backup of PVM, so we don’t have to
>     > >
>     > > Re-send the none-dirtied pages.
>     > >
>     > > The reason why the first checkpoint takes longer time is we
>     have to backup
>     > > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
>     > >
>     > > It is time consuming, but I have optimized in the second patch
>     > > “0001-COLO-Optimize-memory-back-up-process.patch” which you
>     can find in my
>     > > previous reply.
>     > >
>     > >
>     > >
>     > > Besides, I found that, In my previous reply “We can only copy
>     the pages
>     > > that dirtied by PVM and SVM in last checkpoint.”,
>     > >
>     > > We have done this optimization in current upstream codes.
>     > >
>     > >
>     > >
>     > > 2．I don’t quite understand this question. For COLO, we always
>     need both
>     > > network packets of PVM’s and SVM’s to compare before send this
>     packets to
>     > > client.
>     > >
>     > > It depends on this to decide whether or not PVM and SVM are in
>     same state.
>     > >
>     > >
>     > >
>     > > Thanks,
>     > >
>     > > hailiang
>     > >
>     > >
>     > >
>     > > *From:* Daniel Cho [mailto:danielcho@qnap.com
>     <mailto:danielcho@qnap.com> <danielcho@qnap.com
>     <mailto:danielcho@qnap.com>>]
>     > > *Sent:* Wednesday, February 12, 2020 4:37 PM
>     > > *To:* Zhang, Chen <chen.zhang@intel.com
>     <mailto:chen.zhang@intel.com>>
>     > > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com
>     <mailto:zhang.zhanghailiang@huawei.com>>; Dr. David Alan
>     > > Gilbert <dgilbert@redhat.com <mailto:dgilbert@redhat.com>>;
>     qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org>
>     > > *Subject:* Re: The issues about architecture of the COLO
>     checkpoint
>     > >
>     > >
>     > >
>     > > Hi Hailiang,
>     > >
>     > >
>     > >
>     > > Thanks for your replaying and explain in detail.
>     > >
>     > > We will try to use the attachments to enhance memory copy.
>     > >
>     > >
>     > >
>     > > However, we have some questions for your replying.
>     > >
>     > >
>     > >
>     > > 1.  As you said, "for each checkpoint, we have to send the
>     whole PVM's
>     > > pages To SVM", why the only first checkpoint will takes more
>     pause time?
>     > >
>     > > In our observing, the first checkpoint will take more time for
>     pausing,
>     > > then other checkpoints will takes a few time for pausing. Does
>     it means
>     > > only the first checkpoint will send the whole pages to SVM,
>     and the other
>     > > checkpoints send the dirty pages to SVM for reloading?
>     > >
>     > >
>     > >
>     > > 2. We notice the COLO-COMPARE component will stuck the packet
>     until
>     > > receive packets from PVM and SVM, as this rule, when we add the
>     > > COLO-COMPARE to PVM, its network will stuck until SVM start.
>     So it is an
>     > > other issue to make PVM stuck while setting COLO feature. With
>     this issue,
>     > > could we let colo-compare to pass the PVM's packet when the
>     SVM's packet
>     > > queue is empty? Then, the PVM's network won't stock, and "if
>     PVM runs
>     > > firstly, it still need to wait for The network packets from SVM to
>     > > compare before send it to client side" won't happened either.
>     > >
>     > >
>     > >
>     > > Best regard,
>     > >
>     > > Daniel Cho
>     > >
>     > >
>     > >
>     > > Zhang, Chen <chen.zhang@intel.com
>     <mailto:chen.zhang@intel.com>> 於 2020年2月12日 週三 下午1:45寫道：
>     > >
>     > >
>     > >
>     > > > -----Original Message-----
>     > > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com
>     <mailto:zhang.zhanghailiang@huawei.com>>
>     > > > Sent: Wednesday, February 12, 2020 11:18 AM
>     > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com>>; Daniel Cho
>     > > > <danielcho@qnap.com <mailto:danielcho@qnap.com>>; Zhang,
>     Chen <chen.zhang@intel.com <mailto:chen.zhang@intel.com>>
>     > > > Cc: qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org>
>     > > > Subject: RE: The issues about architecture of the COLO
>     checkpoint
>     > > >
>     > > > Hi,
>     > > >
>     > > > Thank you Dave,
>     > > >
>     > > > I'll reply here directly.
>     > > >
>     > > > -----Original Message-----
>     > > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com>]
>     > > > Sent: Wednesday, February 12, 2020 1:48 AM
>     > > > To: Daniel Cho <danielcho@qnap.com
>     <mailto:danielcho@qnap.com>>; chen.zhang@intel.com
>     <mailto:chen.zhang@intel.com>;
>     > > > Zhanghailiang <zhang.zhanghailiang@huawei.com
>     <mailto:zhang.zhanghailiang@huawei.com>>
>     > > > Cc: qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org>
>     > > > Subject: Re: The issues about architecture of the COLO
>     checkpoint
>     > > >
>     > > >
>     > > > cc'ing in COLO people:
>     > > >
>     > > >
>     > > > * Daniel Cho (danielcho@qnap.com
>     <mailto:danielcho@qnap.com>) wrote:
>     > > > > Hi everyone,
>     > > > >      We have some issues about setting COLO feature. Hope
>     somebody
>     > > > > could give us some advice.
>     > > > >
>     > > > > Issue 1:
>     > > > >      We dynamic to set COLO feature for PVM(2 core, 16G
>     memory),  but
>     > > > > the Primary VM will pause a long time(based on memory
>     size) for
>     > > > > waiting SVM start. Does it have any idea to reduce the
>     pause time?
>     > > > >
>     > > >
>     > > > Yes, we do have some ideas to optimize this downtime.
>     > > >
>     > > > The main problem for current version is, for each
>     checkpoint, we have to
>     > > > send the whole PVM's pages
>     > > > To SVM, and then copy the whole VM's state into SVM from ram
>     cache, in
>     > > > this process, we need both of them be paused.
>     > > > Just as you said, the downtime is based on memory size.
>     > > >
>     > > > So firstly, we need to reduce the sending data while do
>     checkpoint,
>     > > actually,
>     > > > we can migrate parts of PVM's dirty pages in background
>     > > > While both of VMs are running. And then we load these pages
>     into ram
>     > > > cache (backup memory) in SVM temporarily. While do checkpoint,
>     > > > We just send the last dirty pages of PVM to slave side and
>     then copy the
>     > > ram
>     > > > cache into SVM. Further on, we don't have
>     > > > To send the whole PVM's dirty pages, we can only send the
>     pages that
>     > > > dirtied by PVM or SVM during two checkpoints. (Because
>     > > > If one page is not dirtied by both PVM and SVM, the data of
>     this pages
>     > > will
>     > > > keep same in SVM, PVM, backup memory). This method can reduce
>     > > > the time that consumed in sending data.
>     > > >
>     > > > For the second problem, we can reduce the memory copy by two
>     methods,
>     > > > first one, we don't have to copy the whole pages in ram cache,
>     > > > We can only copy the pages that dirtied by PVM and SVM in last
>     > > checkpoint.
>     > > > Second, we can use userfault missing function to reduce the
>     > > > Time consumed in memory copy. (For the second time, in
>     theory, we can
>     > > > reduce time consumed in memory into ms level).
>     > > >
>     > > > You can find the first optimization in attachment, it is
>     based on an old
>     > > qemu
>     > > > version (qemu-2.6), it should not be difficult to rebase it
>     > > > Into master or your version. And please feel free to send
>     the new
>     > > version if
>     > > > you want into community ;)
>     > > >
>     > > >
>     > >
>     > > Thanks Hailiang!
>     > > By the way, Do you have time to push the patches to upstream?
>     > > I think this is a better and faster option.
>     > >
>     > > Thanks
>     > > Zhang Chen
>     > >
>     > > > >
>     > > > > Issue 2:
>     > > > >      In
>     > > > >
>     https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
>     > > > > could we move start_vm() before Line 488? Because at first
>     checkpoint
>     > > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
>     > > > >
>     > > >
>     > > > No, that makes no sense, because if PVM runs firstly, it
>     still need to
>     > > wait for
>     > > > The network packets from SVM to compare before send it to
>     client side.
>     > > >
>     > > >
>     > > > Thanks,
>     > > > Hailiang
>     > > >
>     > > > >      We set the COLO feature on running VM, so we hope the
>     running VM
>     > > > > could continuous service for users.
>     > > > > Do you have any suggestions for those issues?
>     > > > >
>     > > > > Best regards,
>     > > > > Daniel Cho
>     > > > --
>     > > > Dr. David Alan Gilbert / dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com> / Manchester, UK
>     > >
>     > >
>     --
>     Dr. David Alan Gilbert / dgilbert@redhat.com
>     <mailto:dgilbert@redhat.com> / Manchester, UK
>

[-- Attachment #2: Type: text/html, Size: 22593 bytes --]

Re: The issues about architecture of the COLO checkpoint

[Daniel Cho]

[-- Attachment #1: Type: text/plain, Size: 12638 bytes --]

Hi Hailiang,
Thanks for your help. If we have any problems we will contact you for your
favor.


Hi Zhang,

" If colo-compare got a primary packet without related secondary packet in
a certain time , it will automatically trigger checkpoint.  "
As you said, the colo-compare will trigger checkpoint, but does it need to
limit checkpoint times?
There is a problem about doing many checkpoints while we use fio to random
write files. Then it will cause low throughput on PVM.
Is this situation is normal on COLO?

Best regards,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com> 於 2020年2月17日 週一 下午1:36寫道：

>
> On 2/15/2020 11:35 AM, Daniel Cho wrote:
>
> Hi Dave,
>
> Yes, I agree with you, it does need a timeout.
>
>
> Hi Daniel and Dave,
>
> Current colo-compare already have the timeout mechanism.
>
> Named packet_check_timer,  It will scan primary packet queue to make sure
> all the primary packet not stay too long time.
>
> If colo-compare got a primary packet without related secondary packet in a
> certain time , it will automatic trigger checkpoint.
>
> https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847
>
>
> Thanks
>
> Zhang Chen
>
>
>
> Hi Hailiang,
>
> We base on qemu-4.1.0 for using COLO feature, in your patch, we found a
> lot of difference  between your version and ours.
> Could you give us a latest release version which is close your developing
> code?
>
> Thanks.
>
> Regards
> Daniel Cho
>
> Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2020年2月13日 週四 下午6:38寫道：
>
>> * Daniel Cho (danielcho@qnap.com) wrote:
>> > Hi Hailiang,
>> >
>> > 1.
>> >     OK, we will try the patch
>> > “0001-COLO-Optimize-memory-back-up-process.patch”,
>> > and thanks for your help.
>> >
>> > 2.
>> >     We understand the reason to compare PVM and SVM's packet. However,
>> the
>> > empty of SVM's packet queue might happened on setting COLO feature and
>> SVM
>> > broken.
>> >
>> > On situation 1 ( setting COLO feature ):
>> >     We could force do checkpoint after setting COLO feature finish,
>> then it
>> > will protect the state of PVM and SVM . As the Zhang Chen said.
>> >
>> > On situation 2 ( SVM broken ):
>> >     COLO will do failover for PVM, so it might not cause any wrong on
>> PVM.
>> >
>> > However, those situations are our views, so there might be a big
>> difference
>> > between reality and our views.
>> > If we have any wrong views and opinions, please let us know, and correct
>> > us.
>>
>> It does need a timeout; the SVM being broken or being in a state where
>> it never sends the corresponding packet (because of a state difference)
>> can happen and COLO needs to timeout when the packet hasn't arrived
>> after a while and trigger the checkpoint.
>>
>> Dave
>>
>> > Thanks.
>> >
>> > Best regards,
>> > Daniel Cho
>> >
>> > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月13日 週四 上午10:17寫道：
>> >
>> > > Add cc Jason Wang, he is a network expert.
>> > >
>> > > In case some network things goes wrong.
>> > >
>> > >
>> > >
>> > > Thanks
>> > >
>> > > Zhang Chen
>> > >
>> > >
>> > >
>> > > *From:* Zhang, Chen
>> > > *Sent:* Thursday, February 13, 2020 10:10 AM
>> > > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com>; Daniel Cho <
>> > > danielcho@qnap.com>
>> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
>> qemu-devel@nongnu.org
>> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
>> > >
>> > >
>> > >
>> > > For the issue 2:
>> > >
>> > >
>> > >
>> > > COLO need use the network packets to confirm PVM and SVM in the same
>> state,
>> > >
>> > > Generally speaking, we can’t send PVM packets without compared with
>> SVM
>> > > packets.
>> > >
>> > > But to prevent jamming, I think COLO can do force checkpoint and send
>> the
>> > > PVM packets in this case.
>> > >
>> > >
>> > >
>> > > Thanks
>> > >
>> > > Zhang Chen
>> > >
>> > >
>> > >
>> > > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com>
>> > > *Sent:* Thursday, February 13, 2020 9:45 AM
>> > > *To:* Daniel Cho <danielcho@qnap.com>
>> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
>> qemu-devel@nongnu.org;
>> > > Zhang, Chen <chen.zhang@intel.com>
>> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
>> > >
>> > >
>> > >
>> > > Hi,
>> > >
>> > >
>> > >
>> > > 1.       After re-walked through the codes, yes, you are right,
>> actually,
>> > > after the first migration, we will keep dirty log on in primary side,
>> > >
>> > > And only send the dirty pages in PVM to SVM. The ram cache in
>> secondary
>> > > side is always a backup of PVM, so we don’t have to
>> > >
>> > > Re-send the none-dirtied pages.
>> > >
>> > > The reason why the first checkpoint takes longer time is we have to
>> backup
>> > > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
>> > >
>> > > It is time consuming, but I have optimized in the second patch
>> > > “0001-COLO-Optimize-memory-back-up-process.patch” which you can find
>> in my
>> > > previous reply.
>> > >
>> > >
>> > >
>> > > Besides, I found that, In my previous reply “We can only copy the
>> pages
>> > > that dirtied by PVM and SVM in last checkpoint.”,
>> > >
>> > > We have done this optimization in current upstream codes.
>> > >
>> > >
>> > >
>> > > 2．I don’t quite understand this question. For COLO, we always need
>> both
>> > > network packets of PVM’s and SVM’s to compare before send this
>> packets to
>> > > client.
>> > >
>> > > It depends on this to decide whether or not PVM and SVM are in same
>> state.
>> > >
>> > >
>> > >
>> > > Thanks,
>> > >
>> > > hailiang
>> > >
>> > >
>> > >
>> > > *From:* Daniel Cho [mailto:danielcho@qnap.com <danielcho@qnap.com>]
>> > > *Sent:* Wednesday, February 12, 2020 4:37 PM
>> > > *To:* Zhang, Chen <chen.zhang@intel.com>
>> > > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David Alan
>> > > Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
>> > > *Subject:* Re: The issues about architecture of the COLO checkpoint
>> > >
>> > >
>> > >
>> > > Hi Hailiang,
>> > >
>> > >
>> > >
>> > > Thanks for your replaying and explain in detail.
>> > >
>> > > We will try to use the attachments to enhance memory copy.
>> > >
>> > >
>> > >
>> > > However, we have some questions for your replying.
>> > >
>> > >
>> > >
>> > > 1.  As you said, "for each checkpoint, we have to send the whole PVM's
>> > > pages To SVM", why the only first checkpoint will takes more pause
>> time?
>> > >
>> > > In our observing, the first checkpoint will take more time for
>> pausing,
>> > > then other checkpoints will takes a few time for pausing. Does it
>> means
>> > > only the first checkpoint will send the whole pages to SVM, and the
>> other
>> > > checkpoints send the dirty pages to SVM for reloading?
>> > >
>> > >
>> > >
>> > > 2. We notice the COLO-COMPARE component will stuck the packet until
>> > > receive packets from PVM and SVM, as this rule, when we add the
>> > > COLO-COMPARE to PVM, its network will stuck until SVM start. So it is
>> an
>> > > other issue to make PVM stuck while setting COLO feature. With this
>> issue,
>> > > could we let colo-compare to pass the PVM's packet when the SVM's
>> packet
>> > > queue is empty? Then, the PVM's network won't stock, and "if PVM runs
>> > > firstly, it still need to wait for The network packets from SVM to
>> > > compare before send it to client side" won't happened either.
>> > >
>> > >
>> > >
>> > > Best regard,
>> > >
>> > > Daniel Cho
>> > >
>> > >
>> > >
>> > > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月12日 週三 下午1:45寫道：
>> > >
>> > >
>> > >
>> > > > -----Original Message-----
>> > > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
>> > > > Sent: Wednesday, February 12, 2020 11:18 AM
>> > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
>> > > > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
>> > > > Cc: qemu-devel@nongnu.org
>> > > > Subject: RE: The issues about architecture of the COLO checkpoint
>> > > >
>> > > > Hi,
>> > > >
>> > > > Thank you Dave,
>> > > >
>> > > > I'll reply here directly.
>> > > >
>> > > > -----Original Message-----
>> > > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
>> > > > Sent: Wednesday, February 12, 2020 1:48 AM
>> > > > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
>> > > > Zhanghailiang <zhang.zhanghailiang@huawei.com>
>> > > > Cc: qemu-devel@nongnu.org
>> > > > Subject: Re: The issues about architecture of the COLO checkpoint
>> > > >
>> > > >
>> > > > cc'ing in COLO people:
>> > > >
>> > > >
>> > > > * Daniel Cho (danielcho@qnap.com) wrote:
>> > > > > Hi everyone,
>> > > > >      We have some issues about setting COLO feature. Hope somebody
>> > > > > could give us some advice.
>> > > > >
>> > > > > Issue 1:
>> > > > >      We dynamic to set COLO feature for PVM(2 core, 16G memory),
>> but
>> > > > > the Primary VM will pause a long time(based on memory size) for
>> > > > > waiting SVM start. Does it have any idea to reduce the pause time?
>> > > > >
>> > > >
>> > > > Yes, we do have some ideas to optimize this downtime.
>> > > >
>> > > > The main problem for current version is, for each checkpoint, we
>> have to
>> > > > send the whole PVM's pages
>> > > > To SVM, and then copy the whole VM's state into SVM from ram cache,
>> in
>> > > > this process, we need both of them be paused.
>> > > > Just as you said, the downtime is based on memory size.
>> > > >
>> > > > So firstly, we need to reduce the sending data while do checkpoint,
>> > > actually,
>> > > > we can migrate parts of PVM's dirty pages in background
>> > > > While both of VMs are running. And then we load these pages into ram
>> > > > cache (backup memory) in SVM temporarily. While do checkpoint,
>> > > > We just send the last dirty pages of PVM to slave side and then
>> copy the
>> > > ram
>> > > > cache into SVM. Further on, we don't have
>> > > > To send the whole PVM's dirty pages, we can only send the pages that
>> > > > dirtied by PVM or SVM during two checkpoints. (Because
>> > > > If one page is not dirtied by both PVM and SVM, the data of this
>> pages
>> > > will
>> > > > keep same in SVM, PVM, backup memory). This method can reduce
>> > > > the time that consumed in sending data.
>> > > >
>> > > > For the second problem, we can reduce the memory copy by two
>> methods,
>> > > > first one, we don't have to copy the whole pages in ram cache,
>> > > > We can only copy the pages that dirtied by PVM and SVM in last
>> > > checkpoint.
>> > > > Second, we can use userfault missing function to reduce the
>> > > > Time consumed in memory copy. (For the second time, in theory, we
>> can
>> > > > reduce time consumed in memory into ms level).
>> > > >
>> > > > You can find the first optimization in attachment, it is based on
>> an old
>> > > qemu
>> > > > version (qemu-2.6), it should not be difficult to rebase it
>> > > > Into master or your version. And please feel free to send the new
>> > > version if
>> > > > you want into community ;)
>> > > >
>> > > >
>> > >
>> > > Thanks Hailiang!
>> > > By the way, Do you have time to push the patches to upstream?
>> > > I think this is a better and faster option.
>> > >
>> > > Thanks
>> > > Zhang Chen
>> > >
>> > > > >
>> > > > > Issue 2:
>> > > > >      In
>> > > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
>> > > > > could we move start_vm() before Line 488? Because at first
>> checkpoint
>> > > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
>> > > > >
>> > > >
>> > > > No, that makes no sense, because if PVM runs firstly, it still need
>> to
>> > > wait for
>> > > > The network packets from SVM to compare before send it to client
>> side.
>> > > >
>> > > >
>> > > > Thanks,
>> > > > Hailiang
>> > > >
>> > > > >      We set the COLO feature on running VM, so we hope the
>> running VM
>> > > > > could continuous service for users.
>> > > > > Do you have any suggestions for those issues?
>> > > > >
>> > > > > Best regards,
>> > > > > Daniel Cho
>> > > > --
>> > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>> > >
>> > >
>> --
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>
>>

[-- Attachment #2: Type: text/html, Size: 22485 bytes --]

Re: The issues about architecture of the COLO checkpoint

[Zhang, Chen]

[-- Attachment #1: Type: text/plain, Size: 17138 bytes --]


On 2/18/2020 5:22 PM, Daniel Cho wrote:
> Hi Hailiang,
> Thanks for your help. If we have any problems we will contact you for 
> your favor.
>
>
> Hi Zhang,
>
> " If colo-compare got a primary packet without related secondary 
> packet in a certain time , it will automatically trigger checkpoint.  "
> As you said, the colo-compare will trigger checkpoint, but does it 
> need to limit checkpoint times?
> There is a problem about doing many checkpoints while we use fio to 
> random write files. Then it will cause low throughput on PVM.
> Is this situation is normal on COLO?


Hi Daniel,

The checkpoint time is designed to be user adjustable based on user 
environment(workload/network status/business conditions...).

In net/colo-compare.c

/* TODO: Should be configurable */
#define REGULAR_PACKET_CHECK_MS 3000

If you need, I can send a patch for this issue. Make users can change 
the value by QMP and qemu monitor commands.

Thanks

Zhang Chen


>
> Best regards,
> Daniel Cho
>
> Zhang, Chen <chen.zhang@intel.com <mailto:chen.zhang@intel.com>> 於 
> 2020年2月17日 週一 下午1:36寫道：
>
>
>     On 2/15/2020 11:35 AM, Daniel Cho wrote:
>>     Hi Dave,
>>
>>     Yes, I agree with you, it does need a timeout.
>
>
>     Hi Daniel and Dave,
>
>     Current colo-compare already have the timeout mechanism.
>
>     Named packet_check_timer,  It will scan primary packet queue to
>     make sure all the primary packet not stay too long time.
>
>     If colo-compare got a primary packet without related secondary
>     packet in a certain time , it will automatic trigger checkpoint.
>
>     https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847
>
>
>     Thanks
>
>     Zhang Chen
>
>
>>
>>     Hi Hailiang,
>>
>>     We base on qemu-4.1.0 for using COLO feature, in your patch, we
>>     found a lot of difference  between your version and ours.
>>     Could you give us a latest release version which is close your
>>     developing code?
>>
>>     Thanks.
>>
>>     Regards
>>     Daniel Cho
>>
>>     Dr. David Alan Gilbert <dgilbert@redhat.com
>>     <mailto:dgilbert@redhat.com>> 於 2020年2月13日 週四 下午6:38寫道：
>>
>>         * Daniel Cho (danielcho@qnap.com <mailto:danielcho@qnap.com>)
>>         wrote:
>>         > Hi Hailiang,
>>         >
>>         > 1.
>>         >     OK, we will try the patch
>>         > “0001-COLO-Optimize-memory-back-up-process.patch”,
>>         > and thanks for your help.
>>         >
>>         > 2.
>>         >     We understand the reason to compare PVM and SVM's
>>         packet. However, the
>>         > empty of SVM's packet queue might happened on setting COLO
>>         feature and SVM
>>         > broken.
>>         >
>>         > On situation 1 ( setting COLO feature ):
>>         >     We could force do checkpoint after setting COLO feature
>>         finish, then it
>>         > will protect the state of PVM and SVM . As the Zhang Chen said.
>>         >
>>         > On situation 2 ( SVM broken ):
>>         >     COLO will do failover for PVM, so it might not cause
>>         any wrong on PVM.
>>         >
>>         > However, those situations are our views, so there might be
>>         a big difference
>>         > between reality and our views.
>>         > If we have any wrong views and opinions, please let us
>>         know, and correct
>>         > us.
>>
>>         It does need a timeout; the SVM being broken or being in a
>>         state where
>>         it never sends the corresponding packet (because of a state
>>         difference)
>>         can happen and COLO needs to timeout when the packet hasn't
>>         arrived
>>         after a while and trigger the checkpoint.
>>
>>         Dave
>>
>>         > Thanks.
>>         >
>>         > Best regards,
>>         > Daniel Cho
>>         >
>>         > Zhang, Chen <chen.zhang@intel.com
>>         <mailto:chen.zhang@intel.com>> 於 2020年2月13日 週四
>>         上午10:17寫道：
>>         >
>>         > > Add cc Jason Wang, he is a network expert.
>>         > >
>>         > > In case some network things goes wrong.
>>         > >
>>         > >
>>         > >
>>         > > Thanks
>>         > >
>>         > > Zhang Chen
>>         > >
>>         > >
>>         > >
>>         > > *From:* Zhang, Chen
>>         > > *Sent:* Thursday, February 13, 2020 10:10 AM
>>         > > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com
>>         <mailto:zhang.zhanghailiang@huawei.com>>; Daniel Cho <
>>         > > danielcho@qnap.com <mailto:danielcho@qnap.com>>
>>         > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com
>>         <mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org
>>         <mailto:qemu-devel@nongnu.org>
>>         > > *Subject:* RE: The issues about architecture of the COLO
>>         checkpoint
>>         > >
>>         > >
>>         > >
>>         > > For the issue 2:
>>         > >
>>         > >
>>         > >
>>         > > COLO need use the network packets to confirm PVM and SVM
>>         in the same state,
>>         > >
>>         > > Generally speaking, we can’t send PVM packets without
>>         compared with SVM
>>         > > packets.
>>         > >
>>         > > But to prevent jamming, I think COLO can do force
>>         checkpoint and send the
>>         > > PVM packets in this case.
>>         > >
>>         > >
>>         > >
>>         > > Thanks
>>         > >
>>         > > Zhang Chen
>>         > >
>>         > >
>>         > >
>>         > > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com
>>         <mailto:zhang.zhanghailiang@huawei.com>>
>>         > > *Sent:* Thursday, February 13, 2020 9:45 AM
>>         > > *To:* Daniel Cho <danielcho@qnap.com
>>         <mailto:danielcho@qnap.com>>
>>         > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com
>>         <mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org
>>         <mailto:qemu-devel@nongnu.org>;
>>         > > Zhang, Chen <chen.zhang@intel.com
>>         <mailto:chen.zhang@intel.com>>
>>         > > *Subject:* RE: The issues about architecture of the COLO
>>         checkpoint
>>         > >
>>         > >
>>         > >
>>         > > Hi,
>>         > >
>>         > >
>>         > >
>>         > > 1.       After re-walked through the codes, yes, you are
>>         right, actually,
>>         > > after the first migration, we will keep dirty log on in
>>         primary side,
>>         > >
>>         > > And only send the dirty pages in PVM to SVM. The ram
>>         cache in secondary
>>         > > side is always a backup of PVM, so we don’t have to
>>         > >
>>         > > Re-send the none-dirtied pages.
>>         > >
>>         > > The reason why the first checkpoint takes longer time is
>>         we have to backup
>>         > > the whole VM’s ram into ram cache, that is
>>         colo_init_ram_cache().
>>         > >
>>         > > It is time consuming, but I have optimized in the second
>>         patch
>>         > > “0001-COLO-Optimize-memory-back-up-process.patch” which
>>         you can find in my
>>         > > previous reply.
>>         > >
>>         > >
>>         > >
>>         > > Besides, I found that, In my previous reply “We can only
>>         copy the pages
>>         > > that dirtied by PVM and SVM in last checkpoint.”,
>>         > >
>>         > > We have done this optimization in current upstream codes.
>>         > >
>>         > >
>>         > >
>>         > > 2．I don’t quite understand this question. For COLO, we
>>         always need both
>>         > > network packets of PVM’s and SVM’s to compare before send
>>         this packets to
>>         > > client.
>>         > >
>>         > > It depends on this to decide whether or not PVM and SVM
>>         are in same state.
>>         > >
>>         > >
>>         > >
>>         > > Thanks,
>>         > >
>>         > > hailiang
>>         > >
>>         > >
>>         > >
>>         > > *From:* Daniel Cho [mailto:danielcho@qnap.com
>>         <mailto:danielcho@qnap.com> <danielcho@qnap.com
>>         <mailto:danielcho@qnap.com>>]
>>         > > *Sent:* Wednesday, February 12, 2020 4:37 PM
>>         > > *To:* Zhang, Chen <chen.zhang@intel.com
>>         <mailto:chen.zhang@intel.com>>
>>         > > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com
>>         <mailto:zhang.zhanghailiang@huawei.com>>; Dr. David Alan
>>         > > Gilbert <dgilbert@redhat.com
>>         <mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org
>>         <mailto:qemu-devel@nongnu.org>
>>         > > *Subject:* Re: The issues about architecture of the COLO
>>         checkpoint
>>         > >
>>         > >
>>         > >
>>         > > Hi Hailiang,
>>         > >
>>         > >
>>         > >
>>         > > Thanks for your replaying and explain in detail.
>>         > >
>>         > > We will try to use the attachments to enhance memory copy.
>>         > >
>>         > >
>>         > >
>>         > > However, we have some questions for your replying.
>>         > >
>>         > >
>>         > >
>>         > > 1.  As you said, "for each checkpoint, we have to send
>>         the whole PVM's
>>         > > pages To SVM", why the only first checkpoint will takes
>>         more pause time?
>>         > >
>>         > > In our observing, the first checkpoint will take more
>>         time for pausing,
>>         > > then other checkpoints will takes a few time for pausing.
>>         Does it means
>>         > > only the first checkpoint will send the whole pages to
>>         SVM, and the other
>>         > > checkpoints send the dirty pages to SVM for reloading?
>>         > >
>>         > >
>>         > >
>>         > > 2. We notice the COLO-COMPARE component will stuck the
>>         packet until
>>         > > receive packets from PVM and SVM, as this rule, when we
>>         add the
>>         > > COLO-COMPARE to PVM, its network will stuck until SVM
>>         start. So it is an
>>         > > other issue to make PVM stuck while setting COLO feature.
>>         With this issue,
>>         > > could we let colo-compare to pass the PVM's packet when
>>         the SVM's packet
>>         > > queue is empty? Then, the PVM's network won't stock, and
>>         "if PVM runs
>>         > > firstly, it still need to wait for The network packets
>>         from SVM to
>>         > > compare before send it to client side" won't happened either.
>>         > >
>>         > >
>>         > >
>>         > > Best regard,
>>         > >
>>         > > Daniel Cho
>>         > >
>>         > >
>>         > >
>>         > > Zhang, Chen <chen.zhang@intel.com
>>         <mailto:chen.zhang@intel.com>> 於 2020年2月12日 週三
>>         下午1:45寫道：
>>         > >
>>         > >
>>         > >
>>         > > > -----Original Message-----
>>         > > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com
>>         <mailto:zhang.zhanghailiang@huawei.com>>
>>         > > > Sent: Wednesday, February 12, 2020 11:18 AM
>>         > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com
>>         <mailto:dgilbert@redhat.com>>; Daniel Cho
>>         > > > <danielcho@qnap.com <mailto:danielcho@qnap.com>>;
>>         Zhang, Chen <chen.zhang@intel.com <mailto:chen.zhang@intel.com>>
>>         > > > Cc: qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org>
>>         > > > Subject: RE: The issues about architecture of the COLO
>>         checkpoint
>>         > > >
>>         > > > Hi,
>>         > > >
>>         > > > Thank you Dave,
>>         > > >
>>         > > > I'll reply here directly.
>>         > > >
>>         > > > -----Original Message-----
>>         > > > From: Dr. David Alan Gilbert
>>         [mailto:dgilbert@redhat.com <mailto:dgilbert@redhat.com>]
>>         > > > Sent: Wednesday, February 12, 2020 1:48 AM
>>         > > > To: Daniel Cho <danielcho@qnap.com
>>         <mailto:danielcho@qnap.com>>; chen.zhang@intel.com
>>         <mailto:chen.zhang@intel.com>;
>>         > > > Zhanghailiang <zhang.zhanghailiang@huawei.com
>>         <mailto:zhang.zhanghailiang@huawei.com>>
>>         > > > Cc: qemu-devel@nongnu.org <mailto:qemu-devel@nongnu.org>
>>         > > > Subject: Re: The issues about architecture of the COLO
>>         checkpoint
>>         > > >
>>         > > >
>>         > > > cc'ing in COLO people:
>>         > > >
>>         > > >
>>         > > > * Daniel Cho (danielcho@qnap.com
>>         <mailto:danielcho@qnap.com>) wrote:
>>         > > > > Hi everyone,
>>         > > > >      We have some issues about setting COLO feature.
>>         Hope somebody
>>         > > > > could give us some advice.
>>         > > > >
>>         > > > > Issue 1:
>>         > > > >      We dynamic to set COLO feature for PVM(2 core,
>>         16G memory),  but
>>         > > > > the Primary VM will pause a long time(based on memory
>>         size) for
>>         > > > > waiting SVM start. Does it have any idea to reduce
>>         the pause time?
>>         > > > >
>>         > > >
>>         > > > Yes, we do have some ideas to optimize this downtime.
>>         > > >
>>         > > > The main problem for current version is, for each
>>         checkpoint, we have to
>>         > > > send the whole PVM's pages
>>         > > > To SVM, and then copy the whole VM's state into SVM
>>         from ram cache, in
>>         > > > this process, we need both of them be paused.
>>         > > > Just as you said, the downtime is based on memory size.
>>         > > >
>>         > > > So firstly, we need to reduce the sending data while do
>>         checkpoint,
>>         > > actually,
>>         > > > we can migrate parts of PVM's dirty pages in background
>>         > > > While both of VMs are running. And then we load these
>>         pages into ram
>>         > > > cache (backup memory) in SVM temporarily. While do
>>         checkpoint,
>>         > > > We just send the last dirty pages of PVM to slave side
>>         and then copy the
>>         > > ram
>>         > > > cache into SVM. Further on, we don't have
>>         > > > To send the whole PVM's dirty pages, we can only send
>>         the pages that
>>         > > > dirtied by PVM or SVM during two checkpoints. (Because
>>         > > > If one page is not dirtied by both PVM and SVM, the
>>         data of this pages
>>         > > will
>>         > > > keep same in SVM, PVM, backup memory). This method can
>>         reduce
>>         > > > the time that consumed in sending data.
>>         > > >
>>         > > > For the second problem, we can reduce the memory copy
>>         by two methods,
>>         > > > first one, we don't have to copy the whole pages in ram
>>         cache,
>>         > > > We can only copy the pages that dirtied by PVM and SVM
>>         in last
>>         > > checkpoint.
>>         > > > Second, we can use userfault missing function to reduce the
>>         > > > Time consumed in memory copy. (For the second time, in
>>         theory, we can
>>         > > > reduce time consumed in memory into ms level).
>>         > > >
>>         > > > You can find the first optimization in attachment, it
>>         is based on an old
>>         > > qemu
>>         > > > version (qemu-2.6), it should not be difficult to rebase it
>>         > > > Into master or your version. And please feel free to
>>         send the new
>>         > > version if
>>         > > > you want into community ;)
>>         > > >
>>         > > >
>>         > >
>>         > > Thanks Hailiang!
>>         > > By the way, Do you have time to push the patches to upstream?
>>         > > I think this is a better and faster option.
>>         > >
>>         > > Thanks
>>         > > Zhang Chen
>>         > >
>>         > > > >
>>         > > > > Issue 2:
>>         > > > >      In
>>         > > > >
>>         https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
>>         > > > > could we move start_vm() before Line 488? Because at
>>         first checkpoint
>>         > > > > PVM will wait for SVM's reply, it cause PVM stop for
>>         a while.
>>         > > > >
>>         > > >
>>         > > > No, that makes no sense, because if PVM runs firstly,
>>         it still need to
>>         > > wait for
>>         > > > The network packets from SVM to compare before send it
>>         to client side.
>>         > > >
>>         > > >
>>         > > > Thanks,
>>         > > > Hailiang
>>         > > >
>>         > > > >      We set the COLO feature on running VM, so we
>>         hope the running VM
>>         > > > > could continuous service for users.
>>         > > > > Do you have any suggestions for those issues?
>>         > > > >
>>         > > > > Best regards,
>>         > > > > Daniel Cho
>>         > > > --
>>         > > > Dr. David Alan Gilbert / dgilbert@redhat.com
>>         <mailto:dgilbert@redhat.com> / Manchester, UK
>>         > >
>>         > >
>>         --
>>         Dr. David Alan Gilbert / dgilbert@redhat.com
>>         <mailto:dgilbert@redhat.com> / Manchester, UK
>>

[-- Attachment #2: Type: text/html, Size: 30508 bytes --]

Re: The issues about architecture of the COLO checkpoint

[Daniel Cho]

[-- Attachment #1: Type: text/plain, Size: 13826 bytes --]

Hi Zhang,

Thanks, I will configure on code for testing first.
However, if you have free time, could you please send the patch file to us,
Thanks.

Best Regard,
Daniel Cho


Zhang, Chen <chen.zhang@intel.com> 於 2020年2月20日 週四 上午11:07寫道：

>
> On 2/18/2020 5:22 PM, Daniel Cho wrote:
>
> Hi Hailiang,
> Thanks for your help. If we have any problems we will contact you for your
> favor.
>
>
> Hi Zhang,
>
> " If colo-compare got a primary packet without related secondary packet in
> a certain time , it will automatically trigger checkpoint.  "
> As you said, the colo-compare will trigger checkpoint, but does it need to
> limit checkpoint times?
> There is a problem about doing many checkpoints while we use fio to random
> write files. Then it will cause low throughput on PVM.
> Is this situation is normal on COLO?
>
>
> Hi Daniel,
>
> The checkpoint time is designed to be user adjustable based on user
> environment(workload/network status/business conditions...).
>
> In net/colo-compare.c
>
> /* TODO: Should be configurable */
> #define REGULAR_PACKET_CHECK_MS 3000
>
> If you need, I can send a patch for this issue. Make users can change the
> value by QMP and qemu monitor commands.
>
> Thanks
>
> Zhang Chen
>
>
>
> Best regards,
> Daniel Cho
>
> Zhang, Chen <chen.zhang@intel.com> 於 2020年2月17日 週一 下午1:36寫道：
>
>>
>> On 2/15/2020 11:35 AM, Daniel Cho wrote:
>>
>> Hi Dave,
>>
>> Yes, I agree with you, it does need a timeout.
>>
>>
>> Hi Daniel and Dave,
>>
>> Current colo-compare already have the timeout mechanism.
>>
>> Named packet_check_timer,  It will scan primary packet queue to make sure
>> all the primary packet not stay too long time.
>>
>> If colo-compare got a primary packet without related secondary packet in
>> a certain time , it will automatic trigger checkpoint.
>>
>> https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847
>>
>>
>> Thanks
>>
>> Zhang Chen
>>
>>
>>
>> Hi Hailiang,
>>
>> We base on qemu-4.1.0 for using COLO feature, in your patch, we found a
>> lot of difference  between your version and ours.
>> Could you give us a latest release version which is close your developing
>> code?
>>
>> Thanks.
>>
>> Regards
>> Daniel Cho
>>
>> Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2020年2月13日 週四 下午6:38寫道：
>>
>>> * Daniel Cho (danielcho@qnap.com) wrote:
>>> > Hi Hailiang,
>>> >
>>> > 1.
>>> >     OK, we will try the patch
>>> > “0001-COLO-Optimize-memory-back-up-process.patch”,
>>> > and thanks for your help.
>>> >
>>> > 2.
>>> >     We understand the reason to compare PVM and SVM's packet. However,
>>> the
>>> > empty of SVM's packet queue might happened on setting COLO feature and
>>> SVM
>>> > broken.
>>> >
>>> > On situation 1 ( setting COLO feature ):
>>> >     We could force do checkpoint after setting COLO feature finish,
>>> then it
>>> > will protect the state of PVM and SVM . As the Zhang Chen said.
>>> >
>>> > On situation 2 ( SVM broken ):
>>> >     COLO will do failover for PVM, so it might not cause any wrong on
>>> PVM.
>>> >
>>> > However, those situations are our views, so there might be a big
>>> difference
>>> > between reality and our views.
>>> > If we have any wrong views and opinions, please let us know, and
>>> correct
>>> > us.
>>>
>>> It does need a timeout; the SVM being broken or being in a state where
>>> it never sends the corresponding packet (because of a state difference)
>>> can happen and COLO needs to timeout when the packet hasn't arrived
>>> after a while and trigger the checkpoint.
>>>
>>> Dave
>>>
>>> > Thanks.
>>> >
>>> > Best regards,
>>> > Daniel Cho
>>> >
>>> > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月13日 週四 上午10:17寫道：
>>> >
>>> > > Add cc Jason Wang, he is a network expert.
>>> > >
>>> > > In case some network things goes wrong.
>>> > >
>>> > >
>>> > >
>>> > > Thanks
>>> > >
>>> > > Zhang Chen
>>> > >
>>> > >
>>> > >
>>> > > *From:* Zhang, Chen
>>> > > *Sent:* Thursday, February 13, 2020 10:10 AM
>>> > > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com>; Daniel Cho <
>>> > > danielcho@qnap.com>
>>> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
>>> qemu-devel@nongnu.org
>>> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
>>> > >
>>> > >
>>> > >
>>> > > For the issue 2:
>>> > >
>>> > >
>>> > >
>>> > > COLO need use the network packets to confirm PVM and SVM in the same
>>> state,
>>> > >
>>> > > Generally speaking, we can’t send PVM packets without compared with
>>> SVM
>>> > > packets.
>>> > >
>>> > > But to prevent jamming, I think COLO can do force checkpoint and
>>> send the
>>> > > PVM packets in this case.
>>> > >
>>> > >
>>> > >
>>> > > Thanks
>>> > >
>>> > > Zhang Chen
>>> > >
>>> > >
>>> > >
>>> > > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com>
>>> > > *Sent:* Thursday, February 13, 2020 9:45 AM
>>> > > *To:* Daniel Cho <danielcho@qnap.com>
>>> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
>>> qemu-devel@nongnu.org;
>>> > > Zhang, Chen <chen.zhang@intel.com>
>>> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
>>> > >
>>> > >
>>> > >
>>> > > Hi,
>>> > >
>>> > >
>>> > >
>>> > > 1.       After re-walked through the codes, yes, you are right,
>>> actually,
>>> > > after the first migration, we will keep dirty log on in primary side,
>>> > >
>>> > > And only send the dirty pages in PVM to SVM. The ram cache in
>>> secondary
>>> > > side is always a backup of PVM, so we don’t have to
>>> > >
>>> > > Re-send the none-dirtied pages.
>>> > >
>>> > > The reason why the first checkpoint takes longer time is we have to
>>> backup
>>> > > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
>>> > >
>>> > > It is time consuming, but I have optimized in the second patch
>>> > > “0001-COLO-Optimize-memory-back-up-process.patch” which you can find
>>> in my
>>> > > previous reply.
>>> > >
>>> > >
>>> > >
>>> > > Besides, I found that, In my previous reply “We can only copy the
>>> pages
>>> > > that dirtied by PVM and SVM in last checkpoint.”,
>>> > >
>>> > > We have done this optimization in current upstream codes.
>>> > >
>>> > >
>>> > >
>>> > > 2．I don’t quite understand this question. For COLO, we always need
>>> both
>>> > > network packets of PVM’s and SVM’s to compare before send this
>>> packets to
>>> > > client.
>>> > >
>>> > > It depends on this to decide whether or not PVM and SVM are in same
>>> state.
>>> > >
>>> > >
>>> > >
>>> > > Thanks,
>>> > >
>>> > > hailiang
>>> > >
>>> > >
>>> > >
>>> > > *From:* Daniel Cho [mailto:danielcho@qnap.com <danielcho@qnap.com>]
>>> > > *Sent:* Wednesday, February 12, 2020 4:37 PM
>>> > > *To:* Zhang, Chen <chen.zhang@intel.com>
>>> > > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David Alan
>>> > > Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
>>> > > *Subject:* Re: The issues about architecture of the COLO checkpoint
>>> > >
>>> > >
>>> > >
>>> > > Hi Hailiang,
>>> > >
>>> > >
>>> > >
>>> > > Thanks for your replaying and explain in detail.
>>> > >
>>> > > We will try to use the attachments to enhance memory copy.
>>> > >
>>> > >
>>> > >
>>> > > However, we have some questions for your replying.
>>> > >
>>> > >
>>> > >
>>> > > 1.  As you said, "for each checkpoint, we have to send the whole
>>> PVM's
>>> > > pages To SVM", why the only first checkpoint will takes more pause
>>> time?
>>> > >
>>> > > In our observing, the first checkpoint will take more time for
>>> pausing,
>>> > > then other checkpoints will takes a few time for pausing. Does it
>>> means
>>> > > only the first checkpoint will send the whole pages to SVM, and the
>>> other
>>> > > checkpoints send the dirty pages to SVM for reloading?
>>> > >
>>> > >
>>> > >
>>> > > 2. We notice the COLO-COMPARE component will stuck the packet until
>>> > > receive packets from PVM and SVM, as this rule, when we add the
>>> > > COLO-COMPARE to PVM, its network will stuck until SVM start. So it
>>> is an
>>> > > other issue to make PVM stuck while setting COLO feature. With this
>>> issue,
>>> > > could we let colo-compare to pass the PVM's packet when the SVM's
>>> packet
>>> > > queue is empty? Then, the PVM's network won't stock, and "if PVM runs
>>> > > firstly, it still need to wait for The network packets from SVM to
>>> > > compare before send it to client side" won't happened either.
>>> > >
>>> > >
>>> > >
>>> > > Best regard,
>>> > >
>>> > > Daniel Cho
>>> > >
>>> > >
>>> > >
>>> > > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月12日 週三 下午1:45寫道：
>>> > >
>>> > >
>>> > >
>>> > > > -----Original Message-----
>>> > > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
>>> > > > Sent: Wednesday, February 12, 2020 11:18 AM
>>> > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
>>> > > > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
>>> > > > Cc: qemu-devel@nongnu.org
>>> > > > Subject: RE: The issues about architecture of the COLO checkpoint
>>> > > >
>>> > > > Hi,
>>> > > >
>>> > > > Thank you Dave,
>>> > > >
>>> > > > I'll reply here directly.
>>> > > >
>>> > > > -----Original Message-----
>>> > > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
>>> > > > Sent: Wednesday, February 12, 2020 1:48 AM
>>> > > > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
>>> > > > Zhanghailiang <zhang.zhanghailiang@huawei.com>
>>> > > > Cc: qemu-devel@nongnu.org
>>> > > > Subject: Re: The issues about architecture of the COLO checkpoint
>>> > > >
>>> > > >
>>> > > > cc'ing in COLO people:
>>> > > >
>>> > > >
>>> > > > * Daniel Cho (danielcho@qnap.com) wrote:
>>> > > > > Hi everyone,
>>> > > > >      We have some issues about setting COLO feature. Hope
>>> somebody
>>> > > > > could give us some advice.
>>> > > > >
>>> > > > > Issue 1:
>>> > > > >      We dynamic to set COLO feature for PVM(2 core, 16G
>>> memory),  but
>>> > > > > the Primary VM will pause a long time(based on memory size) for
>>> > > > > waiting SVM start. Does it have any idea to reduce the pause
>>> time?
>>> > > > >
>>> > > >
>>> > > > Yes, we do have some ideas to optimize this downtime.
>>> > > >
>>> > > > The main problem for current version is, for each checkpoint, we
>>> have to
>>> > > > send the whole PVM's pages
>>> > > > To SVM, and then copy the whole VM's state into SVM from ram
>>> cache, in
>>> > > > this process, we need both of them be paused.
>>> > > > Just as you said, the downtime is based on memory size.
>>> > > >
>>> > > > So firstly, we need to reduce the sending data while do checkpoint,
>>> > > actually,
>>> > > > we can migrate parts of PVM's dirty pages in background
>>> > > > While both of VMs are running. And then we load these pages into
>>> ram
>>> > > > cache (backup memory) in SVM temporarily. While do checkpoint,
>>> > > > We just send the last dirty pages of PVM to slave side and then
>>> copy the
>>> > > ram
>>> > > > cache into SVM. Further on, we don't have
>>> > > > To send the whole PVM's dirty pages, we can only send the pages
>>> that
>>> > > > dirtied by PVM or SVM during two checkpoints. (Because
>>> > > > If one page is not dirtied by both PVM and SVM, the data of this
>>> pages
>>> > > will
>>> > > > keep same in SVM, PVM, backup memory). This method can reduce
>>> > > > the time that consumed in sending data.
>>> > > >
>>> > > > For the second problem, we can reduce the memory copy by two
>>> methods,
>>> > > > first one, we don't have to copy the whole pages in ram cache,
>>> > > > We can only copy the pages that dirtied by PVM and SVM in last
>>> > > checkpoint.
>>> > > > Second, we can use userfault missing function to reduce the
>>> > > > Time consumed in memory copy. (For the second time, in theory, we
>>> can
>>> > > > reduce time consumed in memory into ms level).
>>> > > >
>>> > > > You can find the first optimization in attachment, it is based on
>>> an old
>>> > > qemu
>>> > > > version (qemu-2.6), it should not be difficult to rebase it
>>> > > > Into master or your version. And please feel free to send the new
>>> > > version if
>>> > > > you want into community ;)
>>> > > >
>>> > > >
>>> > >
>>> > > Thanks Hailiang!
>>> > > By the way, Do you have time to push the patches to upstream?
>>> > > I think this is a better and faster option.
>>> > >
>>> > > Thanks
>>> > > Zhang Chen
>>> > >
>>> > > > >
>>> > > > > Issue 2:
>>> > > > >      In
>>> > > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
>>> > > > > could we move start_vm() before Line 488? Because at first
>>> checkpoint
>>> > > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
>>> > > > >
>>> > > >
>>> > > > No, that makes no sense, because if PVM runs firstly, it still
>>> need to
>>> > > wait for
>>> > > > The network packets from SVM to compare before send it to client
>>> side.
>>> > > >
>>> > > >
>>> > > > Thanks,
>>> > > > Hailiang
>>> > > >
>>> > > > >      We set the COLO feature on running VM, so we hope the
>>> running VM
>>> > > > > could continuous service for users.
>>> > > > > Do you have any suggestions for those issues?
>>> > > > >
>>> > > > > Best regards,
>>> > > > > Daniel Cho
>>> > > > --
>>> > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>> > >
>>> > >
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>>>

[-- Attachment #2: Type: text/html, Size: 29207 bytes --]

Re: The issues about architecture of the COLO checkpoint

[Daniel Cho]

[-- Attachment #1: Type: text/plain, Size: 14740 bytes --]

Hi Hailiang,

I have already patched the file to my branch, but there is a problem while
doing migration.
Here is the error message from SVM
"qemu-system-x86_64: /root/download/qemu-4.1.0/memory.c:1079:
memory_region_transaction_commit: Assertion `qemu_mutex_iothread_locked()'
failed."

Do you have this problem?

Best regards,
Daniel Cho

Daniel Cho <danielcho@qnap.com> 於 2020年2月20日 週四 上午11:49寫道：

> Hi Zhang,
>
> Thanks, I will configure on code for testing first.
> However, if you have free time, could you please send the patch file to
> us, Thanks.
>
> Best Regard,
> Daniel Cho
>
>
> Zhang, Chen <chen.zhang@intel.com> 於 2020年2月20日 週四 上午11:07寫道：
>
>>
>> On 2/18/2020 5:22 PM, Daniel Cho wrote:
>>
>> Hi Hailiang,
>> Thanks for your help. If we have any problems we will contact you for
>> your favor.
>>
>>
>> Hi Zhang,
>>
>> " If colo-compare got a primary packet without related secondary packet
>> in a certain time , it will automatically trigger checkpoint.  "
>> As you said, the colo-compare will trigger checkpoint, but does it need
>> to limit checkpoint times?
>> There is a problem about doing many checkpoints while we use fio to
>> random write files. Then it will cause low throughput on PVM.
>> Is this situation is normal on COLO?
>>
>>
>> Hi Daniel,
>>
>> The checkpoint time is designed to be user adjustable based on user
>> environment(workload/network status/business conditions...).
>>
>> In net/colo-compare.c
>>
>> /* TODO: Should be configurable */
>> #define REGULAR_PACKET_CHECK_MS 3000
>>
>> If you need, I can send a patch for this issue. Make users can change the
>> value by QMP and qemu monitor commands.
>>
>> Thanks
>>
>> Zhang Chen
>>
>>
>>
>> Best regards,
>> Daniel Cho
>>
>> Zhang, Chen <chen.zhang@intel.com> 於 2020年2月17日 週一 下午1:36寫道：
>>
>>>
>>> On 2/15/2020 11:35 AM, Daniel Cho wrote:
>>>
>>> Hi Dave,
>>>
>>> Yes, I agree with you, it does need a timeout.
>>>
>>>
>>> Hi Daniel and Dave,
>>>
>>> Current colo-compare already have the timeout mechanism.
>>>
>>> Named packet_check_timer,  It will scan primary packet queue to make
>>> sure all the primary packet not stay too long time.
>>>
>>> If colo-compare got a primary packet without related secondary packet in
>>> a certain time , it will automatic trigger checkpoint.
>>>
>>> https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847
>>>
>>>
>>> Thanks
>>>
>>> Zhang Chen
>>>
>>>
>>>
>>> Hi Hailiang,
>>>
>>> We base on qemu-4.1.0 for using COLO feature, in your patch, we found a
>>> lot of difference  between your version and ours.
>>> Could you give us a latest release version which is close your
>>> developing code?
>>>
>>> Thanks.
>>>
>>> Regards
>>> Daniel Cho
>>>
>>> Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2020年2月13日 週四 下午6:38寫道：
>>>
>>>> * Daniel Cho (danielcho@qnap.com) wrote:
>>>> > Hi Hailiang,
>>>> >
>>>> > 1.
>>>> >     OK, we will try the patch
>>>> > “0001-COLO-Optimize-memory-back-up-process.patch”,
>>>> > and thanks for your help.
>>>> >
>>>> > 2.
>>>> >     We understand the reason to compare PVM and SVM's packet.
>>>> However, the
>>>> > empty of SVM's packet queue might happened on setting COLO feature
>>>> and SVM
>>>> > broken.
>>>> >
>>>> > On situation 1 ( setting COLO feature ):
>>>> >     We could force do checkpoint after setting COLO feature finish,
>>>> then it
>>>> > will protect the state of PVM and SVM . As the Zhang Chen said.
>>>> >
>>>> > On situation 2 ( SVM broken ):
>>>> >     COLO will do failover for PVM, so it might not cause any wrong on
>>>> PVM.
>>>> >
>>>> > However, those situations are our views, so there might be a big
>>>> difference
>>>> > between reality and our views.
>>>> > If we have any wrong views and opinions, please let us know, and
>>>> correct
>>>> > us.
>>>>
>>>> It does need a timeout; the SVM being broken or being in a state where
>>>> it never sends the corresponding packet (because of a state difference)
>>>> can happen and COLO needs to timeout when the packet hasn't arrived
>>>> after a while and trigger the checkpoint.
>>>>
>>>> Dave
>>>>
>>>> > Thanks.
>>>> >
>>>> > Best regards,
>>>> > Daniel Cho
>>>> >
>>>> > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月13日 週四 上午10:17寫道：
>>>> >
>>>> > > Add cc Jason Wang, he is a network expert.
>>>> > >
>>>> > > In case some network things goes wrong.
>>>> > >
>>>> > >
>>>> > >
>>>> > > Thanks
>>>> > >
>>>> > > Zhang Chen
>>>> > >
>>>> > >
>>>> > >
>>>> > > *From:* Zhang, Chen
>>>> > > *Sent:* Thursday, February 13, 2020 10:10 AM
>>>> > > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com>; Daniel Cho
>>>> <
>>>> > > danielcho@qnap.com>
>>>> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
>>>> qemu-devel@nongnu.org
>>>> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
>>>> > >
>>>> > >
>>>> > >
>>>> > > For the issue 2:
>>>> > >
>>>> > >
>>>> > >
>>>> > > COLO need use the network packets to confirm PVM and SVM in the
>>>> same state,
>>>> > >
>>>> > > Generally speaking, we can’t send PVM packets without compared with
>>>> SVM
>>>> > > packets.
>>>> > >
>>>> > > But to prevent jamming, I think COLO can do force checkpoint and
>>>> send the
>>>> > > PVM packets in this case.
>>>> > >
>>>> > >
>>>> > >
>>>> > > Thanks
>>>> > >
>>>> > > Zhang Chen
>>>> > >
>>>> > >
>>>> > >
>>>> > > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>> > > *Sent:* Thursday, February 13, 2020 9:45 AM
>>>> > > *To:* Daniel Cho <danielcho@qnap.com>
>>>> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
>>>> qemu-devel@nongnu.org;
>>>> > > Zhang, Chen <chen.zhang@intel.com>
>>>> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
>>>> > >
>>>> > >
>>>> > >
>>>> > > Hi,
>>>> > >
>>>> > >
>>>> > >
>>>> > > 1.       After re-walked through the codes, yes, you are right,
>>>> actually,
>>>> > > after the first migration, we will keep dirty log on in primary
>>>> side,
>>>> > >
>>>> > > And only send the dirty pages in PVM to SVM. The ram cache in
>>>> secondary
>>>> > > side is always a backup of PVM, so we don’t have to
>>>> > >
>>>> > > Re-send the none-dirtied pages.
>>>> > >
>>>> > > The reason why the first checkpoint takes longer time is we have to
>>>> backup
>>>> > > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
>>>> > >
>>>> > > It is time consuming, but I have optimized in the second patch
>>>> > > “0001-COLO-Optimize-memory-back-up-process.patch” which you can
>>>> find in my
>>>> > > previous reply.
>>>> > >
>>>> > >
>>>> > >
>>>> > > Besides, I found that, In my previous reply “We can only copy the
>>>> pages
>>>> > > that dirtied by PVM and SVM in last checkpoint.”,
>>>> > >
>>>> > > We have done this optimization in current upstream codes.
>>>> > >
>>>> > >
>>>> > >
>>>> > > 2．I don’t quite understand this question. For COLO, we always need
>>>> both
>>>> > > network packets of PVM’s and SVM’s to compare before send this
>>>> packets to
>>>> > > client.
>>>> > >
>>>> > > It depends on this to decide whether or not PVM and SVM are in same
>>>> state.
>>>> > >
>>>> > >
>>>> > >
>>>> > > Thanks,
>>>> > >
>>>> > > hailiang
>>>> > >
>>>> > >
>>>> > >
>>>> > > *From:* Daniel Cho [mailto:danielcho@qnap.com <danielcho@qnap.com>]
>>>> > > *Sent:* Wednesday, February 12, 2020 4:37 PM
>>>> > > *To:* Zhang, Chen <chen.zhang@intel.com>
>>>> > > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David
>>>> Alan
>>>> > > Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
>>>> > > *Subject:* Re: The issues about architecture of the COLO checkpoint
>>>> > >
>>>> > >
>>>> > >
>>>> > > Hi Hailiang,
>>>> > >
>>>> > >
>>>> > >
>>>> > > Thanks for your replaying and explain in detail.
>>>> > >
>>>> > > We will try to use the attachments to enhance memory copy.
>>>> > >
>>>> > >
>>>> > >
>>>> > > However, we have some questions for your replying.
>>>> > >
>>>> > >
>>>> > >
>>>> > > 1.  As you said, "for each checkpoint, we have to send the whole
>>>> PVM's
>>>> > > pages To SVM", why the only first checkpoint will takes more pause
>>>> time?
>>>> > >
>>>> > > In our observing, the first checkpoint will take more time for
>>>> pausing,
>>>> > > then other checkpoints will takes a few time for pausing. Does it
>>>> means
>>>> > > only the first checkpoint will send the whole pages to SVM, and the
>>>> other
>>>> > > checkpoints send the dirty pages to SVM for reloading?
>>>> > >
>>>> > >
>>>> > >
>>>> > > 2. We notice the COLO-COMPARE component will stuck the packet until
>>>> > > receive packets from PVM and SVM, as this rule, when we add the
>>>> > > COLO-COMPARE to PVM, its network will stuck until SVM start. So it
>>>> is an
>>>> > > other issue to make PVM stuck while setting COLO feature. With this
>>>> issue,
>>>> > > could we let colo-compare to pass the PVM's packet when the SVM's
>>>> packet
>>>> > > queue is empty? Then, the PVM's network won't stock, and "if PVM
>>>> runs
>>>> > > firstly, it still need to wait for The network packets from SVM to
>>>> > > compare before send it to client side" won't happened either.
>>>> > >
>>>> > >
>>>> > >
>>>> > > Best regard,
>>>> > >
>>>> > > Daniel Cho
>>>> > >
>>>> > >
>>>> > >
>>>> > > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月12日 週三 下午1:45寫道：
>>>> > >
>>>> > >
>>>> > >
>>>> > > > -----Original Message-----
>>>> > > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>> > > > Sent: Wednesday, February 12, 2020 11:18 AM
>>>> > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
>>>> > > > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
>>>> > > > Cc: qemu-devel@nongnu.org
>>>> > > > Subject: RE: The issues about architecture of the COLO checkpoint
>>>> > > >
>>>> > > > Hi,
>>>> > > >
>>>> > > > Thank you Dave,
>>>> > > >
>>>> > > > I'll reply here directly.
>>>> > > >
>>>> > > > -----Original Message-----
>>>> > > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
>>>> > > > Sent: Wednesday, February 12, 2020 1:48 AM
>>>> > > > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
>>>> > > > Zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>> > > > Cc: qemu-devel@nongnu.org
>>>> > > > Subject: Re: The issues about architecture of the COLO checkpoint
>>>> > > >
>>>> > > >
>>>> > > > cc'ing in COLO people:
>>>> > > >
>>>> > > >
>>>> > > > * Daniel Cho (danielcho@qnap.com) wrote:
>>>> > > > > Hi everyone,
>>>> > > > >      We have some issues about setting COLO feature. Hope
>>>> somebody
>>>> > > > > could give us some advice.
>>>> > > > >
>>>> > > > > Issue 1:
>>>> > > > >      We dynamic to set COLO feature for PVM(2 core, 16G
>>>> memory),  but
>>>> > > > > the Primary VM will pause a long time(based on memory size) for
>>>> > > > > waiting SVM start. Does it have any idea to reduce the pause
>>>> time?
>>>> > > > >
>>>> > > >
>>>> > > > Yes, we do have some ideas to optimize this downtime.
>>>> > > >
>>>> > > > The main problem for current version is, for each checkpoint, we
>>>> have to
>>>> > > > send the whole PVM's pages
>>>> > > > To SVM, and then copy the whole VM's state into SVM from ram
>>>> cache, in
>>>> > > > this process, we need both of them be paused.
>>>> > > > Just as you said, the downtime is based on memory size.
>>>> > > >
>>>> > > > So firstly, we need to reduce the sending data while do
>>>> checkpoint,
>>>> > > actually,
>>>> > > > we can migrate parts of PVM's dirty pages in background
>>>> > > > While both of VMs are running. And then we load these pages into
>>>> ram
>>>> > > > cache (backup memory) in SVM temporarily. While do checkpoint,
>>>> > > > We just send the last dirty pages of PVM to slave side and then
>>>> copy the
>>>> > > ram
>>>> > > > cache into SVM. Further on, we don't have
>>>> > > > To send the whole PVM's dirty pages, we can only send the pages
>>>> that
>>>> > > > dirtied by PVM or SVM during two checkpoints. (Because
>>>> > > > If one page is not dirtied by both PVM and SVM, the data of this
>>>> pages
>>>> > > will
>>>> > > > keep same in SVM, PVM, backup memory). This method can reduce
>>>> > > > the time that consumed in sending data.
>>>> > > >
>>>> > > > For the second problem, we can reduce the memory copy by two
>>>> methods,
>>>> > > > first one, we don't have to copy the whole pages in ram cache,
>>>> > > > We can only copy the pages that dirtied by PVM and SVM in last
>>>> > > checkpoint.
>>>> > > > Second, we can use userfault missing function to reduce the
>>>> > > > Time consumed in memory copy. (For the second time, in theory, we
>>>> can
>>>> > > > reduce time consumed in memory into ms level).
>>>> > > >
>>>> > > > You can find the first optimization in attachment, it is based on
>>>> an old
>>>> > > qemu
>>>> > > > version (qemu-2.6), it should not be difficult to rebase it
>>>> > > > Into master or your version. And please feel free to send the new
>>>> > > version if
>>>> > > > you want into community ;)
>>>> > > >
>>>> > > >
>>>> > >
>>>> > > Thanks Hailiang!
>>>> > > By the way, Do you have time to push the patches to upstream?
>>>> > > I think this is a better and faster option.
>>>> > >
>>>> > > Thanks
>>>> > > Zhang Chen
>>>> > >
>>>> > > > >
>>>> > > > > Issue 2:
>>>> > > > >      In
>>>> > > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
>>>> > > > > could we move start_vm() before Line 488? Because at first
>>>> checkpoint
>>>> > > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
>>>> > > > >
>>>> > > >
>>>> > > > No, that makes no sense, because if PVM runs firstly, it still
>>>> need to
>>>> > > wait for
>>>> > > > The network packets from SVM to compare before send it to client
>>>> side.
>>>> > > >
>>>> > > >
>>>> > > > Thanks,
>>>> > > > Hailiang
>>>> > > >
>>>> > > > >      We set the COLO feature on running VM, so we hope the
>>>> running VM
>>>> > > > > could continuous service for users.
>>>> > > > > Do you have any suggestions for those issues?
>>>> > > > >
>>>> > > > > Best regards,
>>>> > > > > Daniel Cho
>>>> > > > --
>>>> > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>> > >
>>>> > >
>>>> --
>>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>>
>>>>

[-- Attachment #2: Type: text/html, Size: 30050 bytes --]

Re: The issues about architecture of the COLO checkpoint

[Dr. David Alan Gilbert]

* Daniel Cho (danielcho@qnap.com) wrote:
> Hi Hailiang,
> 
> I have already patched the file to my branch, but there is a problem while
> doing migration.
> Here is the error message from SVM
> "qemu-system-x86_64: /root/download/qemu-4.1.0/memory.c:1079:
> memory_region_transaction_commit: Assertion `qemu_mutex_iothread_locked()'
> failed."

It's probably worth getting the full backtrace.

Dave

> Do you have this problem?
> 
> Best regards,
> Daniel Cho
> 
> Daniel Cho <danielcho@qnap.com> 於 2020年2月20日 週四 上午11:49寫道：
> 
> > Hi Zhang,
> >
> > Thanks, I will configure on code for testing first.
> > However, if you have free time, could you please send the patch file to
> > us, Thanks.
> >
> > Best Regard,
> > Daniel Cho
> >
> >
> > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月20日 週四 上午11:07寫道：
> >
> >>
> >> On 2/18/2020 5:22 PM, Daniel Cho wrote:
> >>
> >> Hi Hailiang,
> >> Thanks for your help. If we have any problems we will contact you for
> >> your favor.
> >>
> >>
> >> Hi Zhang,
> >>
> >> " If colo-compare got a primary packet without related secondary packet
> >> in a certain time , it will automatically trigger checkpoint.  "
> >> As you said, the colo-compare will trigger checkpoint, but does it need
> >> to limit checkpoint times?
> >> There is a problem about doing many checkpoints while we use fio to
> >> random write files. Then it will cause low throughput on PVM.
> >> Is this situation is normal on COLO?
> >>
> >>
> >> Hi Daniel,
> >>
> >> The checkpoint time is designed to be user adjustable based on user
> >> environment(workload/network status/business conditions...).
> >>
> >> In net/colo-compare.c
> >>
> >> /* TODO: Should be configurable */
> >> #define REGULAR_PACKET_CHECK_MS 3000
> >>
> >> If you need, I can send a patch for this issue. Make users can change the
> >> value by QMP and qemu monitor commands.
> >>
> >> Thanks
> >>
> >> Zhang Chen
> >>
> >>
> >>
> >> Best regards,
> >> Daniel Cho
> >>
> >> Zhang, Chen <chen.zhang@intel.com> 於 2020年2月17日 週一 下午1:36寫道：
> >>
> >>>
> >>> On 2/15/2020 11:35 AM, Daniel Cho wrote:
> >>>
> >>> Hi Dave,
> >>>
> >>> Yes, I agree with you, it does need a timeout.
> >>>
> >>>
> >>> Hi Daniel and Dave,
> >>>
> >>> Current colo-compare already have the timeout mechanism.
> >>>
> >>> Named packet_check_timer,  It will scan primary packet queue to make
> >>> sure all the primary packet not stay too long time.
> >>>
> >>> If colo-compare got a primary packet without related secondary packet in
> >>> a certain time , it will automatic trigger checkpoint.
> >>>
> >>> https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847
> >>>
> >>>
> >>> Thanks
> >>>
> >>> Zhang Chen
> >>>
> >>>
> >>>
> >>> Hi Hailiang,
> >>>
> >>> We base on qemu-4.1.0 for using COLO feature, in your patch, we found a
> >>> lot of difference  between your version and ours.
> >>> Could you give us a latest release version which is close your
> >>> developing code?
> >>>
> >>> Thanks.
> >>>
> >>> Regards
> >>> Daniel Cho
> >>>
> >>> Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2020年2月13日 週四 下午6:38寫道：
> >>>
> >>>> * Daniel Cho (danielcho@qnap.com) wrote:
> >>>> > Hi Hailiang,
> >>>> >
> >>>> > 1.
> >>>> >     OK, we will try the patch
> >>>> > “0001-COLO-Optimize-memory-back-up-process.patch”,
> >>>> > and thanks for your help.
> >>>> >
> >>>> > 2.
> >>>> >     We understand the reason to compare PVM and SVM's packet.
> >>>> However, the
> >>>> > empty of SVM's packet queue might happened on setting COLO feature
> >>>> and SVM
> >>>> > broken.
> >>>> >
> >>>> > On situation 1 ( setting COLO feature ):
> >>>> >     We could force do checkpoint after setting COLO feature finish,
> >>>> then it
> >>>> > will protect the state of PVM and SVM . As the Zhang Chen said.
> >>>> >
> >>>> > On situation 2 ( SVM broken ):
> >>>> >     COLO will do failover for PVM, so it might not cause any wrong on
> >>>> PVM.
> >>>> >
> >>>> > However, those situations are our views, so there might be a big
> >>>> difference
> >>>> > between reality and our views.
> >>>> > If we have any wrong views and opinions, please let us know, and
> >>>> correct
> >>>> > us.
> >>>>
> >>>> It does need a timeout; the SVM being broken or being in a state where
> >>>> it never sends the corresponding packet (because of a state difference)
> >>>> can happen and COLO needs to timeout when the packet hasn't arrived
> >>>> after a while and trigger the checkpoint.
> >>>>
> >>>> Dave
> >>>>
> >>>> > Thanks.
> >>>> >
> >>>> > Best regards,
> >>>> > Daniel Cho
> >>>> >
> >>>> > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月13日 週四 上午10:17寫道：
> >>>> >
> >>>> > > Add cc Jason Wang, he is a network expert.
> >>>> > >
> >>>> > > In case some network things goes wrong.
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Thanks
> >>>> > >
> >>>> > > Zhang Chen
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > *From:* Zhang, Chen
> >>>> > > *Sent:* Thursday, February 13, 2020 10:10 AM
> >>>> > > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com>; Daniel Cho
> >>>> <
> >>>> > > danielcho@qnap.com>
> >>>> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
> >>>> qemu-devel@nongnu.org
> >>>> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > For the issue 2:
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > COLO need use the network packets to confirm PVM and SVM in the
> >>>> same state,
> >>>> > >
> >>>> > > Generally speaking, we can’t send PVM packets without compared with
> >>>> SVM
> >>>> > > packets.
> >>>> > >
> >>>> > > But to prevent jamming, I think COLO can do force checkpoint and
> >>>> send the
> >>>> > > PVM packets in this case.
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Thanks
> >>>> > >
> >>>> > > Zhang Chen
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com>
> >>>> > > *Sent:* Thursday, February 13, 2020 9:45 AM
> >>>> > > *To:* Daniel Cho <danielcho@qnap.com>
> >>>> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
> >>>> qemu-devel@nongnu.org;
> >>>> > > Zhang, Chen <chen.zhang@intel.com>
> >>>> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Hi,
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1.       After re-walked through the codes, yes, you are right,
> >>>> actually,
> >>>> > > after the first migration, we will keep dirty log on in primary
> >>>> side,
> >>>> > >
> >>>> > > And only send the dirty pages in PVM to SVM. The ram cache in
> >>>> secondary
> >>>> > > side is always a backup of PVM, so we don’t have to
> >>>> > >
> >>>> > > Re-send the none-dirtied pages.
> >>>> > >
> >>>> > > The reason why the first checkpoint takes longer time is we have to
> >>>> backup
> >>>> > > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
> >>>> > >
> >>>> > > It is time consuming, but I have optimized in the second patch
> >>>> > > “0001-COLO-Optimize-memory-back-up-process.patch” which you can
> >>>> find in my
> >>>> > > previous reply.
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Besides, I found that, In my previous reply “We can only copy the
> >>>> pages
> >>>> > > that dirtied by PVM and SVM in last checkpoint.”,
> >>>> > >
> >>>> > > We have done this optimization in current upstream codes.
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2．I don’t quite understand this question. For COLO, we always need
> >>>> both
> >>>> > > network packets of PVM’s and SVM’s to compare before send this
> >>>> packets to
> >>>> > > client.
> >>>> > >
> >>>> > > It depends on this to decide whether or not PVM and SVM are in same
> >>>> state.
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Thanks,
> >>>> > >
> >>>> > > hailiang
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > *From:* Daniel Cho [mailto:danielcho@qnap.com <danielcho@qnap.com>]
> >>>> > > *Sent:* Wednesday, February 12, 2020 4:37 PM
> >>>> > > *To:* Zhang, Chen <chen.zhang@intel.com>
> >>>> > > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David
> >>>> Alan
> >>>> > > Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> >>>> > > *Subject:* Re: The issues about architecture of the COLO checkpoint
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Hi Hailiang,
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Thanks for your replaying and explain in detail.
> >>>> > >
> >>>> > > We will try to use the attachments to enhance memory copy.
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > However, we have some questions for your replying.
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 1.  As you said, "for each checkpoint, we have to send the whole
> >>>> PVM's
> >>>> > > pages To SVM", why the only first checkpoint will takes more pause
> >>>> time?
> >>>> > >
> >>>> > > In our observing, the first checkpoint will take more time for
> >>>> pausing,
> >>>> > > then other checkpoints will takes a few time for pausing. Does it
> >>>> means
> >>>> > > only the first checkpoint will send the whole pages to SVM, and the
> >>>> other
> >>>> > > checkpoints send the dirty pages to SVM for reloading?
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > 2. We notice the COLO-COMPARE component will stuck the packet until
> >>>> > > receive packets from PVM and SVM, as this rule, when we add the
> >>>> > > COLO-COMPARE to PVM, its network will stuck until SVM start. So it
> >>>> is an
> >>>> > > other issue to make PVM stuck while setting COLO feature. With this
> >>>> issue,
> >>>> > > could we let colo-compare to pass the PVM's packet when the SVM's
> >>>> packet
> >>>> > > queue is empty? Then, the PVM's network won't stock, and "if PVM
> >>>> runs
> >>>> > > firstly, it still need to wait for The network packets from SVM to
> >>>> > > compare before send it to client side" won't happened either.
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Best regard,
> >>>> > >
> >>>> > > Daniel Cho
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月12日 週三 下午1:45寫道：
> >>>> > >
> >>>> > >
> >>>> > >
> >>>> > > > -----Original Message-----
> >>>> > > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
> >>>> > > > Sent: Wednesday, February 12, 2020 11:18 AM
> >>>> > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
> >>>> > > > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
> >>>> > > > Cc: qemu-devel@nongnu.org
> >>>> > > > Subject: RE: The issues about architecture of the COLO checkpoint
> >>>> > > >
> >>>> > > > Hi,
> >>>> > > >
> >>>> > > > Thank you Dave,
> >>>> > > >
> >>>> > > > I'll reply here directly.
> >>>> > > >
> >>>> > > > -----Original Message-----
> >>>> > > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> >>>> > > > Sent: Wednesday, February 12, 2020 1:48 AM
> >>>> > > > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
> >>>> > > > Zhanghailiang <zhang.zhanghailiang@huawei.com>
> >>>> > > > Cc: qemu-devel@nongnu.org
> >>>> > > > Subject: Re: The issues about architecture of the COLO checkpoint
> >>>> > > >
> >>>> > > >
> >>>> > > > cc'ing in COLO people:
> >>>> > > >
> >>>> > > >
> >>>> > > > * Daniel Cho (danielcho@qnap.com) wrote:
> >>>> > > > > Hi everyone,
> >>>> > > > >      We have some issues about setting COLO feature. Hope
> >>>> somebody
> >>>> > > > > could give us some advice.
> >>>> > > > >
> >>>> > > > > Issue 1:
> >>>> > > > >      We dynamic to set COLO feature for PVM(2 core, 16G
> >>>> memory),  but
> >>>> > > > > the Primary VM will pause a long time(based on memory size) for
> >>>> > > > > waiting SVM start. Does it have any idea to reduce the pause
> >>>> time?
> >>>> > > > >
> >>>> > > >
> >>>> > > > Yes, we do have some ideas to optimize this downtime.
> >>>> > > >
> >>>> > > > The main problem for current version is, for each checkpoint, we
> >>>> have to
> >>>> > > > send the whole PVM's pages
> >>>> > > > To SVM, and then copy the whole VM's state into SVM from ram
> >>>> cache, in
> >>>> > > > this process, we need both of them be paused.
> >>>> > > > Just as you said, the downtime is based on memory size.
> >>>> > > >
> >>>> > > > So firstly, we need to reduce the sending data while do
> >>>> checkpoint,
> >>>> > > actually,
> >>>> > > > we can migrate parts of PVM's dirty pages in background
> >>>> > > > While both of VMs are running. And then we load these pages into
> >>>> ram
> >>>> > > > cache (backup memory) in SVM temporarily. While do checkpoint,
> >>>> > > > We just send the last dirty pages of PVM to slave side and then
> >>>> copy the
> >>>> > > ram
> >>>> > > > cache into SVM. Further on, we don't have
> >>>> > > > To send the whole PVM's dirty pages, we can only send the pages
> >>>> that
> >>>> > > > dirtied by PVM or SVM during two checkpoints. (Because
> >>>> > > > If one page is not dirtied by both PVM and SVM, the data of this
> >>>> pages
> >>>> > > will
> >>>> > > > keep same in SVM, PVM, backup memory). This method can reduce
> >>>> > > > the time that consumed in sending data.
> >>>> > > >
> >>>> > > > For the second problem, we can reduce the memory copy by two
> >>>> methods,
> >>>> > > > first one, we don't have to copy the whole pages in ram cache,
> >>>> > > > We can only copy the pages that dirtied by PVM and SVM in last
> >>>> > > checkpoint.
> >>>> > > > Second, we can use userfault missing function to reduce the
> >>>> > > > Time consumed in memory copy. (For the second time, in theory, we
> >>>> can
> >>>> > > > reduce time consumed in memory into ms level).
> >>>> > > >
> >>>> > > > You can find the first optimization in attachment, it is based on
> >>>> an old
> >>>> > > qemu
> >>>> > > > version (qemu-2.6), it should not be difficult to rebase it
> >>>> > > > Into master or your version. And please feel free to send the new
> >>>> > > version if
> >>>> > > > you want into community ;)
> >>>> > > >
> >>>> > > >
> >>>> > >
> >>>> > > Thanks Hailiang!
> >>>> > > By the way, Do you have time to push the patches to upstream?
> >>>> > > I think this is a better and faster option.
> >>>> > >
> >>>> > > Thanks
> >>>> > > Zhang Chen
> >>>> > >
> >>>> > > > >
> >>>> > > > > Issue 2:
> >>>> > > > >      In
> >>>> > > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> >>>> > > > > could we move start_vm() before Line 488? Because at first
> >>>> checkpoint
> >>>> > > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
> >>>> > > > >
> >>>> > > >
> >>>> > > > No, that makes no sense, because if PVM runs firstly, it still
> >>>> need to
> >>>> > > wait for
> >>>> > > > The network packets from SVM to compare before send it to client
> >>>> side.
> >>>> > > >
> >>>> > > >
> >>>> > > > Thanks,
> >>>> > > > Hailiang
> >>>> > > >
> >>>> > > > >      We set the COLO feature on running VM, so we hope the
> >>>> running VM
> >>>> > > > > could continuous service for users.
> >>>> > > > > Do you have any suggestions for those issues?
> >>>> > > > >
> >>>> > > > > Best regards,
> >>>> > > > > Daniel Cho
> >>>> > > > --
> >>>> > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >>>> > >
> >>>> > >
> >>>> --
> >>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >>>>
> >>>>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

RE: The issues about architecture of the COLO checkpoint

[Zhanghailiang]

[-- Attachment #1: Type: text/plain, Size: 14163 bytes --]

Hi Daniel,

I have fixed this problem, and send V2, please refer to that series.

Thanks,

From: Daniel Cho [mailto:danielcho@qnap.com]
Sent: Thursday, February 20, 2020 11:52 AM
To: Zhang, Chen <chen.zhang@intel.com>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; Zhanghailiang <zhang.zhanghailiang@huawei.com>; qemu-devel@nongnu.org; Jason Wang <jasowang@redhat.com>
Subject: Re: The issues about architecture of the COLO checkpoint

Hi Hailiang,

I have already patched the file to my branch, but there is a problem while doing migration.
Here is the error message from SVM
"qemu-system-x86_64: /root/download/qemu-4.1.0/memory.c:1079: memory_region_transaction_commit: Assertion `qemu_mutex_iothread_locked()' failed."

Do you have this problem?

Best regards,
Daniel Cho

Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>> 於 2020年2月20日 週四 上午11:49寫道：
Hi Zhang,

Thanks, I will configure on code for testing first.
However, if you have free time, could you please send the patch file to us, Thanks.

Best Regard,
Daniel Cho


Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月20日 週四 上午11:07寫道：


On 2/18/2020 5:22 PM, Daniel Cho wrote:
Hi Hailiang,
Thanks for your help. If we have any problems we will contact you for your favor.


Hi Zhang,

" If colo-compare got a primary packet without related secondary packet in a certain time , it will automatically trigger checkpoint.  "
As you said, the colo-compare will trigger checkpoint, but does it need to limit checkpoint times?
There is a problem about doing many checkpoints while we use fio to random write files. Then it will cause low throughput on PVM.
Is this situation is normal on COLO?



Hi Daniel,

The checkpoint time is designed to be user adjustable based on user environment(workload/network status/business conditions...).

In net/colo-compare.c

/* TODO: Should be configurable */
#define REGULAR_PACKET_CHECK_MS 3000

If you need, I can send a patch for this issue. Make users can change the value by QMP and qemu monitor commands.

Thanks

Zhang Chen



Best regards,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月17日 週一 下午1:36寫道：


On 2/15/2020 11:35 AM, Daniel Cho wrote:
Hi Dave,

Yes, I agree with you, it does need a timeout.



Hi Daniel and Dave,

Current colo-compare already have the timeout mechanism.

Named packet_check_timer,  It will scan primary packet queue to make sure all the primary packet not stay too long time.

If colo-compare got a primary packet without related secondary packet in a certain time , it will automatic trigger checkpoint.

https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847



Thanks

Zhang Chen



Hi Hailiang,

We base on qemu-4.1.0 for using COLO feature, in your patch, we found a lot of difference  between your version and ours.
Could you give us a latest release version which is close your developing code?

Thanks.

Regards
Daniel Cho

Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>> 於 2020年2月13日 週四 下午6:38寫道：
* Daniel Cho (danielcho@qnap.com<mailto:danielcho@qnap.com>) wrote:
> Hi Hailiang,
>
> 1.
>     OK, we will try the patch
> “0001-COLO-Optimize-memory-back-up-process.patch”,
> and thanks for your help.
>
> 2.
>     We understand the reason to compare PVM and SVM's packet. However, the
> empty of SVM's packet queue might happened on setting COLO feature and SVM
> broken.
>
> On situation 1 ( setting COLO feature ):
>     We could force do checkpoint after setting COLO feature finish, then it
> will protect the state of PVM and SVM . As the Zhang Chen said.
>
> On situation 2 ( SVM broken ):
>     COLO will do failover for PVM, so it might not cause any wrong on PVM.
>
> However, those situations are our views, so there might be a big difference
> between reality and our views.
> If we have any wrong views and opinions, please let us know, and correct
> us.

It does need a timeout; the SVM being broken or being in a state where
it never sends the corresponding packet (because of a state difference)
can happen and COLO needs to timeout when the packet hasn't arrived
after a while and trigger the checkpoint.

Dave

> Thanks.
>
> Best regards,
> Daniel Cho
>
> Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月13日 週四 上午10:17寫道：
>
> > Add cc Jason Wang, he is a network expert.
> >
> > In case some network things goes wrong.
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> > *From:* Zhang, Chen
> > *Sent:* Thursday, February 13, 2020 10:10 AM
> > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>; Daniel Cho <
> > danielcho@qnap.com<mailto:danielcho@qnap.com>>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > For the issue 2:
> >
> >
> >
> > COLO need use the network packets to confirm PVM and SVM in the same state,
> >
> > Generally speaking, we can’t send PVM packets without compared with SVM
> > packets.
> >
> > But to prevent jamming, I think COLO can do force checkpoint and send the
> > PVM packets in this case.
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> > *Sent:* Thursday, February 13, 2020 9:45 AM
> > *To:* Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>;
> > Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi,
> >
> >
> >
> > 1.       After re-walked through the codes, yes, you are right, actually,
> > after the first migration, we will keep dirty log on in primary side,
> >
> > And only send the dirty pages in PVM to SVM. The ram cache in secondary
> > side is always a backup of PVM, so we don’t have to
> >
> > Re-send the none-dirtied pages.
> >
> > The reason why the first checkpoint takes longer time is we have to backup
> > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
> >
> > It is time consuming, but I have optimized in the second patch
> > “0001-COLO-Optimize-memory-back-up-process.patch” which you can find in my
> > previous reply.
> >
> >
> >
> > Besides, I found that, In my previous reply “We can only copy the pages
> > that dirtied by PVM and SVM in last checkpoint.”,
> >
> > We have done this optimization in current upstream codes.
> >
> >
> >
> > 2．I don’t quite understand this question. For COLO, we always need both
> > network packets of PVM’s and SVM’s to compare before send this packets to
> > client.
> >
> > It depends on this to decide whether or not PVM and SVM are in same state.
> >
> >
> >
> > Thanks,
> >
> > hailiang
> >
> >
> >
> > *From:* Daniel Cho [mailto:danielcho@qnap.com<mailto:danielcho@qnap.com> <danielcho@qnap.com<mailto:danielcho@qnap.com>>]
> > *Sent:* Wednesday, February 12, 2020 4:37 PM
> > *To:* Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>; Dr. David Alan
> > Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > *Subject:* Re: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi Hailiang,
> >
> >
> >
> > Thanks for your replaying and explain in detail.
> >
> > We will try to use the attachments to enhance memory copy.
> >
> >
> >
> > However, we have some questions for your replying.
> >
> >
> >
> > 1.  As you said, "for each checkpoint, we have to send the whole PVM's
> > pages To SVM", why the only first checkpoint will takes more pause time?
> >
> > In our observing, the first checkpoint will take more time for pausing,
> > then other checkpoints will takes a few time for pausing. Does it means
> > only the first checkpoint will send the whole pages to SVM, and the other
> > checkpoints send the dirty pages to SVM for reloading?
> >
> >
> >
> > 2. We notice the COLO-COMPARE component will stuck the packet until
> > receive packets from PVM and SVM, as this rule, when we add the
> > COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an
> > other issue to make PVM stuck while setting COLO feature. With this issue,
> > could we let colo-compare to pass the PVM's packet when the SVM's packet
> > queue is empty? Then, the PVM's network won't stock, and "if PVM runs
> > firstly, it still need to wait for The network packets from SVM to
> > compare before send it to client side" won't happened either.
> >
> >
> >
> > Best regard,
> >
> > Daniel Cho
> >
> >
> >
> > Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月12日 週三 下午1:45寫道：
> >
> >
> >
> > > -----Original Message-----
> > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> > > Sent: Wednesday, February 12, 2020 11:18 AM
> > > To: Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; Daniel Cho
> > > <danielcho@qnap.com<mailto:danielcho@qnap.com>>; Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> > > Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > > Subject: RE: The issues about architecture of the COLO checkpoint
> > >
> > > Hi,
> > >
> > > Thank you Dave,
> > >
> > > I'll reply here directly.
> > >
> > > -----Original Message-----
> > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com<mailto:dgilbert@redhat.com>]
> > > Sent: Wednesday, February 12, 2020 1:48 AM
> > > To: Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>; chen.zhang@intel.com<mailto:chen.zhang@intel.com>;
> > > Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> > > Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > > Subject: Re: The issues about architecture of the COLO checkpoint
> > >
> > >
> > > cc'ing in COLO people:
> > >
> > >
> > > * Daniel Cho (danielcho@qnap.com<mailto:danielcho@qnap.com>) wrote:
> > > > Hi everyone,
> > > >      We have some issues about setting COLO feature. Hope somebody
> > > > could give us some advice.
> > > >
> > > > Issue 1:
> > > >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > > > the Primary VM will pause a long time(based on memory size) for
> > > > waiting SVM start. Does it have any idea to reduce the pause time?
> > > >
> > >
> > > Yes, we do have some ideas to optimize this downtime.
> > >
> > > The main problem for current version is, for each checkpoint, we have to
> > > send the whole PVM's pages
> > > To SVM, and then copy the whole VM's state into SVM from ram cache, in
> > > this process, we need both of them be paused.
> > > Just as you said, the downtime is based on memory size.
> > >
> > > So firstly, we need to reduce the sending data while do checkpoint,
> > actually,
> > > we can migrate parts of PVM's dirty pages in background
> > > While both of VMs are running. And then we load these pages into ram
> > > cache (backup memory) in SVM temporarily. While do checkpoint,
> > > We just send the last dirty pages of PVM to slave side and then copy the
> > ram
> > > cache into SVM. Further on, we don't have
> > > To send the whole PVM's dirty pages, we can only send the pages that
> > > dirtied by PVM or SVM during two checkpoints. (Because
> > > If one page is not dirtied by both PVM and SVM, the data of this pages
> > will
> > > keep same in SVM, PVM, backup memory). This method can reduce
> > > the time that consumed in sending data.
> > >
> > > For the second problem, we can reduce the memory copy by two methods,
> > > first one, we don't have to copy the whole pages in ram cache,
> > > We can only copy the pages that dirtied by PVM and SVM in last
> > checkpoint.
> > > Second, we can use userfault missing function to reduce the
> > > Time consumed in memory copy. (For the second time, in theory, we can
> > > reduce time consumed in memory into ms level).
> > >
> > > You can find the first optimization in attachment, it is based on an old
> > qemu
> > > version (qemu-2.6), it should not be difficult to rebase it
> > > Into master or your version. And please feel free to send the new
> > version if
> > > you want into community ;)
> > >
> > >
> >
> > Thanks Hailiang!
> > By the way, Do you have time to push the patches to upstream?
> > I think this is a better and faster option.
> >
> > Thanks
> > Zhang Chen
> >
> > > >
> > > > Issue 2:
> > > >      In
> > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > > > could we move start_vm() before Line 488? Because at first checkpoint
> > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
> > > >
> > >
> > > No, that makes no sense, because if PVM runs firstly, it still need to
> > wait for
> > > The network packets from SVM to compare before send it to client side.
> > >
> > >
> > > Thanks,
> > > Hailiang
> > >
> > > >      We set the COLO feature on running VM, so we hope the running VM
> > > > could continuous service for users.
> > > > Do you have any suggestions for those issues?
> > > >
> > > > Best regards,
> > > > Daniel Cho
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK
> >
> >
--
Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK

[-- Attachment #2: Type: text/html, Size: 29308 bytes --]

RE: The issues about architecture of the COLO checkpoint

[Zhang, Chen]

[-- Attachment #1: Type: text/plain, Size: 13704 bytes --]



From: Daniel Cho <danielcho@qnap.com>
Sent: Thursday, February 20, 2020 11:49 AM
To: Zhang, Chen <chen.zhang@intel.com>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; Zhanghailiang <zhang.zhanghailiang@huawei.com>; qemu-devel@nongnu.org; Jason Wang <jasowang@redhat.com>
Subject: Re: The issues about architecture of the COLO checkpoint

Hi Zhang,

Thanks, I will configure on code for testing first.
However, if you have free time, could you please send the patch file to us, Thanks.

OK, I will send this patch recently.
By the way, can you share QNAP’s plan and status for COLO?

Best Regard,
Daniel Cho


Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月20日 週四 上午11:07寫道：


On 2/18/2020 5:22 PM, Daniel Cho wrote:
Hi Hailiang,
Thanks for your help. If we have any problems we will contact you for your favor.


Hi Zhang,

" If colo-compare got a primary packet without related secondary packet in a certain time , it will automatically trigger checkpoint.  "
As you said, the colo-compare will trigger checkpoint, but does it need to limit checkpoint times?
There is a problem about doing many checkpoints while we use fio to random write files. Then it will cause low throughput on PVM.
Is this situation is normal on COLO?



Hi Daniel,

The checkpoint time is designed to be user adjustable based on user environment(workload/network status/business conditions...).

In net/colo-compare.c

/* TODO: Should be configurable */
#define REGULAR_PACKET_CHECK_MS 3000

If you need, I can send a patch for this issue. Make users can change the value by QMP and qemu monitor commands.

Thanks

Zhang Chen



Best regards,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月17日 週一 下午1:36寫道：


On 2/15/2020 11:35 AM, Daniel Cho wrote:
Hi Dave,

Yes, I agree with you, it does need a timeout.



Hi Daniel and Dave,

Current colo-compare already have the timeout mechanism.

Named packet_check_timer,  It will scan primary packet queue to make sure all the primary packet not stay too long time.

If colo-compare got a primary packet without related secondary packet in a certain time , it will automatic trigger checkpoint.

https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847



Thanks

Zhang Chen



Hi Hailiang,

We base on qemu-4.1.0 for using COLO feature, in your patch, we found a lot of difference  between your version and ours.
Could you give us a latest release version which is close your developing code?

Thanks.

Regards
Daniel Cho

Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>> 於 2020年2月13日 週四 下午6:38寫道：
* Daniel Cho (danielcho@qnap.com<mailto:danielcho@qnap.com>) wrote:
> Hi Hailiang,
>
> 1.
>     OK, we will try the patch
> “0001-COLO-Optimize-memory-back-up-process.patch”,
> and thanks for your help.
>
> 2.
>     We understand the reason to compare PVM and SVM's packet. However, the
> empty of SVM's packet queue might happened on setting COLO feature and SVM
> broken.
>
> On situation 1 ( setting COLO feature ):
>     We could force do checkpoint after setting COLO feature finish, then it
> will protect the state of PVM and SVM . As the Zhang Chen said.
>
> On situation 2 ( SVM broken ):
>     COLO will do failover for PVM, so it might not cause any wrong on PVM.
>
> However, those situations are our views, so there might be a big difference
> between reality and our views.
> If we have any wrong views and opinions, please let us know, and correct
> us.

It does need a timeout; the SVM being broken or being in a state where
it never sends the corresponding packet (because of a state difference)
can happen and COLO needs to timeout when the packet hasn't arrived
after a while and trigger the checkpoint.

Dave

> Thanks.
>
> Best regards,
> Daniel Cho
>
> Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月13日 週四 上午10:17寫道：
>
> > Add cc Jason Wang, he is a network expert.
> >
> > In case some network things goes wrong.
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> > *From:* Zhang, Chen
> > *Sent:* Thursday, February 13, 2020 10:10 AM
> > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>; Daniel Cho <
> > danielcho@qnap.com<mailto:danielcho@qnap.com>>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > For the issue 2:
> >
> >
> >
> > COLO need use the network packets to confirm PVM and SVM in the same state,
> >
> > Generally speaking, we can’t send PVM packets without compared with SVM
> > packets.
> >
> > But to prevent jamming, I think COLO can do force checkpoint and send the
> > PVM packets in this case.
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> > *Sent:* Thursday, February 13, 2020 9:45 AM
> > *To:* Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>
> > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>;
> > Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> > *Subject:* RE: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi,
> >
> >
> >
> > 1.       After re-walked through the codes, yes, you are right, actually,
> > after the first migration, we will keep dirty log on in primary side,
> >
> > And only send the dirty pages in PVM to SVM. The ram cache in secondary
> > side is always a backup of PVM, so we don’t have to
> >
> > Re-send the none-dirtied pages.
> >
> > The reason why the first checkpoint takes longer time is we have to backup
> > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
> >
> > It is time consuming, but I have optimized in the second patch
> > “0001-COLO-Optimize-memory-back-up-process.patch” which you can find in my
> > previous reply.
> >
> >
> >
> > Besides, I found that, In my previous reply “We can only copy the pages
> > that dirtied by PVM and SVM in last checkpoint.”,
> >
> > We have done this optimization in current upstream codes.
> >
> >
> >
> > 2．I don’t quite understand this question. For COLO, we always need both
> > network packets of PVM’s and SVM’s to compare before send this packets to
> > client.
> >
> > It depends on this to decide whether or not PVM and SVM are in same state.
> >
> >
> >
> > Thanks,
> >
> > hailiang
> >
> >
> >
> > *From:* Daniel Cho [mailto:danielcho@qnap.com<mailto:danielcho@qnap.com> <danielcho@qnap.com<mailto:danielcho@qnap.com>>]
> > *Sent:* Wednesday, February 12, 2020 4:37 PM
> > *To:* Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>; Dr. David Alan
> > Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > *Subject:* Re: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi Hailiang,
> >
> >
> >
> > Thanks for your replaying and explain in detail.
> >
> > We will try to use the attachments to enhance memory copy.
> >
> >
> >
> > However, we have some questions for your replying.
> >
> >
> >
> > 1.  As you said, "for each checkpoint, we have to send the whole PVM's
> > pages To SVM", why the only first checkpoint will takes more pause time?
> >
> > In our observing, the first checkpoint will take more time for pausing,
> > then other checkpoints will takes a few time for pausing. Does it means
> > only the first checkpoint will send the whole pages to SVM, and the other
> > checkpoints send the dirty pages to SVM for reloading?
> >
> >
> >
> > 2. We notice the COLO-COMPARE component will stuck the packet until
> > receive packets from PVM and SVM, as this rule, when we add the
> > COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an
> > other issue to make PVM stuck while setting COLO feature. With this issue,
> > could we let colo-compare to pass the PVM's packet when the SVM's packet
> > queue is empty? Then, the PVM's network won't stock, and "if PVM runs
> > firstly, it still need to wait for The network packets from SVM to
> > compare before send it to client side" won't happened either.
> >
> >
> >
> > Best regard,
> >
> > Daniel Cho
> >
> >
> >
> > Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>> 於 2020年2月12日 週三 下午1:45寫道：
> >
> >
> >
> > > -----Original Message-----
> > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> > > Sent: Wednesday, February 12, 2020 11:18 AM
> > > To: Dr. David Alan Gilbert <dgilbert@redhat.com<mailto:dgilbert@redhat.com>>; Daniel Cho
> > > <danielcho@qnap.com<mailto:danielcho@qnap.com>>; Zhang, Chen <chen.zhang@intel.com<mailto:chen.zhang@intel.com>>
> > > Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > > Subject: RE: The issues about architecture of the COLO checkpoint
> > >
> > > Hi,
> > >
> > > Thank you Dave,
> > >
> > > I'll reply here directly.
> > >
> > > -----Original Message-----
> > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com<mailto:dgilbert@redhat.com>]
> > > Sent: Wednesday, February 12, 2020 1:48 AM
> > > To: Daniel Cho <danielcho@qnap.com<mailto:danielcho@qnap.com>>; chen.zhang@intel.com<mailto:chen.zhang@intel.com>;
> > > Zhanghailiang <zhang.zhanghailiang@huawei.com<mailto:zhang.zhanghailiang@huawei.com>>
> > > Cc: qemu-devel@nongnu.org<mailto:qemu-devel@nongnu.org>
> > > Subject: Re: The issues about architecture of the COLO checkpoint
> > >
> > >
> > > cc'ing in COLO people:
> > >
> > >
> > > * Daniel Cho (danielcho@qnap.com<mailto:danielcho@qnap.com>) wrote:
> > > > Hi everyone,
> > > >      We have some issues about setting COLO feature. Hope somebody
> > > > could give us some advice.
> > > >
> > > > Issue 1:
> > > >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > > > the Primary VM will pause a long time(based on memory size) for
> > > > waiting SVM start. Does it have any idea to reduce the pause time?
> > > >
> > >
> > > Yes, we do have some ideas to optimize this downtime.
> > >
> > > The main problem for current version is, for each checkpoint, we have to
> > > send the whole PVM's pages
> > > To SVM, and then copy the whole VM's state into SVM from ram cache, in
> > > this process, we need both of them be paused.
> > > Just as you said, the downtime is based on memory size.
> > >
> > > So firstly, we need to reduce the sending data while do checkpoint,
> > actually,
> > > we can migrate parts of PVM's dirty pages in background
> > > While both of VMs are running. And then we load these pages into ram
> > > cache (backup memory) in SVM temporarily. While do checkpoint,
> > > We just send the last dirty pages of PVM to slave side and then copy the
> > ram
> > > cache into SVM. Further on, we don't have
> > > To send the whole PVM's dirty pages, we can only send the pages that
> > > dirtied by PVM or SVM during two checkpoints. (Because
> > > If one page is not dirtied by both PVM and SVM, the data of this pages
> > will
> > > keep same in SVM, PVM, backup memory). This method can reduce
> > > the time that consumed in sending data.
> > >
> > > For the second problem, we can reduce the memory copy by two methods,
> > > first one, we don't have to copy the whole pages in ram cache,
> > > We can only copy the pages that dirtied by PVM and SVM in last
> > checkpoint.
> > > Second, we can use userfault missing function to reduce the
> > > Time consumed in memory copy. (For the second time, in theory, we can
> > > reduce time consumed in memory into ms level).
> > >
> > > You can find the first optimization in attachment, it is based on an old
> > qemu
> > > version (qemu-2.6), it should not be difficult to rebase it
> > > Into master or your version. And please feel free to send the new
> > version if
> > > you want into community ;)
> > >
> > >
> >
> > Thanks Hailiang!
> > By the way, Do you have time to push the patches to upstream?
> > I think this is a better and faster option.
> >
> > Thanks
> > Zhang Chen
> >
> > > >
> > > > Issue 2:
> > > >      In
> > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > > > could we move start_vm() before Line 488? Because at first checkpoint
> > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
> > > >
> > >
> > > No, that makes no sense, because if PVM runs firstly, it still need to
> > wait for
> > > The network packets from SVM to compare before send it to client side.
> > >
> > >
> > > Thanks,
> > > Hailiang
> > >
> > > >      We set the COLO feature on running VM, so we hope the running VM
> > > > could continuous service for users.
> > > > Do you have any suggestions for those issues?
> > > >
> > > > Best regards,
> > > > Daniel Cho
> > > --
> > > Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK
> >
> >
--
Dr. David Alan Gilbert / dgilbert@redhat.com<mailto:dgilbert@redhat.com> / Manchester, UK

[-- Attachment #2: Type: text/html, Size: 26713 bytes --]

Re: The issues about architecture of the COLO checkpoint

[Daniel Cho]

Hi Zhang,

Thanks for your help.
However, did you occur the error which the function qemu_hexdump in
colo-compare.c will crash the qemu process while doing operation with
network?

We are working on VM fault tolerance study and COLO function
evalutation first. Currently we did not have a confirmed plan on it.

Best regard,
Daniel Cho

Zhang, Chen <chen.zhang@intel.com> 於 2020年2月24日 週一 上午2:43寫道：

>
>
>
>
>
> From: Daniel Cho <danielcho@qnap.com>
> Sent: Thursday, February 20, 2020 11:49 AM
> To: Zhang, Chen <chen.zhang@intel.com>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; Zhanghailiang <zhang.zhanghailiang@huawei.com>; qemu-devel@nongnu.org; Jason Wang <jasowang@redhat.com>
> Subject: Re: The issues about architecture of the COLO checkpoint
>
>
>
> Hi Zhang,
>
>
>
> Thanks, I will configure on code for testing first.
>
> However, if you have free time, could you please send the patch file to us, Thanks.
>
>
>
> OK, I will send this patch recently.
>
> By the way, can you share QNAP’s plan and status for COLO?
>
>
>
> Best Regard,
>
> Daniel Cho
>
>
>
>
>
> Zhang, Chen <chen.zhang@intel.com> 於 2020年2月20日 週四 上午11:07寫道：
>
>
>
> On 2/18/2020 5:22 PM, Daniel Cho wrote:
>
> Hi Hailiang,
>
> Thanks for your help. If we have any problems we will contact you for your favor.
>
>
>
>
>
> Hi Zhang,
>
>
>
> " If colo-compare got a primary packet without related secondary packet in a certain time , it will automatically trigger checkpoint.  "
>
> As you said, the colo-compare will trigger checkpoint, but does it need to limit checkpoint times?
>
> There is a problem about doing many checkpoints while we use fio to random write files. Then it will cause low throughput on PVM.
>
> Is this situation is normal on COLO?
>
>
>
> Hi Daniel,
>
> The checkpoint time is designed to be user adjustable based on user environment(workload/network status/business conditions...).
>
> In net/colo-compare.c
>
> /* TODO: Should be configurable */
> #define REGULAR_PACKET_CHECK_MS 3000
>
> If you need, I can send a patch for this issue. Make users can change the value by QMP and qemu monitor commands.
>
> Thanks
>
> Zhang Chen
>
>
>
>
>
> Best regards,
>
> Daniel Cho
>
>
>
> Zhang, Chen <chen.zhang@intel.com> 於 2020年2月17日 週一 下午1:36寫道：
>
>
>
> On 2/15/2020 11:35 AM, Daniel Cho wrote:
>
> Hi Dave,
>
>
>
> Yes, I agree with you, it does need a timeout.
>
>
>
> Hi Daniel and Dave,
>
> Current colo-compare already have the timeout mechanism.
>
> Named packet_check_timer,  It will scan primary packet queue to make sure all the primary packet not stay too long time.
>
> If colo-compare got a primary packet without related secondary packet in a certain time , it will automatic trigger checkpoint.
>
> https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847
>
>
>
> Thanks
>
> Zhang Chen
>
>
>
>
>
> Hi Hailiang,
>
>
>
> We base on qemu-4.1.0 for using COLO feature, in your patch, we found a lot of difference  between your version and ours.
>
> Could you give us a latest release version which is close your developing code?
>
>
>
> Thanks.
>
>
>
> Regards
>
> Daniel Cho
>
>
>
> Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2020年2月13日 週四 下午6:38寫道：
>
> * Daniel Cho (danielcho@qnap.com) wrote:
> > Hi Hailiang,
> >
> > 1.
> >     OK, we will try the patch
> > “0001-COLO-Optimize-memory-back-up-process.patch”,
> > and thanks for your help.
> >
> > 2.
> >     We understand the reason to compare PVM and SVM's packet. However, the
> > empty of SVM's packet queue might happened on setting COLO feature and SVM
> > broken.
> >
> > On situation 1 ( setting COLO feature ):
> >     We could force do checkpoint after setting COLO feature finish, then it
> > will protect the state of PVM and SVM . As the Zhang Chen said.
> >
> > On situation 2 ( SVM broken ):
> >     COLO will do failover for PVM, so it might not cause any wrong on PVM.
> >
> > However, those situations are our views, so there might be a big difference
> > between reality and our views.
> > If we have any wrong views and opinions, please let us know, and correct
> > us.
>
> It does need a timeout; the SVM being broken or being in a state where
> it never sends the corresponding packet (because of a state difference)
> can happen and COLO needs to timeout when the packet hasn't arrived
> after a while and trigger the checkpoint.
>
> Dave
>
> > Thanks.
> >
> > Best regards,
> > Daniel Cho
> >
> > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月13日 週四 上午10:17寫道：
> >
> > > Add cc Jason Wang, he is a network expert.
> > >
> > > In case some network things goes wrong.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Zhang Chen
> > >
> > >
> > >
> > > *From:* Zhang, Chen
> > > *Sent:* Thursday, February 13, 2020 10:10 AM
> > > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com>; Daniel Cho <
> > > danielcho@qnap.com>
> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
> > >
> > >
> > >
> > > For the issue 2:
> > >
> > >
> > >
> > > COLO need use the network packets to confirm PVM and SVM in the same state,
> > >
> > > Generally speaking, we can’t send PVM packets without compared with SVM
> > > packets.
> > >
> > > But to prevent jamming, I think COLO can do force checkpoint and send the
> > > PVM packets in this case.
> > >
> > >
> > >
> > > Thanks
> > >
> > > Zhang Chen
> > >
> > >
> > >
> > > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > *Sent:* Thursday, February 13, 2020 9:45 AM
> > > *To:* Daniel Cho <danielcho@qnap.com>
> > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org;
> > > Zhang, Chen <chen.zhang@intel.com>
> > > *Subject:* RE: The issues about architecture of the COLO checkpoint
> > >
> > >
> > >
> > > Hi,
> > >
> > >
> > >
> > > 1.       After re-walked through the codes, yes, you are right, actually,
> > > after the first migration, we will keep dirty log on in primary side,
> > >
> > > And only send the dirty pages in PVM to SVM. The ram cache in secondary
> > > side is always a backup of PVM, so we don’t have to
> > >
> > > Re-send the none-dirtied pages.
> > >
> > > The reason why the first checkpoint takes longer time is we have to backup
> > > the whole VM’s ram into ram cache, that is colo_init_ram_cache().
> > >
> > > It is time consuming, but I have optimized in the second patch
> > > “0001-COLO-Optimize-memory-back-up-process.patch” which you can find in my
> > > previous reply.
> > >
> > >
> > >
> > > Besides, I found that, In my previous reply “We can only copy the pages
> > > that dirtied by PVM and SVM in last checkpoint.”,
> > >
> > > We have done this optimization in current upstream codes.
> > >
> > >
> > >
> > > 2．I don’t quite understand this question. For COLO, we always need both
> > > network packets of PVM’s and SVM’s to compare before send this packets to
> > > client.
> > >
> > > It depends on this to decide whether or not PVM and SVM are in same state.
> > >
> > >
> > >
> > > Thanks,
> > >
> > > hailiang
> > >
> > >
> > >
> > > *From:* Daniel Cho [mailto:danielcho@qnap.com <danielcho@qnap.com>]
> > > *Sent:* Wednesday, February 12, 2020 4:37 PM
> > > *To:* Zhang, Chen <chen.zhang@intel.com>
> > > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David Alan
> > > Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> > > *Subject:* Re: The issues about architecture of the COLO checkpoint
> > >
> > >
> > >
> > > Hi Hailiang,
> > >
> > >
> > >
> > > Thanks for your replaying and explain in detail.
> > >
> > > We will try to use the attachments to enhance memory copy.
> > >
> > >
> > >
> > > However, we have some questions for your replying.
> > >
> > >
> > >
> > > 1.  As you said, "for each checkpoint, we have to send the whole PVM's
> > > pages To SVM", why the only first checkpoint will takes more pause time?
> > >
> > > In our observing, the first checkpoint will take more time for pausing,
> > > then other checkpoints will takes a few time for pausing. Does it means
> > > only the first checkpoint will send the whole pages to SVM, and the other
> > > checkpoints send the dirty pages to SVM for reloading?
> > >
> > >
> > >
> > > 2. We notice the COLO-COMPARE component will stuck the packet until
> > > receive packets from PVM and SVM, as this rule, when we add the
> > > COLO-COMPARE to PVM, its network will stuck until SVM start. So it is an
> > > other issue to make PVM stuck while setting COLO feature. With this issue,
> > > could we let colo-compare to pass the PVM's packet when the SVM's packet
> > > queue is empty? Then, the PVM's network won't stock, and "if PVM runs
> > > firstly, it still need to wait for The network packets from SVM to
> > > compare before send it to client side" won't happened either.
> > >
> > >
> > >
> > > Best regard,
> > >
> > > Daniel Cho
> > >
> > >
> > >
> > > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月12日 週三 下午1:45寫道：
> > >
> > >
> > >
> > > > -----Original Message-----
> > > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > > Sent: Wednesday, February 12, 2020 11:18 AM
> > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
> > > > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
> > > > Cc: qemu-devel@nongnu.org
> > > > Subject: RE: The issues about architecture of the COLO checkpoint
> > > >
> > > > Hi,
> > > >
> > > > Thank you Dave,
> > > >
> > > > I'll reply here directly.
> > > >
> > > > -----Original Message-----
> > > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > > > Sent: Wednesday, February 12, 2020 1:48 AM
> > > > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
> > > > Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > > Cc: qemu-devel@nongnu.org
> > > > Subject: Re: The issues about architecture of the COLO checkpoint
> > > >
> > > >
> > > > cc'ing in COLO people:
> > > >
> > > >
> > > > * Daniel Cho (danielcho@qnap.com) wrote:
> > > > > Hi everyone,
> > > > >      We have some issues about setting COLO feature. Hope somebody
> > > > > could give us some advice.
> > > > >
> > > > > Issue 1:
> > > > >      We dynamic to set COLO feature for PVM(2 core, 16G memory),  but
> > > > > the Primary VM will pause a long time(based on memory size) for
> > > > > waiting SVM start. Does it have any idea to reduce the pause time?
> > > > >
> > > >
> > > > Yes, we do have some ideas to optimize this downtime.
> > > >
> > > > The main problem for current version is, for each checkpoint, we have to
> > > > send the whole PVM's pages
> > > > To SVM, and then copy the whole VM's state into SVM from ram cache, in
> > > > this process, we need both of them be paused.
> > > > Just as you said, the downtime is based on memory size.
> > > >
> > > > So firstly, we need to reduce the sending data while do checkpoint,
> > > actually,
> > > > we can migrate parts of PVM's dirty pages in background
> > > > While both of VMs are running. And then we load these pages into ram
> > > > cache (backup memory) in SVM temporarily. While do checkpoint,
> > > > We just send the last dirty pages of PVM to slave side and then copy the
> > > ram
> > > > cache into SVM. Further on, we don't have
> > > > To send the whole PVM's dirty pages, we can only send the pages that
> > > > dirtied by PVM or SVM during two checkpoints. (Because
> > > > If one page is not dirtied by both PVM and SVM, the data of this pages
> > > will
> > > > keep same in SVM, PVM, backup memory). This method can reduce
> > > > the time that consumed in sending data.
> > > >
> > > > For the second problem, we can reduce the memory copy by two methods,
> > > > first one, we don't have to copy the whole pages in ram cache,
> > > > We can only copy the pages that dirtied by PVM and SVM in last
> > > checkpoint.
> > > > Second, we can use userfault missing function to reduce the
> > > > Time consumed in memory copy. (For the second time, in theory, we can
> > > > reduce time consumed in memory into ms level).
> > > >
> > > > You can find the first optimization in attachment, it is based on an old
> > > qemu
> > > > version (qemu-2.6), it should not be difficult to rebase it
> > > > Into master or your version. And please feel free to send the new
> > > version if
> > > > you want into community ;)
> > > >
> > > >
> > >
> > > Thanks Hailiang!
> > > By the way, Do you have time to push the patches to upstream?
> > > I think this is a better and faster option.
> > >
> > > Thanks
> > > Zhang Chen
> > >
> > > > >
> > > > > Issue 2:
> > > > >      In
> > > > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > > > > could we move start_vm() before Line 488? Because at first checkpoint
> > > > > PVM will wait for SVM's reply, it cause PVM stop for a while.
> > > > >
> > > >
> > > > No, that makes no sense, because if PVM runs firstly, it still need to
> > > wait for
> > > > The network packets from SVM to compare before send it to client side.
> > > >
> > > >
> > > > Thanks,
> > > > Hailiang
> > > >
> > > > >      We set the COLO feature on running VM, so we hope the running VM
> > > > > could continuous service for users.
> > > > > Do you have any suggestions for those issues?
> > > > >
> > > > > Best regards,
> > > > > Daniel Cho
> > > > --
> > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > >
> > >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

RE: The issues about architecture of the COLO checkpoint

[Zhang, Chen]

> -----Original Message-----
> From: Daniel Cho <danielcho@qnap.com>
> Sent: Monday, February 24, 2020 3:15 PM
> To: Zhang, Chen <chen.zhang@intel.com>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; Zhanghailiang
> <zhang.zhanghailiang@huawei.com>; qemu-devel@nongnu.org; Jason
> Wang <jasowang@redhat.com>
> Subject: Re: The issues about architecture of the COLO checkpoint
> 
> Hi Zhang,
> 
> Thanks for your help.
> However, did you occur the error which the function qemu_hexdump in
> colo-compare.c will crash the qemu process while doing operation with
> network?
> 

No, qemu_hexdump looks no relationship with network...
Do you means it will crashed in qemu_hexdump sometimes? 

> We are working on VM fault tolerance study and COLO function evalutation
> first. Currently we did not have a confirmed plan on it.

OK, keep connection.

Thanks
Zhang Chen

> 
> Best regard,
> Daniel Cho
> 
> Zhang, Chen <chen.zhang@intel.com> 於 2020年2月24日 週一 上午2:43
> 寫道：
> 
> >
> >
> >
> >
> >
> > From: Daniel Cho <danielcho@qnap.com>
> > Sent: Thursday, February 20, 2020 11:49 AM
> > To: Zhang, Chen <chen.zhang@intel.com>
> > Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; Zhanghailiang
> > <zhang.zhanghailiang@huawei.com>; qemu-devel@nongnu.org; Jason
> Wang
> > <jasowang@redhat.com>
> > Subject: Re: The issues about architecture of the COLO checkpoint
> >
> >
> >
> > Hi Zhang,
> >
> >
> >
> > Thanks, I will configure on code for testing first.
> >
> > However, if you have free time, could you please send the patch file to us,
> Thanks.
> >
> >
> >
> > OK, I will send this patch recently.
> >
> > By the way, can you share QNAP’s plan and status for COLO?
> >
> >
> >
> > Best Regard,
> >
> > Daniel Cho
> >
> >
> >
> >
> >
> > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月20日 週四 上午
> 11:07寫道：
> >
> >
> >
> > On 2/18/2020 5:22 PM, Daniel Cho wrote:
> >
> > Hi Hailiang,
> >
> > Thanks for your help. If we have any problems we will contact you for your
> favor.
> >
> >
> >
> >
> >
> > Hi Zhang,
> >
> >
> >
> > " If colo-compare got a primary packet without related secondary packet in
> a certain time , it will automatically trigger checkpoint.  "
> >
> > As you said, the colo-compare will trigger checkpoint, but does it need to
> limit checkpoint times?
> >
> > There is a problem about doing many checkpoints while we use fio to
> random write files. Then it will cause low throughput on PVM.
> >
> > Is this situation is normal on COLO?
> >
> >
> >
> > Hi Daniel,
> >
> > The checkpoint time is designed to be user adjustable based on user
> environment(workload/network status/business conditions...).
> >
> > In net/colo-compare.c
> >
> > /* TODO: Should be configurable */
> > #define REGULAR_PACKET_CHECK_MS 3000
> >
> > If you need, I can send a patch for this issue. Make users can change the
> value by QMP and qemu monitor commands.
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> >
> >
> > Best regards,
> >
> > Daniel Cho
> >
> >
> >
> > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月17日 週一 下午
> 1:36寫道：
> >
> >
> >
> > On 2/15/2020 11:35 AM, Daniel Cho wrote:
> >
> > Hi Dave,
> >
> >
> >
> > Yes, I agree with you, it does need a timeout.
> >
> >
> >
> > Hi Daniel and Dave,
> >
> > Current colo-compare already have the timeout mechanism.
> >
> > Named packet_check_timer,  It will scan primary packet queue to make
> sure all the primary packet not stay too long time.
> >
> > If colo-compare got a primary packet without related secondary packet in a
> certain time , it will automatic trigger checkpoint.
> >
> > https://github.com/qemu/qemu/blob/master/net/colo-compare.c#L847
> >
> >
> >
> > Thanks
> >
> > Zhang Chen
> >
> >
> >
> >
> >
> > Hi Hailiang,
> >
> >
> >
> > We base on qemu-4.1.0 for using COLO feature, in your patch, we found a
> lot of difference  between your version and ours.
> >
> > Could you give us a latest release version which is close your developing
> code?
> >
> >
> >
> > Thanks.
> >
> >
> >
> > Regards
> >
> > Daniel Cho
> >
> >
> >
> > Dr. David Alan Gilbert <dgilbert@redhat.com> 於 2020年2月13日 週四
> 下午6:38寫道：
> >
> > * Daniel Cho (danielcho@qnap.com) wrote:
> > > Hi Hailiang,
> > >
> > > 1.
> > >     OK, we will try the patch
> > > “0001-COLO-Optimize-memory-back-up-process.patch”,
> > > and thanks for your help.
> > >
> > > 2.
> > >     We understand the reason to compare PVM and SVM's packet.
> > > However, the empty of SVM's packet queue might happened on setting
> > > COLO feature and SVM broken.
> > >
> > > On situation 1 ( setting COLO feature ):
> > >     We could force do checkpoint after setting COLO feature finish,
> > > then it will protect the state of PVM and SVM . As the Zhang Chen said.
> > >
> > > On situation 2 ( SVM broken ):
> > >     COLO will do failover for PVM, so it might not cause any wrong on PVM.
> > >
> > > However, those situations are our views, so there might be a big
> > > difference between reality and our views.
> > > If we have any wrong views and opinions, please let us know, and
> > > correct us.
> >
> > It does need a timeout; the SVM being broken or being in a state where
> > it never sends the corresponding packet (because of a state
> > difference) can happen and COLO needs to timeout when the packet
> > hasn't arrived after a while and trigger the checkpoint.
> >
> > Dave
> >
> > > Thanks.
> > >
> > > Best regards,
> > > Daniel Cho
> > >
> > > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月13日 週四 上午
> 10:17寫道：
> > >
> > > > Add cc Jason Wang, he is a network expert.
> > > >
> > > > In case some network things goes wrong.
> > > >
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Zhang Chen
> > > >
> > > >
> > > >
> > > > *From:* Zhang, Chen
> > > > *Sent:* Thursday, February 13, 2020 10:10 AM
> > > > *To:* 'Zhanghailiang' <zhang.zhanghailiang@huawei.com>; Daniel Cho
> > > > < danielcho@qnap.com>
> > > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
> > > > qemu-devel@nongnu.org
> > > > *Subject:* RE: The issues about architecture of the COLO
> > > > checkpoint
> > > >
> > > >
> > > >
> > > > For the issue 2:
> > > >
> > > >
> > > >
> > > > COLO need use the network packets to confirm PVM and SVM in the
> > > > same state,
> > > >
> > > > Generally speaking, we can’t send PVM packets without compared
> > > > with SVM packets.
> > > >
> > > > But to prevent jamming, I think COLO can do force checkpoint and
> > > > send the PVM packets in this case.
> > > >
> > > >
> > > >
> > > > Thanks
> > > >
> > > > Zhang Chen
> > > >
> > > >
> > > >
> > > > *From:* Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > > *Sent:* Thursday, February 13, 2020 9:45 AM
> > > > *To:* Daniel Cho <danielcho@qnap.com>
> > > > *Cc:* Dr. David Alan Gilbert <dgilbert@redhat.com>;
> > > > qemu-devel@nongnu.org; Zhang, Chen <chen.zhang@intel.com>
> > > > *Subject:* RE: The issues about architecture of the COLO
> > > > checkpoint
> > > >
> > > >
> > > >
> > > > Hi,
> > > >
> > > >
> > > >
> > > > 1.       After re-walked through the codes, yes, you are right, actually,
> > > > after the first migration, we will keep dirty log on in primary
> > > > side,
> > > >
> > > > And only send the dirty pages in PVM to SVM. The ram cache in
> > > > secondary side is always a backup of PVM, so we don’t have to
> > > >
> > > > Re-send the none-dirtied pages.
> > > >
> > > > The reason why the first checkpoint takes longer time is we have
> > > > to backup the whole VM’s ram into ram cache, that is
> colo_init_ram_cache().
> > > >
> > > > It is time consuming, but I have optimized in the second patch
> > > > “0001-COLO-Optimize-memory-back-up-process.patch” which you can
> > > > find in my previous reply.
> > > >
> > > >
> > > >
> > > > Besides, I found that, In my previous reply “We can only copy the
> > > > pages that dirtied by PVM and SVM in last checkpoint.”,
> > > >
> > > > We have done this optimization in current upstream codes.
> > > >
> > > >
> > > >
> > > > 2．I don’t quite understand this question. For COLO, we always need
> > > > both network packets of PVM’s and SVM’s to compare before send
> > > > this packets to client.
> > > >
> > > > It depends on this to decide whether or not PVM and SVM are in same
> state.
> > > >
> > > >
> > > >
> > > > Thanks,
> > > >
> > > > hailiang
> > > >
> > > >
> > > >
> > > > *From:* Daniel Cho [mailto:danielcho@qnap.com
> > > > <danielcho@qnap.com>]
> > > > *Sent:* Wednesday, February 12, 2020 4:37 PM
> > > > *To:* Zhang, Chen <chen.zhang@intel.com>
> > > > *Cc:* Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David
> > > > Alan Gilbert <dgilbert@redhat.com>; qemu-devel@nongnu.org
> > > > *Subject:* Re: The issues about architecture of the COLO
> > > > checkpoint
> > > >
> > > >
> > > >
> > > > Hi Hailiang,
> > > >
> > > >
> > > >
> > > > Thanks for your replaying and explain in detail.
> > > >
> > > > We will try to use the attachments to enhance memory copy.
> > > >
> > > >
> > > >
> > > > However, we have some questions for your replying.
> > > >
> > > >
> > > >
> > > > 1.  As you said, "for each checkpoint, we have to send the whole
> > > > PVM's pages To SVM", why the only first checkpoint will takes more
> pause time?
> > > >
> > > > In our observing, the first checkpoint will take more time for
> > > > pausing, then other checkpoints will takes a few time for pausing.
> > > > Does it means only the first checkpoint will send the whole pages
> > > > to SVM, and the other checkpoints send the dirty pages to SVM for
> reloading?
> > > >
> > > >
> > > >
> > > > 2. We notice the COLO-COMPARE component will stuck the packet
> > > > until receive packets from PVM and SVM, as this rule, when we add
> > > > the COLO-COMPARE to PVM, its network will stuck until SVM start.
> > > > So it is an other issue to make PVM stuck while setting COLO
> > > > feature. With this issue, could we let colo-compare to pass the
> > > > PVM's packet when the SVM's packet queue is empty? Then, the
> PVM's
> > > > network won't stock, and "if PVM runs firstly, it still need to
> > > > wait for The network packets from SVM to compare before send it to
> client side" won't happened either.
> > > >
> > > >
> > > >
> > > > Best regard,
> > > >
> > > > Daniel Cho
> > > >
> > > >
> > > >
> > > > Zhang, Chen <chen.zhang@intel.com> 於 2020年2月12日 週三 下午
> 1:45寫道：
> > > >
> > > >
> > > >
> > > > > -----Original Message-----
> > > > > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > > > Sent: Wednesday, February 12, 2020 11:18 AM
> > > > > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
> > > > > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
> > > > > Cc: qemu-devel@nongnu.org
> > > > > Subject: RE: The issues about architecture of the COLO
> > > > > checkpoint
> > > > >
> > > > > Hi,
> > > > >
> > > > > Thank you Dave,
> > > > >
> > > > > I'll reply here directly.
> > > > >
> > > > > -----Original Message-----
> > > > > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > > > > Sent: Wednesday, February 12, 2020 1:48 AM
> > > > > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
> > > > > Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > > > > Cc: qemu-devel@nongnu.org
> > > > > Subject: Re: The issues about architecture of the COLO
> > > > > checkpoint
> > > > >
> > > > >
> > > > > cc'ing in COLO people:
> > > > >
> > > > >
> > > > > * Daniel Cho (danielcho@qnap.com) wrote:
> > > > > > Hi everyone,
> > > > > >      We have some issues about setting COLO feature. Hope
> > > > > > somebody could give us some advice.
> > > > > >
> > > > > > Issue 1:
> > > > > >      We dynamic to set COLO feature for PVM(2 core, 16G
> > > > > > memory),  but the Primary VM will pause a long time(based on
> > > > > > memory size) for waiting SVM start. Does it have any idea to reduce
> the pause time?
> > > > > >
> > > > >
> > > > > Yes, we do have some ideas to optimize this downtime.
> > > > >
> > > > > The main problem for current version is, for each checkpoint, we
> > > > > have to send the whole PVM's pages To SVM, and then copy the
> > > > > whole VM's state into SVM from ram cache, in this process, we
> > > > > need both of them be paused.
> > > > > Just as you said, the downtime is based on memory size.
> > > > >
> > > > > So firstly, we need to reduce the sending data while do
> > > > > checkpoint,
> > > > actually,
> > > > > we can migrate parts of PVM's dirty pages in background While
> > > > > both of VMs are running. And then we load these pages into ram
> > > > > cache (backup memory) in SVM temporarily. While do checkpoint,
> > > > > We just send the last dirty pages of PVM to slave side and then
> > > > > copy the
> > > > ram
> > > > > cache into SVM. Further on, we don't have To send the whole
> > > > > PVM's dirty pages, we can only send the pages that dirtied by
> > > > > PVM or SVM during two checkpoints. (Because If one page is not
> > > > > dirtied by both PVM and SVM, the data of this pages
> > > > will
> > > > > keep same in SVM, PVM, backup memory). This method can reduce
> > > > > the time that consumed in sending data.
> > > > >
> > > > > For the second problem, we can reduce the memory copy by two
> > > > > methods, first one, we don't have to copy the whole pages in ram
> > > > > cache, We can only copy the pages that dirtied by PVM and SVM in
> > > > > last
> > > > checkpoint.
> > > > > Second, we can use userfault missing function to reduce the Time
> > > > > consumed in memory copy. (For the second time, in theory, we can
> > > > > reduce time consumed in memory into ms level).
> > > > >
> > > > > You can find the first optimization in attachment, it is based
> > > > > on an old
> > > > qemu
> > > > > version (qemu-2.6), it should not be difficult to rebase it Into
> > > > > master or your version. And please feel free to send the new
> > > > version if
> > > > > you want into community ;)
> > > > >
> > > > >
> > > >
> > > > Thanks Hailiang!
> > > > By the way, Do you have time to push the patches to upstream?
> > > > I think this is a better and faster option.
> > > >
> > > > Thanks
> > > > Zhang Chen
> > > >
> > > > > >
> > > > > > Issue 2:
> > > > > >      In
> > > > > >
> https://github.com/qemu/qemu/blob/master/migration/colo.c#L503
> > > > > > , could we move start_vm() before Line 488? Because at first
> > > > > > checkpoint PVM will wait for SVM's reply, it cause PVM stop
> > > > > > for a while.
> > > > > >
> > > > >
> > > > > No, that makes no sense, because if PVM runs firstly, it still
> > > > > need to
> > > > wait for
> > > > > The network packets from SVM to compare before send it to client
> side.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Hailiang
> > > > >
> > > > > >      We set the COLO feature on running VM, so we hope the
> > > > > > running VM could continuous service for users.
> > > > > > Do you have any suggestions for those issues?
> > > > > >
> > > > > > Best regards,
> > > > > > Daniel Cho
> > > > > --
> > > > > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> > > >
> > > >
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

Re: The issues about architecture of the COLO checkpoint

[Lukas Straub]

On Wed, 4 Mar 2020 07:44:11 +0000
"Zhang, Chen" <chen.zhang@intel.com> wrote:

> > -----Original Message-----
> > From: Daniel Cho <danielcho@qnap.com>
> > Sent: Monday, February 24, 2020 3:15 PM
> > To: Zhang, Chen <chen.zhang@intel.com>
> > Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; Zhanghailiang
> > <zhang.zhanghailiang@huawei.com>; qemu-devel@nongnu.org; Jason
> > Wang <jasowang@redhat.com>
> > Subject: Re: The issues about architecture of the COLO checkpoint
> >
> > Hi Zhang,
> >
> > Thanks for your help.
> > However, did you occur the error which the function qemu_hexdump in
> > colo-compare.c will crash the qemu process while doing operation with
> > network?
> >
>
> No, qemu_hexdump looks no relationship with network...
> Do you means it will crashed in qemu_hexdump sometimes?

Jeah, I hit that bug too, but it was fixed with
1e907a32b77e5d418538453df5945242e43224fa "COLO-compare: Fix incorrect `if` logic"
in qemu 4.2.

Regards,
Lukas Straub

> > We are working on VM fault tolerance study and COLO function evalutation
> > first. Currently we did not have a confirmed plan on it.
>
> OK, keep connection.
>
> Thanks
> Zhang Chen

Re: The issues about architecture of the COLO checkpoint

[Dr. David Alan Gilbert]

* Lukas Straub (lukasstraub2@web.de) wrote:
> On Wed, 4 Mar 2020 07:44:11 +0000
> "Zhang, Chen" <chen.zhang@intel.com> wrote:
> 
> > > -----Original Message-----
> > > From: Daniel Cho <danielcho@qnap.com>
> > > Sent: Monday, February 24, 2020 3:15 PM
> > > To: Zhang, Chen <chen.zhang@intel.com>
> > > Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; Zhanghailiang
> > > <zhang.zhanghailiang@huawei.com>; qemu-devel@nongnu.org; Jason
> > > Wang <jasowang@redhat.com>
> > > Subject: Re: The issues about architecture of the COLO checkpoint
> > >
> > > Hi Zhang,
> > >
> > > Thanks for your help.
> > > However, did you occur the error which the function qemu_hexdump in
> > > colo-compare.c will crash the qemu process while doing operation with
> > > network?
> > >
> >
> > No, qemu_hexdump looks no relationship with network...
> > Do you means it will crashed in qemu_hexdump sometimes?
> 
> Jeah, I hit that bug too, but it was fixed with
> 1e907a32b77e5d418538453df5945242e43224fa "COLO-compare: Fix incorrect `if` logic"
> in qemu 4.2.

Is anyone running any CI stuff for COLO?  I'm not, but bugs like that
make it clear we probably should be;  it's not trivial to setup, but it
should be doable in a nest setup these days.

Dave

> Regards,
> Lukas Straub
> 
> > > We are working on VM fault tolerance study and COLO function evalutation
> > > first. Currently we did not have a confirmed plan on it.
> >
> > OK, keep connection.
> >
> > Thanks
> > Zhang Chen
> 
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

RE: The issues about architecture of the COLO checkpoint

[Zhang, Chen]

> -----Original Message-----
> From: Dr. David Alan Gilbert <dgilbert@redhat.com>
> Sent: Friday, March 13, 2020 12:39 AM
> To: Lukas Straub <lukasstraub2@web.de>
> Cc: Zhang, Chen <chen.zhang@intel.com>; Daniel Cho
> <danielcho@qnap.com>; qemu-devel@nongnu.org; Jason Wang
> <jasowang@redhat.com>; Zhanghailiang <zhang.zhanghailiang@huawei.com>
> Subject: Re: The issues about architecture of the COLO checkpoint
> 
> * Lukas Straub (lukasstraub2@web.de) wrote:
> > On Wed, 4 Mar 2020 07:44:11 +0000
> > "Zhang, Chen" <chen.zhang@intel.com> wrote:
> >
> > > > -----Original Message-----
> > > > From: Daniel Cho <danielcho@qnap.com>
> > > > Sent: Monday, February 24, 2020 3:15 PM
> > > > To: Zhang, Chen <chen.zhang@intel.com>
> > > > Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>; Zhanghailiang
> > > > <zhang.zhanghailiang@huawei.com>; qemu-devel@nongnu.org; Jason
> > > > Wang <jasowang@redhat.com>
> > > > Subject: Re: The issues about architecture of the COLO checkpoint
> > > >
> > > > Hi Zhang,
> > > >
> > > > Thanks for your help.
> > > > However, did you occur the error which the function qemu_hexdump
> > > > in colo-compare.c will crash the qemu process while doing
> > > > operation with network?
> > > >
> > >
> > > No, qemu_hexdump looks no relationship with network...
> > > Do you means it will crashed in qemu_hexdump sometimes?
> >
> > Jeah, I hit that bug too, but it was fixed with
> > 1e907a32b77e5d418538453df5945242e43224fa "COLO-compare: Fix
> incorrect `if` logic"
> > in qemu 4.2.
> 
> Is anyone running any CI stuff for COLO?  I'm not, but bugs like that make it
> clear we probably should be;  it's not trivial to setup, but it should be doable
> in a nest setup these days.

Currently, No CI stuff for COLO,  I'm busy on COLO landing with CSPs.
We will be happy If someone want to take this work.

Thanks
Zhang Chen

> 
> Dave
> 
> > Regards,
> > Lukas Straub
> >
> > > > We are working on VM fault tolerance study and COLO function
> > > > evalutation first. Currently we did not have a confirmed plan on it.
> > >
> > > OK, keep connection.
> > >
> > > Thanks
> > > Zhang Chen
> >
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK

RE: The issues about architecture of the COLO checkpoint

[Zhanghailiang]

Hi Zhang Chen,

> -----Original Message-----
> From: Zhang, Chen [mailto:chen.zhang@intel.com]
> Sent: Wednesday, February 12, 2020 1:45 PM
> To: Zhanghailiang <zhang.zhanghailiang@huawei.com>; Dr. David Alan
> Gilbert <dgilbert@redhat.com>; Daniel Cho <danielcho@qnap.com>
> Cc: qemu-devel@nongnu.org
> Subject: RE: The issues about architecture of the COLO checkpoint
> 
> 
> 
> > -----Original Message-----
> > From: Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > Sent: Wednesday, February 12, 2020 11:18 AM
> > To: Dr. David Alan Gilbert <dgilbert@redhat.com>; Daniel Cho
> > <danielcho@qnap.com>; Zhang, Chen <chen.zhang@intel.com>
> > Cc: qemu-devel@nongnu.org
> > Subject: RE: The issues about architecture of the COLO checkpoint
> >
> > Hi,
> >
> > Thank you Dave,
> >
> > I'll reply here directly.
> >
> > -----Original Message-----
> > From: Dr. David Alan Gilbert [mailto:dgilbert@redhat.com]
> > Sent: Wednesday, February 12, 2020 1:48 AM
> > To: Daniel Cho <danielcho@qnap.com>; chen.zhang@intel.com;
> > Zhanghailiang <zhang.zhanghailiang@huawei.com>
> > Cc: qemu-devel@nongnu.org
> > Subject: Re: The issues about architecture of the COLO checkpoint
> >
> >
> > cc'ing in COLO people:
> >
> >
> > * Daniel Cho (danielcho@qnap.com) wrote:
> > > Hi everyone,
> > >      We have some issues about setting COLO feature. Hope somebody
> > > could give us some advice.
> > >
> > > Issue 1:
> > >      We dynamic to set COLO feature for PVM(2 core, 16G memory),
> > > but the Primary VM will pause a long time(based on memory size) for
> > > waiting SVM start. Does it have any idea to reduce the pause time?
> > >
> >
> > Yes, we do have some ideas to optimize this downtime.
> >
> > The main problem for current version is, for each checkpoint, we have
> > to send the whole PVM's pages To SVM, and then copy the whole VM's
> > state into SVM from ram cache, in this process, we need both of them
> > be paused.
> > Just as you said, the downtime is based on memory size.
> >
> > So firstly, we need to reduce the sending data while do checkpoint,
> > actually, we can migrate parts of PVM's dirty pages in background
> > While both of VMs are running. And then we load these pages into ram
> > cache (backup memory) in SVM temporarily. While do checkpoint, We just
> > send the last dirty pages of PVM to slave side and then copy the ram
> > cache into SVM. Further on, we don't have To send the whole PVM's
> > dirty pages, we can only send the pages that dirtied by PVM or SVM
> > during two checkpoints. (Because If one page is not dirtied by both
> > PVM and SVM, the data of this pages will keep same in SVM, PVM, backup
> > memory). This method can reduce the time that consumed in sending
> > data.
> >
> > For the second problem, we can reduce the memory copy by two methods,
> > first one, we don't have to copy the whole pages in ram cache, We can
> > only copy the pages that dirtied by PVM and SVM in last checkpoint.
> > Second, we can use userfault missing function to reduce the Time
> > consumed in memory copy. (For the second time, in theory, we can
> > reduce time consumed in memory into ms level).
> >
> > You can find the first optimization in attachment, it is based on an
> > old qemu version (qemu-2.6), it should not be difficult to rebase it
> > Into master or your version. And please feel free to send the new
> > version if you want into community ;)
> >
> >
> 
> Thanks Hailiang!
> By the way, Do you have time to push the patches to upstream?
> I think this is a better and faster option.
> 

Yes, I can do this, for the second optimization, we need time to realize and test

Thanks

> Thanks
> Zhang Chen
> 
> > >
> > > Issue 2:
> > >      In
> > > https://github.com/qemu/qemu/blob/master/migration/colo.c#L503,
> > > could we move start_vm() before Line 488? Because at first
> > > checkpoint PVM will wait for SVM's reply, it cause PVM stop for a while.
> > >
> >
> > No, that makes no sense, because if PVM runs firstly, it still need to
> > wait for The network packets from SVM to compare before send it to client
> side.
> >
> >
> > Thanks,
> > Hailiang
> >
> > >      We set the COLO feature on running VM, so we hope the running
> > > VM could continuous service for users.
> > > Do you have any suggestions for those issues?
> > >
> > > Best regards,
> > > Daniel Cho
> > --
> > Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK