[Qemu-devel] [PATCH V2 0/5] Add Xen COLO support

[Qemu-devel] [PATCH V2 0/5] Add Xen COLO support

[Zhang Chen]

From: Zhang Chen <chen.zhang@intel.com>

Xen COLO based on KVM COLO architecture, it shared COLO proxy and block
replication with KVM COLO. The only differece is Xen COLO have own
COLO-frame to handle live migration related function, so we need this
series make Xen COLO frame can communicate with other COLO modules in
qemu. Xen side related patches have been merged.

V2:
 - Rebase on upstream code.
 - Optimize code by Zhijian's comments in patch 4/5.
 - Remove origin 5/6 patch.

V1:
 - Initial patch.

Zhang Chen (5):
  COLO-compare: Add new parameter to communicate with remote colo-frame
  COLO-compare: Add remote notification chardev handler frame
  COLO-compare: Make the compare_chr_send() can send notification
    message.
  COLO-compare: Add colo-compare remote notify support
  migration/colo.c: Add missed filter notify for Xen COLO.

 migration/colo.c   |   2 +
 net/colo-compare.c | 155 +++++++++++++++++++++++++++++++++++++++------
 qemu-options.hx    |  33 +++++++++-
 3 files changed, 171 insertions(+), 19 deletions(-)

-- 
2.17.GIT

[Qemu-devel] [PATCH V2 1/5] COLO-compare: Add new parameter to communicate with remote colo-frame

[Zhang Chen]

From: Zhang Chen <chen.zhang@intel.com>

We add the "notify_dev=chardevID" parameter. After that colo-compare can connect with
remote(currently just for Xen, KVM-COLO didn't need it.) colo-frame through chardev socket,
it can notify remote(Xen) colo-frame to handle checkpoint event.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 net/colo-compare.c | 21 +++++++++++++++++++++
 qemu-options.hx    | 33 ++++++++++++++++++++++++++++++++-
 2 files changed, 53 insertions(+), 1 deletion(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index fcb491121b..21849748b6 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -83,6 +83,7 @@ typedef struct CompareState {
     char *pri_indev;
     char *sec_indev;
     char *outdev;
+    char *notify_dev;
     CharBackend chr_pri_in;
     CharBackend chr_sec_in;
     CharBackend chr_out;
@@ -897,6 +898,21 @@ static void compare_set_vnet_hdr(Object *obj,
     s->vnet_hdr = value;
 }
 
+static char *compare_get_notify_dev(Object *obj, Error **errp)
+{
+    CompareState *s = COLO_COMPARE(obj);
+
+    return g_strdup(s->notify_dev);
+}
+
+static void compare_set_notify_dev(Object *obj, const char *value, Error **errp)
+{
+    CompareState *s = COLO_COMPARE(obj);
+
+    g_free(s->notify_dev);
+    s->notify_dev = g_strdup(value);
+}
+
 static void compare_pri_rs_finalize(SocketReadState *pri_rs)
 {
     CompareState *s = container_of(pri_rs, CompareState, pri_rs);
@@ -1057,6 +1073,10 @@ static void colo_compare_init(Object *obj)
                             (Object **)&s->iothread,
                             object_property_allow_set_link,
                             OBJ_PROP_LINK_STRONG, NULL);
+    /* This parameter just for Xen COLO */
+    object_property_add_str(obj, "notify_dev",
+                            compare_get_notify_dev, compare_set_notify_dev,
+                            NULL);
 
     s->vnet_hdr = false;
     object_property_add_bool(obj, "vnet_hdr_support", compare_get_vnet_hdr,
@@ -1103,6 +1123,7 @@ static void colo_compare_finalize(Object *obj)
     g_free(s->pri_indev);
     g_free(s->sec_indev);
     g_free(s->outdev);
+    g_free(s->notify_dev);
 }
 
 static const TypeInfo colo_compare_info = {
diff --git a/qemu-options.hx b/qemu-options.hx
index 0d8beb4afd..c18b79099a 100644
--- a/qemu-options.hx
+++ b/qemu-options.hx
@@ -4477,7 +4477,7 @@ Dump the network traffic on netdev @var{dev} to the file specified by
 The file format is libpcap, so it can be analyzed with tools such as tcpdump
 or Wireshark.
 
-@item -object colo-compare,id=@var{id},primary_in=@var{chardevid},secondary_in=@var{chardevid},outdev=@var{chardevid},iothread=@var{id}[,vnet_hdr_support]
+@item -object colo-compare,id=@var{id},primary_in=@var{chardevid},secondary_in=@var{chardevid},outdev=@var{chardevid},iothread=@var{id}[,vnet_hdr_support][,notify_dev=@var{id}]
 
 Colo-compare gets packet from primary_in@var{chardevid} and secondary_in@var{chardevid}, than compare primary packet with
 secondary packet. If the packets are same, we will output primary
@@ -4486,11 +4486,15 @@ do checkpoint and send primary packet to outdev@var{chardevid}.
 In order to improve efficiency, we need to put the task of comparison
 in another thread. If it has the vnet_hdr_support flag, colo compare
 will send/recv packet with vnet_hdr_len.
+If you want to use Xen COLO, will need the notify_dev to notify Xen
+colo-frame to do checkpoint.
 
 we must use it with the help of filter-mirror and filter-redirector.
 
 @example
 
+KVM COLO
+
 primary:
 -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
 -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
@@ -4514,6 +4518,33 @@ secondary:
 -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
 -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
 
+
+Xen COLO
+
+primary:
+-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
+-device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66
+-chardev socket,id=mirror0,host=3.3.3.3,port=9003,server,nowait
+-chardev socket,id=compare1,host=3.3.3.3,port=9004,server,nowait
+-chardev socket,id=compare0,host=3.3.3.3,port=9001,server,nowait
+-chardev socket,id=compare0-0,host=3.3.3.3,port=9001
+-chardev socket,id=compare_out,host=3.3.3.3,port=9005,server,nowait
+-chardev socket,id=compare_out0,host=3.3.3.3,port=9005
+-chardev socket,id=notify_way,host=3.3.3.3,port=9009,server,nowait
+-object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0
+-object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out
+-object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0
+-object iothread,id=iothread1
+-object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,notify_dev=nofity_way,iothread=iothread1
+
+secondary:
+-netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,down script=/etc/qemu-ifdown
+-device e1000,netdev=hn0,mac=52:a4:00:12:78:66
+-chardev socket,id=red0,host=3.3.3.3,port=9003
+-chardev socket,id=red1,host=3.3.3.3,port=9004
+-object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0
+-object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1
+
 @end example
 
 If you want to know the detail of above command line, you can read
-- 
2.17.GIT

[Qemu-devel] [PATCH V2 2/5] COLO-compare: Add remote notification chardev handler frame

[Zhang Chen]

From: Zhang Chen <chen.zhang@intel.com>

Add chardev handler to send notification to remote(current from Xen) colo-frame.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 net/colo-compare.c | 39 +++++++++++++++++++++++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 21849748b6..8b7bd9f0c7 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -87,8 +87,10 @@ typedef struct CompareState {
     CharBackend chr_pri_in;
     CharBackend chr_sec_in;
     CharBackend chr_out;
+    CharBackend chr_notify_dev;
     SocketReadState pri_rs;
     SocketReadState sec_rs;
+    SocketReadState notify_rs;
     bool vnet_hdr;
 
     /*
@@ -745,6 +747,19 @@ static void compare_sec_chr_in(void *opaque, const uint8_t *buf, int size)
     }
 }
 
+static void compare_notify_chr(void *opaque, const uint8_t *buf, int size)
+{
+    CompareState *s = COLO_COMPARE(opaque);
+    int ret;
+
+    ret = net_fill_rstate(&s->notify_rs, buf, size);
+    if (ret == -1) {
+        qemu_chr_fe_set_handlers(&s->chr_notify_dev, NULL, NULL, NULL, NULL,
+                                 NULL, NULL, true);
+        error_report("colo-compare notify_dev error");
+    }
+}
+
 /*
  * Check old packet regularly so it can watch for any packets
  * that the secondary hasn't produced equivalents of.
@@ -832,6 +847,11 @@ static void colo_compare_iothread(CompareState *s)
     qemu_chr_fe_set_handlers(&s->chr_sec_in, compare_chr_can_read,
                              compare_sec_chr_in, NULL, NULL,
                              s, s->worker_context, true);
+    if (s->notify_dev) {
+        qemu_chr_fe_set_handlers(&s->chr_notify_dev, compare_chr_can_read,
+                                 compare_notify_chr, NULL, NULL,
+                                 s, s->worker_context, true);
+    }
 
     colo_compare_timer_init(s);
     s->event_bh = qemu_bh_new(colo_compare_handle_event, s);
@@ -943,6 +963,10 @@ static void compare_sec_rs_finalize(SocketReadState *sec_rs)
     }
 }
 
+static void compare_notify_rs_finalize(SocketReadState *notify_rs)
+{
+    /* Get Xen colo-frame's notify and handle the message */
+}
 
 /*
  * Return 0 is success.
@@ -1013,6 +1037,17 @@ static void colo_compare_complete(UserCreatable *uc, Error **errp)
     net_socket_rs_init(&s->pri_rs, compare_pri_rs_finalize, s->vnet_hdr);
     net_socket_rs_init(&s->sec_rs, compare_sec_rs_finalize, s->vnet_hdr);
 
+    /* Try to enable remote notify chardev, currently just for Xen COLO */
+    if (s->notify_dev) {
+        if (find_and_check_chardev(&chr, s->notify_dev, errp) ||
+            !qemu_chr_fe_init(&s->chr_notify_dev, chr, errp)) {
+            return;
+        }
+
+        net_socket_rs_init(&s->notify_rs, compare_notify_rs_finalize,
+                           s->vnet_hdr);
+    }
+
     QTAILQ_INSERT_TAIL(&net_compares, s, next);
 
     g_queue_init(&s->conn_list);
@@ -1091,6 +1126,10 @@ static void colo_compare_finalize(Object *obj)
     qemu_chr_fe_deinit(&s->chr_pri_in, false);
     qemu_chr_fe_deinit(&s->chr_sec_in, false);
     qemu_chr_fe_deinit(&s->chr_out, false);
+    if (s->notify_dev) {
+        qemu_chr_fe_deinit(&s->chr_notify_dev, false);
+    }
+
     if (s->iothread) {
         colo_compare_timer_del(s);
     }
-- 
2.17.GIT

[Qemu-devel] [PATCH V2 3/5] COLO-compare: Make the compare_chr_send() can send notification message.

[Zhang Chen]

From: Zhang Chen <chen.zhang@intel.com>

We need use this function to send notification message for remote colo-frame(Xen).
So we add new parameter for this job.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 net/colo-compare.c | 41 +++++++++++++++++++++++++++++++++--------
 1 file changed, 33 insertions(+), 8 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 8b7bd9f0c7..16285f4a96 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -129,7 +129,8 @@ static void colo_compare_inconsistency_notify(void)
 static int compare_chr_send(CompareState *s,
                             const uint8_t *buf,
                             uint32_t size,
-                            uint32_t vnet_hdr_len);
+                            uint32_t vnet_hdr_len,
+                            bool notify_remote_frame);
 
 static gint seq_sorter(Packet *a, Packet *b, gpointer data)
 {
@@ -241,7 +242,8 @@ static void colo_release_primary_pkt(CompareState *s, Packet *pkt)
     ret = compare_chr_send(s,
                            pkt->data,
                            pkt->size,
-                           pkt->vnet_hdr_len);
+                           pkt->vnet_hdr_len,
+                           false);
     if (ret < 0) {
         error_report("colo send primary packet failed");
     }
@@ -671,7 +673,8 @@ static void colo_compare_connection(void *opaque, void *user_data)
 static int compare_chr_send(CompareState *s,
                             const uint8_t *buf,
                             uint32_t size,
-                            uint32_t vnet_hdr_len)
+                            uint32_t vnet_hdr_len,
+                            bool notify_remote_frame)
 {
     int ret = 0;
     uint32_t len = htonl(size);
@@ -680,7 +683,14 @@ static int compare_chr_send(CompareState *s,
         return 0;
     }
 
-    ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
+    if (notify_remote_frame) {
+        ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
+                                    (uint8_t *)&len,
+                                    sizeof(len));
+    } else {
+        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
+    }
+
     if (ret != sizeof(len)) {
         goto err;
     }
@@ -691,13 +701,26 @@ static int compare_chr_send(CompareState *s,
          * know how to parse net packet correctly.
          */
         len = htonl(vnet_hdr_len);
-        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)&len, sizeof(len));
+
+        if (!notify_remote_frame) {
+            ret = qemu_chr_fe_write_all(&s->chr_out,
+                                        (uint8_t *)&len,
+                                        sizeof(len));
+        }
+
         if (ret != sizeof(len)) {
             goto err;
         }
     }
 
-    ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)buf, size);
+    if (notify_remote_frame) {
+        ret = qemu_chr_fe_write_all(&s->chr_notify_dev,
+                                    (uint8_t *)buf,
+                                    size);
+    } else {
+        ret = qemu_chr_fe_write_all(&s->chr_out, (uint8_t *)buf, size);
+    }
+
     if (ret != size) {
         goto err;
     }
@@ -943,7 +966,8 @@ static void compare_pri_rs_finalize(SocketReadState *pri_rs)
         compare_chr_send(s,
                          pri_rs->buf,
                          pri_rs->packet_len,
-                         pri_rs->vnet_hdr_len);
+                         pri_rs->vnet_hdr_len,
+                         false);
     } else {
         /* compare packet in the specified connection */
         colo_compare_connection(conn, s);
@@ -1075,7 +1099,8 @@ static void colo_flush_packets(void *opaque, void *user_data)
         compare_chr_send(s,
                          pkt->data,
                          pkt->size,
-                         pkt->vnet_hdr_len);
+                         pkt->vnet_hdr_len,
+                         false);
         packet_destroy(pkt, NULL);
     }
     while (!g_queue_is_empty(&conn->secondary_list)) {
-- 
2.17.GIT

[Qemu-devel] [PATCH V2 4/5] COLO-compare: Add colo-compare remote notify support

[Zhang Chen]

From: Zhang Chen <chen.zhang@intel.com>

This patch make colo-compare can send message to remote COLO frame(Xen) when occur checkpoint.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 net/colo-compare.c | 54 +++++++++++++++++++++++++++++++++++++---------
 1 file changed, 44 insertions(+), 10 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 16285f4a96..516b651ecd 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -120,11 +120,6 @@ enum {
     SECONDARY_IN,
 };
 
-static void colo_compare_inconsistency_notify(void)
-{
-    notifier_list_notify(&colo_compare_notifiers,
-                migrate_get_current());
-}
 
 static int compare_chr_send(CompareState *s,
                             const uint8_t *buf,
@@ -132,6 +127,27 @@ static int compare_chr_send(CompareState *s,
                             uint32_t vnet_hdr_len,
                             bool notify_remote_frame);
 
+static void notify_remote_frame(CompareState *s)
+{
+    char msg[] = "DO_CHECKPOINT";
+    int ret = 0;
+
+    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
+    if (ret < 0) {
+        error_report("Notify Xen COLO-frame failed");
+    }
+}
+
+static void colo_compare_inconsistency_notify(CompareState *s)
+{
+    if (s->notify_dev) {
+        notify_remote_frame(s);
+    } else {
+        notifier_list_notify(&colo_compare_notifiers,
+                             migrate_get_current());
+    }
+}
+
 static gint seq_sorter(Packet *a, Packet *b, gpointer data)
 {
     struct tcp_hdr *atcp, *btcp;
@@ -435,7 +451,7 @@ sec:
         qemu_hexdump((char *)spkt->data, stderr,
                      "colo-compare spkt", spkt->size);
 
-        colo_compare_inconsistency_notify();
+        colo_compare_inconsistency_notify(s);
     }
 }
 
@@ -577,7 +593,7 @@ void colo_compare_unregister_notifier(Notifier *notify)
 }
 
 static int colo_old_packet_check_one_conn(Connection *conn,
-                                           void *user_data)
+                                          CompareState *s)
 {
     GList *result = NULL;
     int64_t check_time = REGULAR_PACKET_CHECK_MS;
@@ -588,7 +604,7 @@ static int colo_old_packet_check_one_conn(Connection *conn,
 
     if (result) {
         /* Do checkpoint will flush old packet */
-        colo_compare_inconsistency_notify();
+        colo_compare_inconsistency_notify(s);
         return 0;
     }
 
@@ -608,7 +624,7 @@ static void colo_old_packet_check(void *opaque)
      * If we find one old packet, stop finding job and notify
      * COLO frame do checkpoint.
      */
-    g_queue_find_custom(&s->conn_list, NULL,
+    g_queue_find_custom(&s->conn_list, s,
                         (GCompareFunc)colo_old_packet_check_one_conn);
 }
 
@@ -637,7 +653,8 @@ static void colo_compare_packet(CompareState *s, Connection *conn,
              */
             trace_colo_compare_main("packet different");
             g_queue_push_head(&conn->primary_list, pkt);
-            colo_compare_inconsistency_notify();
+
+            colo_compare_inconsistency_notify(s);
             break;
         }
     }
@@ -989,7 +1006,24 @@ static void compare_sec_rs_finalize(SocketReadState *sec_rs)
 
 static void compare_notify_rs_finalize(SocketReadState *notify_rs)
 {
+    CompareState *s = container_of(notify_rs, CompareState, notify_rs);
+
     /* Get Xen colo-frame's notify and handle the message */
+    char *data = g_memdup(notify_rs->buf, notify_rs->packet_len);
+    char msg[] = "COLO_COMPARE_GET_XEN_INIT";
+    int ret;
+
+    if (!strcmp(data, "COLO_USERSPACE_PROXY_INIT")) {
+        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
+        if (ret < 0) {
+            error_report("Notify Xen COLO-frame INIT failed");
+        }
+    }
+
+    if (!strcmp(data, "COLO_CHECKPOINT")) {
+        /* colo-compare do checkpoint, flush pri packet and remove sec packet */
+        g_queue_foreach(&s->conn_list, colo_flush_packets, s);
+    }
 }
 
 /*
-- 
2.17.GIT

[Qemu-devel] [PATCH V2 5/5] migration/colo.c: Add missed filter notify for Xen COLO.

[Zhang Chen]

From: Zhang Chen <chen.zhang@intel.com>

We need to notify net filter to do checkpoint for Xen COLO, like KVM side.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
---
 migration/colo.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/migration/colo.c b/migration/colo.c
index 8c1644091f..9f84b1fa3c 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -259,6 +259,8 @@ ReplicationStatus *qmp_query_xen_replication_status(Error **errp)
 void qmp_xen_colo_do_checkpoint(Error **errp)
 {
     replication_do_checkpoint_all(errp);
+    /* Notify all filters of all NIC to do checkpoint */
+    colo_notify_filters_event(COLO_EVENT_CHECKPOINT, errp);
 }
 #endif
 
-- 
2.17.GIT

Re: [Qemu-devel] [PATCH V2 0/5] Add Xen COLO support

[Jason Wang]

On 2019/6/10 上午12:44, Zhang Chen wrote:
> From: Zhang Chen <chen.zhang@intel.com>
>
> Xen COLO based on KVM COLO architecture, it shared COLO proxy and block
> replication with KVM COLO. The only differece is Xen COLO have own
> COLO-frame to handle live migration related function, so we need this
> series make Xen COLO frame can communicate with other COLO modules in
> qemu. Xen side related patches have been merged.


Any pointer to this?

Thanks


>
> V2:
>   - Rebase on upstream code.
>   - Optimize code by Zhijian's comments in patch 4/5.
>   - Remove origin 5/6 patch.
>
> V1:
>   - Initial patch.
>
> Zhang Chen (5):
>    COLO-compare: Add new parameter to communicate with remote colo-frame
>    COLO-compare: Add remote notification chardev handler frame
>    COLO-compare: Make the compare_chr_send() can send notification
>      message.
>    COLO-compare: Add colo-compare remote notify support
>    migration/colo.c: Add missed filter notify for Xen COLO.
>
>   migration/colo.c   |   2 +
>   net/colo-compare.c | 155 +++++++++++++++++++++++++++++++++++++++------
>   qemu-options.hx    |  33 +++++++++-
>   3 files changed, 171 insertions(+), 19 deletions(-)
>

Re: [Qemu-devel] [PATCH V2 0/5] Add Xen COLO support

[Zhang, Chen]

> -----Original Message-----
> From: Jason Wang [mailto:jasowang@redhat.com]
> Sent: Monday, June 10, 2019 12:09 PM
> To: Zhang, Chen <chen.zhang@intel.com>; Li Zhijian <lizhijian@cn.fujitsu.com>;
> Dr. David Alan Gilbert <dgilbert@redhat.com>; Juan Quintela
> <quintela@redhat.com>; zhanghailiang <zhang.zhanghailiang@huawei.com>;
> qemu-dev <qemu-devel@nongnu.org>; Stefano Stabellini
> <sstabellini@kernel.org>; Paul Durrant <paul.durrant@citrix.com>
> Cc: Zhang Chen <zhangckid@gmail.com>
> Subject: Re: [PATCH V2 0/5] Add Xen COLO support
> 
> 
> On 2019/6/10 上午12:44, Zhang Chen wrote:
> > From: Zhang Chen <chen.zhang@intel.com>
> >
> > Xen COLO based on KVM COLO architecture, it shared COLO proxy and
> > block replication with KVM COLO. The only differece is Xen COLO have
> > own COLO-frame to handle live migration related function, so we need
> > this series make Xen COLO frame can communicate with other COLO
> > modules in qemu. Xen side related patches have been merged.
> 
> 
> Any pointer to this?
> 

Yes, please see this thread:

https://lists.xen.org/archives/html/xen-devel/2017-03/msg00718.html

Thanks
Zhang Chen

> Thanks
> 
> 
> >
> > V2:
> >   - Rebase on upstream code.
> >   - Optimize code by Zhijian's comments in patch 4/5.
> >   - Remove origin 5/6 patch.
> >
> > V1:
> >   - Initial patch.
> >
> > Zhang Chen (5):
> >    COLO-compare: Add new parameter to communicate with remote colo-
> frame
> >    COLO-compare: Add remote notification chardev handler frame
> >    COLO-compare: Make the compare_chr_send() can send notification
> >      message.
> >    COLO-compare: Add colo-compare remote notify support
> >    migration/colo.c: Add missed filter notify for Xen COLO.
> >
> >   migration/colo.c   |   2 +
> >   net/colo-compare.c | 155 +++++++++++++++++++++++++++++++++++++++---
> ---
> >   qemu-options.hx    |  33 +++++++++-
> >   3 files changed, 171 insertions(+), 19 deletions(-)
> >

Re: [Qemu-devel] [PATCH V2 0/5] Add Xen COLO support

[Zhang, Chen]

Anyone have any comments about this series?

Thanks
Zhang Chen


> -----Original Message-----
> From: Qemu-devel [mailto:qemu-devel-
> bounces+chen.zhang=intel.com@nongnu.org] On Behalf Of Zhang, Chen
> Sent: Monday, June 10, 2019 1:56 PM
> To: Jason Wang <jasowang@redhat.com>; Li Zhijian <lizhijian@cn.fujitsu.com>;
> Dr. David Alan Gilbert <dgilbert@redhat.com>; Juan Quintela
> <quintela@redhat.com>; zhanghailiang <zhang.zhanghailiang@huawei.com>;
> qemu-dev <qemu-devel@nongnu.org>; Stefano Stabellini
> <sstabellini@kernel.org>; Paul Durrant <paul.durrant@citrix.com>
> Cc: Zhang Chen <zhangckid@gmail.com>
> Subject: Re: [Qemu-devel] [PATCH V2 0/5] Add Xen COLO support
> 
> 
> 
> > -----Original Message-----
> > From: Jason Wang [mailto:jasowang@redhat.com]
> > Sent: Monday, June 10, 2019 12:09 PM
> > To: Zhang, Chen <chen.zhang@intel.com>; Li Zhijian
> > <lizhijian@cn.fujitsu.com>; Dr. David Alan Gilbert
> > <dgilbert@redhat.com>; Juan Quintela <quintela@redhat.com>;
> > zhanghailiang <zhang.zhanghailiang@huawei.com>; qemu-dev
> > <qemu-devel@nongnu.org>; Stefano Stabellini <sstabellini@kernel.org>;
> > Paul Durrant <paul.durrant@citrix.com>
> > Cc: Zhang Chen <zhangckid@gmail.com>
> > Subject: Re: [PATCH V2 0/5] Add Xen COLO support
> >
> >
> > On 2019/6/10 上午12:44, Zhang Chen wrote:
> > > From: Zhang Chen <chen.zhang@intel.com>
> > >
> > > Xen COLO based on KVM COLO architecture, it shared COLO proxy and
> > > block replication with KVM COLO. The only differece is Xen COLO have
> > > own COLO-frame to handle live migration related function, so we need
> > > this series make Xen COLO frame can communicate with other COLO
> > > modules in qemu. Xen side related patches have been merged.
> >
> >
> > Any pointer to this?
> >
> 
> Yes, please see this thread:
> 
> https://lists.xen.org/archives/html/xen-devel/2017-03/msg00718.html
> 
> Thanks
> Zhang Chen
> 
> > Thanks
> >
> >
> > >
> > > V2:
> > >   - Rebase on upstream code.
> > >   - Optimize code by Zhijian's comments in patch 4/5.
> > >   - Remove origin 5/6 patch.
> > >
> > > V1:
> > >   - Initial patch.
> > >
> > > Zhang Chen (5):
> > >    COLO-compare: Add new parameter to communicate with remote colo-
> > frame
> > >    COLO-compare: Add remote notification chardev handler frame
> > >    COLO-compare: Make the compare_chr_send() can send notification
> > >      message.
> > >    COLO-compare: Add colo-compare remote notify support
> > >    migration/colo.c: Add missed filter notify for Xen COLO.
> > >
> > >   migration/colo.c   |   2 +
> > >   net/colo-compare.c | 155
> > > +++++++++++++++++++++++++++++++++++++++---
> > ---
> > >   qemu-options.hx    |  33 +++++++++-
> > >   3 files changed, 171 insertions(+), 19 deletions(-)
> > >

Re: [Qemu-devel] [PATCH V2 0/5] Add Xen COLO support

[Jason Wang]

On 2019/6/21 下午2:43, Zhang, Chen wrote:
> Anyone have any comments about this series?
>
> Thanks
> Zhang Chen


I will probably review and give feedback sometime next week.

Thanks


>
>
>> -----Original Message-----
>> From: Qemu-devel [mailto:qemu-devel-
>> bounces+chen.zhang=intel.com@nongnu.org] On Behalf Of Zhang, Chen
>> Sent: Monday, June 10, 2019 1:56 PM
>> To: Jason Wang <jasowang@redhat.com>; Li Zhijian <lizhijian@cn.fujitsu.com>;
>> Dr. David Alan Gilbert <dgilbert@redhat.com>; Juan Quintela
>> <quintela@redhat.com>; zhanghailiang <zhang.zhanghailiang@huawei.com>;
>> qemu-dev <qemu-devel@nongnu.org>; Stefano Stabellini
>> <sstabellini@kernel.org>; Paul Durrant <paul.durrant@citrix.com>
>> Cc: Zhang Chen <zhangckid@gmail.com>
>> Subject: Re: [Qemu-devel] [PATCH V2 0/5] Add Xen COLO support
>>
>>
>>
>>> -----Original Message-----
>>> From: Jason Wang [mailto:jasowang@redhat.com]
>>> Sent: Monday, June 10, 2019 12:09 PM
>>> To: Zhang, Chen <chen.zhang@intel.com>; Li Zhijian
>>> <lizhijian@cn.fujitsu.com>; Dr. David Alan Gilbert
>>> <dgilbert@redhat.com>; Juan Quintela <quintela@redhat.com>;
>>> zhanghailiang <zhang.zhanghailiang@huawei.com>; qemu-dev
>>> <qemu-devel@nongnu.org>; Stefano Stabellini <sstabellini@kernel.org>;
>>> Paul Durrant <paul.durrant@citrix.com>
>>> Cc: Zhang Chen <zhangckid@gmail.com>
>>> Subject: Re: [PATCH V2 0/5] Add Xen COLO support
>>>
>>>
>>> On 2019/6/10 上午12:44, Zhang Chen wrote:
>>>> From: Zhang Chen <chen.zhang@intel.com>
>>>>
>>>> Xen COLO based on KVM COLO architecture, it shared COLO proxy and
>>>> block replication with KVM COLO. The only differece is Xen COLO have
>>>> own COLO-frame to handle live migration related function, so we need
>>>> this series make Xen COLO frame can communicate with other COLO
>>>> modules in qemu. Xen side related patches have been merged.
>>>
>>> Any pointer to this?
>>>
>> Yes, please see this thread:
>>
>> https://lists.xen.org/archives/html/xen-devel/2017-03/msg00718.html
>>
>> Thanks
>> Zhang Chen
>>
>>> Thanks
>>>
>>>
>>>> V2:
>>>>    - Rebase on upstream code.
>>>>    - Optimize code by Zhijian's comments in patch 4/5.
>>>>    - Remove origin 5/6 patch.
>>>>
>>>> V1:
>>>>    - Initial patch.
>>>>
>>>> Zhang Chen (5):
>>>>     COLO-compare: Add new parameter to communicate with remote colo-
>>> frame
>>>>     COLO-compare: Add remote notification chardev handler frame
>>>>     COLO-compare: Make the compare_chr_send() can send notification
>>>>       message.
>>>>     COLO-compare: Add colo-compare remote notify support
>>>>     migration/colo.c: Add missed filter notify for Xen COLO.
>>>>
>>>>    migration/colo.c   |   2 +
>>>>    net/colo-compare.c | 155
>>>> +++++++++++++++++++++++++++++++++++++++---
>>> ---
>>>>    qemu-options.hx    |  33 +++++++++-
>>>>    3 files changed, 171 insertions(+), 19 deletions(-)
>>>>

Re: [Qemu-devel] [PATCH V2 0/5] Add Xen COLO support

[Zhang, Chen]

> -----Original Message-----
> From: Jason Wang [mailto:jasowang@redhat.com]
> Sent: Friday, June 21, 2019 3:25 PM
> To: Zhang, Chen <chen.zhang@intel.com>; Li Zhijian <lizhijian@cn.fujitsu.com>;
> Dr. David Alan Gilbert <dgilbert@redhat.com>; Juan Quintela
> <quintela@redhat.com>; zhanghailiang <zhang.zhanghailiang@huawei.com>;
> qemu-dev <qemu-devel@nongnu.org>; Stefano Stabellini
> <sstabellini@kernel.org>; Paul Durrant <paul.durrant@citrix.com>
> Cc: Zhang Chen <zhangckid@gmail.com>
> Subject: Re: [PATCH V2 0/5] Add Xen COLO support
> 
> 
> On 2019/6/21 下午2:43, Zhang, Chen wrote:
> > Anyone have any comments about this series?
> >
> > Thanks
> > Zhang Chen
> 
> 
> I will probably review and give feedback sometime next week.
> 

Thank you Jason~ 

Thanks
Zhang Chen

> Thanks
> 
> 
> >
> >
> >> -----Original Message-----
> >> From: Qemu-devel [mailto:qemu-devel-
> >> bounces+chen.zhang=intel.com@nongnu.org] On Behalf Of Zhang, Chen
> >> Sent: Monday, June 10, 2019 1:56 PM
> >> To: Jason Wang <jasowang@redhat.com>; Li Zhijian
> >> <lizhijian@cn.fujitsu.com>; Dr. David Alan Gilbert
> >> <dgilbert@redhat.com>; Juan Quintela <quintela@redhat.com>;
> >> zhanghailiang <zhang.zhanghailiang@huawei.com>; qemu-dev
> >> <qemu-devel@nongnu.org>; Stefano Stabellini <sstabellini@kernel.org>;
> >> Paul Durrant <paul.durrant@citrix.com>
> >> Cc: Zhang Chen <zhangckid@gmail.com>
> >> Subject: Re: [Qemu-devel] [PATCH V2 0/5] Add Xen COLO support
> >>
> >>
> >>
> >>> -----Original Message-----
> >>> From: Jason Wang [mailto:jasowang@redhat.com]
> >>> Sent: Monday, June 10, 2019 12:09 PM
> >>> To: Zhang, Chen <chen.zhang@intel.com>; Li Zhijian
> >>> <lizhijian@cn.fujitsu.com>; Dr. David Alan Gilbert
> >>> <dgilbert@redhat.com>; Juan Quintela <quintela@redhat.com>;
> >>> zhanghailiang <zhang.zhanghailiang@huawei.com>; qemu-dev
> >>> <qemu-devel@nongnu.org>; Stefano Stabellini
> >>> <sstabellini@kernel.org>; Paul Durrant <paul.durrant@citrix.com>
> >>> Cc: Zhang Chen <zhangckid@gmail.com>
> >>> Subject: Re: [PATCH V2 0/5] Add Xen COLO support
> >>>
> >>>
> >>> On 2019/6/10 上午12:44, Zhang Chen wrote:
> >>>> From: Zhang Chen <chen.zhang@intel.com>
> >>>>
> >>>> Xen COLO based on KVM COLO architecture, it shared COLO proxy and
> >>>> block replication with KVM COLO. The only differece is Xen COLO
> >>>> have own COLO-frame to handle live migration related function, so
> >>>> we need this series make Xen COLO frame can communicate with other
> >>>> COLO modules in qemu. Xen side related patches have been merged.
> >>>
> >>> Any pointer to this?
> >>>
> >> Yes, please see this thread:
> >>
> >> https://lists.xen.org/archives/html/xen-devel/2017-03/msg00718.html
> >>
> >> Thanks
> >> Zhang Chen
> >>
> >>> Thanks
> >>>
> >>>
> >>>> V2:
> >>>>    - Rebase on upstream code.
> >>>>    - Optimize code by Zhijian's comments in patch 4/5.
> >>>>    - Remove origin 5/6 patch.
> >>>>
> >>>> V1:
> >>>>    - Initial patch.
> >>>>
> >>>> Zhang Chen (5):
> >>>>     COLO-compare: Add new parameter to communicate with remote
> >>>> colo-
> >>> frame
> >>>>     COLO-compare: Add remote notification chardev handler frame
> >>>>     COLO-compare: Make the compare_chr_send() can send notification
> >>>>       message.
> >>>>     COLO-compare: Add colo-compare remote notify support
> >>>>     migration/colo.c: Add missed filter notify for Xen COLO.
> >>>>
> >>>>    migration/colo.c   |   2 +
> >>>>    net/colo-compare.c | 155
> >>>> +++++++++++++++++++++++++++++++++++++++---
> >>> ---
> >>>>    qemu-options.hx    |  33 +++++++++-
> >>>>    3 files changed, 171 insertions(+), 19 deletions(-)
> >>>>

Re: [Qemu-devel] [PATCH V2 4/5] COLO-compare: Add colo-compare remote notify support

[Jason Wang]

On 2019/6/10 上午12:44, Zhang Chen wrote:
> From: Zhang Chen <chen.zhang@intel.com>
>
> This patch make colo-compare can send message to remote COLO frame(Xen) when occur checkpoint.
>
> Signed-off-by: Zhang Chen <chen.zhang@intel.com>
> ---
>   net/colo-compare.c | 54 +++++++++++++++++++++++++++++++++++++---------
>   1 file changed, 44 insertions(+), 10 deletions(-)
>
> diff --git a/net/colo-compare.c b/net/colo-compare.c
> index 16285f4a96..516b651ecd 100644
> --- a/net/colo-compare.c
> +++ b/net/colo-compare.c
> @@ -120,11 +120,6 @@ enum {
>       SECONDARY_IN,
>   };
>   
> -static void colo_compare_inconsistency_notify(void)
> -{
> -    notifier_list_notify(&colo_compare_notifiers,
> -                migrate_get_current());
> -}
>   
>   static int compare_chr_send(CompareState *s,
>                               const uint8_t *buf,
> @@ -132,6 +127,27 @@ static int compare_chr_send(CompareState *s,
>                               uint32_t vnet_hdr_len,
>                               bool notify_remote_frame);
>   
> +static void notify_remote_frame(CompareState *s)
> +{
> +    char msg[] = "DO_CHECKPOINT";
> +    int ret = 0;
> +
> +    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
> +    if (ret < 0) {
> +        error_report("Notify Xen COLO-frame failed");
> +    }
> +}
> +
> +static void colo_compare_inconsistency_notify(CompareState *s)
> +{
> +    if (s->notify_dev) {
> +        notify_remote_frame(s);
> +    } else {
> +        notifier_list_notify(&colo_compare_notifiers,
> +                             migrate_get_current());
> +    }
> +}
> +
>   static gint seq_sorter(Packet *a, Packet *b, gpointer data)
>   {
>       struct tcp_hdr *atcp, *btcp;
> @@ -435,7 +451,7 @@ sec:
>           qemu_hexdump((char *)spkt->data, stderr,
>                        "colo-compare spkt", spkt->size);
>   
> -        colo_compare_inconsistency_notify();
> +        colo_compare_inconsistency_notify(s);
>       }
>   }
>   
> @@ -577,7 +593,7 @@ void colo_compare_unregister_notifier(Notifier *notify)
>   }
>   
>   static int colo_old_packet_check_one_conn(Connection *conn,
> -                                           void *user_data)
> +                                          CompareState *s)
>   {
>       GList *result = NULL;
>       int64_t check_time = REGULAR_PACKET_CHECK_MS;
> @@ -588,7 +604,7 @@ static int colo_old_packet_check_one_conn(Connection *conn,
>   
>       if (result) {
>           /* Do checkpoint will flush old packet */
> -        colo_compare_inconsistency_notify();
> +        colo_compare_inconsistency_notify(s);
>           return 0;
>       }
>   
> @@ -608,7 +624,7 @@ static void colo_old_packet_check(void *opaque)
>        * If we find one old packet, stop finding job and notify
>        * COLO frame do checkpoint.
>        */
> -    g_queue_find_custom(&s->conn_list, NULL,
> +    g_queue_find_custom(&s->conn_list, s,
>                           (GCompareFunc)colo_old_packet_check_one_conn);
>   }
>   
> @@ -637,7 +653,8 @@ static void colo_compare_packet(CompareState *s, Connection *conn,
>                */
>               trace_colo_compare_main("packet different");
>               g_queue_push_head(&conn->primary_list, pkt);
> -            colo_compare_inconsistency_notify();
> +
> +            colo_compare_inconsistency_notify(s);
>               break;
>           }
>       }
> @@ -989,7 +1006,24 @@ static void compare_sec_rs_finalize(SocketReadState *sec_rs)
>   
>   static void compare_notify_rs_finalize(SocketReadState *notify_rs)
>   {
> +    CompareState *s = container_of(notify_rs, CompareState, notify_rs);
> +
>       /* Get Xen colo-frame's notify and handle the message */
> +    char *data = g_memdup(notify_rs->buf, notify_rs->packet_len);
> +    char msg[] = "COLO_COMPARE_GET_XEN_INIT";
> +    int ret;
> +
> +    if (!strcmp(data, "COLO_USERSPACE_PROXY_INIT")) {
> +        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
> +        if (ret < 0) {
> +            error_report("Notify Xen COLO-frame INIT failed");
> +        }
> +    }
> +
> +    if (!strcmp(data, "COLO_CHECKPOINT")) {
> +        /* colo-compare do checkpoint, flush pri packet and remove sec packet */
> +        g_queue_foreach(&s->conn_list, colo_flush_packets, s);
> +    }
>   }


This protocol looks too simple, is this accepted by Xen?

Thanks


>   
>   /*

Re: [Qemu-devel] [PATCH V2 4/5] COLO-compare: Add colo-compare remote notify support

[Zhang, Chen]

> -----Original Message-----
> From: Jason Wang [mailto:jasowang@redhat.com]
> Sent: Thursday, June 27, 2019 9:58 AM
> To: Zhang, Chen <chen.zhang@intel.com>; Li Zhijian <lizhijian@cn.fujitsu.com>;
> Dr. David Alan Gilbert <dgilbert@redhat.com>; Juan Quintela
> <quintela@redhat.com>; zhanghailiang <zhang.zhanghailiang@huawei.com>;
> qemu-dev <qemu-devel@nongnu.org>; Stefano Stabellini
> <sstabellini@kernel.org>; Paul Durrant <paul.durrant@citrix.com>
> Cc: Zhang Chen <zhangckid@gmail.com>
> Subject: Re: [Qemu-devel] [PATCH V2 4/5] COLO-compare: Add colo-compare
> remote notify support
> 
> 
> On 2019/6/10 上午12:44, Zhang Chen wrote:
> > From: Zhang Chen <chen.zhang@intel.com>
> >
> > This patch make colo-compare can send message to remote COLO frame(Xen)
> when occur checkpoint.
> >
> > Signed-off-by: Zhang Chen <chen.zhang@intel.com>
> > ---
> >   net/colo-compare.c | 54 +++++++++++++++++++++++++++++++++++++--------
> -
> >   1 file changed, 44 insertions(+), 10 deletions(-)
> >
> > diff --git a/net/colo-compare.c b/net/colo-compare.c index
> > 16285f4a96..516b651ecd 100644
> > --- a/net/colo-compare.c
> > +++ b/net/colo-compare.c
> > @@ -120,11 +120,6 @@ enum {
> >       SECONDARY_IN,
> >   };
> >
> > -static void colo_compare_inconsistency_notify(void)
> > -{
> > -    notifier_list_notify(&colo_compare_notifiers,
> > -                migrate_get_current());
> > -}
> >
> >   static int compare_chr_send(CompareState *s,
> >                               const uint8_t *buf, @@ -132,6 +127,27 @@
> > static int compare_chr_send(CompareState *s,
> >                               uint32_t vnet_hdr_len,
> >                               bool notify_remote_frame);
> >
> > +static void notify_remote_frame(CompareState *s) {
> > +    char msg[] = "DO_CHECKPOINT";
> > +    int ret = 0;
> > +
> > +    ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
> > +    if (ret < 0) {
> > +        error_report("Notify Xen COLO-frame failed");
> > +    }
> > +}
> > +
> > +static void colo_compare_inconsistency_notify(CompareState *s) {
> > +    if (s->notify_dev) {
> > +        notify_remote_frame(s);
> > +    } else {
> > +        notifier_list_notify(&colo_compare_notifiers,
> > +                             migrate_get_current());
> > +    }
> > +}
> > +
> >   static gint seq_sorter(Packet *a, Packet *b, gpointer data)
> >   {
> >       struct tcp_hdr *atcp, *btcp;
> > @@ -435,7 +451,7 @@ sec:
> >           qemu_hexdump((char *)spkt->data, stderr,
> >                        "colo-compare spkt", spkt->size);
> >
> > -        colo_compare_inconsistency_notify();
> > +        colo_compare_inconsistency_notify(s);
> >       }
> >   }
> >
> > @@ -577,7 +593,7 @@ void colo_compare_unregister_notifier(Notifier
> *notify)
> >   }
> >
> >   static int colo_old_packet_check_one_conn(Connection *conn,
> > -                                           void *user_data)
> > +                                          CompareState *s)
> >   {
> >       GList *result = NULL;
> >       int64_t check_time = REGULAR_PACKET_CHECK_MS; @@ -588,7 +604,7
> > @@ static int colo_old_packet_check_one_conn(Connection *conn,
> >
> >       if (result) {
> >           /* Do checkpoint will flush old packet */
> > -        colo_compare_inconsistency_notify();
> > +        colo_compare_inconsistency_notify(s);
> >           return 0;
> >       }
> >
> > @@ -608,7 +624,7 @@ static void colo_old_packet_check(void *opaque)
> >        * If we find one old packet, stop finding job and notify
> >        * COLO frame do checkpoint.
> >        */
> > -    g_queue_find_custom(&s->conn_list, NULL,
> > +    g_queue_find_custom(&s->conn_list, s,
> >                           (GCompareFunc)colo_old_packet_check_one_conn);
> >   }
> >
> > @@ -637,7 +653,8 @@ static void colo_compare_packet(CompareState *s,
> Connection *conn,
> >                */
> >               trace_colo_compare_main("packet different");
> >               g_queue_push_head(&conn->primary_list, pkt);
> > -            colo_compare_inconsistency_notify();
> > +
> > +            colo_compare_inconsistency_notify(s);
> >               break;
> >           }
> >       }
> > @@ -989,7 +1006,24 @@ static void
> > compare_sec_rs_finalize(SocketReadState *sec_rs)
> >
> >   static void compare_notify_rs_finalize(SocketReadState *notify_rs)
> >   {
> > +    CompareState *s = container_of(notify_rs, CompareState,
> > + notify_rs);
> > +
> >       /* Get Xen colo-frame's notify and handle the message */
> > +    char *data = g_memdup(notify_rs->buf, notify_rs->packet_len);
> > +    char msg[] = "COLO_COMPARE_GET_XEN_INIT";
> > +    int ret;
> > +
> > +    if (!strcmp(data, "COLO_USERSPACE_PROXY_INIT")) {
> > +        ret = compare_chr_send(s, (uint8_t *)msg, strlen(msg), 0, true);
> > +        if (ret < 0) {
> > +            error_report("Notify Xen COLO-frame INIT failed");
> > +        }
> > +    }
> > +
> > +    if (!strcmp(data, "COLO_CHECKPOINT")) {
> > +        /* colo-compare do checkpoint, flush pri packet and remove sec packet
> */
> > +        g_queue_foreach(&s->conn_list, colo_flush_packets, s);
> > +    }
> >   }
> 
> 
> This protocol looks too simple, is this accepted by Xen?

Yes, that Xen patch from long time ago....
Currently, Our target is make Xen COLO can running, and we can optimize the protocol in the future.

Related Xen code:
https://github.com/xen-project/xen/blob/master/tools/libxl/libxl_colo_proxy.c


Thanks
Zhang Chen

> 
> Thanks
> 
> 
> >
> >   /*

Re: [Qemu-devel] [PATCH V2 0/5] Add Xen COLO support

[Jason Wang]

On 2019/6/10 上午12:44, Zhang Chen wrote:
> From: Zhang Chen <chen.zhang@intel.com>
>
> Xen COLO based on KVM COLO architecture, it shared COLO proxy and block
> replication with KVM COLO. The only differece is Xen COLO have own
> COLO-frame to handle live migration related function, so we need this
> series make Xen COLO frame can communicate with other COLO modules in
> qemu. Xen side related patches have been merged.
>
> V2:
>   - Rebase on upstream code.
>   - Optimize code by Zhijian's comments in patch 4/5.
>   - Remove origin 5/6 patch.
>
> V1:
>   - Initial patch.
>
> Zhang Chen (5):
>    COLO-compare: Add new parameter to communicate with remote colo-frame
>    COLO-compare: Add remote notification chardev handler frame
>    COLO-compare: Make the compare_chr_send() can send notification
>      message.
>    COLO-compare: Add colo-compare remote notify support
>    migration/colo.c: Add missed filter notify for Xen COLO.
>
>   migration/colo.c   |   2 +
>   net/colo-compare.c | 155 +++++++++++++++++++++++++++++++++++++++------
>   qemu-options.hx    |  33 +++++++++-
>   3 files changed, 171 insertions(+), 19 deletions(-)


Applied.

Thanks

>