[PATCH 00/10] COLO project queued patches 20-Oct

[PATCH 00/10] COLO project queued patches 20-Oct

[Zhang Chen]

From: Zhang Chen <chen.zhang@intel.com>

Hi Jason, this series include latest COLO related patches.
please check and merge it.

Thanks
Zhang Chen

Li Zhijian (2):
  colo-compare: fix missing compare_seq initialization
  colo-compare: check mark in mutual exclusion

Pan Nengyuan (1):
  net/filter-rewriter: destroy g_hash_table in colo_rewriter_cleanup

Rao, Lei (3):
  Optimize seq_sorter function for colo-compare
  Reduce the time of checkpoint for COLO
  Fix the qemu crash when guest shutdown in COLO mode

Zhang Chen (4):
  net/colo-compare.c: Fix compare_timeout format issue
  net/colo-compare.c: Change the timer clock type
  net/colo-compare.c: Add secondary old packet detection
  net/colo-compare.c: Increase default queued packet scan frequency

 migration/ram.c       | 14 ++++++++++-
 net/colo-compare.c    | 57 ++++++++++++++++++++++---------------------
 net/colo.c            |  5 +---
 net/filter-rewriter.c |  2 ++
 softmmu/vl.c          |  1 +
 5 files changed, 46 insertions(+), 33 deletions(-)

-- 
2.17.1

[PATCH 01/10] net/filter-rewriter: destroy g_hash_table in colo_rewriter_cleanup

[Zhang Chen]

From: Pan Nengyuan <pannengyuan@huawei.com>

s->connection_track_table forgot to destroy in colo_rewriter_cleanup. Fix it.

Reported-by: Euler Robot <euler.robot@huawei.com>
Signed-off-by: Pan Nengyuan <pannengyuan@huawei.com>
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Reviewed-by: Zhang Chen <chen.zhang@intel.com>
Reviewed-by: Li Qiang <liq3ea@gmail.com>
---
 net/filter-rewriter.c | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/net/filter-rewriter.c b/net/filter-rewriter.c
index dc3c27a489..e063a818b7 100644
--- a/net/filter-rewriter.c
+++ b/net/filter-rewriter.c
@@ -381,6 +381,8 @@ static void colo_rewriter_cleanup(NetFilterState *nf)
         filter_rewriter_flush(nf);
         g_free(s->incoming_queue);
     }
+
+    g_hash_table_destroy(s->connection_track_table);
 }
 
 static void colo_rewriter_setup(NetFilterState *nf, Error **errp)
-- 
2.17.1

[PATCH 02/10] Optimize seq_sorter function for colo-compare

[Zhang Chen]

From: "Rao, Lei" <lei.rao@intel.com>

The seq of tcp has been filled in fill_pkt_tcp_info, it
can be used directly here.

Signed-off-by: leirao <lei.rao@intel.com>
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Reviewed-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Reviewed-by: Zhang Chen <chen.zhang@intel.com>
---
 net/colo-compare.c | 6 +-----
 1 file changed, 1 insertion(+), 5 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 3a45d64175..86980cef5e 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -196,11 +196,7 @@ static void colo_compare_inconsistency_notify(CompareState *s)
 
 static gint seq_sorter(Packet *a, Packet *b, gpointer data)
 {
-    struct tcp_hdr *atcp, *btcp;
-
-    atcp = (struct tcp_hdr *)(a->transport_header);
-    btcp = (struct tcp_hdr *)(b->transport_header);
-    return ntohl(atcp->th_seq) - ntohl(btcp->th_seq);
+    return a->tcp_seq - b->tcp_seq;
 }
 
 static void fill_pkt_tcp_info(void *data, uint32_t *max_ack)
-- 
2.17.1

Re: [PATCH 02/10] Optimize seq_sorter function for colo-compare

[Philippe Mathieu-Daudé]

On 10/14/20 9:25 AM, Zhang Chen wrote:
> From: "Rao, Lei" <lei.rao@intel.com>
> 
> The seq of tcp has been filled in fill_pkt_tcp_info, it
> can be used directly here.
> 
> Signed-off-by: leirao <lei.rao@intel.com>

"Signed-off-by: Lei Rao <lei.rao@intel.com>"?

> Signed-off-by: Zhang Chen <chen.zhang@intel.com>
> Reviewed-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Reviewed-by: Zhang Chen <chen.zhang@intel.com>
> ---
>   net/colo-compare.c | 6 +-----
>   1 file changed, 1 insertion(+), 5 deletions(-)
> 
> diff --git a/net/colo-compare.c b/net/colo-compare.c
> index 3a45d64175..86980cef5e 100644
> --- a/net/colo-compare.c
> +++ b/net/colo-compare.c
> @@ -196,11 +196,7 @@ static void colo_compare_inconsistency_notify(CompareState *s)
>   

Maybe add a comment /* Use restricted to colo_insert_packet() */

Because if someone use it elsewhere it will now be buggy.

>   static gint seq_sorter(Packet *a, Packet *b, gpointer data)
>   {
> -    struct tcp_hdr *atcp, *btcp;
> -
> -    atcp = (struct tcp_hdr *)(a->transport_header);
> -    btcp = (struct tcp_hdr *)(b->transport_header);
> -    return ntohl(atcp->th_seq) - ntohl(btcp->th_seq);
> +    return a->tcp_seq - b->tcp_seq;
>   }
>   
>   static void fill_pkt_tcp_info(void *data, uint32_t *max_ack)
> 

[PATCH 03/10] Reduce the time of checkpoint for COLO

[Zhang Chen]

From: "Rao, Lei" <lei.rao@intel.com>

we should set ram_bulk_stage to false after ram_state_init,
otherwise the bitmap will be unused in migration_bitmap_find_dirty.
all pages in ram cache will be flushed to the ram of secondary guest
for each checkpoint.

Signed-off-by: leirao <lei.rao@intel.com>
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Reviewed-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Reviewed-by: Zhang Chen <chen.zhang@intel.com>
---
 migration/ram.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/migration/ram.c b/migration/ram.c
index 433489d633..9cfac3d9ba 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -3009,6 +3009,18 @@ static void decompress_data_with_multi_threads(QEMUFile *f,
     qemu_mutex_unlock(&decomp_done_lock);
 }
 
+ /*
+  * we must set ram_bulk_stage to false, otherwise in
+  * migation_bitmap_find_dirty the bitmap will be unused and
+  * all the pages in ram cache wil be flushed to the ram of
+  * secondary VM.
+  */
+static void colo_init_ram_state(void)
+{
+    ram_state_init(&ram_state);
+    ram_state->ram_bulk_stage = false;
+}
+
 /*
  * colo cache: this is for secondary VM, we cache the whole
  * memory of the secondary VM, it is need to hold the global lock
@@ -3052,7 +3064,7 @@ int colo_init_ram_cache(void)
         }
     }
 
-    ram_state_init(&ram_state);
+    colo_init_ram_state();
     return 0;
 }
 
-- 
2.17.1

RE: [PATCH 03/10] Reduce the time of checkpoint for COLO

[Zhang, Chen]

> -----Original Message-----
> From: Zhang Chen <chen.zhang@intel.com>
> Sent: Wednesday, October 14, 2020 3:26 PM
> To: Jason Wang <jasowang@redhat.com>; qemu-dev <qemu-
> devel@nongnu.org>
> Cc: Zhang Chen <zhangckid@gmail.com>; Rao, Lei <lei.rao@intel.com>;
> Zhang, Chen <chen.zhang@intel.com>
> Subject: [PATCH 03/10] Reduce the time of checkpoint for COLO
> 
> From: "Rao, Lei" <lei.rao@intel.com>
> 
> we should set ram_bulk_stage to false after ram_state_init, otherwise the
> bitmap will be unused in migration_bitmap_find_dirty.
> all pages in ram cache will be flushed to the ram of secondary guest for each
> checkpoint.
> 
> Signed-off-by: leirao <lei.rao@intel.com>
> Signed-off-by: Zhang Chen <chen.zhang@intel.com>
> Reviewed-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Reviewed-by: Zhang Chen <chen.zhang@intel.com>

Sorry, I forgot to add:
Signed-off-by: Derek Su <dereksu@qnap.com>

Thanks
Zhang Chen

> ---
>  migration/ram.c | 14 +++++++++++++-
>  1 file changed, 13 insertions(+), 1 deletion(-)
> 
> diff --git a/migration/ram.c b/migration/ram.c index 433489d633..9cfac3d9ba
> 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -3009,6 +3009,18 @@ static void
> decompress_data_with_multi_threads(QEMUFile *f,
>      qemu_mutex_unlock(&decomp_done_lock);
>  }
> 
> + /*
> +  * we must set ram_bulk_stage to false, otherwise in
> +  * migation_bitmap_find_dirty the bitmap will be unused and
> +  * all the pages in ram cache wil be flushed to the ram of
> +  * secondary VM.
> +  */
> +static void colo_init_ram_state(void)
> +{
> +    ram_state_init(&ram_state);
> +    ram_state->ram_bulk_stage = false;
> +}
> +
>  /*
>   * colo cache: this is for secondary VM, we cache the whole
>   * memory of the secondary VM, it is need to hold the global lock @@ -
> 3052,7 +3064,7 @@ int colo_init_ram_cache(void)
>          }
>      }
> 
> -    ram_state_init(&ram_state);
> +    colo_init_ram_state();
>      return 0;
>  }
> 
> --
> 2.17.1

[PATCH 04/10] Fix the qemu crash when guest shutdown in COLO mode

[Zhang Chen]

From: "Rao, Lei" <lei.rao@intel.com>

In COLO mode, if the startup parameters of QEMU include "no-shutdown",
QEMU will crash when the guest shutdown. The root cause is when the
guest shutdown, the state of VM will switch COLO to SHUTDOWN. When do
checkpoint again, the state will be changed to COLO. But the state
switch is undefined in runstate_transitions_def, we should add it.
This patch fixes the following:
qemu-system-x86_64: invalid runstate transition: 'shutdown' -> 'colo'
Aborted

Signed-off-by: leirao <lei.rao@intel.com>
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Reviewed-by: Zhang Chen <chen.zhang@intel.com>
---
 softmmu/vl.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/softmmu/vl.c b/softmmu/vl.c
index 5a11a62f78..bafe7c5b70 100644
--- a/softmmu/vl.c
+++ b/softmmu/vl.c
@@ -632,6 +632,7 @@ static const RunStateTransition runstate_transitions_def[] = {
     { RUN_STATE_SHUTDOWN, RUN_STATE_PAUSED },
     { RUN_STATE_SHUTDOWN, RUN_STATE_FINISH_MIGRATE },
     { RUN_STATE_SHUTDOWN, RUN_STATE_PRELAUNCH },
+    { RUN_STATE_SHUTDOWN, RUN_STATE_COLO },
 
     { RUN_STATE_DEBUG, RUN_STATE_SUSPENDED },
     { RUN_STATE_RUNNING, RUN_STATE_SUSPENDED },
-- 
2.17.1

[PATCH 05/10] colo-compare: fix missing compare_seq initialization

[Zhang Chen]

From: Li Zhijian <lizhijian@cn.fujitsu.com>

Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Reviewed-by: Zhang Chen <chen.zhang@intel.com>
---
 net/colo.c | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/net/colo.c b/net/colo.c
index a6c66d829a..ef00609848 100644
--- a/net/colo.c
+++ b/net/colo.c
@@ -133,14 +133,11 @@ void reverse_connection_key(ConnectionKey *key)
 
 Connection *connection_new(ConnectionKey *key)
 {
-    Connection *conn = g_slice_new(Connection);
+    Connection *conn = g_slice_new0(Connection);
 
     conn->ip_proto = key->ip_proto;
     conn->processing = false;
-    conn->offset = 0;
     conn->tcp_state = TCPS_CLOSED;
-    conn->pack = 0;
-    conn->sack = 0;
     g_queue_init(&conn->primary_list);
     g_queue_init(&conn->secondary_list);
 
-- 
2.17.1

Re: [PATCH 05/10] colo-compare: fix missing compare_seq initialization

[Philippe Mathieu-Daudé]

On 10/14/20 9:25 AM, Zhang Chen wrote:
> From: Li Zhijian <lizhijian@cn.fujitsu.com>
> 

Please include:

Fixes: f449c9e549c ("colo: compare the packet based on the tcp sequence 
number")
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Signed-off-by: Zhang Chen <chen.zhang@intel.com>
> Reviewed-by: Zhang Chen <chen.zhang@intel.com>
> ---
>   net/colo.c | 5 +----
>   1 file changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/net/colo.c b/net/colo.c
> index a6c66d829a..ef00609848 100644
> --- a/net/colo.c
> +++ b/net/colo.c
> @@ -133,14 +133,11 @@ void reverse_connection_key(ConnectionKey *key)
>   
>   Connection *connection_new(ConnectionKey *key)
>   {
> -    Connection *conn = g_slice_new(Connection);
> +    Connection *conn = g_slice_new0(Connection);
>   
>       conn->ip_proto = key->ip_proto;
>       conn->processing = false;
> -    conn->offset = 0;
>       conn->tcp_state = TCPS_CLOSED;
> -    conn->pack = 0;
> -    conn->sack = 0;
>       g_queue_init(&conn->primary_list);
>       g_queue_init(&conn->secondary_list);
>   
> 

[PATCH 06/10] colo-compare: check mark in mutual exclusion

[Zhang Chen]

From: Li Zhijian <lizhijian@cn.fujitsu.com>

Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Reviewed-by: Zhang Chen <chen.zhang@intel.com>
---
 net/colo-compare.c | 6 ++----
 1 file changed, 2 insertions(+), 4 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 86980cef5e..24c366eec0 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -476,13 +476,11 @@ sec:
             colo_release_primary_pkt(s, ppkt);
             g_queue_push_head(&conn->secondary_list, spkt);
             goto pri;
-        }
-        if (mark == COLO_COMPARE_FREE_SECONDARY) {
+        } else if (mark == COLO_COMPARE_FREE_SECONDARY) {
             conn->compare_seq = spkt->seq_end;
             packet_destroy(spkt, NULL);
             goto sec;
-        }
-        if (mark == (COLO_COMPARE_FREE_PRIMARY | COLO_COMPARE_FREE_SECONDARY)) {
+        } else if (mark == (COLO_COMPARE_FREE_PRIMARY | COLO_COMPARE_FREE_SECONDARY)) {
             conn->compare_seq = ppkt->seq_end;
             colo_release_primary_pkt(s, ppkt);
             packet_destroy(spkt, NULL);
-- 
2.17.1

[PATCH 07/10] net/colo-compare.c: Fix compare_timeout format issue

[Zhang Chen]

From: Zhang Chen <chen.zhang@intel.com>

This parameter need compare with the return of qemu_clock_get_ms(),
it is uinit64_t. So we need fix this issue here.

Reported-by: Derek Su <dereksu@qnap.com>
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Reviewed-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
 net/colo-compare.c | 12 ++++++------
 1 file changed, 6 insertions(+), 6 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 24c366eec0..f4814c5f09 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -120,7 +120,7 @@ struct CompareState {
     SendCo out_sendco;
     SendCo notify_sendco;
     bool vnet_hdr;
-    uint32_t compare_timeout;
+    uint64_t compare_timeout;
     uint32_t expired_scan_cycle;
 
     /*
@@ -1075,9 +1075,9 @@ static void compare_get_timeout(Object *obj, Visitor *v,
                                 Error **errp)
 {
     CompareState *s = COLO_COMPARE(obj);
-    uint32_t value = s->compare_timeout;
+    uint64_t value = s->compare_timeout;
 
-    visit_type_uint32(v, name, &value, errp);
+    visit_type_uint64(v, name, &value, errp);
 }
 
 static void compare_set_timeout(Object *obj, Visitor *v,
@@ -1140,9 +1140,9 @@ static void set_max_queue_size(Object *obj, Visitor *v,
                                Error **errp)
 {
     Error *local_err = NULL;
-    uint32_t value;
+    uint64_t value;
 
-    visit_type_uint32(v, name, &value, &local_err);
+    visit_type_uint64(v, name, &value, &local_err);
     if (local_err) {
         goto out;
     }
@@ -1390,7 +1390,7 @@ static void colo_compare_init(Object *obj)
     object_property_add_str(obj, "notify_dev",
                             compare_get_notify_dev, compare_set_notify_dev);
 
-    object_property_add(obj, "compare_timeout", "uint32",
+    object_property_add(obj, "compare_timeout", "uint64",
                         compare_get_timeout,
                         compare_set_timeout, NULL, NULL);
 
-- 
2.17.1

Re: [PATCH 07/10] net/colo-compare.c: Fix compare_timeout format issue

[Philippe Mathieu-Daudé]

On 10/14/20 9:25 AM, Zhang Chen wrote:
> From: Zhang Chen <chen.zhang@intel.com>
> 
> This parameter need compare with the return of qemu_clock_get_ms(),
> it is uinit64_t. So we need fix this issue here.

Typo "uint64_t"

Please add:

Fixes: 9cc43c94b31 ("net/colo-compare.c: Expose "compare_timeout" to users")
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> 
> Reported-by: Derek Su <dereksu@qnap.com>
> Signed-off-by: Zhang Chen <chen.zhang@intel.com>
> Reviewed-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> ---
>   net/colo-compare.c | 12 ++++++------
>   1 file changed, 6 insertions(+), 6 deletions(-)
> 
> diff --git a/net/colo-compare.c b/net/colo-compare.c
> index 24c366eec0..f4814c5f09 100644
> --- a/net/colo-compare.c
> +++ b/net/colo-compare.c
> @@ -120,7 +120,7 @@ struct CompareState {
>       SendCo out_sendco;
>       SendCo notify_sendco;
>       bool vnet_hdr;
> -    uint32_t compare_timeout;
> +    uint64_t compare_timeout;
>       uint32_t expired_scan_cycle;
>   
>       /*
> @@ -1075,9 +1075,9 @@ static void compare_get_timeout(Object *obj, Visitor *v,
>                                   Error **errp)
>   {
>       CompareState *s = COLO_COMPARE(obj);
> -    uint32_t value = s->compare_timeout;
> +    uint64_t value = s->compare_timeout;
>   
> -    visit_type_uint32(v, name, &value, errp);
> +    visit_type_uint64(v, name, &value, errp);
>   }
>   
>   static void compare_set_timeout(Object *obj, Visitor *v,
> @@ -1140,9 +1140,9 @@ static void set_max_queue_size(Object *obj, Visitor *v,
>                                  Error **errp)
>   {
>       Error *local_err = NULL;
> -    uint32_t value;
> +    uint64_t value;
>   
> -    visit_type_uint32(v, name, &value, &local_err);
> +    visit_type_uint64(v, name, &value, &local_err);
>       if (local_err) {
>           goto out;
>       }
> @@ -1390,7 +1390,7 @@ static void colo_compare_init(Object *obj)
>       object_property_add_str(obj, "notify_dev",
>                               compare_get_notify_dev, compare_set_notify_dev);
>   
> -    object_property_add(obj, "compare_timeout", "uint32",
> +    object_property_add(obj, "compare_timeout", "uint64",
>                           compare_get_timeout,
>                           compare_set_timeout, NULL, NULL);
>   
> 

[PATCH 08/10] net/colo-compare.c: Change the timer clock type

[Zhang Chen]

From: Zhang Chen <chen.zhang@intel.com>

The virtual clock only runs during the emulation. It stops
when the virtual machine is stopped.
The host clock should be used for device models that emulate accurate
real time sources. It will continue to run when the virtual machine
is suspended. COLO need to know the host time here.

Reported-by: Derek Su <dereksu@qnap.com>
Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Reviewed-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
 net/colo-compare.c | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index f4814c5f09..61c95fe7e9 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -899,7 +899,7 @@ static void check_old_packet_regular(void *opaque)
 
     /* if have old packet we will notify checkpoint */
     colo_old_packet_check(s);
-    timer_mod(s->packet_check_timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) +
+    timer_mod(s->packet_check_timer, qemu_clock_get_ms(QEMU_CLOCK_HOST) +
               s->expired_scan_cycle);
 }
 
@@ -933,10 +933,10 @@ static void colo_compare_timer_init(CompareState *s)
 {
     AioContext *ctx = iothread_get_aio_context(s->iothread);
 
-    s->packet_check_timer = aio_timer_new(ctx, QEMU_CLOCK_VIRTUAL,
+    s->packet_check_timer = aio_timer_new(ctx, QEMU_CLOCK_HOST,
                                 SCALE_MS, check_old_packet_regular,
                                 s);
-    timer_mod(s->packet_check_timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) +
+    timer_mod(s->packet_check_timer, qemu_clock_get_ms(QEMU_CLOCK_HOST) +
               s->expired_scan_cycle);
 }
 
-- 
2.17.1

Re: [PATCH 08/10] net/colo-compare.c: Change the timer clock type

[Philippe Mathieu-Daudé]

On 10/14/20 9:25 AM, Zhang Chen wrote:
> From: Zhang Chen <chen.zhang@intel.com>
> 
> The virtual clock only runs during the emulation. It stops
> when the virtual machine is stopped.
> The host clock should be used for device models that emulate accurate
> real time sources. It will continue to run when the virtual machine
> is suspended. COLO need to know the host time here.
> 

Please add:

Fixes: dd321ecfc2e ("colo-compare: Use IOThread to Check old packet 
regularly and Process packets of the primary")
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>

> Reported-by: Derek Su <dereksu@qnap.com>
> Signed-off-by: Zhang Chen <chen.zhang@intel.com>
> Reviewed-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> ---
>   net/colo-compare.c | 6 +++---
>   1 file changed, 3 insertions(+), 3 deletions(-)
> 
> diff --git a/net/colo-compare.c b/net/colo-compare.c
> index f4814c5f09..61c95fe7e9 100644
> --- a/net/colo-compare.c
> +++ b/net/colo-compare.c
> @@ -899,7 +899,7 @@ static void check_old_packet_regular(void *opaque)
>   
>       /* if have old packet we will notify checkpoint */
>       colo_old_packet_check(s);
> -    timer_mod(s->packet_check_timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) +
> +    timer_mod(s->packet_check_timer, qemu_clock_get_ms(QEMU_CLOCK_HOST) +
>                 s->expired_scan_cycle);
>   }
>   
> @@ -933,10 +933,10 @@ static void colo_compare_timer_init(CompareState *s)
>   {
>       AioContext *ctx = iothread_get_aio_context(s->iothread);
>   
> -    s->packet_check_timer = aio_timer_new(ctx, QEMU_CLOCK_VIRTUAL,
> +    s->packet_check_timer = aio_timer_new(ctx, QEMU_CLOCK_HOST,
>                                   SCALE_MS, check_old_packet_regular,
>                                   s);
> -    timer_mod(s->packet_check_timer, qemu_clock_get_ms(QEMU_CLOCK_VIRTUAL) +
> +    timer_mod(s->packet_check_timer, qemu_clock_get_ms(QEMU_CLOCK_HOST) +
>                 s->expired_scan_cycle);
>   }
>   
> 

[PATCH 09/10] net/colo-compare.c: Add secondary old packet detection

[Zhang Chen]

From: Zhang Chen <chen.zhang@intel.com>

Detect queued secondary packet to sync VM state in time.

Signed-off-by: Zhang Chen <chen.zhang@intel.com>
Reviewed-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
 net/colo-compare.c | 25 ++++++++++++++++---------
 1 file changed, 16 insertions(+), 9 deletions(-)

diff --git a/net/colo-compare.c b/net/colo-compare.c
index 61c95fe7e9..c22f2af941 100644
--- a/net/colo-compare.c
+++ b/net/colo-compare.c
@@ -635,19 +635,26 @@ void colo_compare_unregister_notifier(Notifier *notify)
 static int colo_old_packet_check_one_conn(Connection *conn,
                                           CompareState *s)
 {
-    GList *result = NULL;
-
-    result = g_queue_find_custom(&conn->primary_list,
-                                 &s->compare_timeout,
-                                 (GCompareFunc)colo_old_packet_check_one);
+    if (!g_queue_is_empty(&conn->primary_list)) {
+        if (g_queue_find_custom(&conn->primary_list,
+                                &s->compare_timeout,
+                                (GCompareFunc)colo_old_packet_check_one))
+            goto out;
+    }
 
-    if (result) {
-        /* Do checkpoint will flush old packet */
-        colo_compare_inconsistency_notify(s);
-        return 0;
+    if (!g_queue_is_empty(&conn->secondary_list)) {
+        if (g_queue_find_custom(&conn->secondary_list,
+                                &s->compare_timeout,
+                                (GCompareFunc)colo_old_packet_check_one))
+            goto out;
     }
 
     return 1;
+
+out:
+    /* Do checkpoint will flush old packet */
+    colo_compare_inconsistency_notify(s);
+    return 0;
 }
 
 /*
-- 
2.17.1

Re: [PATCH 00/10] COLO project queued patches 20-Oct

[Jason Wang]

On 2020/10/14 下午3:25, Zhang Chen wrote:
> From: Zhang Chen <chen.zhang@intel.com>
>
> Hi Jason, this series include latest COLO related patches.
> please check and merge it.
>
> Thanks
> Zhang Chen


Hi:

I want to merge but I don't get patch 10/10.

Is that lost during posting?

Thanks


>
> Li Zhijian (2):
>    colo-compare: fix missing compare_seq initialization
>    colo-compare: check mark in mutual exclusion
>
> Pan Nengyuan (1):
>    net/filter-rewriter: destroy g_hash_table in colo_rewriter_cleanup
>
> Rao, Lei (3):
>    Optimize seq_sorter function for colo-compare
>    Reduce the time of checkpoint for COLO
>    Fix the qemu crash when guest shutdown in COLO mode
>
> Zhang Chen (4):
>    net/colo-compare.c: Fix compare_timeout format issue
>    net/colo-compare.c: Change the timer clock type
>    net/colo-compare.c: Add secondary old packet detection
>    net/colo-compare.c: Increase default queued packet scan frequency
>
>   migration/ram.c       | 14 ++++++++++-
>   net/colo-compare.c    | 57 ++++++++++++++++++++++---------------------
>   net/colo.c            |  5 +---
>   net/filter-rewriter.c |  2 ++
>   softmmu/vl.c          |  1 +
>   5 files changed, 46 insertions(+), 33 deletions(-)
>

RE: [PATCH 00/10] COLO project queued patches 20-Oct

[Zhang, Chen]

> -----Original Message-----
> From: Jason Wang <jasowang@redhat.com>
> Sent: Thursday, October 15, 2020 3:56 PM
> To: Zhang, Chen <chen.zhang@intel.com>; qemu-dev <qemu-
> devel@nongnu.org>
> Cc: Zhang Chen <zhangckid@gmail.com>
> Subject: Re: [PATCH 00/10] COLO project queued patches 20-Oct
> 
> 
> On 2020/10/14 下午3:25, Zhang Chen wrote:
> > From: Zhang Chen <chen.zhang@intel.com>
> >
> > Hi Jason, this series include latest COLO related patches.
> > please check and merge it.
> >
> > Thanks
> > Zhang Chen
> 
> 
> Hi:
> 
> I want to merge but I don't get patch 10/10.
> 
> Is that lost during posting?

Oh, Sorry I missed it.
Already resend it:
[PATCH 10/10] net/colo-compare.c: Increase default queued packet scan frequency

Thanks
Chen

> 
> Thanks
> 
> 
> >
> > Li Zhijian (2):
> >    colo-compare: fix missing compare_seq initialization
> >    colo-compare: check mark in mutual exclusion
> >
> > Pan Nengyuan (1):
> >    net/filter-rewriter: destroy g_hash_table in colo_rewriter_cleanup
> >
> > Rao, Lei (3):
> >    Optimize seq_sorter function for colo-compare
> >    Reduce the time of checkpoint for COLO
> >    Fix the qemu crash when guest shutdown in COLO mode
> >
> > Zhang Chen (4):
> >    net/colo-compare.c: Fix compare_timeout format issue
> >    net/colo-compare.c: Change the timer clock type
> >    net/colo-compare.c: Add secondary old packet detection
> >    net/colo-compare.c: Increase default queued packet scan frequency
> >
> >   migration/ram.c       | 14 ++++++++++-
> >   net/colo-compare.c    | 57 ++++++++++++++++++++++---------------------
> >   net/colo.c            |  5 +---
> >   net/filter-rewriter.c |  2 ++
> >   softmmu/vl.c          |  1 +
> >   5 files changed, 46 insertions(+), 33 deletions(-)
> >

Re: [PATCH 00/10] COLO project queued patches 20-Oct

[Jason Wang]

On 2020/10/15 下午3:58, Zhang, Chen wrote:
>
>> -----Original Message-----
>> From: Jason Wang <jasowang@redhat.com>
>> Sent: Thursday, October 15, 2020 3:56 PM
>> To: Zhang, Chen <chen.zhang@intel.com>; qemu-dev <qemu-
>> devel@nongnu.org>
>> Cc: Zhang Chen <zhangckid@gmail.com>
>> Subject: Re: [PATCH 00/10] COLO project queued patches 20-Oct
>>
>>
>> On 2020/10/14 下午3:25, Zhang Chen wrote:
>>> From: Zhang Chen <chen.zhang@intel.com>
>>>
>>> Hi Jason, this series include latest COLO related patches.
>>> please check and merge it.
>>>
>>> Thanks
>>> Zhang Chen
>>
>> Hi:
>>
>> I want to merge but I don't get patch 10/10.
>>
>> Is that lost during posting?
> Oh, Sorry I missed it.
> Already resend it:
> [PATCH 10/10] net/colo-compare.c: Increase default queued packet scan frequency
>
> Thanks
> Chen


It looks to me Philippe has some minor comments on some patches, please 
address them and send a new version.

Thanks


>
>> Thanks
>>
>>
>>> Li Zhijian (2):
>>>     colo-compare: fix missing compare_seq initialization
>>>     colo-compare: check mark in mutual exclusion
>>>
>>> Pan Nengyuan (1):
>>>     net/filter-rewriter: destroy g_hash_table in colo_rewriter_cleanup
>>>
>>> Rao, Lei (3):
>>>     Optimize seq_sorter function for colo-compare
>>>     Reduce the time of checkpoint for COLO
>>>     Fix the qemu crash when guest shutdown in COLO mode
>>>
>>> Zhang Chen (4):
>>>     net/colo-compare.c: Fix compare_timeout format issue
>>>     net/colo-compare.c: Change the timer clock type
>>>     net/colo-compare.c: Add secondary old packet detection
>>>     net/colo-compare.c: Increase default queued packet scan frequency
>>>
>>>    migration/ram.c       | 14 ++++++++++-
>>>    net/colo-compare.c    | 57 ++++++++++++++++++++++---------------------
>>>    net/colo.c            |  5 +---
>>>    net/filter-rewriter.c |  2 ++
>>>    softmmu/vl.c          |  1 +
>>>    5 files changed, 46 insertions(+), 33 deletions(-)
>>>

RE: [PATCH 00/10] COLO project queued patches 20-Oct

[Zhang, Chen]

> -----Original Message-----
> From: Jason Wang <jasowang@redhat.com>
> Sent: Friday, October 16, 2020 10:24 AM
> To: Zhang, Chen <chen.zhang@intel.com>; qemu-dev <qemu-
> devel@nongnu.org>
> Cc: Zhang Chen <zhangckid@gmail.com>
> Subject: Re: [PATCH 00/10] COLO project queued patches 20-Oct
> 
> 
> On 2020/10/15 下午3:58, Zhang, Chen wrote:
> >
> >> -----Original Message-----
> >> From: Jason Wang <jasowang@redhat.com>
> >> Sent: Thursday, October 15, 2020 3:56 PM
> >> To: Zhang, Chen <chen.zhang@intel.com>; qemu-dev <qemu-
> >> devel@nongnu.org>
> >> Cc: Zhang Chen <zhangckid@gmail.com>
> >> Subject: Re: [PATCH 00/10] COLO project queued patches 20-Oct
> >>
> >>
> >> On 2020/10/14 下午3:25, Zhang Chen wrote:
> >>> From: Zhang Chen <chen.zhang@intel.com>
> >>>
> >>> Hi Jason, this series include latest COLO related patches.
> >>> please check and merge it.
> >>>
> >>> Thanks
> >>> Zhang Chen
> >>
> >> Hi:
> >>
> >> I want to merge but I don't get patch 10/10.
> >>
> >> Is that lost during posting?
> > Oh, Sorry I missed it.
> > Already resend it:
> > [PATCH 10/10] net/colo-compare.c: Increase default queued packet scan
> > frequency
> >
> > Thanks
> > Chen
> 
> 
> It looks to me Philippe has some minor comments on some patches, please
> address them and send a new version.

Sure. I will update to V2.

Thanks
Chen

> 
> Thanks
> 
> 
> >
> >> Thanks
> >>
> >>
> >>> Li Zhijian (2):
> >>>     colo-compare: fix missing compare_seq initialization
> >>>     colo-compare: check mark in mutual exclusion
> >>>
> >>> Pan Nengyuan (1):
> >>>     net/filter-rewriter: destroy g_hash_table in
> >>> colo_rewriter_cleanup
> >>>
> >>> Rao, Lei (3):
> >>>     Optimize seq_sorter function for colo-compare
> >>>     Reduce the time of checkpoint for COLO
> >>>     Fix the qemu crash when guest shutdown in COLO mode
> >>>
> >>> Zhang Chen (4):
> >>>     net/colo-compare.c: Fix compare_timeout format issue
> >>>     net/colo-compare.c: Change the timer clock type
> >>>     net/colo-compare.c: Add secondary old packet detection
> >>>     net/colo-compare.c: Increase default queued packet scan
> >>> frequency
> >>>
> >>>    migration/ram.c       | 14 ++++++++++-
> >>>    net/colo-compare.c    | 57 ++++++++++++++++++++++-------------------
> --
> >>>    net/colo.c            |  5 +---
> >>>    net/filter-rewriter.c |  2 ++
> >>>    softmmu/vl.c          |  1 +
> >>>    5 files changed, 46 insertions(+), 33 deletions(-)
> >>>