[Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence

[Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence

[Chegu Vinod]

Busy enterprise workloads hosted on large sized VM's tend to dirty
memory faster than the transfer rate achieved via live guest migration.
Despite some good recent improvements (& using dedicated 10Gig NICs
between hosts) the live migration does NOT converge.

If a user chooses to force convergence of their migration via a new
migration capability "auto-converge" then this change will auto-detect
lack of convergence scenario and trigger a slow down of the workload
by explicitly disallowing the VCPUs from spending much time in the VM
context.

The migration thread tries to catchup and this eventually leads
to convergence in some "deterministic" amount of time. Yes it does
impact the performance of all the VCPUs but in my observation that
lasts only for a short duration of time. i.e. end up entering
stage 3 (downtime phase) soon after that. No external trigger is
required.

Thanks to Juan and Paolo for their useful suggestions.

---
Changes from v7:
- added a missing else to patch 3/3.

Changes from v6:
- incorporated feedback from Paolo.
- rebased to latest qemu.git and removing RFC

Changes from v5:
- incorporated feedback from Paolo & Igor.
- rebased to latest qemu.git

Changes from v4:
- incorporated feedback from Paolo.
- split into 3 patches.

Changes from v3:
- incorporated feedback from Paolo and Eric
- rebased to latest qemu.git

Changes from v2:
- incorporated feedback from Orit, Juan and Eric
- stop the throttling thread at the start of stage 3
- rebased to latest qemu.git

Changes from v1:
- rebased to latest qemu.git
- added auto-converge capability(default off) - suggested by Anthony Liguori &
                                                Eric Blake.

Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
---

Chegu Vinod (3):
  Introduce async_run_on_cpu()
  Add 'auto-converge' migration capability
  Force auto-convegence of live migration

 arch_init.c                   |   85 +++++++++++++++++++++++++++++++++++++++++
 cpus.c                        |   29 ++++++++++++++
 include/migration/migration.h |    2 +
 include/qemu-common.h         |    1 +
 include/qom/cpu.h             |   10 +++++
 migration.c                   |    9 ++++
 qapi-schema.json              |    5 ++-
 7 files changed, 140 insertions(+), 1 deletions(-)

[Qemu-devel] [PATCH v8 1/3] Introduce async_run_on_cpu()

[Chegu Vinod]

Introduce an asynchronous version of run_on_cpu() i.e. the caller
doesn't have to block till the call back routine finishes execution
on the target vcpu.

Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
 cpus.c                |   29 +++++++++++++++++++++++++++++
 include/qemu-common.h |    1 +
 include/qom/cpu.h     |   10 ++++++++++
 3 files changed, 40 insertions(+), 0 deletions(-)

diff --git a/cpus.c b/cpus.c
index c8bc8ad..c7c90d0 100644
--- a/cpus.c
+++ b/cpus.c
@@ -653,6 +653,7 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
 
     wi.func = func;
     wi.data = data;
+    wi.free = false;
     if (cpu->queued_work_first == NULL) {
         cpu->queued_work_first = &wi;
     } else {
@@ -671,6 +672,31 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
     }
 }
 
+void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
+{
+    struct qemu_work_item *wi;
+
+    if (qemu_cpu_is_self(cpu)) {
+        func(data);
+        return;
+    }
+
+    wi = g_malloc0(sizeof(struct qemu_work_item));
+    wi->func = func;
+    wi->data = data;
+    wi->free = true;
+    if (cpu->queued_work_first == NULL) {
+        cpu->queued_work_first = wi;
+    } else {
+        cpu->queued_work_last->next = wi;
+    }
+    cpu->queued_work_last = wi;
+    wi->next = NULL;
+    wi->done = false;
+
+    qemu_cpu_kick(cpu);
+}
+
 static void flush_queued_work(CPUState *cpu)
 {
     struct qemu_work_item *wi;
@@ -683,6 +709,9 @@ static void flush_queued_work(CPUState *cpu)
         cpu->queued_work_first = wi->next;
         wi->func(wi->data);
         wi->done = true;
+        if (wi->free) {
+            g_free(wi);
+        }
     }
     cpu->queued_work_last = NULL;
     qemu_cond_broadcast(&qemu_work_cond);
diff --git a/include/qemu-common.h b/include/qemu-common.h
index 3c91375..9834dcb 100644
--- a/include/qemu-common.h
+++ b/include/qemu-common.h
@@ -291,6 +291,7 @@ struct qemu_work_item {
     void (*func)(void *data);
     void *data;
     int done;
+    bool free;
 };
 
 #ifdef CONFIG_USER_ONLY
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index a5bb515..b555c22 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -288,6 +288,16 @@ bool cpu_is_stopped(CPUState *cpu);
 void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
 
 /**
+ * async_run_on_cpu:
+ * @cpu: The vCPU to run on.
+ * @func: The function to be executed.
+ * @data: Data to pass to the function.
+ *
+ * Schedules the function @func for execution on the vCPU @cpu asynchronously.
+ */
+void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
+
+/**
  * qemu_for_each_cpu:
  * @func: The function to be executed.
  * @data: Data to pass to the function.
-- 
1.7.1

[Qemu-devel] [PATCH v8 2/3] Add 'auto-converge' migration capability

[Chegu Vinod]

The auto-converge migration capability allows the user to specify if they
choose live migration seqeunce to automatically detect and force convergence.

Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
 include/migration/migration.h |    2 ++
 migration.c                   |    9 +++++++++
 qapi-schema.json              |    5 ++++-
 3 files changed, 15 insertions(+), 1 deletions(-)

diff --git a/include/migration/migration.h b/include/migration/migration.h
index e2acec6..ace91b0 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -127,4 +127,6 @@ int migrate_use_xbzrle(void);
 int64_t migrate_xbzrle_cache_size(void);
 
 int64_t xbzrle_cache_resize(int64_t new_size);
+
+bool migrate_auto_converge(void);
 #endif
diff --git a/migration.c b/migration.c
index 058f9e6..d0759c1 100644
--- a/migration.c
+++ b/migration.c
@@ -473,6 +473,15 @@ void qmp_migrate_set_downtime(double value, Error **errp)
     max_downtime = (uint64_t)value;
 }
 
+bool migrate_auto_converge(void)
+{
+    MigrationState *s;
+
+    s = migrate_get_current();
+
+    return s->enabled_capabilities[MIGRATION_CAPABILITY_AUTO_CONVERGE];
+}
+
 int migrate_use_xbzrle(void)
 {
     MigrationState *s;
diff --git a/qapi-schema.json b/qapi-schema.json
index a80ee40..c019fec 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -605,10 +605,13 @@
 #          This feature allows us to minimize migration traffic for certain work
 #          loads, by sending compressed difference of the pages
 #
+# @auto-converge: If enabled, QEMU will automatically throttle down the guest
+#          to speed up convergence of RAM migration. (since 1.6)
+#
 # Since: 1.2
 ##
 { 'enum': 'MigrationCapability',
-  'data': ['xbzrle'] }
+  'data': ['xbzrle', 'auto-converge'] }
 
 ##
 # @MigrationCapabilityStatus
-- 
1.7.1

Re: [Qemu-devel] [PATCH v8 1/3] Introduce async_run_on_cpu()

[Juan Quintela]

Chegu Vinod <chegu_vinod@hp.com> wrote:
> Introduce an asynchronous version of run_on_cpu() i.e. the caller
> doesn't have to block till the call back routine finishes execution
> on the target vcpu.
>
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

Re: [Qemu-devel] [PATCH v8 2/3] Add 'auto-converge' migration capability

[Juan Quintela]

Chegu Vinod <chegu_vinod@hp.com> wrote:
> The auto-converge migration capability allows the user to specify if they
> choose live migration seqeunce to automatically detect and force convergence.
>
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> Reviewed-by: Eric Blake <eblake@redhat.com>

Reviewed-by: Juan Quintela <quintela@redhat.com>

Re: [Qemu-devel] [PATCH v8 1/3] Introduce async_run_on_cpu()

[Orit Wasserman]

On 06/24/2013 12:49 PM, Chegu Vinod wrote:
> Introduce an asynchronous version of run_on_cpu() i.e. the caller
> doesn't have to block till the call back routine finishes execution
> on the target vcpu.
> 
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
>  cpus.c                |   29 +++++++++++++++++++++++++++++
>  include/qemu-common.h |    1 +
>  include/qom/cpu.h     |   10 ++++++++++
>  3 files changed, 40 insertions(+), 0 deletions(-)
> 
> diff --git a/cpus.c b/cpus.c
> index c8bc8ad..c7c90d0 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -653,6 +653,7 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
>  
>      wi.func = func;
>      wi.data = data;
> +    wi.free = false;
>      if (cpu->queued_work_first == NULL) {
>          cpu->queued_work_first = &wi;
>      } else {
> @@ -671,6 +672,31 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
>      }
>  }
>  
> +void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
> +{
> +    struct qemu_work_item *wi;
> +
> +    if (qemu_cpu_is_self(cpu)) {
> +        func(data);
> +        return;
> +    }
> +
> +    wi = g_malloc0(sizeof(struct qemu_work_item));
> +    wi->func = func;
> +    wi->data = data;
> +    wi->free = true;
> +    if (cpu->queued_work_first == NULL) {
> +        cpu->queued_work_first = wi;
> +    } else {
> +        cpu->queued_work_last->next = wi;
> +    }
> +    cpu->queued_work_last = wi;
> +    wi->next = NULL;
> +    wi->done = false;
> +
> +    qemu_cpu_kick(cpu);
> +}
> +
>  static void flush_queued_work(CPUState *cpu)
>  {
>      struct qemu_work_item *wi;
> @@ -683,6 +709,9 @@ static void flush_queued_work(CPUState *cpu)
>          cpu->queued_work_first = wi->next;
>          wi->func(wi->data);
>          wi->done = true;
> +        if (wi->free) {
> +            g_free(wi);
> +        }
>      }
>      cpu->queued_work_last = NULL;
>      qemu_cond_broadcast(&qemu_work_cond);
> diff --git a/include/qemu-common.h b/include/qemu-common.h
> index 3c91375..9834dcb 100644
> --- a/include/qemu-common.h
> +++ b/include/qemu-common.h
> @@ -291,6 +291,7 @@ struct qemu_work_item {
>      void (*func)(void *data);
>      void *data;
>      int done;
> +    bool free;
>  };
>  
>  #ifdef CONFIG_USER_ONLY
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index a5bb515..b555c22 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -288,6 +288,16 @@ bool cpu_is_stopped(CPUState *cpu);
>  void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
>  
>  /**
> + * async_run_on_cpu:
> + * @cpu: The vCPU to run on.
> + * @func: The function to be executed.
> + * @data: Data to pass to the function.
> + *
> + * Schedules the function @func for execution on the vCPU @cpu asynchronously.
> + */
> +void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
> +
> +/**
>   * qemu_for_each_cpu:
>   * @func: The function to be executed.
>   * @data: Data to pass to the function.
> 

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Re: [Qemu-devel] [PATCH v8 2/3] Add 'auto-converge' migration capability

[Orit Wasserman]

On 06/24/2013 12:49 PM, Chegu Vinod wrote:
> The auto-converge migration capability allows the user to specify if they
> choose live migration seqeunce to automatically detect and force convergence.
> 
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> Reviewed-by: Eric Blake <eblake@redhat.com>
> ---
>  include/migration/migration.h |    2 ++
>  migration.c                   |    9 +++++++++
>  qapi-schema.json              |    5 ++++-
>  3 files changed, 15 insertions(+), 1 deletions(-)
> 
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index e2acec6..ace91b0 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -127,4 +127,6 @@ int migrate_use_xbzrle(void);
>  int64_t migrate_xbzrle_cache_size(void);
>  
>  int64_t xbzrle_cache_resize(int64_t new_size);
> +
> +bool migrate_auto_converge(void);
>  #endif
> diff --git a/migration.c b/migration.c
> index 058f9e6..d0759c1 100644
> --- a/migration.c
> +++ b/migration.c
> @@ -473,6 +473,15 @@ void qmp_migrate_set_downtime(double value, Error **errp)
>      max_downtime = (uint64_t)value;
>  }
>  
> +bool migrate_auto_converge(void)
> +{
> +    MigrationState *s;
> +
> +    s = migrate_get_current();
> +
> +    return s->enabled_capabilities[MIGRATION_CAPABILITY_AUTO_CONVERGE];
> +}
> +
>  int migrate_use_xbzrle(void)
>  {
>      MigrationState *s;
> diff --git a/qapi-schema.json b/qapi-schema.json
> index a80ee40..c019fec 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -605,10 +605,13 @@
>  #          This feature allows us to minimize migration traffic for certain work
>  #          loads, by sending compressed difference of the pages
>  #
> +# @auto-converge: If enabled, QEMU will automatically throttle down the guest
> +#          to speed up convergence of RAM migration. (since 1.6)
> +#
>  # Since: 1.2
>  ##
>  { 'enum': 'MigrationCapability',
> -  'data': ['xbzrle'] }
> +  'data': ['xbzrle', 'auto-converge'] }
>  
>  ##
>  # @MigrationCapabilityStatus
> 

Reviewed-by: Orit Wasserman <owasserm@redhat.com>

Re: [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence

[Peter Lieven]

Hi all,

is it possible that not v8 of this patch got merged?
Without checking line-by-line I see at least that this here

+# @auto-converge: If enabled, QEMU will automatically throttle down the guest
+#          to speed up convergence of RAM migration. (since 1.6)
+#

is missing in qapi-schema.json.

BR,
Peter



On 24.06.2013 11:49, Chegu Vinod wrote:
> Busy enterprise workloads hosted on large sized VM's tend to dirty
> memory faster than the transfer rate achieved via live guest migration.
> Despite some good recent improvements (& using dedicated 10Gig NICs
> between hosts) the live migration does NOT converge.
>
> If a user chooses to force convergence of their migration via a new
> migration capability "auto-converge" then this change will auto-detect
> lack of convergence scenario and trigger a slow down of the workload
> by explicitly disallowing the VCPUs from spending much time in the VM
> context.
>
> The migration thread tries to catchup and this eventually leads
> to convergence in some "deterministic" amount of time. Yes it does
> impact the performance of all the VCPUs but in my observation that
> lasts only for a short duration of time. i.e. end up entering
> stage 3 (downtime phase) soon after that. No external trigger is
> required.
>
> Thanks to Juan and Paolo for their useful suggestions.
>
> ---
> Changes from v7:
> - added a missing else to patch 3/3.
>
> Changes from v6:
> - incorporated feedback from Paolo.
> - rebased to latest qemu.git and removing RFC
>
> Changes from v5:
> - incorporated feedback from Paolo & Igor.
> - rebased to latest qemu.git
>
> Changes from v4:
> - incorporated feedback from Paolo.
> - split into 3 patches.
>
> Changes from v3:
> - incorporated feedback from Paolo and Eric
> - rebased to latest qemu.git
>
> Changes from v2:
> - incorporated feedback from Orit, Juan and Eric
> - stop the throttling thread at the start of stage 3
> - rebased to latest qemu.git
>
> Changes from v1:
> - rebased to latest qemu.git
> - added auto-converge capability(default off) - suggested by Anthony Liguori &
>                                                  Eric Blake.
>
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> ---
>
> Chegu Vinod (3):
>    Introduce async_run_on_cpu()
>    Add 'auto-converge' migration capability
>    Force auto-convegence of live migration
>
>   arch_init.c                   |   85 +++++++++++++++++++++++++++++++++++++++++
>   cpus.c                        |   29 ++++++++++++++
>   include/migration/migration.h |    2 +
>   include/qemu-common.h         |    1 +
>   include/qom/cpu.h             |   10 +++++
>   migration.c                   |    9 ++++
>   qapi-schema.json              |    5 ++-
>   7 files changed, 140 insertions(+), 1 deletions(-)
>
>

Re: [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence

[Vinod, Chegu]

Hi Peter,

Thanks for catching this.
Tthis was perhaps accidentally left out during merge to Juan's migration.next tree.  
I have informed Juan and he said he would take care of it.

Vinod

-----Original Message-----
From: Peter Lieven [mailto:lieven-lists@dlhnet.de] 
Sent: Wednesday, July 17, 2013 11:25 PM
To: Vinod, Chegu
Cc: eblake@redhat.com; anthony@codemonkey.ws; quintela@redhat.com; owasserm@redhat.com; qemu-devel@nongnu.org; pbonzini@redhat.com
Subject: Re: [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence

Hi all,

is it possible that not v8 of this patch got merged?
Without checking line-by-line I see at least that this here

+# @auto-converge: If enabled, QEMU will automatically throttle down the guest
+#          to speed up convergence of RAM migration. (since 1.6)
+#

is missing in qapi-schema.json.

BR,
Peter



On 24.06.2013 11:49, Chegu Vinod wrote:
> Busy enterprise workloads hosted on large sized VM's tend to dirty 
> memory faster than the transfer rate achieved via live guest migration.
> Despite some good recent improvements (& using dedicated 10Gig NICs 
> between hosts) the live migration does NOT converge.
>
> If a user chooses to force convergence of their migration via a new 
> migration capability "auto-converge" then this change will auto-detect 
> lack of convergence scenario and trigger a slow down of the workload 
> by explicitly disallowing the VCPUs from spending much time in the VM 
> context.
>
> The migration thread tries to catchup and this eventually leads to 
> convergence in some "deterministic" amount of time. Yes it does impact 
> the performance of all the VCPUs but in my observation that lasts only 
> for a short duration of time. i.e. end up entering stage 3 (downtime 
> phase) soon after that. No external trigger is required.
>
> Thanks to Juan and Paolo for their useful suggestions.
>
> ---
> Changes from v7:
> - added a missing else to patch 3/3.
>
> Changes from v6:
> - incorporated feedback from Paolo.
> - rebased to latest qemu.git and removing RFC
>
> Changes from v5:
> - incorporated feedback from Paolo & Igor.
> - rebased to latest qemu.git
>
> Changes from v4:
> - incorporated feedback from Paolo.
> - split into 3 patches.
>
> Changes from v3:
> - incorporated feedback from Paolo and Eric
> - rebased to latest qemu.git
>
> Changes from v2:
> - incorporated feedback from Orit, Juan and Eric
> - stop the throttling thread at the start of stage 3
> - rebased to latest qemu.git
>
> Changes from v1:
> - rebased to latest qemu.git
> - added auto-converge capability(default off) - suggested by Anthony Liguori &
>                                                  Eric Blake.
>
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> ---
>
> Chegu Vinod (3):
>    Introduce async_run_on_cpu()
>    Add 'auto-converge' migration capability
>    Force auto-convegence of live migration
>
>   arch_init.c                   |   85 +++++++++++++++++++++++++++++++++++++++++
>   cpus.c                        |   29 ++++++++++++++
>   include/migration/migration.h |    2 +
>   include/qemu-common.h         |    1 +
>   include/qom/cpu.h             |   10 +++++
>   migration.c                   |    9 ++++
>   qapi-schema.json              |    5 ++-
>   7 files changed, 140 insertions(+), 1 deletions(-)
>
>