* [Qemu-devel] [PATCH v8 1/3] Introduce async_run_on_cpu()
2013-06-24 9:49 [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence Chegu Vinod
@ 2013-06-24 9:49 ` Chegu Vinod
2013-07-12 11:54 ` Juan Quintela
2013-07-13 9:15 ` Orit Wasserman
2013-06-24 9:49 ` [Qemu-devel] [PATCH v8 2/3] Add 'auto-converge' migration capability Chegu Vinod
2013-07-18 6:24 ` [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence Peter Lieven
2 siblings, 2 replies; 9+ messages in thread
From: Chegu Vinod @ 2013-06-24 9:49 UTC (permalink / raw)
To: eblake, anthony, quintela, owasserm, qemu-devel, pbonzini; +Cc: chegu_vinod
Introduce an asynchronous version of run_on_cpu() i.e. the caller
doesn't have to block till the call back routine finishes execution
on the target vcpu.
Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
---
cpus.c | 29 +++++++++++++++++++++++++++++
include/qemu-common.h | 1 +
include/qom/cpu.h | 10 ++++++++++
3 files changed, 40 insertions(+), 0 deletions(-)
diff --git a/cpus.c b/cpus.c
index c8bc8ad..c7c90d0 100644
--- a/cpus.c
+++ b/cpus.c
@@ -653,6 +653,7 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
wi.func = func;
wi.data = data;
+ wi.free = false;
if (cpu->queued_work_first == NULL) {
cpu->queued_work_first = &wi;
} else {
@@ -671,6 +672,31 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
}
}
+void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
+{
+ struct qemu_work_item *wi;
+
+ if (qemu_cpu_is_self(cpu)) {
+ func(data);
+ return;
+ }
+
+ wi = g_malloc0(sizeof(struct qemu_work_item));
+ wi->func = func;
+ wi->data = data;
+ wi->free = true;
+ if (cpu->queued_work_first == NULL) {
+ cpu->queued_work_first = wi;
+ } else {
+ cpu->queued_work_last->next = wi;
+ }
+ cpu->queued_work_last = wi;
+ wi->next = NULL;
+ wi->done = false;
+
+ qemu_cpu_kick(cpu);
+}
+
static void flush_queued_work(CPUState *cpu)
{
struct qemu_work_item *wi;
@@ -683,6 +709,9 @@ static void flush_queued_work(CPUState *cpu)
cpu->queued_work_first = wi->next;
wi->func(wi->data);
wi->done = true;
+ if (wi->free) {
+ g_free(wi);
+ }
}
cpu->queued_work_last = NULL;
qemu_cond_broadcast(&qemu_work_cond);
diff --git a/include/qemu-common.h b/include/qemu-common.h
index 3c91375..9834dcb 100644
--- a/include/qemu-common.h
+++ b/include/qemu-common.h
@@ -291,6 +291,7 @@ struct qemu_work_item {
void (*func)(void *data);
void *data;
int done;
+ bool free;
};
#ifdef CONFIG_USER_ONLY
diff --git a/include/qom/cpu.h b/include/qom/cpu.h
index a5bb515..b555c22 100644
--- a/include/qom/cpu.h
+++ b/include/qom/cpu.h
@@ -288,6 +288,16 @@ bool cpu_is_stopped(CPUState *cpu);
void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
/**
+ * async_run_on_cpu:
+ * @cpu: The vCPU to run on.
+ * @func: The function to be executed.
+ * @data: Data to pass to the function.
+ *
+ * Schedules the function @func for execution on the vCPU @cpu asynchronously.
+ */
+void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
+
+/**
* qemu_for_each_cpu:
* @func: The function to be executed.
* @data: Data to pass to the function.
--
1.7.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [PATCH v8 1/3] Introduce async_run_on_cpu()
2013-06-24 9:49 ` [Qemu-devel] [PATCH v8 1/3] Introduce async_run_on_cpu() Chegu Vinod
@ 2013-07-12 11:54 ` Juan Quintela
2013-07-13 9:15 ` Orit Wasserman
1 sibling, 0 replies; 9+ messages in thread
From: Juan Quintela @ 2013-07-12 11:54 UTC (permalink / raw)
To: Chegu Vinod; +Cc: owasserm, qemu-devel, anthony, pbonzini
Chegu Vinod <chegu_vinod@hp.com> wrote:
> Introduce an asynchronous version of run_on_cpu() i.e. the caller
> doesn't have to block till the call back routine finishes execution
> on the target vcpu.
>
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [PATCH v8 1/3] Introduce async_run_on_cpu()
2013-06-24 9:49 ` [Qemu-devel] [PATCH v8 1/3] Introduce async_run_on_cpu() Chegu Vinod
2013-07-12 11:54 ` Juan Quintela
@ 2013-07-13 9:15 ` Orit Wasserman
1 sibling, 0 replies; 9+ messages in thread
From: Orit Wasserman @ 2013-07-13 9:15 UTC (permalink / raw)
To: Chegu Vinod; +Cc: pbonzini, qemu-devel, anthony, quintela
On 06/24/2013 12:49 PM, Chegu Vinod wrote:
> Introduce an asynchronous version of run_on_cpu() i.e. the caller
> doesn't have to block till the call back routine finishes execution
> on the target vcpu.
>
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> ---
> cpus.c | 29 +++++++++++++++++++++++++++++
> include/qemu-common.h | 1 +
> include/qom/cpu.h | 10 ++++++++++
> 3 files changed, 40 insertions(+), 0 deletions(-)
>
> diff --git a/cpus.c b/cpus.c
> index c8bc8ad..c7c90d0 100644
> --- a/cpus.c
> +++ b/cpus.c
> @@ -653,6 +653,7 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
>
> wi.func = func;
> wi.data = data;
> + wi.free = false;
> if (cpu->queued_work_first == NULL) {
> cpu->queued_work_first = &wi;
> } else {
> @@ -671,6 +672,31 @@ void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
> }
> }
>
> +void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data)
> +{
> + struct qemu_work_item *wi;
> +
> + if (qemu_cpu_is_self(cpu)) {
> + func(data);
> + return;
> + }
> +
> + wi = g_malloc0(sizeof(struct qemu_work_item));
> + wi->func = func;
> + wi->data = data;
> + wi->free = true;
> + if (cpu->queued_work_first == NULL) {
> + cpu->queued_work_first = wi;
> + } else {
> + cpu->queued_work_last->next = wi;
> + }
> + cpu->queued_work_last = wi;
> + wi->next = NULL;
> + wi->done = false;
> +
> + qemu_cpu_kick(cpu);
> +}
> +
> static void flush_queued_work(CPUState *cpu)
> {
> struct qemu_work_item *wi;
> @@ -683,6 +709,9 @@ static void flush_queued_work(CPUState *cpu)
> cpu->queued_work_first = wi->next;
> wi->func(wi->data);
> wi->done = true;
> + if (wi->free) {
> + g_free(wi);
> + }
> }
> cpu->queued_work_last = NULL;
> qemu_cond_broadcast(&qemu_work_cond);
> diff --git a/include/qemu-common.h b/include/qemu-common.h
> index 3c91375..9834dcb 100644
> --- a/include/qemu-common.h
> +++ b/include/qemu-common.h
> @@ -291,6 +291,7 @@ struct qemu_work_item {
> void (*func)(void *data);
> void *data;
> int done;
> + bool free;
> };
>
> #ifdef CONFIG_USER_ONLY
> diff --git a/include/qom/cpu.h b/include/qom/cpu.h
> index a5bb515..b555c22 100644
> --- a/include/qom/cpu.h
> +++ b/include/qom/cpu.h
> @@ -288,6 +288,16 @@ bool cpu_is_stopped(CPUState *cpu);
> void run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
>
> /**
> + * async_run_on_cpu:
> + * @cpu: The vCPU to run on.
> + * @func: The function to be executed.
> + * @data: Data to pass to the function.
> + *
> + * Schedules the function @func for execution on the vCPU @cpu asynchronously.
> + */
> +void async_run_on_cpu(CPUState *cpu, void (*func)(void *data), void *data);
> +
> +/**
> * qemu_for_each_cpu:
> * @func: The function to be executed.
> * @data: Data to pass to the function.
>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* [Qemu-devel] [PATCH v8 2/3] Add 'auto-converge' migration capability
2013-06-24 9:49 [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence Chegu Vinod
2013-06-24 9:49 ` [Qemu-devel] [PATCH v8 1/3] Introduce async_run_on_cpu() Chegu Vinod
@ 2013-06-24 9:49 ` Chegu Vinod
2013-07-12 11:56 ` Juan Quintela
2013-07-13 9:17 ` Orit Wasserman
2013-07-18 6:24 ` [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence Peter Lieven
2 siblings, 2 replies; 9+ messages in thread
From: Chegu Vinod @ 2013-06-24 9:49 UTC (permalink / raw)
To: eblake, anthony, quintela, owasserm, qemu-devel, pbonzini; +Cc: chegu_vinod
The auto-converge migration capability allows the user to specify if they
choose live migration seqeunce to automatically detect and force convergence.
Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
include/migration/migration.h | 2 ++
migration.c | 9 +++++++++
qapi-schema.json | 5 ++++-
3 files changed, 15 insertions(+), 1 deletions(-)
diff --git a/include/migration/migration.h b/include/migration/migration.h
index e2acec6..ace91b0 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -127,4 +127,6 @@ int migrate_use_xbzrle(void);
int64_t migrate_xbzrle_cache_size(void);
int64_t xbzrle_cache_resize(int64_t new_size);
+
+bool migrate_auto_converge(void);
#endif
diff --git a/migration.c b/migration.c
index 058f9e6..d0759c1 100644
--- a/migration.c
+++ b/migration.c
@@ -473,6 +473,15 @@ void qmp_migrate_set_downtime(double value, Error **errp)
max_downtime = (uint64_t)value;
}
+bool migrate_auto_converge(void)
+{
+ MigrationState *s;
+
+ s = migrate_get_current();
+
+ return s->enabled_capabilities[MIGRATION_CAPABILITY_AUTO_CONVERGE];
+}
+
int migrate_use_xbzrle(void)
{
MigrationState *s;
diff --git a/qapi-schema.json b/qapi-schema.json
index a80ee40..c019fec 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -605,10 +605,13 @@
# This feature allows us to minimize migration traffic for certain work
# loads, by sending compressed difference of the pages
#
+# @auto-converge: If enabled, QEMU will automatically throttle down the guest
+# to speed up convergence of RAM migration. (since 1.6)
+#
# Since: 1.2
##
{ 'enum': 'MigrationCapability',
- 'data': ['xbzrle'] }
+ 'data': ['xbzrle', 'auto-converge'] }
##
# @MigrationCapabilityStatus
--
1.7.1
^ permalink raw reply related [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [PATCH v8 2/3] Add 'auto-converge' migration capability
2013-06-24 9:49 ` [Qemu-devel] [PATCH v8 2/3] Add 'auto-converge' migration capability Chegu Vinod
@ 2013-07-12 11:56 ` Juan Quintela
2013-07-13 9:17 ` Orit Wasserman
1 sibling, 0 replies; 9+ messages in thread
From: Juan Quintela @ 2013-07-12 11:56 UTC (permalink / raw)
To: Chegu Vinod; +Cc: owasserm, qemu-devel, anthony, pbonzini
Chegu Vinod <chegu_vinod@hp.com> wrote:
> The auto-converge migration capability allows the user to specify if they
> choose live migration seqeunce to automatically detect and force convergence.
>
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> Reviewed-by: Eric Blake <eblake@redhat.com>
Reviewed-by: Juan Quintela <quintela@redhat.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [PATCH v8 2/3] Add 'auto-converge' migration capability
2013-06-24 9:49 ` [Qemu-devel] [PATCH v8 2/3] Add 'auto-converge' migration capability Chegu Vinod
2013-07-12 11:56 ` Juan Quintela
@ 2013-07-13 9:17 ` Orit Wasserman
1 sibling, 0 replies; 9+ messages in thread
From: Orit Wasserman @ 2013-07-13 9:17 UTC (permalink / raw)
To: Chegu Vinod; +Cc: pbonzini, qemu-devel, anthony, quintela
On 06/24/2013 12:49 PM, Chegu Vinod wrote:
> The auto-converge migration capability allows the user to specify if they
> choose live migration seqeunce to automatically detect and force convergence.
>
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> Reviewed-by: Paolo Bonzini <pbonzini@redhat.com>
> Reviewed-by: Eric Blake <eblake@redhat.com>
> ---
> include/migration/migration.h | 2 ++
> migration.c | 9 +++++++++
> qapi-schema.json | 5 ++++-
> 3 files changed, 15 insertions(+), 1 deletions(-)
>
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index e2acec6..ace91b0 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -127,4 +127,6 @@ int migrate_use_xbzrle(void);
> int64_t migrate_xbzrle_cache_size(void);
>
> int64_t xbzrle_cache_resize(int64_t new_size);
> +
> +bool migrate_auto_converge(void);
> #endif
> diff --git a/migration.c b/migration.c
> index 058f9e6..d0759c1 100644
> --- a/migration.c
> +++ b/migration.c
> @@ -473,6 +473,15 @@ void qmp_migrate_set_downtime(double value, Error **errp)
> max_downtime = (uint64_t)value;
> }
>
> +bool migrate_auto_converge(void)
> +{
> + MigrationState *s;
> +
> + s = migrate_get_current();
> +
> + return s->enabled_capabilities[MIGRATION_CAPABILITY_AUTO_CONVERGE];
> +}
> +
> int migrate_use_xbzrle(void)
> {
> MigrationState *s;
> diff --git a/qapi-schema.json b/qapi-schema.json
> index a80ee40..c019fec 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -605,10 +605,13 @@
> # This feature allows us to minimize migration traffic for certain work
> # loads, by sending compressed difference of the pages
> #
> +# @auto-converge: If enabled, QEMU will automatically throttle down the guest
> +# to speed up convergence of RAM migration. (since 1.6)
> +#
> # Since: 1.2
> ##
> { 'enum': 'MigrationCapability',
> - 'data': ['xbzrle'] }
> + 'data': ['xbzrle', 'auto-converge'] }
>
> ##
> # @MigrationCapabilityStatus
>
Reviewed-by: Orit Wasserman <owasserm@redhat.com>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence
2013-06-24 9:49 [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence Chegu Vinod
2013-06-24 9:49 ` [Qemu-devel] [PATCH v8 1/3] Introduce async_run_on_cpu() Chegu Vinod
2013-06-24 9:49 ` [Qemu-devel] [PATCH v8 2/3] Add 'auto-converge' migration capability Chegu Vinod
@ 2013-07-18 6:24 ` Peter Lieven
2013-07-18 17:42 ` Vinod, Chegu
2 siblings, 1 reply; 9+ messages in thread
From: Peter Lieven @ 2013-07-18 6:24 UTC (permalink / raw)
To: Chegu Vinod; +Cc: quintela, qemu-devel, owasserm, anthony, pbonzini
Hi all,
is it possible that not v8 of this patch got merged?
Without checking line-by-line I see at least that this here
+# @auto-converge: If enabled, QEMU will automatically throttle down the guest
+# to speed up convergence of RAM migration. (since 1.6)
+#
is missing in qapi-schema.json.
BR,
Peter
On 24.06.2013 11:49, Chegu Vinod wrote:
> Busy enterprise workloads hosted on large sized VM's tend to dirty
> memory faster than the transfer rate achieved via live guest migration.
> Despite some good recent improvements (& using dedicated 10Gig NICs
> between hosts) the live migration does NOT converge.
>
> If a user chooses to force convergence of their migration via a new
> migration capability "auto-converge" then this change will auto-detect
> lack of convergence scenario and trigger a slow down of the workload
> by explicitly disallowing the VCPUs from spending much time in the VM
> context.
>
> The migration thread tries to catchup and this eventually leads
> to convergence in some "deterministic" amount of time. Yes it does
> impact the performance of all the VCPUs but in my observation that
> lasts only for a short duration of time. i.e. end up entering
> stage 3 (downtime phase) soon after that. No external trigger is
> required.
>
> Thanks to Juan and Paolo for their useful suggestions.
>
> ---
> Changes from v7:
> - added a missing else to patch 3/3.
>
> Changes from v6:
> - incorporated feedback from Paolo.
> - rebased to latest qemu.git and removing RFC
>
> Changes from v5:
> - incorporated feedback from Paolo & Igor.
> - rebased to latest qemu.git
>
> Changes from v4:
> - incorporated feedback from Paolo.
> - split into 3 patches.
>
> Changes from v3:
> - incorporated feedback from Paolo and Eric
> - rebased to latest qemu.git
>
> Changes from v2:
> - incorporated feedback from Orit, Juan and Eric
> - stop the throttling thread at the start of stage 3
> - rebased to latest qemu.git
>
> Changes from v1:
> - rebased to latest qemu.git
> - added auto-converge capability(default off) - suggested by Anthony Liguori &
> Eric Blake.
>
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> ---
>
> Chegu Vinod (3):
> Introduce async_run_on_cpu()
> Add 'auto-converge' migration capability
> Force auto-convegence of live migration
>
> arch_init.c | 85 +++++++++++++++++++++++++++++++++++++++++
> cpus.c | 29 ++++++++++++++
> include/migration/migration.h | 2 +
> include/qemu-common.h | 1 +
> include/qom/cpu.h | 10 +++++
> migration.c | 9 ++++
> qapi-schema.json | 5 ++-
> 7 files changed, 140 insertions(+), 1 deletions(-)
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence
2013-07-18 6:24 ` [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence Peter Lieven
@ 2013-07-18 17:42 ` Vinod, Chegu
0 siblings, 0 replies; 9+ messages in thread
From: Vinod, Chegu @ 2013-07-18 17:42 UTC (permalink / raw)
To: Peter Lieven; +Cc: quintela, qemu-devel, owasserm, anthony, pbonzini
Hi Peter,
Thanks for catching this.
Tthis was perhaps accidentally left out during merge to Juan's migration.next tree.
I have informed Juan and he said he would take care of it.
Vinod
-----Original Message-----
From: Peter Lieven [mailto:lieven-lists@dlhnet.de]
Sent: Wednesday, July 17, 2013 11:25 PM
To: Vinod, Chegu
Cc: eblake@redhat.com; anthony@codemonkey.ws; quintela@redhat.com; owasserm@redhat.com; qemu-devel@nongnu.org; pbonzini@redhat.com
Subject: Re: [Qemu-devel] [PATCH v8 0/3] Throttle-down guest to help with live migration convergence
Hi all,
is it possible that not v8 of this patch got merged?
Without checking line-by-line I see at least that this here
+# @auto-converge: If enabled, QEMU will automatically throttle down the guest
+# to speed up convergence of RAM migration. (since 1.6)
+#
is missing in qapi-schema.json.
BR,
Peter
On 24.06.2013 11:49, Chegu Vinod wrote:
> Busy enterprise workloads hosted on large sized VM's tend to dirty
> memory faster than the transfer rate achieved via live guest migration.
> Despite some good recent improvements (& using dedicated 10Gig NICs
> between hosts) the live migration does NOT converge.
>
> If a user chooses to force convergence of their migration via a new
> migration capability "auto-converge" then this change will auto-detect
> lack of convergence scenario and trigger a slow down of the workload
> by explicitly disallowing the VCPUs from spending much time in the VM
> context.
>
> The migration thread tries to catchup and this eventually leads to
> convergence in some "deterministic" amount of time. Yes it does impact
> the performance of all the VCPUs but in my observation that lasts only
> for a short duration of time. i.e. end up entering stage 3 (downtime
> phase) soon after that. No external trigger is required.
>
> Thanks to Juan and Paolo for their useful suggestions.
>
> ---
> Changes from v7:
> - added a missing else to patch 3/3.
>
> Changes from v6:
> - incorporated feedback from Paolo.
> - rebased to latest qemu.git and removing RFC
>
> Changes from v5:
> - incorporated feedback from Paolo & Igor.
> - rebased to latest qemu.git
>
> Changes from v4:
> - incorporated feedback from Paolo.
> - split into 3 patches.
>
> Changes from v3:
> - incorporated feedback from Paolo and Eric
> - rebased to latest qemu.git
>
> Changes from v2:
> - incorporated feedback from Orit, Juan and Eric
> - stop the throttling thread at the start of stage 3
> - rebased to latest qemu.git
>
> Changes from v1:
> - rebased to latest qemu.git
> - added auto-converge capability(default off) - suggested by Anthony Liguori &
> Eric Blake.
>
> Signed-off-by: Chegu Vinod <chegu_vinod@hp.com>
> ---
>
> Chegu Vinod (3):
> Introduce async_run_on_cpu()
> Add 'auto-converge' migration capability
> Force auto-convegence of live migration
>
> arch_init.c | 85 +++++++++++++++++++++++++++++++++++++++++
> cpus.c | 29 ++++++++++++++
> include/migration/migration.h | 2 +
> include/qemu-common.h | 1 +
> include/qom/cpu.h | 10 +++++
> migration.c | 9 ++++
> qapi-schema.json | 5 ++-
> 7 files changed, 140 insertions(+), 1 deletions(-)
>
>
^ permalink raw reply [flat|nested] 9+ messages in thread