* [PATCH v11 01/27] tools/libxl: introduction of libxl__qmp_restore to load qemu state
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 16:30 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 02/27] tools/libxl: introduce libxl__domain_common_switch_qemu_logdirty() Changlong Xie
` (26 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
In normal migration, the qemu state is passed to qemu as a parameter.
With COLO, secondary vm is running. So we will do the following steps
at every checkpoint:
1. suspend both primary vm and secondary vm
2. sync the state
3. resume both primary vm and secondary vm
Primary will send qemu's state in step2, and secondary's qemu should
read it and restore the state before it is resumed. We can not pass the
state to qemu as a parameter because secondary QEMU is already started
at this point, so we introduce libxl__qmp_restore() to do it.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
Cc: Anthony Perard <anthony.perard@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
tools/libxl/libxl_internal.h | 2 ++
tools/libxl/libxl_qmp.c | 10 ++++++++++
2 files changed, 12 insertions(+)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 005fe53..7801385 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1762,6 +1762,8 @@ _hidden int libxl__qmp_stop(libxl__gc *gc, int domid);
_hidden int libxl__qmp_resume(libxl__gc *gc, int domid);
/* Save current QEMU state into fd. */
_hidden int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename);
+/* Load current QEMU state from file. */
+_hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char *filename);
/* Set dirty bitmap logging status */
_hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool enable);
_hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const libxl_device_disk *disk);
diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
index c45702e..c0bdfcb 100644
--- a/tools/libxl/libxl_qmp.c
+++ b/tools/libxl/libxl_qmp.c
@@ -906,6 +906,16 @@ int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename)
NULL, NULL);
}
+int libxl__qmp_restore(libxl__gc *gc, int domid, const char *state_file)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_string(gc, &args, "filename", state_file);
+
+ return qmp_run_command(gc, domid, "xen-load-devices-state", args,
+ NULL, NULL);
+}
+
static int qmp_change(libxl__gc *gc, libxl__qmp_handler *qmp,
char *device, char *target, char *arg)
{
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 01/27] tools/libxl: introduction of libxl__qmp_restore to load qemu state
2016-03-04 8:41 ` [PATCH v11 01/27] tools/libxl: introduction of libxl__qmp_restore to load qemu state Changlong Xie
@ 2016-03-04 16:30 ` Ian Jackson
2016-03-14 9:03 ` Changlong Xie
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 16:30 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 01/27] tools/libxl: introduction of libxl__qmp_restore to load qemu state"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> In normal migration, the qemu state is passed to qemu as a parameter.
> With COLO, secondary vm is running. So we will do the following steps
> at every checkpoint:
> 1. suspend both primary vm and secondary vm
> 2. sync the state
> 3. resume both primary vm and secondary vm
> Primary will send qemu's state in step2, and secondary's qemu should
> read it and restore the state before it is resumed. We can not pass the
> state to qemu as a parameter because secondary QEMU is already started
> at this point, so we introduce libxl__qmp_restore() to do it.
We should accept both the Xen part and the Qemu part, or neither. So
I think this proposal needs (in the Xen commit message) a reference to
the Qemu upstream commit, or Qemu upstream patches, so that we can see
that they've got the appropriate Qemu upstream acks.
But, subject to that, from a Xen point of view this approach looks
good to me.
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 01/27] tools/libxl: introduction of libxl__qmp_restore to load qemu state
2016-03-04 16:30 ` Ian Jackson
@ 2016-03-14 9:03 ` Changlong Xie
0 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-14 9:03 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 12:30 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 01/27] tools/libxl: introduction of libxl__qmp_restore to load qemu state"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> In normal migration, the qemu state is passed to qemu as a parameter.
>> With COLO, secondary vm is running. So we will do the following steps
>> at every checkpoint:
>> 1. suspend both primary vm and secondary vm
>> 2. sync the state
>> 3. resume both primary vm and secondary vm
>> Primary will send qemu's state in step2, and secondary's qemu should
>> read it and restore the state before it is resumed. We can not pass the
>> state to qemu as a parameter because secondary QEMU is already started
>> at this point, so we introduce libxl__qmp_restore() to do it.
>
> We should accept both the Xen part and the Qemu part, or neither. So
> I think this proposal needs (in the Xen commit message) a reference to
> the Qemu upstream commit, or Qemu upstream patches, so that we can see
> that they've got the appropriate Qemu upstream acks.
>
> But, subject to that, from a Xen point of view this approach looks
> good to me.
>
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
>
Hi Ian
We now push this qemu relevant patch in
http://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg02955.html
Will update the commit messege when it's merged.
Thanks
-Xie
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 02/27] tools/libxl: introduce libxl__domain_common_switch_qemu_logdirty()
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
2016-03-04 8:41 ` [PATCH v11 01/27] tools/libxl: introduction of libxl__qmp_restore to load qemu state Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 8:41 ` [PATCH v11 03/27] tools/libxl: Add back channel to allow migration target send data back Changlong Xie
` (25 subsequent siblings)
27 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Secondary vm is running in COLO mode, we need to send secondary
vm's dirty page information to primary host at checkpoint, so we
have to enable qemu logdirty on secondary.
libxl__domain_suspend_common_switch_qemu_logdirty() is to enable
qemu logdirty. But it uses libxl__domain_save_state, and calls
libxl__xc_domain_saverestore_async_callback_done() before exits.
This can not be used for secondary vm.
Update libxl__domain_suspend_common_switch_qemu_logdirty() to
introduce a new API libxl__domain_common_switch_qemu_logdirty().
This API only uses libxl__logdirty_switch, and calls
lds->callback before exits. This new API will be used by the patch:
secondary vm suspend/resume/checkpoint code
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl_dom_save.c | 95 ++++++++++++++++++++++++--------------------
tools/libxl/libxl_internal.h | 8 ++++
2 files changed, 60 insertions(+), 43 deletions(-)
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index f3288b9..4df86c3 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -42,7 +42,7 @@ static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev,
static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch*,
const char *watch_path, const char *event_path);
static void switch_logdirty_done(libxl__egc *egc,
- libxl__domain_save_state *dss, int rc);
+ libxl__logdirty_switch *lds, int rc);
void libxl__logdirty_init(libxl__logdirty_switch *lds)
{
@@ -52,13 +52,10 @@ void libxl__logdirty_init(libxl__logdirty_switch *lds)
}
static void domain_suspend_switch_qemu_xen_traditional_logdirty
- (int domid, unsigned enable,
- libxl__save_helper_state *shs)
+ (libxl__egc *egc, int domid, unsigned enable,
+ libxl__logdirty_switch *lds)
{
- libxl__egc *egc = shs->egc;
- libxl__domain_save_state *dss = shs->caller_state;
- libxl__logdirty_switch *lds = &dss->logdirty;
- STATE_AO_GC(dss->ao);
+ STATE_AO_GC(lds->ao);
int rc;
xs_transaction_t t = 0;
const char *got;
@@ -120,26 +117,34 @@ static void domain_suspend_switch_qemu_xen_traditional_logdirty
out:
LOG(ERROR,"logdirty switch failed (rc=%d), abandoning suspend",rc);
libxl__xs_transaction_abort(gc, &t);
- switch_logdirty_done(egc,dss,rc);
+ switch_logdirty_done(egc,lds,rc);
}
static void domain_suspend_switch_qemu_xen_logdirty
- (int domid, unsigned enable,
- libxl__save_helper_state *shs)
+ (libxl__egc *egc, int domid, unsigned enable,
+ libxl__logdirty_switch *lds)
{
- libxl__egc *egc = shs->egc;
- libxl__domain_save_state *dss = shs->caller_state;
- STATE_AO_GC(dss->ao);
+ STATE_AO_GC(lds->ao);
int rc;
rc = libxl__qmp_set_global_dirty_log(gc, domid, enable);
- if (!rc) {
- libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
- } else {
+ if (rc)
LOG(ERROR,"logdirty switch failed (rc=%d), abandoning suspend",rc);
+
+ lds->callback(egc, lds, rc);
+}
+
+static void domain_suspend_switch_qemu_logdirty_done
+ (libxl__egc *egc, libxl__logdirty_switch *lds, int rc)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(lds, *dss, logdirty);
+
+ if (rc) {
dss->rc = rc;
- libxl__xc_domain_saverestore_async_callback_done(egc, shs, -1);
- }
+ libxl__xc_domain_saverestore_async_callback_done(egc,
+ &dss->sws.shs, -1);
+ } else
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
}
void libxl__domain_suspend_common_switch_qemu_logdirty
@@ -148,42 +153,52 @@ void libxl__domain_suspend_common_switch_qemu_logdirty
libxl__save_helper_state *shs = user;
libxl__egc *egc = shs->egc;
libxl__domain_save_state *dss = shs->caller_state;
- STATE_AO_GC(dss->ao);
+
+ /* Convenience aliases. */
+ libxl__logdirty_switch *const lds = &dss->logdirty;
+
+ lds->callback = domain_suspend_switch_qemu_logdirty_done;
+ libxl__domain_common_switch_qemu_logdirty(egc, domid, enable, lds);
+}
+
+void libxl__domain_common_switch_qemu_logdirty(libxl__egc *egc,
+ int domid, unsigned enable,
+ libxl__logdirty_switch *lds)
+{
+ STATE_AO_GC(lds->ao);
switch (libxl__device_model_version_running(gc, domid)) {
case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
- domain_suspend_switch_qemu_xen_traditional_logdirty(domid, enable, shs);
+ domain_suspend_switch_qemu_xen_traditional_logdirty(egc, domid, enable,
+ lds);
break;
case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
- domain_suspend_switch_qemu_xen_logdirty(domid, enable, shs);
+ domain_suspend_switch_qemu_xen_logdirty(egc, domid, enable, lds);
break;
case LIBXL_DEVICE_MODEL_VERSION_NONE:
- libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+ lds->callback(egc, lds, 0);
break;
default:
LOG(ERROR,"logdirty switch failed"
", no valid device model version found, abandoning suspend");
- dss->rc = ERROR_FAIL;
- libxl__xc_domain_saverestore_async_callback_done(egc, shs, -1);
+ lds->callback(egc, lds, ERROR_FAIL);
}
}
static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev,
const struct timeval *requested_abs,
int rc)
{
- libxl__domain_save_state *dss = CONTAINER_OF(ev, *dss, logdirty.timeout);
- STATE_AO_GC(dss->ao);
+ libxl__logdirty_switch *lds = CONTAINER_OF(ev, *lds, timeout);
+ STATE_AO_GC(lds->ao);
LOG(ERROR,"logdirty switch: wait for device model timed out");
- switch_logdirty_done(egc,dss,ERROR_FAIL);
+ switch_logdirty_done(egc,lds,ERROR_FAIL);
}
static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch *watch,
const char *watch_path, const char *event_path)
{
- libxl__domain_save_state *dss =
- CONTAINER_OF(watch, *dss, logdirty.watch);
- libxl__logdirty_switch *lds = &dss->logdirty;
- STATE_AO_GC(dss->ao);
+ libxl__logdirty_switch *lds = CONTAINER_OF(watch, *lds, watch);
+ STATE_AO_GC(lds->ao);
const char *got;
xs_transaction_t t = 0;
int rc;
@@ -229,28 +244,20 @@ static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch *watch,
if (rc <= 0) {
if (rc < 0)
LOG(ERROR,"logdirty switch: failed (rc=%d)",rc);
- switch_logdirty_done(egc,dss,rc);
+ switch_logdirty_done(egc,lds,rc);
}
}
static void switch_logdirty_done(libxl__egc *egc,
- libxl__domain_save_state *dss,
+ libxl__logdirty_switch *lds,
int rc)
{
- STATE_AO_GC(dss->ao);
- libxl__logdirty_switch *lds = &dss->logdirty;
+ STATE_AO_GC(lds->ao);
libxl__ev_xswatch_deregister(gc, &lds->watch);
libxl__ev_time_deregister(gc, &lds->timeout);
- int broke;
- if (rc) {
- broke = -1;
- dss->rc = rc;
- } else {
- broke = 0;
- }
- libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, broke);
+ lds->callback(egc, lds, rc);
}
/*----- callbacks, called by xc_domain_save -----*/
@@ -347,6 +354,8 @@ void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
dss->rc = 0;
libxl__logdirty_init(&dss->logdirty);
+ dss->logdirty.ao = ao;
+
dsps->ao = ao;
dsps->domid = domid;
rc = libxl__domain_suspend_init(egc, dsps, type);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 7801385..15b9508 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3086,6 +3086,11 @@ libxl__stream_write_inuse(const libxl__stream_write_state *stream)
}
typedef struct libxl__logdirty_switch {
+ /* Set by caller of libxl__domain_common_switch_qemu_logdirty */
+ libxl__ao *ao;
+ void (*callback)(libxl__egc *egc, struct libxl__logdirty_switch *lds,
+ int rc);
+
const char *cmd;
const char *cmd_path;
const char *ret_path;
@@ -3504,6 +3509,9 @@ void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
_hidden void libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned int enable, void *data);
+_hidden void libxl__domain_common_switch_qemu_logdirty(libxl__egc *egc,
+ int domid, unsigned enable,
+ libxl__logdirty_switch *lds);
_hidden int libxl__save_emulator_xenstore_data(libxl__domain_save_state *dss,
char **buf, uint32_t *len);
_hidden int libxl__restore_emulator_xenstore_data
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* [PATCH v11 03/27] tools/libxl: Add back channel to allow migration target send data back
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
2016-03-04 8:41 ` [PATCH v11 01/27] tools/libxl: introduction of libxl__qmp_restore to load qemu state Changlong Xie
2016-03-04 8:41 ` [PATCH v11 02/27] tools/libxl: introduce libxl__domain_common_switch_qemu_logdirty() Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 16:38 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 04/27] tools/libxl: Introduce new helper function dup_fd_helper() Changlong Xie
` (24 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
In COLO mode, secondary needs to send the following data to primary:
1. In libxl
Secondary sends the following CHECKPOINT_CONTEXT to primary:
CHECKPOINT_SVM_SUSPENDED, CHECKPOINT_SVM_READY and CHECKPOINT_SVM_RESUMED
2. In libxc
Secondary sends the dirty pfn list to primary
But the io_fd only can be written in primary, and only can be read in
secondary. Save recv_fd in domain_suspend_state, and send_fd in
domain_create_state. Extend libxl_domain_create_restore API, add a
send_fd param to it. Add LIBXL_HAVE_CREATE_RESTORE_SEND_FD to indicate
the API change.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl.c | 3 +--
tools/libxl/libxl.h | 29 ++++++++++++++++++++++++++++-
tools/libxl/libxl_create.c | 10 ++++++----
tools/libxl/libxl_internal.h | 2 ++
tools/libxl/xl_cmdimpl.c | 8 +++++++-
tools/ocaml/libs/xl/xenlight_stubs.c | 2 +-
6 files changed, 45 insertions(+), 9 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 4cdc169..e9ab78c 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -834,7 +834,6 @@ out:
static void remus_failover_cb(libxl__egc *egc,
libxl__domain_save_state *dss, int rc);
-/* TODO: Explicit Checkpoint acknowledgements via recv_fd. */
int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
uint32_t domid, int send_fd, int recv_fd,
const libxl_asyncop_how *ao_how)
@@ -871,7 +870,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
dss->callback = remus_failover_cb;
dss->domid = domid;
dss->fd = send_fd;
- /* TODO do something with recv_fd */
+ dss->recv_fd = recv_fd;
dss->type = type;
dss->live = 1;
dss->debug = 0;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index f9e3ef5..5a286ec 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -639,6 +639,15 @@ typedef struct libxl__ctx libxl_ctx;
#define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_PARAMS 1
/*
+ * LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
+ *
+ * If this is defined, libxl_domain_create_restore()'s API has changed to
+ * include a send_back_fd param which used for libxl migration back channel
+ * during COLO.
+ */
+#define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
+
+/*
* LIBXL_HAVE_CREATEINFO_PVH
* If this is defined, then libxl supports creation of a PVH guest.
*/
@@ -1165,6 +1174,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid, int restore_fd,
+ int send_back_fd,
const libxl_domain_restore_params *params,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
@@ -1185,7 +1195,7 @@ int static inline libxl_domain_create_restore_0x040200(
libxl_domain_restore_params_init(¶ms);
ret = libxl_domain_create_restore(
- ctx, d_config, domid, restore_fd, ¶ms, ao_how, aop_console_how);
+ ctx, d_config, domid, restore_fd, -1, ¶ms, ao_how, aop_console_how);
libxl_domain_restore_params_dispose(¶ms);
return ret;
@@ -1193,6 +1203,23 @@ int static inline libxl_domain_create_restore_0x040200(
#define libxl_domain_create_restore libxl_domain_create_restore_0x040200
+#elif defined(LIBXL_API_VERSION) && LIBXL_API_VERSION >= 0x040400 \
+ && LIBXL_API_VERSION < 0x040700
+
+int static inline libxl_domain_create_restore_0x040400(
+ libxl_ctx *ctx, libxl_domain_config *d_config,
+ uint32_t *domid, int restore_fd,
+ const libxl_domain_restore_params *params,
+ const libxl_asyncop_how *ao_how,
+ const libxl_asyncprogress_how *aop_console_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY
+{
+ return libxl_domain_create_restore(ctx, d_config, domid, restore_fd,
+ -1, params, ao_how, aop_console_how);
+}
+
+#define libxl_domain_create_restore libxl_domain_create_restore_0x040400
+
#endif
int libxl_domain_soft_reset(libxl_ctx *ctx,
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index f1028bc..525bf85 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1572,7 +1572,7 @@ static void domain_create_cb(libxl__egc *egc,
int rc, uint32_t domid);
static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
- uint32_t *domid, int restore_fd,
+ uint32_t *domid, int restore_fd, int send_back_fd,
const libxl_domain_restore_params *params,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
@@ -1587,6 +1587,7 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
libxl_domain_config_init(&cdcs->dcs.guest_config_saved);
libxl_domain_config_copy(ctx, &cdcs->dcs.guest_config_saved, d_config);
cdcs->dcs.restore_fd = cdcs->dcs.libxc_fd = restore_fd;
+ cdcs->dcs.send_back_fd = send_back_fd;
if (restore_fd > -1) {
cdcs->dcs.restore_params = *params;
rc = libxl__fd_flags_modify_save(gc, cdcs->dcs.restore_fd,
@@ -1765,18 +1766,19 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
- return do_domain_create(ctx, d_config, domid, -1, NULL,
+ return do_domain_create(ctx, d_config, domid, -1, -1, NULL,
ao_how, aop_console_how);
}
int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid, int restore_fd,
+ int send_back_fd,
const libxl_domain_restore_params *params,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
- return do_domain_create(ctx, d_config, domid, restore_fd, params,
- ao_how, aop_console_how);
+ return do_domain_create(ctx, d_config, domid, restore_fd, send_back_fd,
+ params, ao_how, aop_console_how);
}
int libxl_domain_soft_reset(libxl_ctx *ctx,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 15b9508..5314522 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3132,6 +3132,7 @@ struct libxl__domain_save_state {
uint32_t domid;
int fd;
int fdfl; /* original flags on fd */
+ int recv_fd;
libxl_domain_type type;
int live;
int debug;
@@ -3465,6 +3466,7 @@ struct libxl__domain_create_state {
libxl_domain_config guest_config_saved; /* vanilla config */
int restore_fd, libxc_fd;
int restore_fdfl; /* original flags of restore_fd */
+ int send_back_fd;
libxl_domain_restore_params restore_params;
uint32_t domid_soft_reset;
libxl__domain_create_cb *callback;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 990d3c9..523e0e9 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -159,6 +159,7 @@ struct domain_create {
char *extra_config; /* extra config string */
const char *restore_file;
int migrate_fd; /* -1 means none */
+ int send_back_fd; /* -1 means none */
char **migration_domname_r; /* from malloc */
};
@@ -2691,6 +2692,7 @@ static uint32_t create_domain(struct domain_create *dom_info)
int config_len = 0;
int restore_fd = -1;
int restore_fd_to_close = -1;
+ int send_back_fd = -1;
const libxl_asyncprogress_how *autoconnect_console_how;
struct save_file_header hdr;
uint32_t domid_soft_reset = INVALID_DOMID;
@@ -2708,6 +2710,7 @@ static uint32_t create_domain(struct domain_create *dom_info)
if (migrate_fd >= 0) {
restore_source = "<incoming migration stream>";
restore_fd = migrate_fd;
+ send_back_fd = dom_info->send_back_fd;
} else {
restore_source = restore_file;
restore_fd = open(restore_file, O_RDONLY);
@@ -2896,7 +2899,7 @@ start:
ret = libxl_domain_create_restore(ctx, &d_config,
&domid, restore_fd,
- ¶ms,
+ send_back_fd, ¶ms,
0, autoconnect_console_how);
libxl_domain_restore_params_dispose(¶ms);
@@ -4459,6 +4462,7 @@ static void migrate_receive(int debug, int daemonize, int monitor,
dom_info.monitor = monitor;
dom_info.paused = 1;
dom_info.migrate_fd = recv_fd;
+ dom_info.send_back_fd = send_fd;
dom_info.migration_domname_r = &migration_domname;
dom_info.checkpointed_stream = checkpointed;
@@ -4632,6 +4636,7 @@ int main_restore(int argc, char **argv)
dom_info.config_file = config_file;
dom_info.restore_file = checkpoint_file;
dom_info.migrate_fd = -1;
+ dom_info.send_back_fd = -1;
dom_info.vnc = vnc;
dom_info.vncautopass = vncautopass;
dom_info.console_autoconnect = console_autoconnect;
@@ -5099,6 +5104,7 @@ int main_create(int argc, char **argv)
dom_info.quiet = quiet;
dom_info.config_file = filename;
dom_info.migrate_fd = -1;
+ dom_info.send_back_fd = -1;
dom_info.vnc = vnc;
dom_info.vncautopass = vncautopass;
dom_info.console_autoconnect = console_autoconnect;
diff --git a/tools/ocaml/libs/xl/xenlight_stubs.c b/tools/ocaml/libs/xl/xenlight_stubs.c
index 4133527..98b52b9 100644
--- a/tools/ocaml/libs/xl/xenlight_stubs.c
+++ b/tools/ocaml/libs/xl/xenlight_stubs.c
@@ -538,7 +538,7 @@ value stub_libxl_domain_create_restore(value ctx, value domain_config, value par
caml_enter_blocking_section();
ret = libxl_domain_create_restore(CTX, &c_dconfig, &c_domid, restore_fd,
- &c_params, ao_how, NULL);
+ -1, &c_params, ao_how, NULL);
caml_leave_blocking_section();
free(ao_how);
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 03/27] tools/libxl: Add back channel to allow migration target send data back
2016-03-04 8:41 ` [PATCH v11 03/27] tools/libxl: Add back channel to allow migration target send data back Changlong Xie
@ 2016-03-04 16:38 ` Ian Jackson
2016-03-08 16:38 ` Wei Liu
2016-03-17 8:07 ` Changlong Xie
0 siblings, 2 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 16:38 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 03/27] tools/libxl: Add back channel to allow migration target send data back"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> In COLO mode, secondary needs to send the following data to primary:
> 1. In libxl
> Secondary sends the following CHECKPOINT_CONTEXT to primary:
> CHECKPOINT_SVM_SUSPENDED, CHECKPOINT_SVM_READY and CHECKPOINT_SVM_RESUMED
> 2. In libxc
> Secondary sends the dirty pfn list to primary
The overall API approach here seems good to me.
But, my reading of the code is that this new fd is currently ignored.
This is, AFAICT, intentional in the non-colo case, and we have no colo
case yet.
So I think that this new parameter needs to be slightly better
documented. I suggest:
* In this patch, add a comment next to it saying "always pass -1".
* In the patch were the fd actually starts to do something, change
this comment to something more meaningful.
> /*
> + * LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
> + *
> + * If this is defined, libxl_domain_create_restore()'s API has changed to
> + * include a send_back_fd param which used for libxl migration back channel
> + * during COLO.
> + */
> +#define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
I have a minor grammar quibble with this. I would write:
"If this is defined, libxl_domain_create_restore()'s API
includes the send_back_fd param. This is used only with
COLO, for the libxl migration back channel; other callers
should pass -1."
And, with this definition of the API, I think that the code should
actually check that -1 is passed. Personally I would be happy with
the error case either failing assert() or returning ERROR_INVAL, but
maybe other maintainers have a specific view.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 03/27] tools/libxl: Add back channel to allow migration target send data back
2016-03-04 16:38 ` Ian Jackson
@ 2016-03-08 16:38 ` Wei Liu
2016-03-17 8:07 ` Changlong Xie
1 sibling, 0 replies; 100+ messages in thread
From: Wei Liu @ 2016-03-08 16:38 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Anthony Perard, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On Fri, Mar 04, 2016 at 04:38:23PM +0000, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 03/27] tools/libxl: Add back channel to allow migration target send data back"):
> > From: Wen Congyang <wency@cn.fujitsu.com>
> >
> > In COLO mode, secondary needs to send the following data to primary:
> > 1. In libxl
> > Secondary sends the following CHECKPOINT_CONTEXT to primary:
> > CHECKPOINT_SVM_SUSPENDED, CHECKPOINT_SVM_READY and CHECKPOINT_SVM_RESUMED
> > 2. In libxc
> > Secondary sends the dirty pfn list to primary
>
> The overall API approach here seems good to me.
>
> But, my reading of the code is that this new fd is currently ignored.
> This is, AFAICT, intentional in the non-colo case, and we have no colo
> case yet.
>
> So I think that this new parameter needs to be slightly better
> documented. I suggest:
>
> * In this patch, add a comment next to it saying "always pass -1".
> * In the patch were the fd actually starts to do something, change
> this comment to something more meaningful.
>
> > /*
> > + * LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
> > + *
> > + * If this is defined, libxl_domain_create_restore()'s API has changed to
> > + * include a send_back_fd param which used for libxl migration back channel
> > + * during COLO.
> > + */
> > +#define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
>
> I have a minor grammar quibble with this. I would write:
>
> "If this is defined, libxl_domain_create_restore()'s API
> includes the send_back_fd param. This is used only with
> COLO, for the libxl migration back channel; other callers
> should pass -1."
>
> And, with this definition of the API, I think that the code should
> actually check that -1 is passed. Personally I would be happy with
> the error case either failing assert() or returning ERROR_INVAL, but
> maybe other maintainers have a specific view.
>
I have no preference on this issue. Either calling assert or
returning ERROR_INVAL is fine by me.
Wei.
> Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 03/27] tools/libxl: Add back channel to allow migration target send data back
2016-03-04 16:38 ` Ian Jackson
2016-03-08 16:38 ` Wei Liu
@ 2016-03-17 8:07 ` Changlong Xie
1 sibling, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-17 8:07 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 12:38 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 03/27] tools/libxl: Add back channel to allow migration target send data back"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> In COLO mode, secondary needs to send the following data to primary:
>> 1. In libxl
>> Secondary sends the following CHECKPOINT_CONTEXT to primary:
>> CHECKPOINT_SVM_SUSPENDED, CHECKPOINT_SVM_READY and CHECKPOINT_SVM_RESUMED
>> 2. In libxc
>> Secondary sends the dirty pfn list to primary
>
> The overall API approach here seems good to me.
>
> But, my reading of the code is that this new fd is currently ignored.
> This is, AFAICT, intentional in the non-colo case, and we have no colo
> case yet.
>
> So I think that this new parameter needs to be slightly better
> documented. I suggest:
>
> * In this patch, add a comment next to it saying "always pass -1".
> * In the patch were the fd actually starts to do something, change
> this comment to something more meaningful.
>
Ok
>> /*
>> + * LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
>> + *
>> + * If this is defined, libxl_domain_create_restore()'s API has changed to
>> + * include a send_back_fd param which used for libxl migration back channel
>> + * during COLO.
>> + */
>> +#define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
>
> I have a minor grammar quibble with this. I would write:
>
> "If this is defined, libxl_domain_create_restore()'s API
> includes the send_back_fd param. This is used only with
> COLO, for the libxl migration back channel; other callers
> should pass -1."
>
> And, with this definition of the API, I think that the code should
> actually check that -1 is passed. Personally I would be happy with
> the error case either failing assert() or returning ERROR_INVAL, but
>maybe other maintainers have a specific view.
>
I prefer assert(), will fix in next version
Thanks
-Xie
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 04/27] tools/libxl: Introduce new helper function dup_fd_helper()
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (2 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 03/27] tools/libxl: Add back channel to allow migration target send data back Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 16:42 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 05/27] tools/libx{l, c}: add back channel to libxc Changlong Xie
` (23 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
It is pure refactoring and no functional changes.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl_save_callout.c | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 7f1f5d4..a885504 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -118,6 +118,21 @@ void libxl__save_helper_init(libxl__save_helper_state *shs)
}
/*----- helper execution -----*/
+static int dup_fd_helper(libxl__gc *gc, int fd, const char *what)
+{
+ int dup_fd = fd;
+
+ if (fd <= 2) {
+ dup_fd = dup(fd);
+ if (dup_fd < 0) {
+ LOGE(ERROR,"dup %s", what);
+ exit(-1);
+ }
+ }
+ libxl_fd_set_cloexec(CTX, dup_fd, 0);
+
+ return dup_fd;
+}
/*
* Both save and restore share four parameters:
@@ -186,14 +201,7 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
pid_t pid = libxl__ev_child_fork(gc, &shs->child, helper_exited);
if (!pid) {
- if (stream_fd <= 2) {
- stream_fd = dup(stream_fd);
- if (stream_fd < 0) {
- LOGE(ERROR,"dup migration stream fd");
- exit(-1);
- }
- }
- libxl_fd_set_cloexec(CTX, stream_fd, 0);
+ stream_fd = dup_fd_helper(gc, stream_fd, "migration stream fd");
*stream_fd_arg = GCSPRINTF("%d", stream_fd);
for (i=0; i<num_preserve_fds; i++)
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 04/27] tools/libxl: Introduce new helper function dup_fd_helper()
2016-03-04 8:41 ` [PATCH v11 04/27] tools/libxl: Introduce new helper function dup_fd_helper() Changlong Xie
@ 2016-03-04 16:42 ` Ian Jackson
2016-03-17 8:08 ` Changlong Xie
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 16:42 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 04/27] tools/libxl: Introduce new helper function dup_fd_helper()"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> It is pure refactoring and no functional changes.
...
> /*----- helper execution -----*/
> +static int dup_fd_helper(libxl__gc *gc, int fd, const char *what)
> +{
This is a fine thing to do. But the function name is less than
informative. I think I would call it something like `dup_cloexec'.
The use of the term `helper' here is particularly confusing because it
also refers to the `save helper' of which this is part.
And
- missing newline after the /*---*/ comment
- this function should have a doc comment saying it cannot fail
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 04/27] tools/libxl: Introduce new helper function dup_fd_helper()
2016-03-04 16:42 ` Ian Jackson
@ 2016-03-17 8:08 ` Changlong Xie
0 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-17 8:08 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 12:42 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 04/27] tools/libxl: Introduce new helper function dup_fd_helper()"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> It is pure refactoring and no functional changes.
> ...
>> /*----- helper execution -----*/
>> +static int dup_fd_helper(libxl__gc *gc, int fd, const char *what)
>> +{
>
> This is a fine thing to do. But the function name is less than
> informative. I think I would call it something like `dup_cloexec'.
ok
> The use of the term `helper' here is particularly confusing because it
> also refers to the `save helper' of which this is part.
>
> And
> - missing newline after the /*---*/ comment
> - this function should have a doc comment saying it cannot fail
>
ok, will fix in next version.
Thanks
-Xie
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 05/27] tools/libx{l, c}: add back channel to libxc
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (3 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 04/27] tools/libxl: Introduce new helper function dup_fd_helper() Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 16:45 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 06/27] docs: add colo readme Changlong Xie
` (22 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Wen Congyang,
Gui Jianfeng, Jiang Yunhong, Dong Eddie, Anthony Perard,
Shriram Rajagopalan, Ian Jackson, Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
In COLO mode, both VMs are running, and are considered in sync if the
visible network traffic is identical. After some time, they fall out of
sync.
At this point, the two VMs have definitely diverged. Lets call the
primary dirty bitmap set A, while the secondary dirty bitmap set B.
Sets A and B are different.
Under normal migration, the page data for set A will be sent from the
primary to the secondary.
However, the set difference B - A (the one in B but not in A, lets
call this C) is out-of-date on the secondary (with respect to the
primary) and will not be sent by the primary (to secondary), as it
was not memory dirtied by the primary. The secondary needs C page data
to reconstruct an exact copy of the primary at the checkpoint.
The secondary cannot calculate C as it doesn't know A. Instead, the
secondary must send B to the primary, at which point the primary
calculates the union of A and B (lets call this D) which is all the
pages dirtied by both the primary and the secondary, and sends all page
data covered by D.
In the general case, D is a superset of both A and B. Without the
backchannel dirty bitmap, a COLO checkpoint can't reconstruct a valid
copy of the primary.
We transfer the dirty bitmap on libxc side, so we need to introduce back
channel to libxc.
Note: it is different from the paper. We change the original design to
the current one, according to our following concerns:
1. The original design needs extra memory on Secondary host. When there's
multiple backups on one host, the memory cost is high.
2. The memory cache code will be another 1k+, it will make the review
more time consuming.
Note: this patch merely adds new parameters to various prototypes and
functions. The new parameters are used in later patch called
"libxc/restore: send dirty pfn list to primary when checkpoint under
COLO".
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
tools/libxc/include/xenguest.h | 4 ++--
tools/libxc/xc_nomigrate.c | 4 ++--
tools/libxc/xc_sr_restore.c | 2 +-
tools/libxc/xc_sr_save.c | 2 +-
tools/libxl/libxl_save_callout.c | 21 +++++++++++++++------
tools/libxl/libxl_save_helper.c | 8 ++++++--
6 files changed, 27 insertions(+), 14 deletions(-)
diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..8040ac7 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -88,7 +88,7 @@ struct save_callbacks {
int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
struct save_callbacks* callbacks, int hvm,
- int checkpointed_stream);
+ int checkpointed_stream, int recv_fd);
/* callbacks provided by xc_domain_restore */
struct restore_callbacks {
@@ -127,7 +127,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
unsigned long *console_mfn, domid_t console_domid,
unsigned int hvm, unsigned int pae, int superpages,
int checkpointed_stream,
- struct restore_callbacks *callbacks);
+ struct restore_callbacks *callbacks, int send_back_fd);
/**
* This function will create a domain for a paravirtualized Linux
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index c9124df..3634743 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -23,7 +23,7 @@
int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iters,
uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
- int checkpointed_stream)
+ int checkpointed_stream, int recv_fd)
{
errno = ENOSYS;
return -1;
@@ -35,7 +35,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
unsigned long *console_mfn, domid_t console_domid,
unsigned int hvm, unsigned int pae, int superpages,
int checkpointed_stream,
- struct restore_callbacks *callbacks)
+ struct restore_callbacks *callbacks, int send_back_fd)
{
errno = ENOSYS;
return -1;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index d4d33fd..e24560e 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -726,7 +726,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
unsigned long *console_gfn, domid_t console_domid,
unsigned int hvm, unsigned int pae, int superpages,
int checkpointed_stream,
- struct restore_callbacks *callbacks)
+ struct restore_callbacks *callbacks, int send_back_fd)
{
struct xc_sr_context ctx =
{
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index e258b7c..fe210cc 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -830,7 +830,7 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type)
int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
uint32_t max_iters, uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
- int checkpointed_stream)
+ int checkpointed_stream, int recv_fd)
{
struct xc_sr_context ctx =
{
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index a885504..0d6949a 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -27,7 +27,7 @@
*/
static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
const char *mode_arg,
- int stream_fd,
+ int stream_fd, int back_channel_fd,
const int *preserve_fds, int num_preserve_fds,
const unsigned long *argnums, int num_argnums);
@@ -50,6 +50,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
/* Convenience aliases */
const uint32_t domid = dcs->guest_domid;
const int restore_fd = dcs->libxc_fd;
+ const int send_back_fd = dcs->send_back_fd;
libxl__domain_build_state *const state = &dcs->build_state;
unsigned cbflags =
@@ -71,7 +72,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
shs->caller_state = dcs;
shs->need_results = 1;
- run_helper(egc, shs, "--restore-domain", restore_fd, 0, 0,
+ run_helper(egc, shs, "--restore-domain", restore_fd, send_back_fd, 0, 0,
argnums, ARRAY_SIZE(argnums));
}
@@ -95,7 +96,7 @@ void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_save_state *dss,
shs->caller_state = dss;
shs->need_results = 0;
- run_helper(egc, shs, "--save-domain", dss->fd,
+ run_helper(egc, shs, "--save-domain", dss->fd, dss->recv_fd,
NULL, 0,
argnums, ARRAY_SIZE(argnums));
return;
@@ -139,12 +140,14 @@ static int dup_fd_helper(libxl__gc *gc, int fd, const char *what)
* 1) Path to libxl-save-helper.
* 2) --[restore|save]-domain.
* 3) stream file descriptor.
+ * 4) back channel file descriptor.
* n) save/restore specific parameters.
- * 4) A \0 at the end.
+ * 5) A \0 at the end.
*/
-#define HELPER_NR_ARGS 4
+#define HELPER_NR_ARGS 5
static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
- const char *mode_arg, int stream_fd,
+ const char *mode_arg,
+ int stream_fd, int back_channel_fd,
const int *preserve_fds, int num_preserve_fds,
const unsigned long *argnums, int num_argnums)
{
@@ -177,6 +180,7 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
*arg++ = getenv("LIBXL_SAVE_HELPER") ?: LIBEXEC_BIN "/" "libxl-save-helper";
*arg++ = mode_arg;
const char **stream_fd_arg = arg++;
+ const char **back_channel_fd_arg = arg++;
for (i=0; i<num_argnums; i++)
*arg++ = GCSPRINTF("%lu", argnums[i]);
*arg++ = 0;
@@ -204,6 +208,11 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
stream_fd = dup_fd_helper(gc, stream_fd, "migration stream fd");
*stream_fd_arg = GCSPRINTF("%d", stream_fd);
+ if (back_channel_fd >= 0)
+ back_channel_fd = dup_fd_helper(gc, back_channel_fd,
+ "migration back channel fd");
+ *back_channel_fd_arg = GCSPRINTF("%d", back_channel_fd);
+
for (i=0; i<num_preserve_fds; i++)
if (preserve_fds[i] >= 0) {
assert(preserve_fds[i] > 2);
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index 6bdcf13..2ff4c6e 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -238,6 +238,7 @@ static struct restore_callbacks helper_restore_callbacks;
int main(int argc, char **argv)
{
int r;
+ int send_back_fd, recv_fd;
#define NEXTARG (++argv, assert(*argv), *argv)
@@ -247,6 +248,7 @@ int main(int argc, char **argv)
if (!strcmp(mode,"--save-domain")) {
io_fd = atoi(NEXTARG);
+ recv_fd = atoi(NEXTARG);
uint32_t dom = strtoul(NEXTARG,0,10);
uint32_t max_iters = strtoul(NEXTARG,0,10);
uint32_t max_factor = strtoul(NEXTARG,0,10);
@@ -262,12 +264,14 @@ int main(int argc, char **argv)
setup_signals(save_signal_handler);
r = xc_domain_save(xch, io_fd, dom, max_iters, max_factor, flags,
- &helper_save_callbacks, hvm, checkpointed_stream);
+ &helper_save_callbacks, hvm, checkpointed_stream,
+ recv_fd);
complete(r);
} else if (!strcmp(mode,"--restore-domain")) {
io_fd = atoi(NEXTARG);
+ send_back_fd = atoi(NEXTARG);
uint32_t dom = strtoul(NEXTARG,0,10);
unsigned store_evtchn = strtoul(NEXTARG,0,10);
domid_t store_domid = strtoul(NEXTARG,0,10);
@@ -292,7 +296,7 @@ int main(int argc, char **argv)
store_domid, console_evtchn, &console_mfn,
console_domid, hvm, pae, superpages,
checkpointed,
- &helper_restore_callbacks);
+ &helper_restore_callbacks, send_back_fd);
helper_stub_restore_results(store_mfn,console_mfn,0);
complete(r);
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 05/27] tools/libx{l, c}: add back channel to libxc
2016-03-04 8:41 ` [PATCH v11 05/27] tools/libx{l, c}: add back channel to libxc Changlong Xie
@ 2016-03-04 16:45 ` Ian Jackson
0 siblings, 0 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 16:45 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 05/27] tools/libx{l,c}: add back channel to libxc"):
> [stuff]
A good explanation, thanks.
> Note: it is different from the paper. We change the original design to
> the current one, according to our following concerns:
> 1. The original design needs extra memory on Secondary host. When there's
> multiple backups on one host, the memory cost is high.
> 2. The memory cache code will be another 1k+, it will make the review
> more time consuming.
Again, good.
> Note: this patch merely adds new parameters to various prototypes and
> functions. The new parameters are used in later patch called
> "libxc/restore: send dirty pfn list to primary when checkpoint under
> COLO".
Thanks for splitting this off like that; it makes review much easier.
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 06/27] docs: add colo readme
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (4 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 05/27] tools/libx{l, c}: add back channel to libxc Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 8:41 ` [PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams Changlong Xie
` (21 subsequent siblings)
27 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
add colo readme, refer to
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
docs/README.colo | 9 +++++++++
1 file changed, 9 insertions(+)
create mode 100644 docs/README.colo
diff --git a/docs/README.colo b/docs/README.colo
new file mode 100644
index 0000000..466eb72
--- /dev/null
+++ b/docs/README.colo
@@ -0,0 +1,9 @@
+COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
+project is a high availability solution. Both primary VM (PVM) and secondary VM
+(SVM) run in parallel. They receive the same request from client, and generate
+response in parallel too. If the response packets from PVM and SVM are
+identical, they are released immediately. Otherwise, a VM checkpoint (on demand)
+is conducted.
+
+See the website at http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
+for details.
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* [PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (5 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 06/27] docs: add colo readme Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 16:51 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 08/27] libxc/migration: Specification update for DIRTY_PFN_LIST records Changlong Xie
` (20 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
It is the negotiation record for COLO.
Primary->Secondary:
control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
Secondary->Primary:
0x00000001: Secondary VM is suspended
0x00000002: Secondary VM is ready
0x00000003: Secondary VM is resumed
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
docs/specs/libxl-migration-stream.pandoc | 31 +++++++++++++++++++++++++++++--
tools/libxl/libxl_sr_stream_format.h | 11 +++++++++++
tools/python/xen/migration/libxl.py | 9 +++++++++
3 files changed, 49 insertions(+), 2 deletions(-)
diff --git a/docs/specs/libxl-migration-stream.pandoc b/docs/specs/libxl-migration-stream.pandoc
index 2c97d86..5d577ac 100644
--- a/docs/specs/libxl-migration-stream.pandoc
+++ b/docs/specs/libxl-migration-stream.pandoc
@@ -1,6 +1,8 @@
% LibXenLight Domain Image Format
% Andrew Cooper <<andrew.cooper3@citrix.com>>
-% Revision 1
+ Wen Congyang <<wency@cn.fujitsu.com>>
+ Yang Hongyang <<hongyang.yang@easystack.cn>>
+% Revision 2
Introduction
============
@@ -119,7 +121,9 @@ type 0x00000000: END
0x00000004: CHECKPOINT_END
- 0x00000005 - 0x7FFFFFFF: Reserved for future _mandatory_
+ 0x00000005: CHECKPOINT_STATE
+
+ 0x00000006 - 0x7FFFFFFF: Reserved for future _mandatory_
records.
0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -249,6 +253,29 @@ A checkpoint end record marks the end of a checkpoint in the image.
The end record contains no fields; its body_length is 0.
+CHECKPOINT\_STATE
+--------------
+
+A checkpoint state record contains the control information for checkpoint.
+
+ 0 1 2 3 4 5 6 7 octet
+ +------------------------+------------------------+
+ | control_id | padding |
+ +------------------------+------------------------+
+
+--------------------------------------------------------------------
+Field Description
+------------ ---------------------------------------------------
+control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
+ (Primary -> Secondary)
+
+ 0x00000001: Secondary VM is suspended (Secondary -> Primary)
+
+ 0x00000002: Secondary VM is ready (Secondary -> Primary)
+
+ 0x00000003: Secondary VM is resumed (Secondary -> Primary)
+--------------------------------------------------------------------
+
Future Extensions
=================
diff --git a/tools/libxl/libxl_sr_stream_format.h b/tools/libxl/libxl_sr_stream_format.h
index 54da360..75f5190 100644
--- a/tools/libxl/libxl_sr_stream_format.h
+++ b/tools/libxl/libxl_sr_stream_format.h
@@ -36,6 +36,7 @@ typedef struct libxl__sr_rec_hdr
#define REC_TYPE_EMULATOR_XENSTORE_DATA 0x00000002U
#define REC_TYPE_EMULATOR_CONTEXT 0x00000003U
#define REC_TYPE_CHECKPOINT_END 0x00000004U
+#define REC_TYPE_CHECKPOINT_STATE 0x00000005U
typedef struct libxl__sr_emulator_hdr
{
@@ -47,6 +48,16 @@ typedef struct libxl__sr_emulator_hdr
#define EMULATOR_QEMU_TRADITIONAL 0x00000001U
#define EMULATOR_QEMU_UPSTREAM 0x00000002U
+typedef struct libxl_sr_checkpoint_state
+{
+ uint32_t id;
+} libxl_sr_checkpoint_state;
+
+#define CHECKPOINT_NEW 0x00000000U
+#define CHECKPOINT_SVM_SUSPENDED 0x00000001U
+#define CHECKPOINT_SVM_READY 0x00000002U
+#define CHECKPOINT_SVM_RESUMED 0x00000003U
+
#endif /* LIBXL__SR_STREAM_FORMAT_H */
/*
diff --git a/tools/python/xen/migration/libxl.py b/tools/python/xen/migration/libxl.py
index fc0acf6..d5f54dc 100644
--- a/tools/python/xen/migration/libxl.py
+++ b/tools/python/xen/migration/libxl.py
@@ -37,6 +37,7 @@ REC_TYPE_libxc_context = 0x00000001
REC_TYPE_emulator_xenstore_data = 0x00000002
REC_TYPE_emulator_context = 0x00000003
REC_TYPE_checkpoint_end = 0x00000004
+REC_TYPE_checkpoint_state = 0x00000005
rec_type_to_str = {
REC_TYPE_end : "End",
@@ -44,6 +45,7 @@ rec_type_to_str = {
REC_TYPE_emulator_xenstore_data : "Emulator xenstore data",
REC_TYPE_emulator_context : "Emulator context",
REC_TYPE_checkpoint_end : "Checkpoint end",
+ REC_TYPE_checkpoint_state : "Checkpoint state"
}
# emulator_* header
@@ -212,6 +214,11 @@ class VerifyLibxl(VerifyBase):
if len(content) != 0:
raise RecordError("Checkpoint end record with non-zero length")
+ def verify_record_checkpoint_state(self, content):
+ """ Checkpoint state """
+ if len(content) == 0:
+ raise RecordError("Checkpoint state record with zero length")
+
record_verifiers = {
REC_TYPE_end:
@@ -224,4 +231,6 @@ record_verifiers = {
VerifyLibxl.verify_record_emulator_context,
REC_TYPE_checkpoint_end:
VerifyLibxl.verify_record_checkpoint_end,
+ REC_TYPE_checkpoint_state:
+ VerifyLibxl.verify_record_checkpoint_state,
}
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams
2016-03-04 8:41 ` [PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams Changlong Xie
@ 2016-03-04 16:51 ` Ian Jackson
2016-03-08 16:38 ` Wei Liu
2016-03-11 7:13 ` Wen Congyang
0 siblings, 2 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 16:51 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams"):
> From: Wen Congyang <wency@cn.fujitsu.com>
I think we will want to see an ack from Andy Cooper on this, in due
course.
> It is the negotiation record for COLO.
> Primary->Secondary:
> control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
> Secondary->Primary:
> 0x00000001: Secondary VM is suspended
> 0x00000002: Secondary VM is ready
> 0x00000003: Secondary VM is resumed
I don't think it is necessary to repeat the enum assignment here in
the commit message.
> +CHECKPOINT\_STATE
> +--------------
This documentation patch ought to explicitly mention COLO, and have
cross-references to the various documents (eg, the README added in the
previous patch).
> +A checkpoint state record contains the control information for checkpoint.
> +
> + 0 1 2 3 4 5 6 7 octet
> + +------------------------+------------------------+
> + | control_id | padding |
> + +------------------------+------------------------+
> +
> +--------------------------------------------------------------------
> +Field Description
> +------------ ---------------------------------------------------
> +control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
> + (Primary -> Secondary)
> +
> + 0x00000001: Secondary VM is suspended (Secondary -> Primary)
> +
> + 0x00000002: Secondary VM is ready (Secondary -> Primary)
> +
> + 0x00000003: Secondary VM is resumed (Secondary -> Primary)
I think this should be accompanied by an explanation of what order
these messages are sent in, and what both ends may or may not do
during that time.
> @@ -212,6 +214,11 @@ class VerifyLibxl(VerifyBase):
> if len(content) != 0:
> raise RecordError("Checkpoint end record with non-zero length")
>
> + def verify_record_checkpoint_state(self, content):
> + """ Checkpoint state """
> + if len(content) == 0:
> + raise RecordError("Checkpoint state record with zero length")
> +
I'm not verify familiar with this area of the code, but I think that
this should probably check that the control_id is as expected. Can it
know what the right sequencing is ?
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams
2016-03-04 16:51 ` Ian Jackson
@ 2016-03-08 16:38 ` Wei Liu
2016-03-11 7:13 ` Wen Congyang
1 sibling, 0 replies; 100+ messages in thread
From: Wei Liu @ 2016-03-08 16:38 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Anthony Perard, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On Fri, Mar 04, 2016 at 04:51:20PM +0000, Ian Jackson wrote:
[...]
>
> > @@ -212,6 +214,11 @@ class VerifyLibxl(VerifyBase):
> > if len(content) != 0:
> > raise RecordError("Checkpoint end record with non-zero length")
> >
> > + def verify_record_checkpoint_state(self, content):
> > + """ Checkpoint state """
> > + if len(content) == 0:
> > + raise RecordError("Checkpoint state record with zero length")
> > +
>
> I'm not verify familiar with this area of the code, but I think that
> this should probably check that the control_id is as expected. Can it
> know what the right sequencing is ?
>
FWIW this script is not used in live system -- so it probably doesn't
have information on the control id and the sequence on a live system.
Wei.
> Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams
2016-03-04 16:51 ` Ian Jackson
2016-03-08 16:38 ` Wei Liu
@ 2016-03-11 7:13 ` Wen Congyang
1 sibling, 0 replies; 100+ messages in thread
From: Wen Congyang @ 2016-03-11 7:13 UTC (permalink / raw)
To: Ian Jackson, Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Andrew Cooper, Jiang Yunhong,
Dong Eddie, xen devel, Anthony Perard, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 12:51 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>
> I think we will want to see an ack from Andy Cooper on this, in due
> course.
>
>> It is the negotiation record for COLO.
>> Primary->Secondary:
>> control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
>> Secondary->Primary:
>> 0x00000001: Secondary VM is suspended
>> 0x00000002: Secondary VM is ready
>> 0x00000003: Secondary VM is resumed
>
> I don't think it is necessary to repeat the enum assignment here in
> the commit message.
OK, will fix it in the next version.
>
>
>> +CHECKPOINT\_STATE
>> +--------------
>
> This documentation patch ought to explicitly mention COLO, and have
> cross-references to the various documents (eg, the README added in the
> previous patch).
>
>> +A checkpoint state record contains the control information for checkpoint.
>> +
>> + 0 1 2 3 4 5 6 7 octet
>> + +------------------------+------------------------+
>> + | control_id | padding |
>> + +------------------------+------------------------+
>> +
>> +--------------------------------------------------------------------
>> +Field Description
>> +------------ ---------------------------------------------------
>> +control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
>> + (Primary -> Secondary)
>> +
>> + 0x00000001: Secondary VM is suspended (Secondary -> Primary)
>> +
>> + 0x00000002: Secondary VM is ready (Secondary -> Primary)
>> +
>> + 0x00000003: Secondary VM is resumed (Secondary -> Primary)
>
> I think this should be accompanied by an explanation of what order
> these messages are sent in, and what both ends may or may not do
> during that time.
OK, will fix it in the next version.
>
>
>> @@ -212,6 +214,11 @@ class VerifyLibxl(VerifyBase):
>> if len(content) != 0:
>> raise RecordError("Checkpoint end record with non-zero length")
>>
>> + def verify_record_checkpoint_state(self, content):
>> + """ Checkpoint state """
>> + if len(content) == 0:
>> + raise RecordError("Checkpoint state record with zero length")
>> +
>
> I'm not verify familiar with this area of the code, but I think that
> this should probably check that the control_id is as expected. Can it
> know what the right sequencing is ?
To Andrew Cooper:
What is the purpost of this script? If it is not used for live system, I think
the stream should not contain checkpoint state record.
Thanks
Wen Congyang
>
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 08/27] libxc/migration: Specification update for DIRTY_PFN_LIST records
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (6 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 07/27] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 16:53 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 09/27] libxc/migration: export read_record for common use Changlong Xie
` (19 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Used by secondary to send it's dirty bitmap to primary under COLO.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
docs/specs/libxc-migration-stream.pandoc | 26 +++++++++++-
tools/libxc/xc_sr_common.c | 31 ++++++++-------
tools/libxc/xc_sr_stream_format.h | 31 ++++++++-------
tools/python/xen/migration/libxc.py | 68 ++++++++++++++++++--------------
4 files changed, 95 insertions(+), 61 deletions(-)
diff --git a/docs/specs/libxc-migration-stream.pandoc b/docs/specs/libxc-migration-stream.pandoc
index 8cd678f..18544e0 100644
--- a/docs/specs/libxc-migration-stream.pandoc
+++ b/docs/specs/libxc-migration-stream.pandoc
@@ -1,6 +1,8 @@
% LibXenCtrl Domain Image Format
% David Vrabel <<david.vrabel@citrix.com>>
Andrew Cooper <<andrew.cooper3@citrix.com>>
+ Wen Congyang <<wency@cn.fujitsu.com>>
+ Yang Hongyang <<hongyang.yang@easystack.cn>>
% Revision 1
Introduction
@@ -227,7 +229,9 @@ type 0x00000000: END
0x0000000E: CHECKPOINT
- 0x0000000F - 0x7FFFFFFF: Reserved for future _mandatory_
+ 0x0000000F: CHECKPOINT_DIRTY_PFN_LIST (Secondary -> Primary)
+
+ 0x00000010 - 0x7FFFFFFF: Reserved for future _mandatory_
records.
0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -599,6 +603,26 @@ CHECKPOINT record or an END record.
\clearpage
+CHECKPOINT_DIRTY_PFN_LIST
+------------
+
+A checkpoint dirty pfn list record is used to convey information about
+dirty memory in the VM. It is an unordered list of PFNs. Currently only
+applicable in the backchannel of a checkpointed stream.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+ | pfn[0] |
+ +-------------------------------------------------+
+ ...
+ +-------------------------------------------------+
+ | pfn[C-1] |
+ +-------------------------------------------------+
+
+The count of pfns is: record->length/sizeof(uint64_t).
+
+\clearpage
+
Layout
======
diff --git a/tools/libxc/xc_sr_common.c b/tools/libxc/xc_sr_common.c
index 945cfa6..3313a90 100644
--- a/tools/libxc/xc_sr_common.c
+++ b/tools/libxc/xc_sr_common.c
@@ -20,21 +20,22 @@ const char *dhdr_type_to_str(uint32_t type)
static const char *mandatory_rec_types[] =
{
- [REC_TYPE_END] = "End",
- [REC_TYPE_PAGE_DATA] = "Page data",
- [REC_TYPE_X86_PV_INFO] = "x86 PV info",
- [REC_TYPE_X86_PV_P2M_FRAMES] = "x86 PV P2M frames",
- [REC_TYPE_X86_PV_VCPU_BASIC] = "x86 PV vcpu basic",
- [REC_TYPE_X86_PV_VCPU_EXTENDED] = "x86 PV vcpu extended",
- [REC_TYPE_X86_PV_VCPU_XSAVE] = "x86 PV vcpu xsave",
- [REC_TYPE_SHARED_INFO] = "Shared info",
- [REC_TYPE_TSC_INFO] = "TSC info",
- [REC_TYPE_HVM_CONTEXT] = "HVM context",
- [REC_TYPE_HVM_PARAMS] = "HVM params",
- [REC_TYPE_TOOLSTACK] = "Toolstack",
- [REC_TYPE_X86_PV_VCPU_MSRS] = "x86 PV vcpu msrs",
- [REC_TYPE_VERIFY] = "Verify",
- [REC_TYPE_CHECKPOINT] = "Checkpoint",
+ [REC_TYPE_END] = "End",
+ [REC_TYPE_PAGE_DATA] = "Page data",
+ [REC_TYPE_X86_PV_INFO] = "x86 PV info",
+ [REC_TYPE_X86_PV_P2M_FRAMES] = "x86 PV P2M frames",
+ [REC_TYPE_X86_PV_VCPU_BASIC] = "x86 PV vcpu basic",
+ [REC_TYPE_X86_PV_VCPU_EXTENDED] = "x86 PV vcpu extended",
+ [REC_TYPE_X86_PV_VCPU_XSAVE] = "x86 PV vcpu xsave",
+ [REC_TYPE_SHARED_INFO] = "Shared info",
+ [REC_TYPE_TSC_INFO] = "TSC info",
+ [REC_TYPE_HVM_CONTEXT] = "HVM context",
+ [REC_TYPE_HVM_PARAMS] = "HVM params",
+ [REC_TYPE_TOOLSTACK] = "Toolstack",
+ [REC_TYPE_X86_PV_VCPU_MSRS] = "x86 PV vcpu msrs",
+ [REC_TYPE_VERIFY] = "Verify",
+ [REC_TYPE_CHECKPOINT] = "Checkpoint",
+ [REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST] = "Checkpoint dirty pfn list",
};
const char *rec_type_to_str(uint32_t type)
diff --git a/tools/libxc/xc_sr_stream_format.h b/tools/libxc/xc_sr_stream_format.h
index 6d0f8fd..3291b25 100644
--- a/tools/libxc/xc_sr_stream_format.h
+++ b/tools/libxc/xc_sr_stream_format.h
@@ -60,21 +60,22 @@ struct xc_sr_rhdr
/* Somewhat arbitrary - 8MB */
#define REC_LENGTH_MAX (8U << 20)
-#define REC_TYPE_END 0x00000000U
-#define REC_TYPE_PAGE_DATA 0x00000001U
-#define REC_TYPE_X86_PV_INFO 0x00000002U
-#define REC_TYPE_X86_PV_P2M_FRAMES 0x00000003U
-#define REC_TYPE_X86_PV_VCPU_BASIC 0x00000004U
-#define REC_TYPE_X86_PV_VCPU_EXTENDED 0x00000005U
-#define REC_TYPE_X86_PV_VCPU_XSAVE 0x00000006U
-#define REC_TYPE_SHARED_INFO 0x00000007U
-#define REC_TYPE_TSC_INFO 0x00000008U
-#define REC_TYPE_HVM_CONTEXT 0x00000009U
-#define REC_TYPE_HVM_PARAMS 0x0000000aU
-#define REC_TYPE_TOOLSTACK 0x0000000bU
-#define REC_TYPE_X86_PV_VCPU_MSRS 0x0000000cU
-#define REC_TYPE_VERIFY 0x0000000dU
-#define REC_TYPE_CHECKPOINT 0x0000000eU
+#define REC_TYPE_END 0x00000000U
+#define REC_TYPE_PAGE_DATA 0x00000001U
+#define REC_TYPE_X86_PV_INFO 0x00000002U
+#define REC_TYPE_X86_PV_P2M_FRAMES 0x00000003U
+#define REC_TYPE_X86_PV_VCPU_BASIC 0x00000004U
+#define REC_TYPE_X86_PV_VCPU_EXTENDED 0x00000005U
+#define REC_TYPE_X86_PV_VCPU_XSAVE 0x00000006U
+#define REC_TYPE_SHARED_INFO 0x00000007U
+#define REC_TYPE_TSC_INFO 0x00000008U
+#define REC_TYPE_HVM_CONTEXT 0x00000009U
+#define REC_TYPE_HVM_PARAMS 0x0000000aU
+#define REC_TYPE_TOOLSTACK 0x0000000bU
+#define REC_TYPE_X86_PV_VCPU_MSRS 0x0000000cU
+#define REC_TYPE_VERIFY 0x0000000dU
+#define REC_TYPE_CHECKPOINT 0x0000000eU
+#define REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST 0x0000000fU
#define REC_TYPE_OPTIONAL 0x80000000U
diff --git a/tools/python/xen/migration/libxc.py b/tools/python/xen/migration/libxc.py
index b0255ac..85a78f4 100644
--- a/tools/python/xen/migration/libxc.py
+++ b/tools/python/xen/migration/libxc.py
@@ -45,38 +45,40 @@ dhdr_type_to_str = {
# Records
RH_FORMAT = "II"
-REC_TYPE_end = 0x00000000
-REC_TYPE_page_data = 0x00000001
-REC_TYPE_x86_pv_info = 0x00000002
-REC_TYPE_x86_pv_p2m_frames = 0x00000003
-REC_TYPE_x86_pv_vcpu_basic = 0x00000004
-REC_TYPE_x86_pv_vcpu_extended = 0x00000005
-REC_TYPE_x86_pv_vcpu_xsave = 0x00000006
-REC_TYPE_shared_info = 0x00000007
-REC_TYPE_tsc_info = 0x00000008
-REC_TYPE_hvm_context = 0x00000009
-REC_TYPE_hvm_params = 0x0000000a
-REC_TYPE_toolstack = 0x0000000b
-REC_TYPE_x86_pv_vcpu_msrs = 0x0000000c
-REC_TYPE_verify = 0x0000000d
-REC_TYPE_checkpoint = 0x0000000e
+REC_TYPE_end = 0x00000000
+REC_TYPE_page_data = 0x00000001
+REC_TYPE_x86_pv_info = 0x00000002
+REC_TYPE_x86_pv_p2m_frames = 0x00000003
+REC_TYPE_x86_pv_vcpu_basic = 0x00000004
+REC_TYPE_x86_pv_vcpu_extended = 0x00000005
+REC_TYPE_x86_pv_vcpu_xsave = 0x00000006
+REC_TYPE_shared_info = 0x00000007
+REC_TYPE_tsc_info = 0x00000008
+REC_TYPE_hvm_context = 0x00000009
+REC_TYPE_hvm_params = 0x0000000a
+REC_TYPE_toolstack = 0x0000000b
+REC_TYPE_x86_pv_vcpu_msrs = 0x0000000c
+REC_TYPE_verify = 0x0000000d
+REC_TYPE_checkpoint = 0x0000000e
+REC_TYPE_checkpoint_dirty_pfn_list = 0x0000000f
rec_type_to_str = {
- REC_TYPE_end : "End",
- REC_TYPE_page_data : "Page data",
- REC_TYPE_x86_pv_info : "x86 PV info",
- REC_TYPE_x86_pv_p2m_frames : "x86 PV P2M frames",
- REC_TYPE_x86_pv_vcpu_basic : "x86 PV vcpu basic",
- REC_TYPE_x86_pv_vcpu_extended : "x86 PV vcpu extended",
- REC_TYPE_x86_pv_vcpu_xsave : "x86 PV vcpu xsave",
- REC_TYPE_shared_info : "Shared info",
- REC_TYPE_tsc_info : "TSC info",
- REC_TYPE_hvm_context : "HVM context",
- REC_TYPE_hvm_params : "HVM params",
- REC_TYPE_toolstack : "Toolstack",
- REC_TYPE_x86_pv_vcpu_msrs : "x86 PV vcpu msrs",
- REC_TYPE_verify : "Verify",
- REC_TYPE_checkpoint : "Checkpoint",
+ REC_TYPE_end : "End",
+ REC_TYPE_page_data : "Page data",
+ REC_TYPE_x86_pv_info : "x86 PV info",
+ REC_TYPE_x86_pv_p2m_frames : "x86 PV P2M frames",
+ REC_TYPE_x86_pv_vcpu_basic : "x86 PV vcpu basic",
+ REC_TYPE_x86_pv_vcpu_extended : "x86 PV vcpu extended",
+ REC_TYPE_x86_pv_vcpu_xsave : "x86 PV vcpu xsave",
+ REC_TYPE_shared_info : "Shared info",
+ REC_TYPE_tsc_info : "TSC info",
+ REC_TYPE_hvm_context : "HVM context",
+ REC_TYPE_hvm_params : "HVM params",
+ REC_TYPE_toolstack : "Toolstack",
+ REC_TYPE_x86_pv_vcpu_msrs : "x86 PV vcpu msrs",
+ REC_TYPE_verify : "Verify",
+ REC_TYPE_checkpoint : "Checkpoint",
+ REC_TYPE_checkpoint_dirty_pfn_list : "Checkpoint dirty pfn list"
}
# page_data
@@ -403,6 +405,10 @@ class VerifyLibxc(VerifyBase):
if len(content) != 0:
raise RecordError("Checkpoint record with non-zero length")
+ def verify_record_checkpoint_dirty_pfn_list(self, content):
+ """ checkpoint dirty pfn list """
+ raise RecordError("Found checkpoint dirty pfn list record in stream")
+
record_verifiers = {
REC_TYPE_end:
@@ -443,4 +449,6 @@ record_verifiers = {
VerifyLibxc.verify_record_verify,
REC_TYPE_checkpoint:
VerifyLibxc.verify_record_checkpoint,
+ REC_TYPE_checkpoint_dirty_pfn_list:
+ VerifyLibxc.verify_record_checkpoint_dirty_pfn_list,
}
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 08/27] libxc/migration: Specification update for DIRTY_PFN_LIST records
2016-03-04 8:41 ` [PATCH v11 08/27] libxc/migration: Specification update for DIRTY_PFN_LIST records Changlong Xie
@ 2016-03-04 16:53 ` Ian Jackson
2016-03-17 8:10 ` Changlong Xie
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 16:53 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 08/27] libxc/migration: Specification update for DIRTY_PFN_LIST records"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> Used by secondary to send it's dirty bitmap to primary under COLO.
Again, I think this will want a review from Andrew Coooper.
Also, again, I think it would benefit from mentioning COLO in the
record description.
But the principle looks good to me.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 08/27] libxc/migration: Specification update for DIRTY_PFN_LIST records
2016-03-04 16:53 ` Ian Jackson
@ 2016-03-17 8:10 ` Changlong Xie
0 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-17 8:10 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 12:53 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 08/27] libxc/migration: Specification update for DIRTY_PFN_LIST records"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> Used by secondary to send it's dirty bitmap to primary under COLO.
>
> Again, I think this will want a review from Andrew Coooper.
>
> Also, again, I think it would benefit from mentioning COLO in the
> record description.
>
Surely.
Thanks
-Xie
> But the principle looks good to me.
>
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 09/27] libxc/migration: export read_record for common use
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (7 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 08/27] libxc/migration: Specification update for DIRTY_PFN_LIST records Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 16:55 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 10/27] tools/libxl: add back channel support to write stream Changlong Xie
` (18 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
read_record() could be used by primary to read dirty bitmap
record sent by secondary under COLO.
When used by save side, we need to pass the backchannel fd
instead of ctx->fd to read_record(), so we added a fd param to
it.
No functional changes.
CC: Andrew Cooper <andrew.cooper3@citrix.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxc/xc_sr_common.c | 49 +++++++++++++++++++++++++++++++++++
tools/libxc/xc_sr_common.h | 14 ++++++++++
tools/libxc/xc_sr_restore.c | 63 +--------------------------------------------
3 files changed, 64 insertions(+), 62 deletions(-)
diff --git a/tools/libxc/xc_sr_common.c b/tools/libxc/xc_sr_common.c
index 3313a90..b228a15 100644
--- a/tools/libxc/xc_sr_common.c
+++ b/tools/libxc/xc_sr_common.c
@@ -89,6 +89,55 @@ int write_split_record(struct xc_sr_context *ctx, struct xc_sr_record *rec,
return -1;
}
+int read_record(struct xc_sr_context *ctx, int fd, struct xc_sr_record *rec)
+{
+ xc_interface *xch = ctx->xch;
+ struct xc_sr_rhdr rhdr;
+ size_t datasz;
+
+ if ( read_exact(fd, &rhdr, sizeof(rhdr)) )
+ {
+ PERROR("Failed to read Record Header from stream");
+ return -1;
+ }
+ else if ( rhdr.length > REC_LENGTH_MAX )
+ {
+ ERROR("Record (0x%08x, %s) length %#x exceeds max (%#x)", rhdr.type,
+ rec_type_to_str(rhdr.type), rhdr.length, REC_LENGTH_MAX);
+ return -1;
+ }
+
+ datasz = ROUNDUP(rhdr.length, REC_ALIGN_ORDER);
+
+ if ( datasz )
+ {
+ rec->data = malloc(datasz);
+
+ if ( !rec->data )
+ {
+ ERROR("Unable to allocate %zu bytes for record data (0x%08x, %s)",
+ datasz, rhdr.type, rec_type_to_str(rhdr.type));
+ return -1;
+ }
+
+ if ( read_exact(fd, rec->data, datasz) )
+ {
+ free(rec->data);
+ rec->data = NULL;
+ PERROR("Failed to read %zu bytes of data for record (0x%08x, %s)",
+ datasz, rhdr.type, rec_type_to_str(rhdr.type));
+ return -1;
+ }
+ }
+ else
+ rec->data = NULL;
+
+ rec->type = rhdr.type;
+ rec->length = rhdr.length;
+
+ return 0;
+};
+
static void __attribute__((unused)) build_assertions(void)
{
XC_BUILD_BUG_ON(sizeof(struct xc_sr_ihdr) != 24);
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..5d9f497 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -382,6 +382,20 @@ static inline int write_record(struct xc_sr_context *ctx,
}
/*
+ * Reads a record from the stream, and fills in the record structure.
+ *
+ * Returns 0 on success and non-0 on failure.
+ *
+ * On success, the records type and size shall be valid.
+ * - If size is 0, data shall be NULL.
+ * - If size is non-0, data shall be a buffer allocated by malloc() which must
+ * be passed to free() by the caller.
+ *
+ * On failure, the contents of the record structure are undefined.
+ */
+int read_record(struct xc_sr_context *ctx, int fd, struct xc_sr_record *rec);
+
+/*
* This would ideally be private in restore.c, but is needed by
* x86_pv_localise_page() if we receive pagetables frames ahead of the
* contents of the frames they point at.
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index e24560e..aef9bca 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -69,67 +69,6 @@ static int read_headers(struct xc_sr_context *ctx)
}
/*
- * Reads a record from the stream, and fills in the record structure.
- *
- * Returns 0 on success and non-0 on failure.
- *
- * On success, the records type and size shall be valid.
- * - If size is 0, data shall be NULL.
- * - If size is non-0, data shall be a buffer allocated by malloc() which must
- * be passed to free() by the caller.
- *
- * On failure, the contents of the record structure are undefined.
- */
-static int read_record(struct xc_sr_context *ctx, struct xc_sr_record *rec)
-{
- xc_interface *xch = ctx->xch;
- struct xc_sr_rhdr rhdr;
- size_t datasz;
-
- if ( read_exact(ctx->fd, &rhdr, sizeof(rhdr)) )
- {
- PERROR("Failed to read Record Header from stream");
- return -1;
- }
- else if ( rhdr.length > REC_LENGTH_MAX )
- {
- ERROR("Record (0x%08x, %s) length %#x exceeds max (%#x)", rhdr.type,
- rec_type_to_str(rhdr.type), rhdr.length, REC_LENGTH_MAX);
- return -1;
- }
-
- datasz = ROUNDUP(rhdr.length, REC_ALIGN_ORDER);
-
- if ( datasz )
- {
- rec->data = malloc(datasz);
-
- if ( !rec->data )
- {
- ERROR("Unable to allocate %zu bytes for record data (0x%08x, %s)",
- datasz, rhdr.type, rec_type_to_str(rhdr.type));
- return -1;
- }
-
- if ( read_exact(ctx->fd, rec->data, datasz) )
- {
- free(rec->data);
- rec->data = NULL;
- PERROR("Failed to read %zu bytes of data for record (0x%08x, %s)",
- datasz, rhdr.type, rec_type_to_str(rhdr.type));
- return -1;
- }
- }
- else
- rec->data = NULL;
-
- rec->type = rhdr.type;
- rec->length = rhdr.length;
-
- return 0;
-};
-
-/*
* Is a pfn populated?
*/
static bool pfn_is_populated(const struct xc_sr_context *ctx, xen_pfn_t pfn)
@@ -650,7 +589,7 @@ static int restore(struct xc_sr_context *ctx)
do
{
- rc = read_record(ctx, &rec);
+ rc = read_record(ctx, ctx->fd, &rec);
if ( rc )
{
if ( ctx->restore.buffer_all_records )
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 09/27] libxc/migration: export read_record for common use
2016-03-04 8:41 ` [PATCH v11 09/27] libxc/migration: export read_record for common use Changlong Xie
@ 2016-03-04 16:55 ` Ian Jackson
0 siblings, 0 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 16:55 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 09/27] libxc/migration: export read_record for common use"):
> From: Wen Congyang <wency@cn.fujitsu.com>
...
> +int read_record(struct xc_sr_context *ctx, int fd, struct xc_sr_record *rec)
> +{
I haven't checked that this is (apart from the specified change) pure
code motion, but on the basis that it is as described:
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 10/27] tools/libxl: add back channel support to write stream
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (8 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 09/27] libxc/migration: export read_record for common use Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:00 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 11/27] tools/libxl: add back channel support to read stream Changlong Xie
` (17 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Add back channel support to write stream. If the write stream is
a back channel stream, this means the write stream is used by
Secondary to send some records back.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_dom_save.c | 1 +
tools/libxl/libxl_internal.h | 6 +++
tools/libxl/libxl_stream_write.c | 99 ++++++++++++++++++++++++++++++++++++----
3 files changed, 96 insertions(+), 10 deletions(-)
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index 4df86c3..cd324bb 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -405,6 +405,7 @@ void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
dss->sws.ao = dss->ao;
dss->sws.dss = dss;
dss->sws.fd = dss->fd;
+ dss->sws.back_channel = false;
dss->sws.completion_callback = stream_done;
libxl__stream_write_start(egc, &dss->sws);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 5314522..fb8b2fa 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3042,6 +3042,7 @@ struct libxl__stream_write_state {
libxl__ao *ao;
libxl__domain_save_state *dss;
int fd;
+ bool back_channel;
void (*completion_callback)(libxl__egc *egc,
libxl__stream_write_state *sws,
int rc);
@@ -3053,6 +3054,7 @@ struct libxl__stream_write_state {
bool running;
bool in_checkpoint;
bool sync_teardown; /* Only used to coordinate shutdown on error path. */
+ bool in_checkpoint_state;
libxl__save_helper_state shs;
/* Main stream-writing data. */
@@ -3076,6 +3078,10 @@ _hidden void libxl__stream_write_start(libxl__egc *egc,
_hidden void
libxl__stream_write_start_checkpoint(libxl__egc *egc,
libxl__stream_write_state *stream);
+_hidden void
+libxl__stream_write_checkpoint_state(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ libxl_sr_checkpoint_state *srcs);
_hidden void libxl__stream_write_abort(libxl__egc *egc,
libxl__stream_write_state *stream,
int rc);
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
index f6ea55d..8c77d52 100644
--- a/tools/libxl/libxl_stream_write.c
+++ b/tools/libxl/libxl_stream_write.c
@@ -49,6 +49,13 @@
* - if (hvm)
* - Emulator context record
* - Checkpoint end record
+ *
+ * For back channel stream:
+ * - libxl__stream_write_start()
+ * - Set up the stream to running state
+ *
+ * - Use libxl__stream_write_checkpoint_state to write the record. When the
+ * record is written out, call stream->checkpoint_callback() to return.
*/
/* Success/error/cleanup handling. */
@@ -91,6 +98,12 @@ static void write_checkpoint_end_record(libxl__egc *egc,
static void checkpoint_end_record_done(libxl__egc *egc,
libxl__stream_write_state *stream);
+/* checkpoint state */
+static void write_checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_write_state *stream, int rc);
+
/*----- Helpers -----*/
static void write_done(libxl__egc *egc,
@@ -225,6 +238,17 @@ void libxl__stream_write_start(libxl__egc *egc,
stream->running = true;
+ dc->ao = ao;
+ dc->readfd = -1;
+ dc->writewhat = "stream header";
+ dc->copywhat = "save v2 stream";
+ dc->writefd = stream->fd;
+ dc->maxsz = -1;
+ dc->callback = stream_header_done;
+
+ if (stream->back_channel)
+ return;
+
if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
stream->device_model_version =
libxl__device_model_version_running(gc, dss->domid);
@@ -249,14 +273,6 @@ void libxl__stream_write_start(libxl__egc *egc,
stream->emu_sub_hdr.index = 0;
}
- dc->ao = ao;
- dc->readfd = -1;
- dc->writewhat = "stream header";
- dc->copywhat = "save v2 stream";
- dc->writefd = stream->fd;
- dc->maxsz = -1;
- dc->callback = stream_header_done;
-
rc = libxl__datacopier_start(dc);
if (rc)
goto err;
@@ -279,6 +295,7 @@ void libxl__stream_write_start_checkpoint(libxl__egc *egc,
{
assert(stream->running);
assert(!stream->in_checkpoint);
+ assert(!stream->back_channel);
stream->in_checkpoint = true;
write_emulator_xenstore_record(egc, stream);
@@ -577,6 +594,21 @@ static void stream_complete(libxl__egc *egc,
return;
}
+ if (stream->in_checkpoint_state) {
+ assert(rc);
+
+ /*
+ * If an error is encountered while in a checkpoint, pass it
+ * back to libxc. The failure will come back around to us via
+ * 1. normal stream
+ * libxl__xc_domain_save_done()
+ * 2. back_channel stream
+ * libxl__stream_write_abort()
+ */
+ checkpoint_state_done(egc, stream, rc);
+ return;
+ }
+
stream_done(egc, stream, rc);
}
@@ -584,13 +616,24 @@ static void stream_done(libxl__egc *egc,
libxl__stream_write_state *stream, int rc)
{
assert(stream->running);
+ assert(!stream->in_checkpoint_state);
stream->running = false;
if (stream->emu_carefd)
libxl__carefd_close(stream->emu_carefd);
free(stream->emu_body);
- check_all_finished(egc, stream, rc);
+ if (!stream->back_channel) {
+ /*
+ * 1. In stream_done(), stream->running is set to false, so
+ * the stream itself is not in use.
+ * 2. Write stream is a back channel stream, this means it
+ * is only used by secondary(restore side) to send records
+ * back, so it doesn't have save helper.
+ * So we don't need invoke check_all_finished here
+ */
+ check_all_finished(egc, stream, rc);
+ }
}
static void checkpoint_done(libxl__egc *egc,
@@ -642,7 +685,43 @@ static void check_all_finished(libxl__egc *egc,
libxl__save_helper_inuse(&stream->shs))
return;
- stream->completion_callback(egc, stream, stream->rc);
+ if (stream->completion_callback)
+ /* back channel stream doesn't have completion_callback() */
+ stream->completion_callback(egc, stream, stream->rc);
+}
+
+/*----- checkpoint state -----*/
+void libxl__stream_write_checkpoint_state(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ libxl_sr_checkpoint_state *srcs)
+{
+ struct libxl__sr_rec_hdr rec;
+
+ assert(stream->running);
+ assert(!stream->in_checkpoint);
+ assert(!stream->in_checkpoint_state);
+ stream->in_checkpoint_state = true;
+
+ FILLZERO(rec);
+ rec.type = REC_TYPE_CHECKPOINT_STATE;
+ rec.length = sizeof(*srcs);
+
+ setup_write(egc, stream, "checkpoint state", &rec,
+ srcs, write_checkpoint_state_done);
+}
+
+static void write_checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ checkpoint_state_done(egc, stream, 0);
+}
+
+static void checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_write_state *stream, int rc)
+{
+ assert(stream->in_checkpoint_state);
+ stream->in_checkpoint_state = false;
+ stream->checkpoint_callback(egc, stream, rc);
}
/*
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 10/27] tools/libxl: add back channel support to write stream
2016-03-04 8:41 ` [PATCH v11 10/27] tools/libxl: add back channel support to write stream Changlong Xie
@ 2016-03-04 17:00 ` Ian Jackson
2016-03-07 2:13 ` Wen Congyang
` (2 more replies)
0 siblings, 3 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:00 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 10/27] tools/libxl: add back channel support to write stream"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> Add back channel support to write stream. If the write stream is
> a back channel stream, this means the write stream is used by
> Secondary to send some records back.
The general idea seems fine to me but I want an opinion from Andrew.
If I'm not mistaken there is no call site for this yet. In which case
this should be mentioned in the commit message.
> +/*----- checkpoint state -----*/
> +void libxl__stream_write_checkpoint_state(libxl__egc *egc,
> + libxl__stream_write_state *stream,
> + libxl_sr_checkpoint_state *srcs)
Firstly, missing blank line.
Secondly, reading all this leads me to think that maybe the
`checkpoint_state' record should be called something different. Is it
only ever going to be used for COLO ? Maybe it should be
`COLOHA_STATE' or something (and all the functions etc. renamed
consequently) ?
What do you think ?
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 10/27] tools/libxl: add back channel support to write stream
2016-03-04 17:00 ` Ian Jackson
@ 2016-03-07 2:13 ` Wen Congyang
2016-03-11 9:05 ` Wen Congyang
2016-03-17 8:11 ` Changlong Xie
2 siblings, 0 replies; 100+ messages in thread
From: Wen Congyang @ 2016-03-07 2:13 UTC (permalink / raw)
To: Ian Jackson, Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Andrew Cooper, Jiang Yunhong,
Dong Eddie, xen devel, Anthony Perard, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:00 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 10/27] tools/libxl: add back channel support to write stream"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> Add back channel support to write stream. If the write stream is
>> a back channel stream, this means the write stream is used by
>> Secondary to send some records back.
>
> The general idea seems fine to me but I want an opinion from Andrew.
>
> If I'm not mistaken there is no call site for this yet. In which case
> this should be mentioned in the commit message.
>
>> +/*----- checkpoint state -----*/
>> +void libxl__stream_write_checkpoint_state(libxl__egc *egc,
>> + libxl__stream_write_state *stream,
>> + libxl_sr_checkpoint_state *srcs)
>
> Firstly, missing blank line.
>
> Secondly, reading all this leads me to think that maybe the
> `checkpoint_state' record should be called something different. Is it
> only ever going to be used for COLO ? Maybe it should be
Yes, it is only used for COLO now.
> `COLOHA_STATE' or something (and all the functions etc. renamed
> consequently) ?
>
> What do you think ?
COLO is FT, not HA. What aboyt COLOFT_STATE?
Thanks
Wen Congyang
>
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 10/27] tools/libxl: add back channel support to write stream
2016-03-04 17:00 ` Ian Jackson
2016-03-07 2:13 ` Wen Congyang
@ 2016-03-11 9:05 ` Wen Congyang
2016-03-17 8:11 ` Changlong Xie
2 siblings, 0 replies; 100+ messages in thread
From: Wen Congyang @ 2016-03-11 9:05 UTC (permalink / raw)
To: Ian Jackson, Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Andrew Cooper, Jiang Yunhong,
Dong Eddie, xen devel, Anthony Perard, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:00 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 10/27] tools/libxl: add back channel support to write stream"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> Add back channel support to write stream. If the write stream is
>> a back channel stream, this means the write stream is used by
>> Secondary to send some records back.
>
> The general idea seems fine to me but I want an opinion from Andrew.
>
> If I'm not mistaken there is no call site for this yet. In which case
> this should be mentioned in the commit message.
>
>> +/*----- checkpoint state -----*/
>> +void libxl__stream_write_checkpoint_state(libxl__egc *egc,
>> + libxl__stream_write_state *stream,
>> + libxl_sr_checkpoint_state *srcs)
>
> Firstly, missing blank line.
>
> Secondly, reading all this leads me to think that maybe the
> `checkpoint_state' record should be called something different. Is it
> only ever going to be used for COLO ? Maybe it should be
> `COLOHA_STATE' or something (and all the functions etc. renamed
> consequently) ?
Another question: what should be renamed? The function name or the type name
or both?
Thanks
Wen Congyang
>
> What do you think ?
>
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 10/27] tools/libxl: add back channel support to write stream
2016-03-04 17:00 ` Ian Jackson
2016-03-07 2:13 ` Wen Congyang
2016-03-11 9:05 ` Wen Congyang
@ 2016-03-17 8:11 ` Changlong Xie
2 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-17 8:11 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:00 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 10/27] tools/libxl: add back channel support to write stream"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> Add back channel support to write stream. If the write stream is
>> a back channel stream, this means the write stream is used by
>> Secondary to send some records back.
>
> The general idea seems fine to me but I want an opinion from Andrew.
>
> If I'm not mistaken there is no call site for this yet. In which case
> this should be mentioned in the commit message.
ok
Thanks
-Xie
>
>> +/*----- checkpoint state -----*/
>> +void libxl__stream_write_checkpoint_state(libxl__egc *egc,
>> + libxl__stream_write_state *stream,
>> + libxl_sr_checkpoint_state *srcs)
>
> Firstly, missing blank line.
>
> Secondly, reading all this leads me to think that maybe the
> `checkpoint_state' record should be called something different. Is it
> only ever going to be used for COLO ? Maybe it should be
> `COLOHA_STATE' or something (and all the functions etc. renamed
> consequently) ?
>
> What do you think ?
>
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 11/27] tools/libxl: add back channel support to read stream
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (9 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 10/27] tools/libxl: add back channel support to write stream Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:01 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 12/27] tools/libx{l, c}: introduce wait_checkpoint callback Changlong Xie
` (16 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
This is used by primay to read records sent by secondary.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_create.c | 1 +
tools/libxl/libxl_internal.h | 4 ++
tools/libxl/libxl_stream_read.c | 94 +++++++++++++++++++++++++++++++++++++----
3 files changed, 91 insertions(+), 8 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 525bf85..fc746fb 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1030,6 +1030,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
dcs->srs.dcs = dcs;
dcs->srs.fd = restore_fd;
dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
+ dcs->srs.back_channel = false;
dcs->srs.completion_callback = domcreate_stream_done;
if (restore_fd >= 0) {
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index fb8b2fa..6307b71 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3418,6 +3418,7 @@ struct libxl__stream_read_state {
libxl__domain_create_state *dcs;
int fd;
bool legacy;
+ bool back_channel;
void (*completion_callback)(libxl__egc *egc,
libxl__stream_read_state *srs,
int rc);
@@ -3429,6 +3430,7 @@ struct libxl__stream_read_state {
bool running;
bool in_checkpoint;
bool sync_teardown; /* Only used to coordinate shutdown on error path. */
+ bool in_checkpoint_state;
libxl__save_helper_state shs;
libxl__conversion_helper_state chs;
@@ -3456,6 +3458,8 @@ _hidden void libxl__stream_read_start(libxl__egc *egc,
libxl__stream_read_state *stream);
_hidden void libxl__stream_read_start_checkpoint(libxl__egc *egc,
libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_checkpoint_state(libxl__egc *egc,
+ libxl__stream_read_state *stream);
_hidden void libxl__stream_read_abort(libxl__egc *egc,
libxl__stream_read_state *stream, int rc);
static inline bool
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index f4781eb..302ae53 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -118,6 +118,15 @@
* record, and therefore the buffered state is inconsistent. In
* libxl__xc_domain_restore_done(), we just complete the stream and
* stream->completion_callback() will be called to resume the guest
+ *
+ * For back channel stream:
+ * - libxl__stream_read_start()
+ * - Set up the stream to running state
+ *
+ * - libxl__stream_read_continue()
+ * - Set up reading the next record from a started stream.
+ * Add some codes to process_record() to handle the record.
+ * Then call stream->checkpoint_callback() to return.
*/
/* Success/error/cleanup handling. */
@@ -157,6 +166,10 @@ static void write_emulator_done(libxl__egc *egc,
libxl__datacopier_state *dc,
int rc, int onwrite, int errnoval);
+/* Handlers for checkpoint state mini-loop */
+static void checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
+
/*----- Helpers -----*/
/* Helper to set up reading some data from the stream. */
@@ -221,6 +234,14 @@ void libxl__stream_read_start(libxl__egc *egc,
stream->running = true;
stream->phase = SRS_PHASE_NORMAL;
+ dc->ao = stream->ao;
+ dc->copywhat = "restore v2 stream";
+ dc->readfd = stream->fd;
+ dc->writefd = -1;
+
+ if (stream->back_channel)
+ return;
+
if (stream->legacy) {
/* Convert the legacy stream. */
libxl__conversion_helper_state *chs = &stream->chs;
@@ -241,12 +262,6 @@ void libxl__stream_read_start(libxl__egc *egc,
stream->fd = libxl__carefd_fd(stream->chs.v2_carefd);
stream->dcs->libxc_fd = stream->fd;
}
- /* stream->fd is now a v2 stream. */
-
- dc->ao = stream->ao;
- dc->copywhat = "restore v2 stream";
- dc->readfd = stream->fd;
- dc->writefd = -1;
/* Start reading the stream header. */
rc = setup_read(stream, "stream header",
@@ -532,6 +547,7 @@ static bool process_record(libxl__egc *egc,
STATE_AO_GC(stream->ao);
libxl__domain_create_state *dcs = stream->dcs;
libxl__sr_record_buf *rec;
+ libxl_sr_checkpoint_state *srcs;
bool further_action_needed = false;
int rc = 0;
@@ -602,6 +618,17 @@ static bool process_record(libxl__egc *egc,
checkpoint_done(egc, stream, 0);
break;
+ case REC_TYPE_CHECKPOINT_STATE:
+ if (!stream->in_checkpoint_state) {
+ LOG(ERROR, "Unexpected CHECKPOINT_STATE record in stream");
+ rc = ERROR_FAIL;
+ goto err;
+ }
+
+ srcs = rec->body;
+ checkpoint_state_done(egc, stream, srcs->id);
+ break;
+
default:
LOG(ERROR, "Unrecognised record 0x%08x", rec->hdr.type);
rc = ERROR_FAIL;
@@ -713,6 +740,21 @@ static void stream_complete(libxl__egc *egc,
return;
}
+ if (stream->in_checkpoint_state) {
+ assert(rc);
+
+ /*
+ * If an error is encountered while in a checkpoint, pass it
+ * back to libxc. The failure will come back around to us via
+ * 1. normal stream
+ * libxl__xc_domain_restore_done()
+ * 2. back_channel stream
+ * libxl__stream_read_abort()
+ */
+ checkpoint_state_done(egc, stream, rc);
+ return;
+ }
+
stream_done(egc, stream, rc);
}
@@ -743,6 +785,7 @@ static void stream_done(libxl__egc *egc,
assert(stream->running);
assert(!stream->in_checkpoint);
+ assert(!stream->in_checkpoint_state);
stream->running = false;
if (stream->incoming_record)
@@ -762,7 +805,19 @@ static void stream_done(libxl__egc *egc,
LIBXL_STAILQ_FOREACH_SAFE(rec, &stream->record_queue, entry, trec)
free_record(rec);
- check_all_finished(egc, stream, rc);
+ if (!stream->back_channel) {
+ /*
+ * 1. In stream_done(), stream->running is set to false, so
+ * the stream itself is not in use.
+ * 2. Read stream is a back channel stream, this means it is
+ * only used by primary(save side) to read records sent by
+ * secondary(restore side), so it doesn't have restore helper.
+ * 3. Back channel stream doesn't support legacy stream, so
+ * there is no conversion helper.
+ * So we don't need invoke check_all_finished here
+ */
+ check_all_finished(egc, stream, rc);
+ }
}
void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
@@ -862,7 +917,30 @@ static void check_all_finished(libxl__egc *egc,
libxl__conversion_helper_inuse(&stream->chs))
return;
- stream->completion_callback(egc, stream, stream->rc);
+ if (stream->completion_callback)
+ /* back channel stream doesn't have completion_callback() */
+ stream->completion_callback(egc, stream, stream->rc);
+}
+
+/*----- Checkpoint state handlers -----*/
+
+void libxl__stream_read_checkpoint_state(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ assert(stream->running);
+ assert(!stream->in_checkpoint);
+ assert(!stream->in_checkpoint_state);
+ stream->in_checkpoint_state = true;
+
+ setup_read_record(egc, stream);
+}
+
+static void checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc)
+{
+ assert(stream->in_checkpoint_state);
+ stream->in_checkpoint_state = false;
+ stream->checkpoint_callback(egc, stream, rc);
}
/*
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 11/27] tools/libxl: add back channel support to read stream
2016-03-04 8:41 ` [PATCH v11 11/27] tools/libxl: add back channel support to read stream Changlong Xie
@ 2016-03-04 17:01 ` Ian Jackson
0 siblings, 0 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:01 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 11/27] tools/libxl: add back channel support to read stream"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> This is used by primay to read records sent by secondary.
I haven't reviewed the flow here in detail but the general idea seems
sound.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 12/27] tools/libx{l, c}: introduce wait_checkpoint callback
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (10 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 11/27] tools/libxl: add back channel support to read stream Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:03 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 13/27] tools/libx{l, c}: add postcopy/suspend callback to restore side Changlong Xie
` (15 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Under COLO, we are doing checkpoint on demand, if this
callback returns 1, we will take another checkpoint.
0 indicates unexpected error.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxc/include/xenguest.h | 18 ++++++++++++++++++
tools/libxl/libxl_save_msgs_gen.pl | 7 ++++---
2 files changed, 22 insertions(+), 3 deletions(-)
diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index 8040ac7..f598bec 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -68,6 +68,15 @@ struct save_callbacks {
* 1: take another checkpoint */
int (*checkpoint)(void* data);
+ /*
+ * Called after the checkpoint callback.
+ *
+ * returns:
+ * 0: terminate checkpointing gracefully
+ * 1: take another checkpoint
+ */
+ int (*wait_checkpoint)(void* data);
+
/* Enable qemu-dm logging dirty pages to xen */
int (*switch_qemu_logdirty)(int domid, unsigned enable, void *data); /* HVM only */
@@ -99,6 +108,15 @@ struct restore_callbacks {
#define XGR_CHECKPOINT_FAILOVER 2 /* Failover and resume VM */
int (*checkpoint)(void* data);
+ /*
+ * Called after the checkpoint callback.
+ *
+ * returns:
+ * 0: terminate checkpointing gracefully
+ * 1: take another checkpoint
+ */
+ int (*wait_checkpoint)(void* data);
+
/* to be provided as the last argument to each callback function */
void* data;
};
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index d6d2967..51d004d 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -26,11 +26,12 @@ our @msgs = (
[ 3, 'scxA', "suspend", [] ],
[ 4, 'scxA', "postcopy", [] ],
[ 5, 'srcxA', "checkpoint", [] ],
- [ 6, 'scxA', "switch_qemu_logdirty", [qw(int domid
+ [ 6, 'srcxA', "wait_checkpoint", [] ],
+ [ 7, 'scxA', "switch_qemu_logdirty", [qw(int domid
unsigned enable)] ],
- [ 7, 'r', "restore_results", ['unsigned long', 'store_mfn',
+ [ 8, 'r', "restore_results", ['unsigned long', 'store_mfn',
'unsigned long', 'console_mfn'] ],
- [ 8, 'srW', "complete", [qw(int retval
+ [ 9, 'srW', "complete", [qw(int retval
int errnoval)] ],
);
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 12/27] tools/libx{l, c}: introduce wait_checkpoint callback
2016-03-04 8:41 ` [PATCH v11 12/27] tools/libx{l, c}: introduce wait_checkpoint callback Changlong Xie
@ 2016-03-04 17:03 ` Ian Jackson
2016-03-04 20:23 ` Konrad Rzeszutek Wilk
2016-03-17 8:16 ` Changlong Xie
0 siblings, 2 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:03 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 12/27] tools/libx{l,c}: introduce wait_checkpoint callback"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> Under COLO, we are doing checkpoint on demand, if this
> callback returns 1, we will take another checkpoint.
> 0 indicates unexpected error.
This doesn't seem to have a corresponding implementation. I think the
implementation ought to be in the same patch.
If 0 is always an `unexpected error', perhaps the return value should
be an error code or something ? I'm not sure.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 12/27] tools/libx{l, c}: introduce wait_checkpoint callback
2016-03-04 17:03 ` Ian Jackson
@ 2016-03-04 20:23 ` Konrad Rzeszutek Wilk
2016-03-07 2:16 ` Wen Congyang
2016-03-17 8:16 ` Changlong Xie
1 sibling, 1 reply; 100+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-03-04 20:23 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Anthony Perard, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On Fri, Mar 04, 2016 at 05:03:16PM +0000, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 12/27] tools/libx{l,c}: introduce wait_checkpoint callback"):
> > From: Wen Congyang <wency@cn.fujitsu.com>
> >
> > Under COLO, we are doing checkpoint on demand, if this
> > callback returns 1, we will take another checkpoint.
> > 0 indicates unexpected error.
>
> This doesn't seem to have a corresponding implementation. I think the
> implementation ought to be in the same patch.
>
> If 0 is always an `unexpected error', perhaps the return value should
> be an error code or something ? I'm not sure.
I struggled with this API.
I like the idea of that negative value would imply 'unexpected error'.
And 1 for 'OK, take another checkpoint'. Not sure if zero would be a valid
return value..
>
> Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 12/27] tools/libx{l, c}: introduce wait_checkpoint callback
2016-03-04 20:23 ` Konrad Rzeszutek Wilk
@ 2016-03-07 2:16 ` Wen Congyang
0 siblings, 0 replies; 100+ messages in thread
From: Wen Congyang @ 2016-03-07 2:16 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, Ian Jackson
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 04:23 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 04, 2016 at 05:03:16PM +0000, Ian Jackson wrote:
>> Changlong Xie writes ("[PATCH v11 12/27] tools/libx{l,c}: introduce wait_checkpoint callback"):
>>> From: Wen Congyang <wency@cn.fujitsu.com>
>>>
>>> Under COLO, we are doing checkpoint on demand, if this
>>> callback returns 1, we will take another checkpoint.
>>> 0 indicates unexpected error.
>>
>> This doesn't seem to have a corresponding implementation. I think the
>> implementation ought to be in the same patch.
>>
>> If 0 is always an `unexpected error', perhaps the return value should
>> be an error code or something ? I'm not sure.
>
> I struggled with this API.
>
> I like the idea of that negative value would imply 'unexpected error'.
> And 1 for 'OK, take another checkpoint'. Not sure if zero would be a valid
> return value..
IIRC, save/restore callback always use 0 for unexpected error, 1 for OK.
negative value for pipe is broken.
Thanks
Wen Congyang
>
>
>>
>> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH v11 12/27] tools/libx{l, c}: introduce wait_checkpoint callback
2016-03-04 17:03 ` Ian Jackson
2016-03-04 20:23 ` Konrad Rzeszutek Wilk
@ 2016-03-17 8:16 ` Changlong Xie
1 sibling, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-17 8:16 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:03 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 12/27] tools/libx{l,c}: introduce wait_checkpoint callback"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> Under COLO, we are doing checkpoint on demand, if this
>> callback returns 1, we will take another checkpoint.
>> 0 indicates unexpected error.
>
> This doesn't seem to have a corresponding implementation. I think the
> implementation ought to be in the same patch.
>
Surely, will merge it in the implement patch in next version
Thanks
-Xie
> If 0 is always an `unexpected error', perhaps the return value should
> be an error code or something ? I'm not sure.
>
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 13/27] tools/libx{l, c}: add postcopy/suspend callback to restore side
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (11 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 12/27] tools/libx{l, c}: introduce wait_checkpoint callback Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:05 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 14/27] secondary vm suspend/resume/checkpoint code Changlong Xie
` (14 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Secondary(restore side) is running under COLO, we also need
postcopy/suspend callbacks.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxc/include/xenguest.h | 10 ++++++++++
tools/libxl/libxl_save_msgs_gen.pl | 4 ++--
2 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index f598bec..cb0bbcf 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -101,6 +101,16 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iter
/* callbacks provided by xc_domain_restore */
struct restore_callbacks {
+ /* Called after a new checkpoint to suspend the guest.
+ */
+ int (*suspend)(void* data);
+
+ /* Called after the secondary vm is ready to resume.
+ * Callback function resumes the guest & the device model,
+ * returns to xc_domain_restore.
+ */
+ int (*postcopy)(void* data);
+
/* A checkpoint record has been found in the stream.
* returns: */
#define XGR_CHECKPOINT_ERROR 0 /* Terminate processing */
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index 51d004d..6016706 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -23,8 +23,8 @@ our @msgs = (
STRING doing_what),
'unsigned long', 'done',
'unsigned long', 'total'] ],
- [ 3, 'scxA', "suspend", [] ],
- [ 4, 'scxA', "postcopy", [] ],
+ [ 3, 'srcxA', "suspend", [] ],
+ [ 4, 'srcxA', "postcopy", [] ],
[ 5, 'srcxA', "checkpoint", [] ],
[ 6, 'srcxA', "wait_checkpoint", [] ],
[ 7, 'scxA', "switch_qemu_logdirty", [qw(int domid
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 13/27] tools/libx{l, c}: add postcopy/suspend callback to restore side
2016-03-04 8:41 ` [PATCH v11 13/27] tools/libx{l, c}: add postcopy/suspend callback to restore side Changlong Xie
@ 2016-03-04 17:05 ` Ian Jackson
2016-03-17 8:17 ` Changlong Xie
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:05 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 13/27] tools/libx{l,c}: add postcopy/suspend callback to restore side"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> Secondary(restore side) is running under COLO, we also need
> postcopy/suspend callbacks.
This patch does not provide an implementation of any of these
callbacks, nor a call site.
I think you need to mention that in the commit message. You should
also explain what would currently happen in libxl if these callbacks
were to be called.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 13/27] tools/libx{l, c}: add postcopy/suspend callback to restore side
2016-03-04 17:05 ` Ian Jackson
@ 2016-03-17 8:17 ` Changlong Xie
0 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-17 8:17 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:05 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 13/27] tools/libx{l,c}: add postcopy/suspend callback to restore side"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> Secondary(restore side) is running under COLO, we also need
>> postcopy/suspend callbacks.
>
> This patch does not provide an implementation of any of these
> callbacks, nor a call site.
>
> I think you need to mention that in the commit message. You should
> also explain what would currently happen in libxl if these callbacks
> were to be called.
I will merge it in the implement patch in next version
Thanks
-Xie
>
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 14/27] secondary vm suspend/resume/checkpoint code
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (12 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 13/27] tools/libx{l, c}: add postcopy/suspend callback to restore side Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:11 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 15/27] primary " Changlong Xie
` (13 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Secondary vm is running in colo mode. So we will do
the following things again and again:
1. Resume secondary vm
a. Send CHECKPOINT_SVM_READY to master.
b. If it is not the first resume, call libxl__checkpoint_devices_preresume().
c. If it is the first resume(resume right after live migration),
- call libxl__xc_domain_restore_done() to build the secondary vm.
- enable secondary vm's logdirty.
- call libxl__domain_resume() to resume secondary vm.
- call libxl__checkpoint_devices_setup() to setup checkpoint devices.
d. Send CHECKPOINT_SVM_RESUMED to master.
2. Wait a new checkpoint
a. Call libxl__checkpoint_devices_commit().
b. Read CHECKPOINT_NEW from master.
3. Suspend secondary vm
a. Suspend secondary vm.
b. Call libxl__checkpoint_devices_postsuspend().
c. Send CHECKPOINT_SVM_SUSPENDED to master.
4. Checkpoint
a. Read emulator xenstore data and emulator context
b. REC_TYPE_CHECKPOINT_END
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxc/xc_sr_common.h | 2 +
tools/libxc/xc_sr_save.c | 3 +-
tools/libxl/Makefile | 1 +
tools/libxl/libxl_colo.h | 55 ++
tools/libxl/libxl_colo_restore.c | 1025 ++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_create.c | 36 ++
tools/libxl/libxl_internal.h | 11 +-
tools/libxl/libxl_save_callout.c | 6 +-
tools/libxl/libxl_stream_read.c | 12 +
tools/libxl/libxl_types.idl | 1 +
10 files changed, 1143 insertions(+), 9 deletions(-)
create mode 100644 tools/libxl/libxl_colo.h
create mode 100644 tools/libxl/libxl_colo_restore.c
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 5d9f497..2bfed64 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -184,10 +184,12 @@ struct xc_sr_context
* migration stream
* 0: Plain VM
* 1: Remus
+ * 2: COLO
*/
enum {
MIG_STREAM_NONE, /* plain stream */
MIG_STREAM_REMUS,
+ MIG_STREAM_COLO,
} migration_stream;
union /* Common save or restore data. */
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index fe210cc..7393355 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -846,7 +846,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
/* If altering migration_stream update this assert too. */
assert(checkpointed_stream == MIG_STREAM_NONE ||
- checkpointed_stream == MIG_STREAM_REMUS);
+ checkpointed_stream == MIG_STREAM_REMUS ||
+ checkpointed_stream == MIG_STREAM_COLO);
/*
* TODO: Find some time to better tweak the live migration algorithm.
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 789a12e..d8612eb 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -65,6 +65,7 @@ LIBXL_OBJS-y += libxl_no_convert_callout.o
endif
LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
+LIBXL_OBJS-y += libxl_colo_restore.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
new file mode 100644
index 0000000..f2b98cc
--- /dev/null
+++ b/tools/libxl/libxl_colo.h
@@ -0,0 +1,55 @@
+/*
+ * Copyright (C) 2016 FUJITSU LIMITED
+ * Author: Wen Congyang <wency@cn.fujitsu.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#ifndef LIBXL_COLO_H
+#define LIBXL_COLO_H
+
+struct libxl__ao;
+struct libxl__egc;
+
+enum {
+ LIBXL_COLO_SETUPED,
+ LIBXL_COLO_SUSPENDED,
+ LIBXL_COLO_RESUMED,
+};
+
+typedef struct libxl__domain_create_state libxl__domain_create_state;
+typedef void libxl__domain_create_cb(struct libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int rc, uint32_t domid);
+
+typedef struct libxl__colo_restore_state libxl__colo_restore_state;
+typedef void libxl__colo_callback(struct libxl__egc *egc,
+ libxl__colo_restore_state *crs, int rc);
+
+struct libxl__colo_restore_state {
+ /* must set by caller of libxl__colo_(setup|teardown) */
+ struct libxl__ao *ao;
+ uint32_t domid;
+ int send_back_fd;
+ int recv_fd;
+ int hvm;
+ libxl__colo_callback *callback;
+
+ /* private, colo restore checkpoint state */
+ libxl__domain_create_cb *saved_cb;
+ void *crcs;
+};
+
+extern void libxl__colo_restore_setup(struct libxl__egc *egc,
+ libxl__colo_restore_state *crs);
+extern void libxl__colo_restore_teardown(struct libxl__egc *egc, void *dcs_void,
+ int ret, int retval, int errnoval);
+#endif
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
new file mode 100644
index 0000000..829687e
--- /dev/null
+++ b/tools/libxl/libxl_colo_restore.c
@@ -0,0 +1,1025 @@
+/*
+ * Copyright (C) 2016 FUJITSU LIMITED
+ * Author: Wen Congyang <wency@cn.fujitsu.com>
+ * Yang Hongyang <hongyang.yang@easystack.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+#include "libxl_sr_stream_format.h"
+
+typedef struct libxl__colo_restore_checkpoint_state libxl__colo_restore_checkpoint_state;
+struct libxl__colo_restore_checkpoint_state {
+ libxl__domain_suspend_state dsps;
+ libxl__logdirty_switch lds;
+ libxl__colo_restore_state *crs;
+ libxl__stream_write_state sws;
+ int status;
+ bool preresume;
+ /* used for teardown */
+ int teardown_devices;
+ int saved_rc;
+ char *state_file;
+
+ void (*callback)(libxl__egc *,
+ libxl__colo_restore_checkpoint_state *,
+ int);
+};
+
+static const libxl__checkpoint_device_instance_ops *colo_restore_ops[] = {
+ NULL,
+};
+
+/* ===================== colo: common functions ===================== */
+static void colo_enable_logdirty(libxl__colo_restore_state *crs, libxl__egc *egc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ const uint32_t domid = crs->domid;
+ libxl__logdirty_switch *const lds = &crcs->lds;
+
+ EGC_GC;
+
+ /* we need to know which pages are dirty to restore the guest */
+ if (xc_shadow_control(CTX->xch, domid,
+ XEN_DOMCTL_SHADOW_OP_ENABLE_LOGDIRTY,
+ NULL, 0, NULL, 0, NULL) < 0) {
+ LOG(ERROR, "cannot enable secondary vm's logdirty");
+ lds->callback(egc, lds, ERROR_FAIL);
+ return;
+ }
+
+ if (crs->hvm) {
+ libxl__domain_common_switch_qemu_logdirty(egc, domid, 1, lds);
+ return;
+ }
+
+ lds->callback(egc, lds, 0);
+}
+
+static void colo_disable_logdirty(libxl__colo_restore_state *crs,
+ libxl__egc *egc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ const uint32_t domid = crs->domid;
+ libxl__logdirty_switch *const lds = &crcs->lds;
+
+ EGC_GC;
+
+ /* we need to know which pages are dirty to restore the guest */
+ if (xc_shadow_control(CTX->xch, domid, XEN_DOMCTL_SHADOW_OP_OFF,
+ NULL, 0, NULL, 0, NULL) < 0)
+ LOG(WARN, "cannot disable secondary vm's logdirty");
+
+ if (crs->hvm) {
+ libxl__domain_common_switch_qemu_logdirty(egc, domid, 0, lds);
+ return;
+ }
+
+ lds->callback(egc, lds, 0);
+}
+
+static void colo_resume_vm(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int restore_device_model)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+ int rc;
+
+ /* Convenience aliases */
+ libxl__colo_restore_state *const crs = crcs->crs;
+
+ EGC_GC;
+
+ if (!crs->saved_cb) {
+ /* TODO: sync mmu for hvm? */
+ if (restore_device_model) {
+ rc = libxl__qmp_restore(gc, crs->domid, crcs->state_file);
+ if (rc) {
+ LOG(ERROR, "cannot restore device model for secondary vm");
+ crcs->callback(egc, crcs, rc);
+ return;
+ }
+ }
+ rc = libxl__domain_resume(gc, crs->domid, 0);
+ if (rc)
+ LOG(ERROR, "cannot resume secondary vm");
+
+ crcs->callback(egc, crcs, rc);
+ return;
+ }
+
+ /*
+ * TODO: get store gfn and console gfn
+ * We should call the callback restore_results in
+ * xc_domain_restore() before resuming the guest.
+ */
+ libxl__xc_domain_restore_done(egc, dcs, 0, 0, 0);
+
+ return;
+}
+
+static int init_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+ /* init device subkind-specific state in the libxl ctx */
+ int rc;
+ STATE_AO_GC(cds->ao);
+
+ rc = 0;
+ return rc;
+}
+
+static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+ /* cleanup device subkind-specific state in the libxl ctx */
+ STATE_AO_GC(cds->ao);
+}
+
+
+/* ================ colo: setup restore environment ================ */
+static void libxl__colo_domain_create_cb(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int rc, uint32_t domid);
+
+static int init_dsps(libxl__domain_suspend_state *dsps)
+{
+ int rc = ERROR_FAIL;
+ libxl_domain_type type;
+
+ STATE_AO_GC(dsps->ao);
+
+ libxl__xswait_init(&dsps->pvcontrol);
+ libxl__ev_evtchn_init(&dsps->guest_evtchn);
+ libxl__ev_xswatch_init(&dsps->guest_watch);
+ libxl__ev_time_init(&dsps->guest_timeout);
+
+ type = libxl__domain_type(gc, dsps->domid);
+ if (type == LIBXL_DOMAIN_TYPE_INVALID)
+ goto out;
+
+ dsps->type = type;
+
+ dsps->guest_evtchn.port = -1;
+ dsps->guest_evtchn_lockfd = -1;
+ dsps->guest_responded = 0;
+ dsps->dm_savefile = libxl__device_model_savefile(gc, dsps->domid);
+
+ /* Secondary vm is not created, so we cannot get evtchn port */
+
+ rc = 0;
+
+out:
+ return rc;
+}
+
+/*
+ * checkpoint callbacks are called in the following order:
+ * 1. resume
+ * 2. wait checkpoint
+ * 3. suspend
+ * 4. checkpoint
+ */
+static void libxl__colo_restore_domain_resume_callback(void *data);
+static void libxl__colo_restore_domain_wait_checkpoint_callback(void *data);
+static void libxl__colo_restore_domain_suspend_callback(void *data);
+static void libxl__colo_restore_domain_checkpoint_callback(void *data);
+
+void libxl__colo_restore_setup(libxl__egc *egc,
+ libxl__colo_restore_state *crs)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs;
+ int rc = ERROR_FAIL;
+
+ /* Convenience aliases */
+ libxl__srm_restore_autogen_callbacks *const callbacks =
+ &dcs->srs.shs.callbacks.restore.a;
+ const int domid = crs->domid;
+
+ STATE_AO_GC(crs->ao);
+
+ GCNEW(crcs);
+ crs->crcs = crcs;
+ crcs->crs = crs;
+
+ /* setup dsps */
+ crcs->dsps.ao = ao;
+ crcs->dsps.domid = domid;
+ if (init_dsps(&crcs->dsps))
+ goto out;
+
+ callbacks->postcopy = libxl__colo_restore_domain_resume_callback;
+ callbacks->wait_checkpoint = libxl__colo_restore_domain_wait_checkpoint_callback;
+ callbacks->suspend = libxl__colo_restore_domain_suspend_callback;
+ callbacks->checkpoint = libxl__colo_restore_domain_checkpoint_callback;
+
+ /*
+ * Secondary vm is running in colo mode, so we need to call
+ * libxl__xc_domain_restore_done() to create secondary vm.
+ * But we will exit in domain_create_cb(). So replace the
+ * callback here.
+ */
+ crs->saved_cb = dcs->callback;
+ dcs->callback = libxl__colo_domain_create_cb;
+ crcs->state_file = GCSPRINTF(LIBXL_DEVICE_MODEL_RESTORE_FILE".%d", domid);
+ crcs->status = LIBXL_COLO_SETUPED;
+
+ libxl__logdirty_init(&crcs->lds);
+ crcs->lds.ao = ao;
+
+ crcs->sws.fd = crs->send_back_fd;
+ crcs->sws.ao = ao;
+ crcs->sws.back_channel = true;
+
+ dcs->cds.concrete_data = crs;
+
+ libxl__stream_write_start(egc, &crcs->sws);
+
+ rc = 0;
+
+out:
+ crs->callback(egc, crs, rc);
+ return;
+}
+
+static void libxl__colo_domain_create_cb(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int rc, uint32_t domid)
+{
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+
+ crcs->callback(egc, crcs, rc);
+}
+
+/* ================ colo: teardown restore environment ================ */
+static void colo_restore_teardown_devices_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds, int rc);
+static void do_failover(libxl__egc *egc, libxl__colo_restore_state *crs);
+static void do_failover_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state* crcs,
+ int rc);
+static void colo_disable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc);
+static void libxl__colo_restore_teardown_done(libxl__egc *egc,
+ libxl__colo_restore_state *crs,
+ int rc);
+
+void libxl__colo_restore_teardown(libxl__egc *egc, void *dcs_void,
+ int ret, int retval, int errnoval)
+{
+ libxl__domain_create_state *dcs = dcs_void;
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+ int rc = 1;
+
+ /* convenience aliases */
+ libxl__colo_restore_state *const crs = &dcs->crs;
+ EGC_GC;
+
+ if (ret == 0 && retval == 0)
+ rc = 0;
+
+ LOG(INFO, "%s", rc ? "colo fails" : "failover");
+
+ libxl__stream_write_abort(egc, &crcs->sws, 1);
+ if (crs->saved_cb) {
+ /* crcs->status is LIBXL_COLO_SETUPED */
+ dcs->srs.completion_callback = NULL;
+ }
+ libxl__xc_domain_restore_done(egc, dcs, ret, retval, errnoval);
+
+ crcs->saved_rc = rc;
+ if (!crcs->teardown_devices) {
+ colo_restore_teardown_devices_done(egc, &dcs->cds, 0);
+ return;
+ }
+
+ dcs->cds.callback = colo_restore_teardown_devices_done;
+ libxl__checkpoint_devices_teardown(egc, &dcs->cds);
+}
+
+static void colo_restore_teardown_devices_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds, int rc)
+{
+ libxl__colo_restore_state *crs = cds->concrete_data;
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+
+ EGC_GC;
+
+ if (rc)
+ LOG(ERROR, "COLO: failed to teardown device for guest with domid %u,"
+ " rc %d", cds->domid, rc);
+
+ if (crcs->teardown_devices)
+ cleanup_device_subkind(cds);
+
+ rc = crcs->saved_rc;
+ if (!rc) {
+ crcs->callback = do_failover_done;
+ do_failover(egc, crs);
+ return;
+ }
+
+ libxl__colo_restore_teardown_done(egc, crs, rc);
+}
+
+static void do_failover(libxl__egc *egc, libxl__colo_restore_state *crs)
+{
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ const int status = crcs->status;
+ libxl__logdirty_switch *const lds = &crcs->lds;
+
+ EGC_GC;
+
+ switch(status) {
+ case LIBXL_COLO_SETUPED:
+ /*
+ * We will come here only when reading emulator xenstore data or
+ * emulator context fails, and libxl__xc_domain_restore_done()
+ * is not called. In this case, the migration is not finished,
+ * so we cannot do failover.
+ */
+ LOG(ERROR, "migration fails");
+ crcs->callback(egc, crcs, ERROR_FAIL);
+ return;
+ case LIBXL_COLO_SUSPENDED:
+ case LIBXL_COLO_RESUMED:
+ /* disable logdirty first */
+ lds->callback = colo_disable_logdirty_done;
+ colo_disable_logdirty(crs, egc);
+ return;
+ default:
+ LOG(ERROR, "invalid status: %d", status);
+ crcs->callback(egc, crcs, ERROR_FAIL);
+ }
+}
+
+static void do_failover_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state* crcs,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__colo_restore_state *const crs = crcs->crs;
+
+ EGC_GC;
+
+ if (rc)
+ LOG(ERROR, "cannot do failover");
+
+ libxl__colo_restore_teardown_done(egc, crs, rc);
+}
+
+static void colo_disable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
+
+ EGC_GC;
+
+ if (rc)
+ LOG(WARN, "cannot disable logdirty");
+
+ if (crcs->status == LIBXL_COLO_SUSPENDED) {
+ /*
+ * failover when reading state from master, so no need to
+ * call libxl__qmp_restore().
+ */
+ colo_resume_vm(egc, crcs, 0);
+ return;
+ }
+
+ /* If we cannot disable logdirty, we still can do failover */
+ crcs->callback(egc, crcs, 0);
+}
+
+static void libxl__colo_restore_teardown_done(libxl__egc *egc,
+ libxl__colo_restore_state *crs,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ EGC_GC;
+
+ /* convenience aliases */
+ const int domid = crs->domid;
+ const libxl_ctx *const ctx = libxl__gc_owner(gc);
+ xc_interface *const xch = ctx->xch;
+
+ if (!rc)
+ /* failover, no need to destroy the secondary vm */
+ goto out;
+
+ xc_domain_destroy(xch, domid);
+
+out:
+ if (crs->saved_cb) {
+ dcs->callback = crs->saved_cb;
+ crs->saved_cb = NULL;
+ }
+
+ dcs->callback(egc, dcs, rc, crs->domid);
+}
+
+static void colo_common_write_stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc);
+static void colo_common_read_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ int rc);
+/* ======================== colo: checkpoint ======================= */
+/*
+ * Do the following things when resuming secondary vm:
+ * 1. read emulator xenstore data
+ * 2. read emulator context
+ * 3. REC_TYPE_CHECKPOINT_END
+ */
+static void libxl__colo_restore_domain_checkpoint_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_read_state *srs = CONTAINER_OF(shs, *srs, shs);
+ libxl__domain_create_state *dcs = CONTAINER_OF(srs, *dcs, srs);
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+
+ crcs->callback = NULL;
+ dcs->srs.checkpoint_callback = colo_common_read_stream_done;
+ libxl__stream_read_start_checkpoint(shs->egc, &dcs->srs);
+}
+
+
+/* ===================== colo: resume secondary vm ===================== */
+/*
+ * Do the following things when resuming secondary vm the first time:
+ * 1. resume secondary vm
+ * 2. enable log dirty
+ * 3. setup checkpoint devices
+ * 4. write CHECKPOINT_SVM_READY
+ * 5. unpause secondary vm
+ * 6. write CHECKPOINT_SVM_RESUMED
+ *
+ * Do the following things when resuming secondary vm:
+ * 1. write CHECKPOINT_SVM_READY
+ * 2. resume secondary vm
+ * 3. write CHECKPOINT_SVM_RESUMED
+ */
+static void colo_send_svm_ready(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs);
+static void colo_send_svm_ready_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int rc);
+static void colo_restore_preresume_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_restore_resume_vm(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs);
+static void colo_resume_vm_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int rc);
+static void colo_write_svm_resumed(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs);
+static void colo_enable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int retval);
+static void colo_reenable_logdirty(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc);
+static void colo_reenable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc);
+static void colo_setup_checkpoint_devices(libxl__egc *egc,
+ libxl__colo_restore_state *crs);
+static void colo_restore_setup_cds_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_unpause_svm(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs);
+
+static void libxl__colo_restore_domain_resume_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_read_state *srs = CONTAINER_OF(shs, *srs, shs);
+ libxl__domain_create_state *dcs = CONTAINER_OF(srs, *dcs, srs);
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+
+ if (crcs->teardown_devices)
+ colo_send_svm_ready(shs->egc, crcs);
+ else
+ colo_restore_resume_vm(shs->egc, crcs);
+}
+
+static void colo_send_svm_ready(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs)
+{
+ libxl_sr_checkpoint_state srcs = { .id = CHECKPOINT_SVM_READY };
+
+ crcs->callback = colo_send_svm_ready_done;
+ crcs->sws.checkpoint_callback = colo_common_write_stream_done;
+ libxl__stream_write_checkpoint_state(egc, &crcs->sws, &srcs);
+}
+
+static void colo_send_svm_ready_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *cds = &dcs->cds;
+
+ if (!crcs->preresume) {
+ crcs->preresume = true;
+ colo_unpause_svm(egc, crcs);
+ return;
+ }
+
+ cds->callback = colo_restore_preresume_cb;
+ libxl__checkpoint_devices_preresume(egc, cds);
+}
+
+static void colo_restore_preresume_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_restore_state *crs = cds->concrete_data;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "preresume fails");
+ goto out;
+ }
+
+ colo_restore_resume_vm(egc, crcs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+static void colo_restore_resume_vm(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs)
+{
+
+ crcs->callback = colo_resume_vm_done;
+ colo_resume_vm(egc, crcs, 1);
+}
+
+static void colo_resume_vm_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__colo_restore_state *const crs = crcs->crs;
+ libxl__logdirty_switch *const lds = &crcs->lds;
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "cannot resume secondary vm");
+ goto out;
+ }
+
+ crcs->status = LIBXL_COLO_RESUMED;
+
+ /* avoid calling stream->completion_callback() more than once */
+ if (crs->saved_cb) {
+ dcs->callback = crs->saved_cb;
+ crs->saved_cb = NULL;
+
+ dcs->srs.completion_callback = NULL;
+
+ lds->callback = colo_enable_logdirty_done;
+ colo_enable_logdirty(crs, egc);
+ return;
+ }
+
+ colo_write_svm_resumed(egc, crcs);
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+static void colo_write_svm_resumed(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs)
+{
+ libxl_sr_checkpoint_state srcs = { .id = CHECKPOINT_SVM_RESUMED };
+
+ crcs->callback = NULL;
+ crcs->sws.checkpoint_callback = colo_common_write_stream_done;
+ libxl__stream_write_checkpoint_state(egc, &crcs->sws, &srcs);
+}
+
+static void colo_enable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
+
+ /* Convenience aliases */
+ libxl__colo_restore_state *const crs = crcs->crs;
+
+ EGC_GC;
+
+ if (rc) {
+ /*
+ * log-dirty already enabled? There's no test op,
+ * so attempt to disable then reenable it
+ */
+ lds->callback = colo_reenable_logdirty;
+ colo_disable_logdirty(crs, egc);
+ return;
+ }
+
+ colo_setup_checkpoint_devices(egc, crs);
+}
+
+static void colo_reenable_logdirty(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__colo_restore_state *const crs = crcs->crs;
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "cannot enable logdirty");
+ goto out;
+ }
+
+ lds->callback = colo_reenable_logdirty_done;
+ colo_enable_logdirty(crs, egc);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+static void colo_reenable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "cannot enable logdirty");
+ goto out;
+ }
+
+ colo_setup_checkpoint_devices(egc, crcs->crs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+/*
+ * We cannot setup checkpoint devices in libxl__colo_restore_setup(),
+ * because the guest is not ready.
+ */
+static void colo_setup_checkpoint_devices(libxl__egc *egc,
+ libxl__colo_restore_state *crs)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *cds = &dcs->cds;
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ STATE_AO_GC(crs->ao);
+
+ /* TODO: disk/nic support */
+ cds->device_kind_flags = 0;
+ cds->callback = colo_restore_setup_cds_done;
+ cds->ao = ao;
+ cds->domid = crs->domid;
+ cds->ops = colo_restore_ops;
+
+ if (init_device_subkind(cds))
+ goto out;
+
+ crcs->teardown_devices = 1;
+
+ libxl__checkpoint_devices_setup(egc, cds);
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+static void colo_restore_setup_cds_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_restore_state *crs = cds->concrete_data;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "COLO: failed to setup device for guest with domid %u",
+ cds->domid);
+ goto out;
+ }
+
+ colo_send_svm_ready(egc, crcs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+static void colo_unpause_svm(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+ int rc;
+
+ /* Convenience aliases */
+ const uint32_t domid = crcs->crs->domid;
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ /* We have enabled secondary vm's logdirty, so we can unpause it now */
+ rc = libxl_domain_unpause(CTX, domid);
+ if (rc) {
+ LOG(ERROR, "cannot unpause secondary vm");
+ goto out;
+ }
+
+ colo_write_svm_resumed(egc, crcs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+
+/* ===================== colo: wait new checkpoint ===================== */
+static void colo_restore_commit_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_stream_read_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int real_size);
+
+static void libxl__colo_restore_domain_wait_checkpoint_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_read_state *srs = CONTAINER_OF(shs, *srs, shs);
+ libxl__domain_create_state *dcs = CONTAINER_OF(srs, *dcs, srs);
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *cds = &dcs->cds;
+
+ cds->callback = colo_restore_commit_cb;
+ libxl__checkpoint_devices_commit(shs->egc, cds);
+}
+
+static void colo_restore_commit_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_restore_state *crs = cds->concrete_data;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "commit fails");
+ goto out;
+ }
+
+ crcs->callback = colo_stream_read_done;
+ dcs->srs.checkpoint_callback = colo_common_read_stream_done;
+ libxl__stream_read_checkpoint_state(egc, &dcs->srs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, 0);
+}
+
+static void colo_stream_read_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int id)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+ int ok = 0;
+
+ EGC_GC;
+
+ if (id != CHECKPOINT_NEW) {
+ LOG(ERROR, "invalid section: %d", id);
+ goto out;
+ }
+
+ ok = 1;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, ok);
+}
+
+
+/* ===================== colo: suspend secondary vm ===================== */
+/*
+ * Do the following things when resuming secondary vm:
+ * 1. suspend secondary vm
+ * 2. send CHECKPOINT_SVM_SUSPENDED
+ */
+static void colo_suspend_vm_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dsps,
+ int ok);
+static void colo_restore_postsuspend_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+
+static void libxl__colo_restore_domain_suspend_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_read_state *srs = CONTAINER_OF(shs, *srs, shs);
+ libxl__domain_create_state *dcs = CONTAINER_OF(srs, *dcs, srs);
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+
+ STATE_AO_GC(dcs->ao);
+
+ /* Convenience aliases */
+ libxl__domain_suspend_state *const dsps = &crcs->dsps;
+
+ /* suspend secondary vm */
+ dsps->callback_common_done = colo_suspend_vm_done;
+
+ libxl__domain_suspend(shs->egc, dsps);
+}
+
+static void colo_suspend_vm_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dsps,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(dsps, *crcs, dsps);
+ libxl__colo_restore_state *crs = crcs->crs;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *cds = &dcs->cds;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "cannot suspend secondary vm");
+ goto out;
+ }
+
+ crcs->status = LIBXL_COLO_SUSPENDED;
+
+ cds->callback = colo_restore_postsuspend_cb;
+ libxl__checkpoint_devices_postsuspend(egc, cds);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, !rc);
+}
+
+static void colo_restore_postsuspend_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_restore_state *crs = cds->concrete_data;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+ libxl_sr_checkpoint_state srcs = { .id = CHECKPOINT_SVM_SUSPENDED };
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "postsuspend fails");
+ goto out;
+ }
+
+ crcs->callback = NULL;
+ crcs->sws.checkpoint_callback = colo_common_write_stream_done;
+ libxl__stream_write_checkpoint_state(egc, &crcs->sws, &srcs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, !rc);
+}
+
+
+/* ===================== colo: common callback ===================== */
+static void colo_common_write_stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs =
+ CONTAINER_OF(stream, *crcs, sws);
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+ int ok;
+
+ EGC_GC;
+
+ if (rc < 0) {
+ /* TODO: it may be a internal error, but we don't know */
+ LOG(ERROR, "sending data fails");
+ ok = 2;
+ goto out;
+ }
+
+ if (!crcs->callback) {
+ /* Everythins is OK */
+ ok = 1;
+ goto out;
+ }
+
+ crcs->callback(egc, crcs, 0);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, ok);
+}
+
+static void colo_common_read_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+ int ok;
+
+ EGC_GC;
+
+ if (rc < 0) {
+ /* TODO: it may be a internal error, but we don't know */
+ LOG(ERROR, "reading data fails");
+ ok = 2;
+ goto out;
+ }
+
+ if (!crcs->callback) {
+ /* Everythins is OK */
+ ok = 1;
+ goto out;
+ }
+
+ /* rc contains the id */
+ crcs->callback(egc, crcs, rc);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, ok);
+}
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index fc746fb..7301ac4 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -981,6 +981,23 @@ static void domcreate_console_available(libxl__egc *egc,
dcs->aop_console_how.for_event));
}
+static void libxl__colo_restore_setup_done(libxl__egc *egc,
+ libxl__colo_restore_state *crs,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "colo restore setup fails: %d", rc);
+ domcreate_stream_done(egc, &dcs->srs, rc);
+ return;
+ }
+
+ libxl__stream_read_start(egc, &dcs->srs);
+}
+
static void domcreate_bootloader_done(libxl__egc *egc,
libxl__bootloader_state *bl,
int rc)
@@ -994,6 +1011,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
+ libxl__colo_restore_state *const crs = &dcs->crs;
+ libxl_domain_build_info *const info = &d_config->b_info;
if (rc) {
domcreate_rebuild_done(egc, dcs, rc);
@@ -1022,6 +1041,13 @@ static void domcreate_bootloader_done(libxl__egc *egc,
/* Restore */
+ /* COLO only supports HVM now */
+ if (info->type != LIBXL_DOMAIN_TYPE_HVM &&
+ checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
rc = libxl__build_pre(gc, domid, d_config, state);
if (rc)
goto out;
@@ -1035,6 +1061,16 @@ static void domcreate_bootloader_done(libxl__egc *egc,
if (restore_fd >= 0) {
switch (checkpointed_stream) {
+ case LIBXL_CHECKPOINTED_STREAM_COLO:
+ /* colo restore setup */
+ crs->ao = ao;
+ crs->domid = domid;
+ crs->send_back_fd = dcs->send_back_fd;
+ crs->recv_fd = restore_fd;
+ crs->hvm = (info->type == LIBXL_DOMAIN_TYPE_HVM);
+ crs->callback = libxl__colo_restore_setup_done;
+ libxl__colo_restore_setup(egc, crs);
+ break;
case LIBXL_CHECKPOINTED_STREAM_REMUS:
libxl__remus_restore_setup(egc, dcs);
/* fall through */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 6307b71..48b4858 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -87,6 +87,8 @@
#include "_libxl_types_internal.h"
#include "_libxl_types_internal_json.h"
+#include "libxl_colo.h"
+
#define LIBXL_INIT_TIMEOUT 10
#define LIBXL_DESTROY_TIMEOUT 10
#define LIBXL_HOTPLUG_TIMEOUT 40
@@ -3396,12 +3398,6 @@ _hidden int libxl__destroy_qdisk_backend(libxl__gc *gc, uint32_t domid);
/*----- Domain creation -----*/
-typedef struct libxl__domain_create_state libxl__domain_create_state;
-
-typedef void libxl__domain_create_cb(libxl__egc *egc,
- libxl__domain_create_state*,
- int rc, uint32_t domid);
-
/* State for manipulating a libxl migration v2 stream */
typedef struct libxl__stream_read_state libxl__stream_read_state;
@@ -3468,7 +3464,6 @@ libxl__stream_read_inuse(const libxl__stream_read_state *stream)
return stream->running;
}
-
struct libxl__domain_create_state {
/* filled in by user */
libxl__ao *ao;
@@ -3484,6 +3479,8 @@ struct libxl__domain_create_state {
/* private to domain_create */
int guest_domid;
libxl__domain_build_state build_state;
+ libxl__colo_restore_state crs;
+ libxl__checkpoint_devices_state cds;
libxl__bootloader_state bl;
libxl__stub_dm_spawn_state dmss;
/* If we're not doing stubdom, we use only dmss.dm,
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 0d6949a..d1d474d 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -68,7 +68,11 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
shs->ao = ao;
shs->domid = domid;
shs->recv_callback = libxl__srm_callout_received_restore;
- shs->completion_callback = libxl__xc_domain_restore_done;
+ if (dcs->restore_params.checkpointed_stream ==
+ LIBXL_CHECKPOINTED_STREAM_COLO)
+ shs->completion_callback = libxl__colo_restore_teardown;
+ else
+ shs->completion_callback = libxl__xc_domain_restore_done;
shs->caller_state = dcs;
shs->need_results = 1;
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index 302ae53..9659051 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -850,6 +850,18 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
*/
if (libxl__stream_read_inuse(stream)) {
switch (checkpointed_stream) {
+ case LIBXL_CHECKPOINTED_STREAM_COLO:
+ if (stream->completion_callback) {
+ /*
+ * restore, just build the secondary vm, don't close
+ * the stream
+ */
+ stream->completion_callback(egc, stream, 0);
+ } else {
+ /* failover, just close the stream */
+ stream_complete(egc, stream, 0);
+ }
+ break;
case LIBXL_CHECKPOINTED_STREAM_REMUS:
/*
* Failover from primary. Domain state is currently at a
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 632c009..33f4a90 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -232,6 +232,7 @@ libxl_hdtype = Enumeration("hdtype", [
libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
(0, "NONE"),
(1, "REMUS"),
+ (2, "COLO"),
])
#
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 14/27] secondary vm suspend/resume/checkpoint code
2016-03-04 8:41 ` [PATCH v11 14/27] secondary vm suspend/resume/checkpoint code Changlong Xie
@ 2016-03-04 17:11 ` Ian Jackson
2016-03-07 2:57 ` Wen Congyang
2016-03-17 9:03 ` Changlong Xie
0 siblings, 2 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:11 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 14/27] secondary vm suspend/resume/checkpoint code"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> Secondary vm is running in colo mode. So we will do
> the following things again and again:
I don't propose to review this in detail. Skimreading it, it looks
plausible. I don't think a detailed review is needed.
I will review the changes to the core code.
> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
> index 5d9f497..2bfed64 100644
> --- a/tools/libxc/xc_sr_common.h
> +++ b/tools/libxc/xc_sr_common.h
> @@ -184,10 +184,12 @@ struct xc_sr_context
> * migration stream
> * 0: Plain VM
> * 1: Remus
> + * 2: COLO
> */
> enum {
> MIG_STREAM_NONE, /* plain stream */
> MIG_STREAM_REMUS,
> + MIG_STREAM_COLO,
I think this shows that the duplicated list (in the comment, above the
enum) is a mistake. I would prefer it to be removed.
> +/* ===================== colo: common functions ===================== */
> +static void colo_enable_logdirty(libxl__colo_restore_state *crs, libxl__egc *egc)
Here's another missing blank line. This seems to be a general theme:
can you change this everywhere ? Thanks.
> @@ -994,6 +1011,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
> const int restore_fd = dcs->restore_fd;
> libxl__domain_build_state *const state = &dcs->build_state;
> const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
> + libxl__colo_restore_state *const crs = &dcs->crs;
> + libxl_domain_build_info *const info = &d_config->b_info;
>
> if (rc) {
> domcreate_rebuild_done(egc, dcs, rc);
> @@ -1022,6 +1041,13 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>
> /* Restore */
>
> + /* COLO only supports HVM now */
> + if (info->type != LIBXL_DOMAIN_TYPE_HVM &&
> + checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
> + rc = ERROR_FAIL;
> + goto out;
Please log something here, or it may be very mysterious.
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 6307b71..48b4858 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -87,6 +87,8 @@
> @@ -3468,7 +3464,6 @@ libxl__stream_read_inuse(const libxl__stream_read_state *stream)
> return stream->running;
> }
>
> -
> struct libxl__domain_create_state {
> /* filled in by user */
> libxl__ao *ao;
> @@ -3484,6 +3479,8 @@ struct libxl__domain_create_state {
Unintentional whitespace change.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 14/27] secondary vm suspend/resume/checkpoint code
2016-03-04 17:11 ` Ian Jackson
@ 2016-03-07 2:57 ` Wen Congyang
2016-03-17 9:03 ` Changlong Xie
1 sibling, 0 replies; 100+ messages in thread
From: Wen Congyang @ 2016-03-07 2:57 UTC (permalink / raw)
To: Ian Jackson, Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Andrew Cooper, Jiang Yunhong,
Dong Eddie, xen devel, Anthony Perard, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:11 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 14/27] secondary vm suspend/resume/checkpoint code"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> Secondary vm is running in colo mode. So we will do
>> the following things again and again:
>
> I don't propose to review this in detail. Skimreading it, it looks
> plausible. I don't think a detailed review is needed.
>
> I will review the changes to the core code.
>
>> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
>> index 5d9f497..2bfed64 100644
>> --- a/tools/libxc/xc_sr_common.h
>> +++ b/tools/libxc/xc_sr_common.h
>> @@ -184,10 +184,12 @@ struct xc_sr_context
>> * migration stream
>> * 0: Plain VM
>> * 1: Remus
>> + * 2: COLO
>> */
>> enum {
>> MIG_STREAM_NONE, /* plain stream */
>> MIG_STREAM_REMUS,
>> + MIG_STREAM_COLO,
>
> I think this shows that the duplicated list (in the comment, above the
> enum) is a mistake. I would prefer it to be removed.
Do you mean remove the comments?
>
>> +/* ===================== colo: common functions ===================== */
Add a blank line here? Will fix it in the next version.
>> +static void colo_enable_logdirty(libxl__colo_restore_state *crs, libxl__egc *egc)
>
> Here's another missing blank line. This seems to be a general theme:
> can you change this everywhere ? Thanks.
>
>> @@ -994,6 +1011,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>> const int restore_fd = dcs->restore_fd;
>> libxl__domain_build_state *const state = &dcs->build_state;
>> const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
>> + libxl__colo_restore_state *const crs = &dcs->crs;
>> + libxl_domain_build_info *const info = &d_config->b_info;
>>
>> if (rc) {
>> domcreate_rebuild_done(egc, dcs, rc);
>> @@ -1022,6 +1041,13 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>>
>> /* Restore */
>>
>> + /* COLO only supports HVM now */
>> + if (info->type != LIBXL_DOMAIN_TYPE_HVM &&
>> + checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
>> + rc = ERROR_FAIL;
>> + goto out;
>
> Please log something here, or it may be very mysterious.
OK. Will add some comments to explain why only support pure HVM now.
>
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 6307b71..48b4858 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -87,6 +87,8 @@
>> @@ -3468,7 +3464,6 @@ libxl__stream_read_inuse(const libxl__stream_read_state *stream)
>> return stream->running;
>> }
>>
>> -
>> struct libxl__domain_create_state {
>> /* filled in by user */
>> libxl__ao *ao;
>> @@ -3484,6 +3479,8 @@ struct libxl__domain_create_state {
>
> Unintentional whitespace change.
Sorry for the mistake. Will fix it in the next version.
Thanks
Wen Congyang
>
>
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 14/27] secondary vm suspend/resume/checkpoint code
2016-03-04 17:11 ` Ian Jackson
2016-03-07 2:57 ` Wen Congyang
@ 2016-03-17 9:03 ` Changlong Xie
2016-03-17 12:19 ` Wei Liu
1 sibling, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-17 9:03 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:11 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 14/27] secondary vm suspend/resume/checkpoint code"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> Secondary vm is running in colo mode. So we will do
>> the following things again and again:
>
> I don't propose to review this in detail. Skimreading it, it looks
> plausible. I don't think a detailed review is needed.
>
> I will review the changes to the core code.
>
>> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
>> index 5d9f497..2bfed64 100644
>> --- a/tools/libxc/xc_sr_common.h
>> +++ b/tools/libxc/xc_sr_common.h
>> @@ -184,10 +184,12 @@ struct xc_sr_context
>> * migration stream
>> * 0: Plain VM
>> * 1: Remus
>> + * 2: COLO
>> */
>> enum {
>> MIG_STREAM_NONE, /* plain stream */
>> MIG_STREAM_REMUS,
>> + MIG_STREAM_COLO,
>
> I think this shows that the duplicated list (in the comment, above the
> enum) is a mistake. I would prefer it to be removed.
We've sent out a patch to fix this issue first and wonder it can be
merged, so we can rebase to it.
http://www.gossamer-threads.com/lists/xen/devel/423093
Thanks
-Xie
>
>> +/* ===================== colo: common functions ===================== */
>> +static void colo_enable_logdirty(libxl__colo_restore_state *crs, libxl__egc *egc)
>
> Here's another missing blank line. This seems to be a general theme:
> can you change this everywhere ? Thanks.
>
>> @@ -994,6 +1011,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>> const int restore_fd = dcs->restore_fd;
>> libxl__domain_build_state *const state = &dcs->build_state;
>> const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
>> + libxl__colo_restore_state *const crs = &dcs->crs;
>> + libxl_domain_build_info *const info = &d_config->b_info;
>>
>> if (rc) {
>> domcreate_rebuild_done(egc, dcs, rc);
>> @@ -1022,6 +1041,13 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>>
>> /* Restore */
>>
>> + /* COLO only supports HVM now */
>> + if (info->type != LIBXL_DOMAIN_TYPE_HVM &&
>> + checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
>> + rc = ERROR_FAIL;
>> + goto out;
>
> Please log something here, or it may be very mysterious.
>
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 6307b71..48b4858 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -87,6 +87,8 @@
>> @@ -3468,7 +3464,6 @@ libxl__stream_read_inuse(const libxl__stream_read_state *stream)
>> return stream->running;
>> }
>>
>> -
>> struct libxl__domain_create_state {
>> /* filled in by user */
>> libxl__ao *ao;
>> @@ -3484,6 +3479,8 @@ struct libxl__domain_create_state {
>
> Unintentional whitespace change.
>
>
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 14/27] secondary vm suspend/resume/checkpoint code
2016-03-17 9:03 ` Changlong Xie
@ 2016-03-17 12:19 ` Wei Liu
0 siblings, 0 replies; 100+ messages in thread
From: Wei Liu @ 2016-03-17 12:19 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Anthony Perard,
Dong Eddie, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On Thu, Mar 17, 2016 at 05:03:12PM +0800, Changlong Xie wrote:
> On 03/05/2016 01:11 AM, Ian Jackson wrote:
> >Changlong Xie writes ("[PATCH v11 14/27] secondary vm suspend/resume/checkpoint code"):
> >>From: Wen Congyang <wency@cn.fujitsu.com>
> >>
> >>Secondary vm is running in colo mode. So we will do
> >>the following things again and again:
> >
> >I don't propose to review this in detail. Skimreading it, it looks
> >plausible. I don't think a detailed review is needed.
> >
> >I will review the changes to the core code.
> >
> >>diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
> >>index 5d9f497..2bfed64 100644
> >>--- a/tools/libxc/xc_sr_common.h
> >>+++ b/tools/libxc/xc_sr_common.h
> >>@@ -184,10 +184,12 @@ struct xc_sr_context
> >> * migration stream
> >> * 0: Plain VM
> >> * 1: Remus
> >>+ * 2: COLO
> >> */
> >> enum {
> >> MIG_STREAM_NONE, /* plain stream */
> >> MIG_STREAM_REMUS,
> >>+ MIG_STREAM_COLO,
> >
> >I think this shows that the duplicated list (in the comment, above the
> >enum) is a mistake. I would prefer it to be removed.
>
> We've sent out a patch to fix this issue first and wonder it can be merged,
> so we can rebase to it.
>
> http://www.gossamer-threads.com/lists/xen/devel/423093
>
> Thanks
> -Xie
They will be committed today.
Sorry for the delay!
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 15/27] primary vm suspend/resume/checkpoint code
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (13 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 14/27] secondary vm suspend/resume/checkpoint code Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:14 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 16/27] libxc/restore: support COLO restore Changlong Xie
` (12 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
We will do the following things again and again:
1. Suspend primary vm
a. Suspend primary vm
b. do postsuspend
c. Read CHECKPOINT_SVM_SUSPENDED sent by secondary
2. Checkpoint
a. Write emulator xenstore data and emulator context
b. Write checkpoint end record
3. Resume primary vm
a. Read CHECKPOINT_SVM_READY from slave
b. Do presume
c. Resume primary vm
d. Read CHECKPOINT_SVM_RESUMED from slave
4. Wait a new checkpoint
a. Wait a new checkpoint(not implemented)
b. Send CHECKPOINT_NEW to slave
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxl/Makefile | 2 +-
tools/libxl/libxl.c | 5 +-
tools/libxl/libxl_colo.h | 6 +
tools/libxl/libxl_colo_save.c | 562 ++++++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_dom_save.c | 7 +-
tools/libxl/libxl_internal.h | 157 ++++++------
tools/libxl/libxl_types.idl | 1 +
7 files changed, 666 insertions(+), 74 deletions(-)
create mode 100644 tools/libxl/libxl_colo_save.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index d8612eb..2016393 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -65,7 +65,7 @@ LIBXL_OBJS-y += libxl_no_convert_callout.o
endif
LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
-LIBXL_OBJS-y += libxl_colo_restore.o
+LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index e9ab78c..305231b 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -880,7 +880,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
assert(info);
/* Point of no return */
- libxl__remus_setup(egc, &dss->rs);
+ if (libxl_defbool_val(info->colo))
+ libxl__colo_save_setup(egc, &dss->css);
+ else
+ libxl__remus_setup(egc, &dss->rs);
return AO_INPROGRESS;
out:
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index f2b98cc..feec7f1 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -18,6 +18,7 @@
struct libxl__ao;
struct libxl__egc;
+struct libxl__colo_save_state;
enum {
LIBXL_COLO_SETUPED,
@@ -52,4 +53,9 @@ extern void libxl__colo_restore_setup(struct libxl__egc *egc,
libxl__colo_restore_state *crs);
extern void libxl__colo_restore_teardown(struct libxl__egc *egc, void *dcs_void,
int ret, int retval, int errnoval);
+extern void libxl__colo_save_setup(struct libxl__egc *egc,
+ struct libxl__colo_save_state *css);
+extern void libxl__colo_save_teardown(struct libxl__egc *egc,
+ struct libxl__colo_save_state *css,
+ int rc);
#endif
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
new file mode 100644
index 0000000..e318ea2
--- /dev/null
+++ b/tools/libxl/libxl_colo_save.c
@@ -0,0 +1,562 @@
+/*
+ * Copyright (C) 2014 FUJITSU LIMITED
+ * Author: Wen Congyang <wency@cn.fujitsu.com>
+ * Yang Hongyang <hongyang.yang@easystack.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
+ NULL,
+};
+
+/* ================= helper functions ================= */
+static int init_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+ /* init device subkind-specific state in the libxl ctx */
+ int rc;
+ STATE_AO_GC(cds->ao);
+
+ rc = 0;
+ return rc;
+}
+
+static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+ /* cleanup device subkind-specific state in the libxl ctx */
+ STATE_AO_GC(cds->ao);
+}
+
+/* ================= colo: setup save environment ================= */
+static void colo_save_setup_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_save_setup_failed(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+/*
+ * checkpoint callbacks are called in the following order:
+ * 1. suspend
+ * 2. checkpoint
+ * 3. resume
+ * 4. wait checkpoint
+ */
+static void libxl__colo_save_domain_suspend_callback(void *data);
+static void libxl__colo_save_domain_checkpoint_callback(void *data);
+static void libxl__colo_save_domain_resume_callback(void *data);
+static void libxl__colo_save_domain_wait_checkpoint_callback(void *data);
+
+void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *const cds = &dss->cds;
+ libxl__srm_save_autogen_callbacks *const callbacks =
+ &dss->sws.shs.callbacks.save.a;
+
+ STATE_AO_GC(dss->ao);
+
+ if (dss->type != LIBXL_DOMAIN_TYPE_HVM) {
+ LOG(ERROR, "COLO only supports hvm now");
+ goto out;
+ }
+
+ css->send_fd = dss->fd;
+ css->recv_fd = dss->recv_fd;
+ css->svm_running = false;
+
+ /* TODO: disk/nic support */
+ cds->device_kind_flags = 0;
+ cds->ops = colo_ops;
+ cds->callback = colo_save_setup_done;
+ cds->ao = ao;
+ cds->domid = dss->domid;
+ cds->concrete_data = css;
+
+ css->srs.ao = ao;
+ css->srs.fd = css->recv_fd;
+ css->srs.back_channel = true;
+ libxl__stream_read_start(egc, &css->srs);
+
+ if (init_device_subkind(cds))
+ goto out;
+
+ callbacks->suspend = libxl__colo_save_domain_suspend_callback;
+ callbacks->checkpoint = libxl__colo_save_domain_checkpoint_callback;
+ callbacks->postcopy = libxl__colo_save_domain_resume_callback;
+ callbacks->wait_checkpoint = libxl__colo_save_domain_wait_checkpoint_callback;
+
+ libxl__checkpoint_devices_setup(egc, &dss->cds);
+
+ return;
+
+out:
+ libxl__ao_complete(egc, ao, ERROR_FAIL);
+}
+
+static void colo_save_setup_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+ EGC_GC;
+
+ if (!rc) {
+ libxl__domain_save(egc, dss);
+ return;
+ }
+
+ LOG(ERROR, "COLO: failed to setup device for guest with domid %u",
+ dss->domid);
+ cds->callback = colo_save_setup_failed;
+ libxl__checkpoint_devices_teardown(egc, cds);
+}
+
+static void colo_save_setup_failed(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ STATE_AO_GC(cds->ao);
+
+ if (rc)
+ LOG(ERROR, "COLO: failed to teardown device after setup failed"
+ " for guest with domid %u, rc %d", cds->domid, rc);
+
+ cleanup_device_subkind(cds);
+ libxl__ao_complete(egc, ao, rc);
+}
+
+
+/* ================= colo: teardown save environment ================= */
+static void colo_teardown_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+
+void libxl__colo_save_teardown(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int rc)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ LOG(WARN, "COLO: Domain suspend terminated with rc %d,"
+ " teardown COLO devices...", rc);
+
+ libxl__stream_read_abort(egc, &css->srs, 1);
+
+ dss->cds.callback = colo_teardown_done;
+ libxl__checkpoint_devices_teardown(egc, &dss->cds);
+ return;
+}
+
+static void colo_teardown_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ cleanup_device_subkind(cds);
+ dss->callback(egc, dss, rc);
+}
+
+static void colo_common_write_stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc);
+static void colo_common_read_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ int rc);
+/* ===================== colo: suspend primary vm ===================== */
+
+static void colo_read_svm_suspended_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id);
+/*
+ * Do the following things when suspending primary vm:
+ * 1. suspend primary vm
+ * 2. do postsuspend
+ * 3. read CHECKPOINT_SVM_SUSPENDED
+ * 4. read secondary vm's dirty pages
+ */
+static void colo_suspend_primary_vm_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dsps,
+ int ok);
+static void colo_postsuspend_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+
+static void libxl__colo_save_domain_suspend_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__egc *egc = shs->egc;
+ libxl__stream_write_state *sws = CONTAINER_OF(shs, *sws, shs);
+ libxl__domain_save_state *dss = sws->dss;
+
+ /* Convenience aliases */
+ libxl__domain_suspend_state *dsps = &dss->dsps;
+
+ dsps->callback_common_done = colo_suspend_primary_vm_done;
+ libxl__domain_suspend(egc, dsps);
+}
+
+static void colo_suspend_primary_vm_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dsps,
+ int rc)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(dsps, *dss, dsps);
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "cannot suspend primary vm");
+ goto out;
+ }
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *const cds = &dss->cds;
+
+ cds->callback = colo_postsuspend_cb;
+ libxl__checkpoint_devices_postsuspend(egc, cds);
+ return;
+
+out:
+ dss->rc = rc;
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, !rc);
+}
+
+static void colo_postsuspend_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "postsuspend fails");
+ goto out;
+ }
+
+ if (!css->svm_running) {
+ rc = 0;
+ goto out;
+ }
+
+ /*
+ * read CHECKPOINT_SVM_SUSPENDED
+ */
+ css->callback = colo_read_svm_suspended_done;
+ css->srs.checkpoint_callback = colo_common_read_stream_done;
+ libxl__stream_read_checkpoint_state(egc, &css->srs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, !rc);
+}
+
+static void colo_read_svm_suspended_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id)
+{
+ int ok = 0;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (id != CHECKPOINT_SVM_SUSPENDED) {
+ LOG(ERROR, "invalid section: %d, expected: %d", id,
+ CHECKPOINT_SVM_SUSPENDED);
+ goto out;
+ }
+
+ ok = 1;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, ok);
+}
+
+
+/* ===================== colo: send tailbuf ========================== */
+static void libxl__colo_save_domain_checkpoint_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_write_state *sws = CONTAINER_OF(shs, *sws, shs);
+ libxl__domain_save_state *dss = sws->dss;
+
+ /* Convenience aliases */
+ libxl__colo_save_state *const css = &dss->css;
+
+ /* write emulator xenstore data, emulator context, and checkpoint end */
+ css->callback = NULL;
+ dss->sws.checkpoint_callback = colo_common_write_stream_done;
+ libxl__stream_write_start_checkpoint(shs->egc, &dss->sws);
+}
+
+/* ===================== colo: resume primary vm ===================== */
+/*
+ * Do the following things when resuming primary vm:
+ * 1. read CHECKPOINT_SVM_READY
+ * 2. do preresume
+ * 3. resume primary vm
+ * 4. read CHECKPOINT_SVM_RESUMED
+ */
+static void colo_read_svm_ready_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id);
+static void colo_preresume_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_read_svm_resumed_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id);
+
+static void libxl__colo_save_domain_resume_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__egc *egc = shs->egc;
+ libxl__stream_write_state *sws = CONTAINER_OF(shs, *sws, shs);
+ libxl__domain_save_state *dss = sws->dss;
+
+ /* Convenience aliases */
+ libxl__colo_save_state *const css = &dss->css;
+
+ EGC_GC;
+
+ /* read CHECKPOINT_SVM_READY */
+ css->callback = colo_read_svm_ready_done;
+ css->srs.checkpoint_callback = colo_common_read_stream_done;
+ libxl__stream_read_checkpoint_state(egc, &css->srs);
+}
+
+static void colo_read_svm_ready_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (id != CHECKPOINT_SVM_READY) {
+ LOG(ERROR, "invalid section: %d, expected: %d", id,
+ CHECKPOINT_SVM_READY);
+ goto out;
+ }
+
+ css->svm_running = true;
+ dss->cds.callback = colo_preresume_cb;
+ libxl__checkpoint_devices_preresume(egc, &dss->cds);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
+}
+
+static void colo_preresume_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "preresume fails");
+ goto out;
+ }
+
+ /* Resumes the domain and the device model */
+ if (libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1)) {
+ LOG(ERROR, "cannot resume primary vm");
+ goto out;
+ }
+
+ /* read CHECKPOINT_SVM_RESUMED */
+ css->callback = colo_read_svm_resumed_done;
+ css->srs.checkpoint_callback = colo_common_read_stream_done;
+ libxl__stream_read_checkpoint_state(egc, &css->srs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
+}
+
+static void colo_read_svm_resumed_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id)
+{
+ int ok = 0;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (id != CHECKPOINT_SVM_RESUMED) {
+ LOG(ERROR, "invalid section: %d, expected: %d", id,
+ CHECKPOINT_SVM_RESUMED);
+ goto out;
+ }
+
+ ok = 1;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, ok);
+}
+
+
+/* ===================== colo: wait new checkpoint ===================== */
+/*
+ * Do the following things:
+ * 1. do commit
+ * 2. wait for a new checkpoint
+ * 3. write CHECKPOINT_NEW
+ */
+static void colo_device_commit_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_start_new_checkpoint(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+
+static void libxl__colo_save_domain_wait_checkpoint_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_write_state *sws = CONTAINER_OF(shs, *sws, shs);
+ libxl__domain_save_state *dss = sws->dss;
+ libxl__egc *egc = dss->sws.shs.egc;
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *const cds = &dss->cds;
+
+ cds->callback = colo_device_commit_cb;
+ libxl__checkpoint_devices_commit(egc, cds);
+}
+
+static void colo_device_commit_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "commit fails");
+ goto out;
+ }
+
+ /* TODO: wait a new checkpoint */
+ colo_start_new_checkpoint(egc, cds, 0);
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
+}
+
+static void colo_start_new_checkpoint(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+ libxl_sr_checkpoint_state srcs = { .id = CHECKPOINT_NEW };
+
+ if (rc)
+ goto out;
+
+ /* write CHECKPOINT_NEW */
+ css->callback = NULL;
+ dss->sws.checkpoint_callback = colo_common_write_stream_done;
+ libxl__stream_write_checkpoint_state(egc, &dss->sws, &srcs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
+}
+
+
+/* ===================== colo: common callback ===================== */
+static void colo_common_write_stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(stream, *dss, sws);
+ int ok;
+
+ /* Convenience aliases */
+ libxl__colo_save_state *const css = &dss->css;
+
+ EGC_GC;
+
+ if (rc < 0) {
+ /* TODO: it may be a internal error, but we don't know */
+ LOG(ERROR, "sending data fails");
+ ok = 0;
+ goto out;
+ }
+
+ if (!css->callback) {
+ /* Everythins is OK */
+ ok = 1;
+ goto out;
+ }
+
+ css->callback(egc, css, 0);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, ok);
+}
+
+static void colo_common_read_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ int rc)
+{
+ libxl__colo_save_state *css = CONTAINER_OF(stream, *css, srs);
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+ int ok;
+
+ EGC_GC;
+
+ if (rc < 0) {
+ /* TODO: it may be a internal error, but we don't know */
+ LOG(ERROR, "reading data fails");
+ ok = 0;
+ goto out;
+ }
+
+ if (!css->callback) {
+ /* Everythins is OK */
+ ok = 1;
+ goto out;
+ }
+
+ /* rc contains the id */
+ css->callback(egc, css, rc);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, ok);
+}
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index cd324bb..821f862 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -438,12 +438,15 @@ static void domain_save_done(libxl__egc *egc,
if (dss->remus) {
/*
- * With Remus, if we reach this point, it means either
+ * With Remus/COLO, if we reach this point, it means either
* backup died or some network error occurred preventing us
* from sending checkpoints. Teardown the network buffers and
* release netlink resources. This is an async op.
*/
- libxl__remus_teardown(egc, &dss->rs, rc);
+ if (libxl_defbool_val(dss->remus->colo))
+ libxl__colo_save_teardown(egc, &dss->css, rc);
+ else
+ libxl__remus_teardown(egc, &dss->rs, rc);
return;
}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 48b4858..5160939 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2802,7 +2802,7 @@ typedef struct libxl__save_helper_state {
/*
* The abstract checkpoint device layer exposes a common
* set of API to [external] libxl for manipulating devices attached to
- * a guest protected by Remus. The device layer also exposes a set of
+ * a guest protected by Remus/COLO. The device layer also exposes a set of
* [internal] interfaces that every device type must implement.
*
* The following API are exposed to libxl:
@@ -2820,7 +2820,7 @@ typedef struct libxl__save_helper_state {
* +libxl__checkpoint_devices_commit
*
* Each device type needs to implement the interfaces specified in
- * the libxl__checkpoint_device_instance_ops if it wishes to support Remus.
+ * the libxl__checkpoint_device_instance_ops if it wishes to support Remus/COLO.
*
* The high-level control flow through the checkpoint device layer is shown
* below:
@@ -2840,7 +2840,7 @@ typedef struct libxl__checkpoint_device_instance_ops libxl__checkpoint_device_in
/*
* Interfaces to be implemented by every device subkind that wishes to
- * support Remus. Functions must be implemented unless otherwise
+ * support Remus/COLO. Functions must be implemented unless otherwise
* stated. Many of these functions are asynchronous. They call
* dev->aodev.callback when done. The actual implementations may be
* synchronous and call dev->aodev.callback directly (as the last
@@ -3017,6 +3017,72 @@ static inline bool libxl__conversion_helper_inuse
(const libxl__conversion_helper_state *chs)
{ return libxl__ev_child_inuse(&chs->child); }
+/* State for manipulating a libxl migration v2 stream */
+
+typedef struct libxl__stream_read_state libxl__stream_read_state;
+
+typedef struct libxl__sr_record_buf {
+ /* private to stream read helper */
+ LIBXL_STAILQ_ENTRY(struct libxl__sr_record_buf) entry;
+ libxl__sr_rec_hdr hdr;
+ void *body; /* iff hdr.length != 0 */
+} libxl__sr_record_buf;
+
+struct libxl__stream_read_state {
+ /* filled by the user */
+ libxl__ao *ao;
+ libxl__domain_create_state *dcs;
+ int fd;
+ bool legacy;
+ bool back_channel;
+ void (*completion_callback)(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int rc);
+ void (*checkpoint_callback)(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int rc);
+ /* Private */
+ int rc;
+ bool running;
+ bool in_checkpoint;
+ bool sync_teardown; /* Only used to coordinate shutdown on error path. */
+ bool in_checkpoint_state;
+ libxl__save_helper_state shs;
+ libxl__conversion_helper_state chs;
+
+ /* Main stream-reading data. */
+ libxl__datacopier_state dc; /* Only used when reading a record */
+ libxl__sr_hdr hdr;
+ LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue; /* NOGC */
+ enum {
+ SRS_PHASE_NORMAL,
+ SRS_PHASE_BUFFERING,
+ SRS_PHASE_UNBUFFERING,
+ } phase;
+ bool recursion_guard;
+
+ /* Only used while actively reading a record from the stream. */
+ libxl__sr_record_buf *incoming_record; /* NOGC */
+
+ /* Both only used when processing an EMULATOR record. */
+ libxl__datacopier_state emu_dc;
+ libxl__carefd *emu_carefd;
+};
+
+_hidden void libxl__stream_read_init(libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_start(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_start_checkpoint(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_checkpoint_state(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_abort(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
+static inline bool
+libxl__stream_read_inuse(const libxl__stream_read_state *stream)
+{
+ return stream->running;
+}
/*----- Domain suspend (save) state structure -----*/
/*
@@ -3093,6 +3159,18 @@ libxl__stream_write_inuse(const libxl__stream_write_state *stream)
return stream->running;
}
+/*----- colo related state structure -----*/
+typedef struct libxl__colo_save_state libxl__colo_save_state;
+struct libxl__colo_save_state {
+ int send_fd;
+ int recv_fd;
+
+ /* private */
+ libxl__stream_read_state srs;
+ void (*callback)(libxl__egc *, libxl__colo_save_state *, int);
+ bool svm_running;
+};
+
typedef struct libxl__logdirty_switch {
/* Set by caller of libxl__domain_common_switch_qemu_logdirty */
libxl__ao *ao;
@@ -3151,7 +3229,12 @@ struct libxl__domain_save_state {
int hvm;
int xcflags;
libxl__domain_suspend_state dsps;
- libxl__remus_state rs;
+ union {
+ /* for Remus */
+ libxl__remus_state rs;
+ /* for COLO */
+ libxl__colo_save_state css;
+ };
libxl__checkpoint_devices_state cds;
libxl__stream_write_state sws;
libxl__logdirty_switch logdirty;
@@ -3398,72 +3481,6 @@ _hidden int libxl__destroy_qdisk_backend(libxl__gc *gc, uint32_t domid);
/*----- Domain creation -----*/
-/* State for manipulating a libxl migration v2 stream */
-typedef struct libxl__stream_read_state libxl__stream_read_state;
-
-typedef struct libxl__sr_record_buf {
- /* private to stream read helper */
- LIBXL_STAILQ_ENTRY(struct libxl__sr_record_buf) entry;
- libxl__sr_rec_hdr hdr;
- void *body; /* iff hdr.length != 0 */
-} libxl__sr_record_buf;
-
-struct libxl__stream_read_state {
- /* filled by the user */
- libxl__ao *ao;
- libxl__domain_create_state *dcs;
- int fd;
- bool legacy;
- bool back_channel;
- void (*completion_callback)(libxl__egc *egc,
- libxl__stream_read_state *srs,
- int rc);
- void (*checkpoint_callback)(libxl__egc *egc,
- libxl__stream_read_state *srs,
- int rc);
- /* Private */
- int rc;
- bool running;
- bool in_checkpoint;
- bool sync_teardown; /* Only used to coordinate shutdown on error path. */
- bool in_checkpoint_state;
- libxl__save_helper_state shs;
- libxl__conversion_helper_state chs;
-
- /* Main stream-reading data. */
- libxl__datacopier_state dc; /* Only used when reading a record */
- libxl__sr_hdr hdr;
- LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue; /* NOGC */
- enum {
- SRS_PHASE_NORMAL,
- SRS_PHASE_BUFFERING,
- SRS_PHASE_UNBUFFERING,
- } phase;
- bool recursion_guard;
-
- /* Only used while actively reading a record from the stream. */
- libxl__sr_record_buf *incoming_record; /* NOGC */
-
- /* Both only used when processing an EMULATOR record. */
- libxl__datacopier_state emu_dc;
- libxl__carefd *emu_carefd;
-};
-
-_hidden void libxl__stream_read_init(libxl__stream_read_state *stream);
-_hidden void libxl__stream_read_start(libxl__egc *egc,
- libxl__stream_read_state *stream);
-_hidden void libxl__stream_read_start_checkpoint(libxl__egc *egc,
- libxl__stream_read_state *stream);
-_hidden void libxl__stream_read_checkpoint_state(libxl__egc *egc,
- libxl__stream_read_state *stream);
-_hidden void libxl__stream_read_abort(libxl__egc *egc,
- libxl__stream_read_state *stream, int rc);
-static inline bool
-libxl__stream_read_inuse(const libxl__stream_read_state *stream)
-{
- return stream->running;
-}
-
struct libxl__domain_create_state {
/* filled in by user */
libxl__ao *ao;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 33f4a90..9b0a537 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -763,6 +763,7 @@ libxl_domain_remus_info = Struct("domain_remus_info",[
("netbuf", libxl_defbool),
("netbufscript", string),
("diskbuf", libxl_defbool),
+ ("colo", libxl_defbool)
])
libxl_event_type = Enumeration("event_type", [
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 15/27] primary vm suspend/resume/checkpoint code
2016-03-04 8:41 ` [PATCH v11 15/27] primary " Changlong Xie
@ 2016-03-04 17:14 ` Ian Jackson
2016-03-07 2:59 ` Wen Congyang
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:14 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 15/27] primary vm suspend/resume/checkpoint code"):
> From: Wen Congyang <wency@cn.fujitsu.com>
I would look at this on the same basis as the previous patch.
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 48b4858..5160939 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> +struct libxl__stream_read_state {
> + /* filled by the user */
> + libxl__ao *ao;
> + libxl__domain_create_state *dcs;
> + int fd;
> + bool legacy;
Can you please split out this code motion into a separate patch ?
As it is it is very difficult to review.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 15/27] primary vm suspend/resume/checkpoint code
2016-03-04 17:14 ` Ian Jackson
@ 2016-03-07 2:59 ` Wen Congyang
0 siblings, 0 replies; 100+ messages in thread
From: Wen Congyang @ 2016-03-07 2:59 UTC (permalink / raw)
To: Ian Jackson, Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Andrew Cooper, Jiang Yunhong,
Dong Eddie, xen devel, Anthony Perard, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:14 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 15/27] primary vm suspend/resume/checkpoint code"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>
> I would look at this on the same basis as the previous patch.
>
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 48b4858..5160939 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>
>> +struct libxl__stream_read_state {
>> + /* filled by the user */
>> + libxl__ao *ao;
>> + libxl__domain_create_state *dcs;
>> + int fd;
>> + bool legacy;
>
> Can you please split out this code motion into a separate patch ?
> As it is it is very difficult to review.
OK, will fix it in the next version.
Thanks
Wen Congyang
>
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 16/27] libxc/restore: support COLO restore
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (14 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 15/27] primary " Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:16 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 17/27] libxc/save: support COLO save Changlong Xie
` (11 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
a. call callbacks resume/checkpoint/suspend while secondary vm
status is consistent with primary
b. send dirty pfn list to primary when checkpoint under colo
c. send store gfn and console gfn to xl before resuming secondary vm
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
---
tools/libxc/include/xenguest.h | 8 ++
tools/libxc/xc_sr_common.h | 8 +-
tools/libxc/xc_sr_restore.c | 181 +++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_colo_restore.c | 5 -
tools/libxl/libxl_create.c | 3 +
tools/libxl/libxl_save_msgs_gen.pl | 4 +-
6 files changed, 200 insertions(+), 9 deletions(-)
diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index cb0bbcf..67cb540 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -127,6 +127,14 @@ struct restore_callbacks {
*/
int (*wait_checkpoint)(void* data);
+ /*
+ * callback to send store gfn and console gfn to xl
+ * if we want to resume vm before xc_domain_save()
+ * exits.
+ */
+ void (*restore_results)(xen_pfn_t store_gfn, xen_pfn_t console_gfn,
+ void *data);
+
/* to be provided as the last argument to each callback function */
void* data;
};
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 2bfed64..29ab4eb 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -226,6 +226,10 @@ struct xc_sr_context
struct xc_sr_restore_ops ops;
struct restore_callbacks *callbacks;
+ int send_back_fd;
+ unsigned long p2m_size;
+ xc_hypercall_buffer_t dirty_bitmap_hbuf;
+
/* From Image Header. */
uint32_t format_version;
@@ -234,13 +238,13 @@ struct xc_sr_context
uint32_t guest_page_size;
/* Plain VM, or checkpoints over time. */
- bool checkpointed;
+ int checkpointed;
/* Currently buffering records between a checkpoint */
bool buffer_all_records;
/*
- * With Remus, we buffer the records sent by the primary at checkpoint,
+ * With Remus/COLO, we buffer the records sent by the primary at checkpoint,
* in case the primary will fail, we can recover from the last
* checkpoint state.
* This should be enough for most of the cases because primary only send
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index aef9bca..24c4719 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -411,6 +411,92 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec)
return rc;
}
+/*
+ * Send checkpoint dirty pfn list to primary.
+ */
+static int send_checkpoint_dirty_pfn_list(struct xc_sr_context *ctx)
+{
+ xc_interface *xch = ctx->xch;
+ int rc = -1;
+ unsigned count, written;
+ uint64_t i, *pfns = NULL;
+ struct iovec *iov = NULL;
+ xc_shadow_op_stats_t stats = { 0, ctx->restore.p2m_size };
+ struct xc_sr_record rec =
+ {
+ .type = REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST,
+ };
+ DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+ &ctx->restore.dirty_bitmap_hbuf);
+
+ if ( xc_shadow_control(
+ xch, ctx->domid, XEN_DOMCTL_SHADOW_OP_CLEAN,
+ HYPERCALL_BUFFER(dirty_bitmap), ctx->restore.p2m_size,
+ NULL, 0, &stats) != ctx->restore.p2m_size )
+ {
+ PERROR("Failed to retrieve logdirty bitmap");
+ goto err;
+ }
+
+ for ( i = 0, count = 0; i < ctx->restore.p2m_size; i++ )
+ {
+ if ( test_bit(i, dirty_bitmap) )
+ count++;
+ }
+
+
+ pfns = malloc(count * sizeof(*pfns));
+ if ( !pfns )
+ {
+ ERROR("Unable to allocate %zu bytes of memory for dirty pfn list",
+ count * sizeof(*pfns));
+ goto err;
+ }
+
+ for ( i = 0, written = 0; i < ctx->restore.p2m_size; ++i )
+ {
+ if ( !test_bit(i, dirty_bitmap) )
+ continue;
+
+ if ( written > count )
+ {
+ ERROR("Dirty pfn list exceed");
+ goto err;
+ }
+
+ pfns[written++] = i;
+ }
+
+ /* iovec[] for writev(). */
+ iov = malloc(3 * sizeof(*iov));
+ if ( !iov )
+ {
+ ERROR("Unable to allocate memory for sending dirty bitmap");
+ goto err;
+ }
+
+ rec.length = count * sizeof(*pfns);
+
+ iov[0].iov_base = &rec.type;
+ iov[0].iov_len = sizeof(rec.type);
+
+ iov[1].iov_base = &rec.length;
+ iov[1].iov_len = sizeof(rec.length);
+
+ iov[2].iov_base = pfns;
+ iov[2].iov_len = count * sizeof(*pfns);
+
+ if ( writev_exact(ctx->restore.send_back_fd, iov, 3) )
+ {
+ PERROR("Failed to write dirty bitmap to stream");
+ goto err;
+ }
+
+ rc = 0;
+ err:
+ return rc;
+}
+
static int process_record(struct xc_sr_context *ctx, struct xc_sr_record *rec);
static int handle_checkpoint(struct xc_sr_context *ctx)
{
@@ -460,6 +546,53 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
else
ctx->restore.buffer_all_records = true;
+ if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+ {
+#define HANDLE_CALLBACK_RETURN_VALUE(ret) \
+ do { \
+ if ( ret == 1 ) \
+ rc = 0; /* Success */ \
+ else \
+ { \
+ if ( ret == 2 ) \
+ rc = BROKEN_CHANNEL; \
+ else \
+ rc = -1; /* Some unspecified error */ \
+ goto err; \
+ } \
+ } while (0)
+
+ /* COLO */
+
+ /* We need to resume guest */
+ rc = ctx->restore.ops.stream_complete(ctx);
+ if ( rc )
+ goto err;
+
+ ctx->restore.callbacks->restore_results(ctx->restore.xenstore_gfn,
+ ctx->restore.console_gfn,
+ ctx->restore.callbacks->data);
+
+ /* Resume secondary vm */
+ ret = ctx->restore.callbacks->postcopy(ctx->restore.callbacks->data);
+ HANDLE_CALLBACK_RETURN_VALUE(ret);
+
+ /* Wait for a new checkpoint */
+ ret = ctx->restore.callbacks->wait_checkpoint(
+ ctx->restore.callbacks->data);
+ HANDLE_CALLBACK_RETURN_VALUE(ret);
+
+ /* suspend secondary vm */
+ ret = ctx->restore.callbacks->suspend(ctx->restore.callbacks->data);
+ HANDLE_CALLBACK_RETURN_VALUE(ret);
+
+#undef HANDLE_CALLBACK_RETURN_VALUE
+
+ rc = send_checkpoint_dirty_pfn_list(ctx);
+ if ( rc )
+ goto err;
+ }
+
err:
return rc;
}
@@ -529,6 +662,21 @@ static int setup(struct xc_sr_context *ctx)
{
xc_interface *xch = ctx->xch;
int rc;
+ DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+ &ctx->restore.dirty_bitmap_hbuf);
+
+ if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+ {
+ dirty_bitmap = xc_hypercall_buffer_alloc_pages(xch, dirty_bitmap,
+ NRPAGES(bitmap_size(ctx->restore.p2m_size)));
+
+ if ( !dirty_bitmap )
+ {
+ ERROR("Unable to allocate memory for dirty bitmap");
+ rc = -1;
+ goto err;
+ }
+ }
rc = ctx->restore.ops.setup(ctx);
if ( rc )
@@ -562,10 +710,15 @@ static void cleanup(struct xc_sr_context *ctx)
{
xc_interface *xch = ctx->xch;
unsigned i;
+ DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+ &ctx->restore.dirty_bitmap_hbuf);
for ( i = 0; i < ctx->restore.buffered_rec_num; i++ )
free(ctx->restore.buffered_records[i].data);
+ if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+ xc_hypercall_buffer_free_pages(xch, dirty_bitmap,
+ NRPAGES(bitmap_size(ctx->restore.p2m_size)));
free(ctx->restore.buffered_records);
free(ctx->restore.populated_pfns);
if ( ctx->restore.ops.cleanup(ctx) )
@@ -631,6 +784,15 @@ static int restore(struct xc_sr_context *ctx)
} while ( rec.type != REC_TYPE_END );
remus_failover:
+
+ if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+ {
+ /* With COLO, we have already called stream_complete */
+ rc = 0;
+ IPRINTF("COLO Failover");
+ goto done;
+ }
+
/*
* With Remus, if we reach here, there must be some error on primary,
* failover from the last checkpoint state.
@@ -667,6 +829,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
int checkpointed_stream,
struct restore_callbacks *callbacks, int send_back_fd)
{
+ xen_pfn_t nr_pfns;
struct xc_sr_context ctx =
{
.xch = xch,
@@ -680,11 +843,21 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
ctx.restore.xenstore_domid = store_domid;
ctx.restore.checkpointed = checkpointed_stream;
ctx.restore.callbacks = callbacks;
+ ctx.restore.send_back_fd = send_back_fd;
/* Sanity checks for callbacks. */
if ( checkpointed_stream )
assert(callbacks->checkpoint);
+ if ( ctx.restore.checkpointed == MIG_STREAM_COLO )
+ {
+ /* this is COLO restore */
+ assert(callbacks->suspend &&
+ callbacks->postcopy &&
+ callbacks->wait_checkpoint &&
+ callbacks->restore_results);
+ }
+
DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
", checkpointed_stream %d", io_fd, dom, hvm, pae,
superpages, checkpointed_stream);
@@ -706,6 +879,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
if ( read_headers(&ctx) )
return -1;
+ if ( xc_domain_nr_gpfns(xch, dom, &nr_pfns) < 0 )
+ {
+ PERROR("Unable to obtain the guest p2m size");
+ return -1;
+ }
+
+ ctx.restore.p2m_size = nr_pfns;
+
if ( ctx.dominfo.hvm )
{
ctx.restore.ops = restore_ops_x86_hvm;
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index 829687e..b0dd52b 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -125,11 +125,6 @@ static void colo_resume_vm(libxl__egc *egc,
return;
}
- /*
- * TODO: get store gfn and console gfn
- * We should call the callback restore_results in
- * xc_domain_restore() before resuming the guest.
- */
libxl__xc_domain_restore_done(egc, dcs, 0, 0, 0);
return;
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 7301ac4..3610a39 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1013,6 +1013,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
libxl__colo_restore_state *const crs = &dcs->crs;
libxl_domain_build_info *const info = &d_config->b_info;
+ libxl__srm_restore_autogen_callbacks *const callbacks =
+ &dcs->srs.shs.callbacks.restore.a;
if (rc) {
domcreate_rebuild_done(egc, dcs, rc);
@@ -1040,6 +1042,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
}
/* Restore */
+ callbacks->restore_results = libxl__srm_callout_callback_restore_results;
/* COLO only supports HVM now */
if (info->type != LIBXL_DOMAIN_TYPE_HVM &&
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index 6016706..c2243f2 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -29,8 +29,8 @@ our @msgs = (
[ 6, 'srcxA', "wait_checkpoint", [] ],
[ 7, 'scxA', "switch_qemu_logdirty", [qw(int domid
unsigned enable)] ],
- [ 8, 'r', "restore_results", ['unsigned long', 'store_mfn',
- 'unsigned long', 'console_mfn'] ],
+ [ 8, 'rcx', "restore_results", ['unsigned long', 'store_gfn',
+ 'unsigned long', 'console_gfn'] ],
[ 9, 'srW', "complete", [qw(int retval
int errnoval)] ],
);
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* [PATCH v11 16/27] libxc/restore: support COLO restore
2016-03-04 8:41 ` [PATCH v11 16/27] libxc/restore: support COLO restore Changlong Xie
@ 2016-03-04 17:16 ` Ian Jackson
0 siblings, 0 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:16 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 16/27] libxc/restore: support COLO restore"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> a. call callbacks resume/checkpoint/suspend while secondary vm
> status is consistent with primary
> b. send dirty pfn list to primary when checkpoint under colo
> c. send store gfn and console gfn to xl before resuming secondary vm
This looks like a plausible approach to me. Again I am not going to
review the new colo-specific code in detail but skimreading it it
looks like the right kind of thing.
I'd like some opinion from Andrew Cooper about whether he's happy with
the interleaving of the colo-specific functionality in the middle of
the restore code, as you have done it. Personally I think it seems
like it's probably the best way, but it may merit questioning, at
least.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 17/27] libxc/save: support COLO save
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (15 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 16/27] libxc/restore: support COLO restore Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:18 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 18/27] implement the cmdline for COLO Changlong Xie
` (10 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
After suspend primary vm, get dirty bitmap on secondary vm,
and send pages both dirty on primary/secondary to secondary.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
---
tools/libxc/xc_sr_common.h | 2 +
tools/libxc/xc_sr_save.c | 95 +++++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 95 insertions(+), 2 deletions(-)
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 29ab4eb..5301746 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -196,6 +196,8 @@ struct xc_sr_context
{
struct /* Save data. */
{
+ int recv_fd;
+
struct xc_sr_save_ops ops;
struct save_callbacks *callbacks;
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index 7393355..c2d30bc 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -517,6 +517,58 @@ static int send_memory_live(struct xc_sr_context *ctx)
return rc;
}
+static int merge_secondary_dirty_bitmap(struct xc_sr_context *ctx)
+{
+ xc_interface *xch = ctx->xch;
+ struct xc_sr_record rec;
+ uint64_t *pfns = NULL;
+ uint64_t pfn;
+ unsigned count, i;
+ int rc;
+ DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+ &ctx->save.dirty_bitmap_hbuf);
+
+ rc = read_record(ctx, ctx->save.recv_fd, &rec);
+ if ( rc )
+ goto err;
+
+ if ( rec.type != REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST )
+ {
+ PERROR("Expect dirty bitmap record, but received %u", rec.type );
+ rc = -1;
+ goto err;
+ }
+
+ if ( rec.length % sizeof(*pfns) )
+ {
+ PERROR("Invalid dirty pfn list record length %u", rec.length );
+ rc = -1;
+ goto err;
+ }
+
+ count = rec.length / sizeof(*pfns);
+ pfns = rec.data;
+
+ for ( i = 0; i < count; i++ )
+ {
+ pfn = pfns[i];
+ if (pfn > ctx->save.p2m_size)
+ {
+ PERROR("Invalid pfn %#lx", pfn );
+ rc = -1;
+ goto err;
+ }
+
+ set_bit(pfn, dirty_bitmap);
+ }
+
+ rc = 0;
+
+ err:
+ free(rec.data);
+ return rc;
+}
+
/*
* Suspend the domain and send dirty memory.
* This is the last iteration of the live migration and the
@@ -558,6 +610,16 @@ static int suspend_and_send_dirty(struct xc_sr_context *ctx)
bitmap_or(dirty_bitmap, ctx->save.deferred_pages, ctx->save.p2m_size);
+ if ( !ctx->save.live && ctx->save.checkpointed == MIG_STREAM_COLO )
+ {
+ rc = merge_secondary_dirty_bitmap(ctx);
+ if ( rc )
+ {
+ PERROR("Failed to get secondary vm's dirty pages");
+ goto out;
+ }
+ }
+
rc = send_dirty_pages(ctx, stats.dirty_count + ctx->save.nr_deferred_pages);
if ( rc )
goto out;
@@ -791,13 +853,39 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type)
if ( rc )
goto err;
+ if ( ctx->save.checkpointed == MIG_STREAM_COLO )
+ {
+ rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
+ if ( !rc )
+ {
+ rc = -1;
+ goto err;
+ }
+ }
+
rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
if ( rc <= 0 )
goto err;
- rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
- if ( rc <= 0 )
+ if ( ctx->save.checkpointed == MIG_STREAM_COLO )
+ {
+ rc = ctx->save.callbacks->wait_checkpoint(
+ ctx->save.callbacks->data);
+ if ( rc <= 0 )
+ goto err;
+ }
+ else if ( ctx->save.checkpointed == MIG_STREAM_REMUS )
+ {
+ rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
+ if ( rc <= 0 )
+ goto err;
+ }
+ else
+ {
+ ERROR("Unknown checkpointed stream");
+ rc = -1;
goto err;
+ }
}
} while ( ctx->save.checkpointed != MIG_STREAM_NONE );
@@ -843,6 +931,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
ctx.save.live = !!(flags & XCFLAGS_LIVE);
ctx.save.debug = !!(flags & XCFLAGS_DEBUG);
ctx.save.checkpointed = checkpointed_stream;
+ ctx.save.recv_fd = recv_fd;
/* If altering migration_stream update this assert too. */
assert(checkpointed_stream == MIG_STREAM_NONE ||
@@ -863,6 +952,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
assert(callbacks->switch_qemu_logdirty);
if ( ctx.save.checkpointed )
assert(callbacks->checkpoint && callbacks->postcopy);
+ if ( ctx.save.checkpointed == MIG_STREAM_COLO )
+ assert(callbacks->wait_checkpoint);
DPRINTF("fd %d, dom %u, max_iters %u, max_factor %u, flags %u, hvm %d",
io_fd, dom, max_iters, max_factor, flags, hvm);
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 17/27] libxc/save: support COLO save
2016-03-04 8:41 ` [PATCH v11 17/27] libxc/save: support COLO save Changlong Xie
@ 2016-03-04 17:18 ` Ian Jackson
2016-03-07 3:00 ` Wen Congyang
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:18 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 17/27] libxc/save: support COLO save"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> After suspend primary vm, get dirty bitmap on secondary vm,
> and send pages both dirty on primary/secondary to secondary.
This patch again seems like a plausible kind of thing. Again, I'd
like to hear from Andrew.
> +static int merge_secondary_dirty_bitmap(struct xc_sr_context *ctx)
> +{
This function might want the word `colo' in its name somewhere.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 17/27] libxc/save: support COLO save
2016-03-04 17:18 ` Ian Jackson
@ 2016-03-07 3:00 ` Wen Congyang
0 siblings, 0 replies; 100+ messages in thread
From: Wen Congyang @ 2016-03-07 3:00 UTC (permalink / raw)
To: Ian Jackson, Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Andrew Cooper, Jiang Yunhong,
Dong Eddie, xen devel, Anthony Perard, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:18 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 17/27] libxc/save: support COLO save"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> After suspend primary vm, get dirty bitmap on secondary vm,
>> and send pages both dirty on primary/secondary to secondary.
>
> This patch again seems like a plausible kind of thing. Again, I'd
> like to hear from Andrew.
>
>> +static int merge_secondary_dirty_bitmap(struct xc_sr_context *ctx)
>> +{
>
> This function might want the word `colo' in its name somewhere.
OK, will fix it in the next version.
Thanks
Wen Congyang
>
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 18/27] implement the cmdline for COLO
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (16 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 17/27] libxc/save: support COLO save Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:22 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication Changlong Xie
` (9 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Add a new option -c to the command 'xl remus'. If you want
to use COLO HA instead of Remus HA, please use -c option.
Update man pages to reflect the addition of a new option to
'xl remus' command.
Also add a new option -c to the internal command 'xl migrate-receive'.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
docs/man/xl.pod.1 | 12 ++++++++--
tools/libxl/libxl.c | 22 ++++++++++++++++--
tools/libxl/xl_cmdimpl.c | 59 ++++++++++++++++++++++++++++++++++++-----------
tools/libxl/xl_cmdtable.c | 4 +++-
4 files changed, 78 insertions(+), 19 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 4279c7c..1c6dd87 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -447,12 +447,15 @@ Print huge (!) amount of debug during the migration process.
=item B<remus> [I<OPTIONS>] I<domain-id> I<host>
-Enable Remus HA for domain. By default B<xl> relies on ssh as a transport
-mechanism between the two hosts.
+Enable Remus HA or COLO HA for domain. By default B<xl> relies on ssh as a
+transport mechanism between the two hosts.
N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
Disk replication support is limited to DRBD disks.
+ COLO support in xl is still in experimental (proof-of-concept) phase.
+ There is no support for network or disk at the moment.
+
B<OPTIONS>
=over 4
@@ -498,6 +501,11 @@ Disable network output buffering. Requires enabling unsafe mode.
Disable disk replication. Requires enabling unsafe mode.
+=item B<-c>
+
+Enable COLO HA. This conflicts with B<-i> and B<-b>, and memory
+checkpoint compression must be disabled.
+
=back
=item B<pause> I<domain-id>
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 305231b..3689dfc 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -848,12 +848,27 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
goto out;
}
+ /* The caller must set this defbool */
+ if (libxl_defbool_is_default(info->colo)) {
+ LOG(ERROR, "colo mode must be enabled/disabled");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
libxl_defbool_setdefault(&info->allow_unsafe, false);
libxl_defbool_setdefault(&info->blackhole, false);
- libxl_defbool_setdefault(&info->compression, true);
+ libxl_defbool_setdefault(&info->compression,
+ !libxl_defbool_val(info->colo));
libxl_defbool_setdefault(&info->netbuf, true);
libxl_defbool_setdefault(&info->diskbuf, true);
+ if (libxl_defbool_val(info->colo) &&
+ libxl_defbool_val(info->compression)) {
+ LOG(ERROR, "cannot use memory checkpoint compression in COLO mode");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
if (!libxl_defbool_val(info->allow_unsafe) &&
(libxl_defbool_val(info->blackhole) ||
!libxl_defbool_val(info->netbuf) ||
@@ -875,7 +890,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
dss->live = 1;
dss->debug = 0;
dss->remus = info;
- dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
+ if (libxl_defbool_val(info->colo))
+ dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
+ else
+ dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
assert(info);
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 523e0e9..30d26bc 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -4445,6 +4445,8 @@ static void migrate_receive(int debug, int daemonize, int monitor,
char rc_buf;
char *migration_domname;
struct domain_create dom_info;
+ const char *ha = checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO ?
+ "COLO" : "Remus";
signal(SIGPIPE, SIG_IGN);
/* if we get SIGPIPE we'd rather just have it as an error */
@@ -4477,11 +4479,12 @@ static void migrate_receive(int debug, int daemonize, int monitor,
switch (checkpointed) {
case LIBXL_CHECKPOINTED_STREAM_REMUS:
+ case LIBXL_CHECKPOINTED_STREAM_COLO:
/* If we are here, it means that the sender (primary) has crashed.
* TODO: Split-Brain Check.
*/
- fprintf(stderr, "migration target: Remus Failover for domain %u\n",
- domid);
+ fprintf(stderr, "migration target: %s Failover for domain %u\n",
+ ha, domid);
/*
* If domain renaming fails, lets just continue (as we need the domain
@@ -4497,16 +4500,20 @@ static void migrate_receive(int debug, int daemonize, int monitor,
rc = libxl_domain_rename(ctx, domid, migration_domname,
common_domname);
if (rc)
- fprintf(stderr, "migration target (Remus): "
+ fprintf(stderr, "migration target (%s): "
"Failed to rename domain from %s to %s:%d\n",
- migration_domname, common_domname, rc);
+ ha, migration_domname, common_domname, rc);
}
+ if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
+ /* The guest is running after failover in COLO mode */
+ exit(rc ? -ERROR_FAIL: 0);
+
rc = libxl_domain_unpause(ctx, domid);
if (rc)
- fprintf(stderr, "migration target (Remus): "
+ fprintf(stderr, "migration target (%s): "
"Failed to unpause domain %s (id: %u):%d\n",
- common_domname, domid, rc);
+ ha, common_domname, domid, rc);
exit(rc ? -ERROR_FAIL: 0);
default:
@@ -4654,7 +4661,7 @@ int main_migrate_receive(int argc, char **argv)
libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
int opt;
- SWITCH_FOREACH_OPT(opt, "Fedr", NULL, "migrate-receive", 0) {
+ SWITCH_FOREACH_OPT(opt, "Fedrc", NULL, "migrate-receive", 0) {
case 'F':
daemonize = 0;
break;
@@ -4668,6 +4675,9 @@ int main_migrate_receive(int argc, char **argv)
case 'r':
checkpointed = LIBXL_CHECKPOINTED_STREAM_REMUS;
break;
+ case 'c':
+ checkpointed = LIBXL_CHECKPOINTED_STREAM_COLO;
+ break;
}
if (argc-optind != 0) {
@@ -8043,11 +8053,8 @@ int main_remus(int argc, char **argv)
int config_len;
memset(&r_info, 0, sizeof(libxl_domain_remus_info));
- /* Defaults */
- r_info.interval = 200;
- libxl_defbool_setdefault(&r_info.blackhole, false);
- SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:e", NULL, "remus", 2) {
+ SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:ec", NULL, "remus", 2) {
case 'i':
r_info.interval = atoi(optarg);
break;
@@ -8075,11 +8082,32 @@ int main_remus(int argc, char **argv)
case 'e':
daemonize = 0;
break;
+ case 'c':
+ libxl_defbool_set(&r_info.colo, true);
}
domid = find_domain(argv[optind]);
host = argv[optind + 1];
+ /* Defaults */
+ libxl_defbool_setdefault(&r_info.blackhole, false);
+ libxl_defbool_setdefault(&r_info.colo, false);
+ if (!libxl_defbool_val(r_info.colo) && !r_info.interval)
+ r_info.interval = 200;
+
+ if (libxl_defbool_val(r_info.colo)) {
+ if (r_info.interval || libxl_defbool_val(r_info.blackhole)) {
+ perror("Option -c conflicts with -i or -b");
+ exit(-1);
+ }
+
+ if (libxl_defbool_is_default(r_info.compression)) {
+ perror("COLO can't be used with memory compression. "
+ "Disable memory checkpoint compression now...");
+ libxl_defbool_set(&r_info.compression, false);
+ }
+ }
+
if (!r_info.netbufscript)
r_info.netbufscript = default_remus_netbufscript;
@@ -8094,8 +8122,9 @@ int main_remus(int argc, char **argv)
if (!ssh_command[0]) {
rune = host;
} else {
- xasprintf(&rune, "exec %s %s xl migrate-receive -r %s",
+ xasprintf(&rune, "exec %s %s xl migrate-receive %s %s",
ssh_command, host,
+ libxl_defbool_val(r_info.colo) ? "-c" : "-r",
daemonize ? "" : " -e");
}
@@ -8123,7 +8152,8 @@ int main_remus(int argc, char **argv)
* domain to force failover
*/
if (libxl_domain_info(ctx, 0, domid)) {
- fprintf(stderr, "Remus: Primary domain has been destroyed.\n");
+ fprintf(stderr, "%s: Primary domain has been destroyed.\n",
+ libxl_defbool_val(r_info.colo) ? "COLO" : "Remus");
close(send_fd);
return 0;
}
@@ -8135,7 +8165,8 @@ int main_remus(int argc, char **argv)
if (rc == ERROR_GUEST_TIMEDOUT)
fprintf(stderr, "Failed to suspend domain at primary.\n");
else {
- fprintf(stderr, "Remus: Backup failed? resuming domain at primary.\n");
+ fprintf(stderr, "%s: Backup failed? resuming domain at primary.\n",
+ libxl_defbool_val(r_info.colo) ? "COLO" : "Remus");
libxl_domain_resume(ctx, domid, 1, 0);
}
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index fdc1ac6..b6b630c 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -499,7 +499,9 @@ struct cmd_spec cmd_table[] = {
"-b Replicate memory checkpoints to /dev/null (blackhole).\n"
" Works only in unsafe mode.\n"
"-n Disable network output buffering. Works only in unsafe mode.\n"
- "-d Disable disk replication. Works only in unsafe mode."
+ "-d Disable disk replication. Works only in unsafe mode.\n"
+ "-c Enable COLO HA. It is conflict with -i and -b, and memory\n"
+ " checkpoint must be disabled"
},
#endif
{ "devd",
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 18/27] implement the cmdline for COLO
2016-03-04 8:41 ` [PATCH v11 18/27] implement the cmdline for COLO Changlong Xie
@ 2016-03-04 17:22 ` Ian Jackson
2016-03-07 3:04 ` Wen Congyang
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:22 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
> --- a/docs/man/xl.pod.1
> +++ b/docs/man/xl.pod.1
...
> + COLO support in xl is still in experimental (proof-of-concept) phase.
> + There is no support for network or disk at the moment.
I think you need to spell out the lack of storage and network handling
means that the guest will corrupt its disk and confuse its network
peers.
> @@ -875,7 +890,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
> dss->live = 1;
> dss->debug = 0;
> dss->remus = info;
> - dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
> + if (libxl_defbool_val(info->colo))
> + dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
> + else
> + dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
If you prefer, a ? : expression would do here as well.
+ dss->checkpointed_stream =
libxl_defbool_val() ? LIBXL_CHECKPOINTED_STREAM_COLO :...
(only completed with sensible formatting). Up to you - it's fine as
it is, too.
Most of this patch looks good to me.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 18/27] implement the cmdline for COLO
2016-03-04 17:22 ` Ian Jackson
@ 2016-03-07 3:04 ` Wen Congyang
0 siblings, 0 replies; 100+ messages in thread
From: Wen Congyang @ 2016-03-07 3:04 UTC (permalink / raw)
To: Ian Jackson, Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Andrew Cooper, Jiang Yunhong,
Dong Eddie, xen devel, Anthony Perard, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:22 AM, Ian Jackson wrote:
>> --- a/docs/man/xl.pod.1
>> +++ b/docs/man/xl.pod.1
> ...
>> + COLO support in xl is still in experimental (proof-of-concept) phase.
>> + There is no support for network or disk at the moment.
>
> I think you need to spell out the lack of storage and network handling
> means that the guest will corrupt its disk and confuse its network
> peers.
OK, will fix it in the next version.
>
>> @@ -875,7 +890,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
>> dss->live = 1;
>> dss->debug = 0;
>> dss->remus = info;
>> - dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>> + if (libxl_defbool_val(info->colo))
>> + dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
>> + else
>> + dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>
> If you prefer, a ? : expression would do here as well.
>
> + dss->checkpointed_stream =
> libxl_defbool_val() ? LIBXL_CHECKPOINTED_STREAM_COLO :...
If so, this line will contains more than 80 characters. So I will not
change it.
Thanks
Wen Congyang
>
> (only completed with sensible formatting). Up to you - it's fine as
> it is, too.
>
> Most of this patch looks good to me.
>
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (17 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 18/27] implement the cmdline for COLO Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:26 ` Ian Jackson
2016-03-04 17:29 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 20/27] Support colo mode for qemu disk Changlong Xie
` (8 subsequent siblings)
27 siblings, 2 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
We will use qemu block replication, and qemu provides some qmp commands
to prepare replication, start replication, get replication error, and
stop replication. Introduce new API to execute these qmp commands.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl_internal.h | 20 +++++++++
tools/libxl/libxl_qmp.c | 96 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 116 insertions(+)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 5160939..8cb9f19 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1771,6 +1771,26 @@ _hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool enabl
_hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const libxl_device_disk *disk);
/* Add a virtual CPU */
_hidden int libxl__qmp_cpu_add(libxl__gc *gc, int domid, int index);
+/* Start NBD server */
+_hidden int libxl__qmp_nbd_server_start(libxl__gc *gc, int domid,
+ const char *host, const char *port);
+/* Add a disk to NBD server */
+_hidden int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char *disk);
+/* Start replication */
+_hidden int libxl__qmp_start_replication(libxl__gc *gc, int domid, bool primary);
+/* Get replication error that occurs when the vm is running */
+_hidden int libxl__qmp_get_replication_error(libxl__gc *gc, int domid);
+/* Do checkpoint */
+_hidden int libxl__qmp_do_checkpoint(libxl__gc *gc, int domid);
+/* Stop replication */
+_hidden int libxl__qmp_stop_replication(libxl__gc *gc, int domid, bool primary);
+/* Stop NBD server */
+_hidden int libxl__qmp_nbd_server_stop(libxl__gc *gc, int domid);
+/* Add or remove a child to/from quorum */
+_hidden int libxl__qmp_x_blockdev_change(libxl__gc *gc, int domid, const char *parant,
+ const char *child, const char *node);
+/* run a hmp command in qmp mode */
+_hidden int libxl__qmp_hmp(libxl__gc *gc, int domid, const char *command_line);
/* close and free the QMP handler */
_hidden void libxl__qmp_close(libxl__qmp_handler *qmp);
/* remove the socket file, if the file has already been removed,
diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
index c0bdfcb..3eb279a 100644
--- a/tools/libxl/libxl_qmp.c
+++ b/tools/libxl/libxl_qmp.c
@@ -979,6 +979,102 @@ int libxl__qmp_cpu_add(libxl__gc *gc, int domid, int idx)
return qmp_run_command(gc, domid, "cpu-add", args, NULL, NULL);
}
+int libxl__qmp_nbd_server_start(libxl__gc *gc, int domid,
+ const char *host, const char *port)
+{
+ libxl__json_object *args = NULL;
+ libxl__json_object *addr = NULL;
+ libxl__json_object *data = NULL;
+
+ /* 'addr': {
+ * 'type': 'inet',
+ * 'data': {
+ * 'host': '$nbd_host',
+ * 'port': '$nbd_port'
+ * }
+ * }
+ */
+ qmp_parameters_add_string(gc, &data, "host", host);
+ qmp_parameters_add_string(gc, &data, "port", port);
+
+ qmp_parameters_add_string(gc, &addr, "type", "inet");
+ qmp_parameters_common_add(gc, &addr, "data", data);
+
+ qmp_parameters_common_add(gc, &args, "addr", addr);
+
+ return qmp_run_command(gc, domid, "nbd-server-start", args, NULL, NULL);
+}
+
+int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char *disk)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_string(gc, &args, "device", disk);
+ qmp_parameters_add_bool(gc, &args, "writable", true);
+
+ return qmp_run_command(gc, domid, "nbd-server-add", args, NULL, NULL);
+}
+
+int libxl__qmp_start_replication(libxl__gc *gc, int domid, bool primary)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_bool(gc, &args, "enable", true);
+ qmp_parameters_add_bool(gc, &args, "primary", primary);
+
+ return qmp_run_command(gc, domid, "xen-set-replication", args, NULL, NULL);
+}
+
+int libxl__qmp_get_replication_error(libxl__gc *gc, int domid)
+{
+ return qmp_run_command(gc, domid, "xen-get-replication-error", NULL,
+ NULL, NULL);
+}
+
+int libxl__qmp_do_checkpoint(libxl__gc *gc, int domid)
+{
+ return qmp_run_command(gc, domid, "xen-do-checkpoint", NULL, NULL, NULL);
+}
+
+int libxl__qmp_stop_replication(libxl__gc *gc, int domid, bool primary)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_bool(gc, &args, "enable", false);
+ qmp_parameters_add_bool(gc, &args, "primary", primary);
+
+ return qmp_run_command(gc, domid, "xen-set-replication", args, NULL, NULL);
+}
+
+int libxl__qmp_nbd_server_stop(libxl__gc *gc, int domid)
+{
+ return qmp_run_command(gc, domid, "nbd-server-stop", NULL, NULL, NULL);
+}
+
+int libxl__qmp_x_blockdev_change(libxl__gc *gc, int domid, const char *parent,
+ const char *child, const char *node)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_string(gc, &args, "parent", parent);
+ if (child)
+ qmp_parameters_add_string(gc, &args, "child", child);
+ if (node)
+ qmp_parameters_add_string(gc, &args, "node", node);
+
+ return qmp_run_command(gc, domid, "x-blockdev-change", args, NULL, NULL);
+}
+
+int libxl__qmp_hmp(libxl__gc *gc, int domid, const char *command_line)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_string(gc, &args, "command-line", command_line);
+
+ return qmp_run_command(gc, domid, "human-monitor-command", args,
+ NULL, NULL);
+}
+
int libxl__qmp_initializations(libxl__gc *gc, uint32_t domid,
const libxl_domain_config *guest_config)
{
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication
2016-03-04 8:41 ` [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication Changlong Xie
@ 2016-03-04 17:26 ` Ian Jackson
2016-03-08 16:46 ` Wei Liu
` (2 more replies)
2016-03-04 17:29 ` Ian Jackson
1 sibling, 3 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:26 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> We will use qemu block replication, and qemu provides some qmp commands
> to prepare replication, start replication, get replication error, and
> stop replication. Introduce new API to execute these qmp commands.
How will this work if in future we want to support HVM (or
hvm-lite-ng) guests ?
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 5160939..8cb9f19 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -1771,6 +1771,26 @@ _hidden int
> libxl__qmp_set_global_dirty_log(libxl__gc ...
> +_hidden int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char *disk);
It's a tiny nit, but can I grumble about the long lines here ? Less
than ~70-75 characters is best.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication
2016-03-04 17:26 ` Ian Jackson
@ 2016-03-08 16:46 ` Wei Liu
2016-03-18 3:44 ` Changlong Xie
2016-03-18 3:45 ` Changlong Xie
2 siblings, 0 replies; 100+ messages in thread
From: Wei Liu @ 2016-03-08 16:46 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Anthony Perard, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On Fri, Mar 04, 2016 at 05:26:44PM +0000, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication"):
> > From: Wen Congyang <wency@cn.fujitsu.com>
> >
> > We will use qemu block replication, and qemu provides some qmp commands
> > to prepare replication, start replication, get replication error, and
> > stop replication. Introduce new API to execute these qmp commands.
>
> How will this work if in future we want to support HVM (or
> hvm-lite-ng) guests ?
>
Just to clarify things: all the new functions in this patch are internal
to libxl. So they have no implication on how PVHv2 COLO is implemented
-- it probably won't be using all these functions anyway.
Wei.
> > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> > index 5160939..8cb9f19 100644
> > --- a/tools/libxl/libxl_internal.h
> > +++ b/tools/libxl/libxl_internal.h
> > @@ -1771,6 +1771,26 @@ _hidden int
> > libxl__qmp_set_global_dirty_log(libxl__gc ...
> > +_hidden int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char *disk);
>
> It's a tiny nit, but can I grumble about the long lines here ? Less
> than ~70-75 characters is best.
>
> Thanks,
> Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication
2016-03-04 17:26 ` Ian Jackson
2016-03-08 16:46 ` Wei Liu
@ 2016-03-18 3:44 ` Changlong Xie
2016-03-18 11:35 ` Wei Liu
2016-03-18 3:45 ` Changlong Xie
2 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-18 3:44 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:26 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> We will use qemu block replication, and qemu provides some qmp commands
>> to prepare replication, start replication, get replication error, and
>> stop replication. Introduce new API to execute these qmp commands.
>
> How will this work if in future we want to support HVM (or
> hvm-lite-ng) guests ?
Sorry for bad english, what does "hvm-lite-ng" mean? I can't figure out
this abbr.
>
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 5160939..8cb9f19 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -1771,6 +1771,26 @@ _hidden int
>> libxl__qmp_set_global_dirty_log(libxl__gc ...
>> +_hidden int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char *disk);
>
> It's a tiny nit, but can I grumble about the long lines here ? Less
> than ~70-75 characters is best.
>
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication
2016-03-18 3:44 ` Changlong Xie
@ 2016-03-18 11:35 ` Wei Liu
0 siblings, 0 replies; 100+ messages in thread
From: Wei Liu @ 2016-03-18 11:35 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Anthony Perard,
Dong Eddie, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On Fri, Mar 18, 2016 at 11:44:34AM +0800, Changlong Xie wrote:
> On 03/05/2016 01:26 AM, Ian Jackson wrote:
> >Changlong Xie writes ("[PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication"):
> >>From: Wen Congyang <wency@cn.fujitsu.com>
> >>
> >>We will use qemu block replication, and qemu provides some qmp commands
> >>to prepare replication, start replication, get replication error, and
> >>stop replication. Introduce new API to execute these qmp commands.
> >
> >How will this work if in future we want to support HVM (or
> >hvm-lite-ng) guests ?
>
> Sorry for bad english, what does "hvm-lite-ng" mean? I can't figure out this
> abbr.
>
Oh, sorry for now having explained this earlier. HVMlite is a new guest
mode. It's basically HVM without QEMU. The official name for that new
mode is not yet coined, so you might see it being referred to as PVHv2,
hvmlite-nodm or some other names, but in essence they point to the same
thing.
For more information:
http://wiki.xen.org/wiki/Virtualization_Spectrum
And scroll down to the bottom for a nice picture. To avoid further
complicating things, I will dwell into the difference between PVHv1 and
PVHv2 (hvmlite).
Wei.
> >
> >>diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> >>index 5160939..8cb9f19 100644
> >>--- a/tools/libxl/libxl_internal.h
> >>+++ b/tools/libxl/libxl_internal.h
> >>@@ -1771,6 +1771,26 @@ _hidden int
> >>libxl__qmp_set_global_dirty_log(libxl__gc ...
> >>+_hidden int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char *disk);
> >
> >It's a tiny nit, but can I grumble about the long lines here ? Less
> >than ~70-75 characters is best.
> >
> >Thanks,
> >Ian.
> >
> >
> >.
> >
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication
2016-03-04 17:26 ` Ian Jackson
2016-03-08 16:46 ` Wei Liu
2016-03-18 3:44 ` Changlong Xie
@ 2016-03-18 3:45 ` Changlong Xie
2 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-18 3:45 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:26 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> We will use qemu block replication, and qemu provides some qmp commands
>> to prepare replication, start replication, get replication error, and
>> stop replication. Introduce new API to execute these qmp commands.
>
> How will this work if in future we want to support HVM (or
> hvm-lite-ng) guests ?
Sorry for bad english, what does "hvm-lite-ng" mean? I can't figure out
this abbr.
>
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 5160939..8cb9f19 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -1771,6 +1771,26 @@ _hidden int
>> libxl__qmp_set_global_dirty_log(libxl__gc ...
>> +_hidden int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char *disk);
>
> It's a tiny nit, but can I grumble about the long lines here ? Less
> than ~70-75 characters is best.
>
Surely
Thanks
-Xie
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication
2016-03-04 8:41 ` [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication Changlong Xie
2016-03-04 17:26 ` Ian Jackson
@ 2016-03-04 17:29 ` Ian Jackson
2016-03-18 3:49 ` Changlong Xie
1 sibling, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:29 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> We will use qemu block replication, and qemu provides some qmp commands
> to prepare replication, start replication, get replication error, and
> stop replication. Introduce new API to execute these qmp commands.
Also, my comments about needing to see the status of the corresponding
Qemu upstream patches, apply here too.
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication
2016-03-04 17:29 ` Ian Jackson
@ 2016-03-18 3:49 ` Changlong Xie
0 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-18 3:49 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:29 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> We will use qemu block replication, and qemu provides some qmp commands
>> to prepare replication, start replication, get replication error, and
>> stop replication. Introduce new API to execute these qmp commands.
>
> Also, my comments about needing to see the status of the corresponding
> Qemu upstream patches, apply here too.
Yes, the lastest version is V16.
http://lists.nongnu.org/archive/html/qemu-devel/2016-03/msg02623.html
Thanks
-Xie
>
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (18 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 19/27] COLO: introduce new API to prepare/start/do/get_error/stop replication Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:44 ` Ian Jackson
2016-03-04 17:52 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 21/27] COLO: use qemu block replication Changlong Xie
` (7 subsequent siblings)
27 siblings, 2 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Usage: disk = ['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
For QEMU block replication details:
http://wiki.qemu.org/Features/BlockReplication
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
docs/man/xl.pod.1 | 36 ++++++-
docs/misc/xl-disk-configuration.txt | 53 +++++++++++
tools/libxl/libxl.c | 64 ++++++++++++-
tools/libxl/libxl_create.c | 25 ++++-
tools/libxl/libxl_device.c | 54 +++++++++++
tools/libxl/libxl_dm.c | 184 ++++++++++++++++++++++++++++++++++--
tools/libxl/libxl_types.idl | 7 ++
tools/libxl/libxlu_disk_l.l | 7 ++
8 files changed, 418 insertions(+), 12 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 1c6dd87..36b093b 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -450,11 +450,39 @@ Print huge (!) amount of debug during the migration process.
Enable Remus HA or COLO HA for domain. By default B<xl> relies on ssh as a
transport mechanism between the two hosts.
-N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
- Disk replication support is limited to DRBD disks.
+B<NOTES>
+
+=over 4
+
+Remus support in xl is still in experimental (proof-of-concept) phase.
+Disk replication support is limited to DRBD disks.
+
+COLO support in xl is still in experimental (proof-of-concept) phase.
+There is no support for network at the moment.
+
+=back
+
+B<EXAMPLE>
- COLO support in xl is still in experimental (proof-of-concept) phase.
- There is no support for network or disk at the moment.
+=over 4
+
+(a) An example for COLO replication's configuration: disk =['...,colo,colo-host
+=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
+
+=item B<colo-host> :Secondary host's ip address.
+
+=item B<colo-port> :Secondary host's port, we will run a nbd server on
+secondary host, and the nbd server will listen this port.
+
+=item B<colo-export> :Nbd server's disk export name of secondary host.
+
+=item B<active-disk> :Secondary's guest write will be buffered in this disk,
+and it's used by secondary.
+
+=item B<hidden-disk> :Primary's modified contents will be buffered in this
+disk, and it's used by secondary.
+
+=back
B<OPTIONS>
diff --git a/docs/misc/xl-disk-configuration.txt b/docs/misc/xl-disk-configuration.txt
index 29f6ddb..6e73975 100644
--- a/docs/misc/xl-disk-configuration.txt
+++ b/docs/misc/xl-disk-configuration.txt
@@ -234,6 +234,59 @@ were intentionally created non-sparse to avoid fragmentation of the
file.
+===============
+COLO PARAMETERS
+===============
+
+
+colo
+----
+
+Enable COLO HA for disk. For better understanding block replication on
+QEMU, please refer to:
+http://wiki.qemu.org/Features/BlockReplication
+
+
+colo-host
+---------
+
+Description: Secondary host's address
+Mandatory: Yes when COLO enabled
+
+
+colo-port
+---------
+
+Description: Secondary port
+ We will run a nbd server on secondary host,
+ and the nbd server will listen this port.
+Mandatory: Yes when COLO enabled
+
+
+colo-export
+-----------
+
+Description: We will run a nbd server on secondary host,
+ exportname is the nbd server's disk export name.
+Mandatory: Yes when COLO enabled
+
+
+active-disk
+-----------
+
+Description: This is used by secondary. Secondary guest's write
+ will be buffered in this disk.
+Mandatory: Yes when COLO enabled
+
+
+hidden-disk
+-----------
+
+Description: This is used by secondary. It buffers the original
+ content that is modified by the primary VM.
+Mandatory: Yes when COLO enabled
+
+
============================================
DEPRECATED PARAMETERS, PREFIXES AND SYNTAXES
============================================
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 3689dfc..eb1b277 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -2307,6 +2307,8 @@ int libxl__device_disk_setdefault(libxl__gc *gc, libxl_device_disk *disk)
int rc;
libxl_defbool_setdefault(&disk->discard_enable, !!disk->readwrite);
+ libxl_defbool_setdefault(&disk->colo_enable, false);
+ libxl_defbool_setdefault(&disk->colo_restore_enable, false);
rc = libxl__resolve_domid(gc, disk->backend_domname, &disk->backend_domid);
if (rc < 0) return rc;
@@ -2505,6 +2507,18 @@ static void device_disk_add(libxl__egc *egc, uint32_t domid,
flexarray_append(back, "params");
flexarray_append(back, GCSPRINTF("%s:%s",
libxl__device_disk_string_of_format(disk->format), disk->pdev_path));
+ if (libxl_defbool_val(disk->colo_enable)) {
+ flexarray_append(back, "colo-host");
+ flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_host));
+ flexarray_append(back, "colo-port");
+ flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_port));
+ flexarray_append(back, "colo-export");
+ flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_export));
+ flexarray_append(back, "active-disk");
+ flexarray_append(back, libxl__sprintf(gc, "%s", disk->active_disk));
+ flexarray_append(back, "hidden-disk");
+ flexarray_append(back, libxl__sprintf(gc, "%s", disk->hidden_disk));
+ }
assert(device->backend_kind == LIBXL__DEVICE_KIND_QDISK);
break;
default:
@@ -2620,7 +2634,12 @@ static int libxl__device_disk_from_xs_be(libxl__gc *gc,
goto cleanup;
}
- /* "params" may not be present; but everything else must be. */
+ /*
+ * "params" may not be present; but everything else must be.
+ * colo releated entries(colo-host, colo-port, colo-export,
+ * active-disk and hidden-disk) are present only if colo is
+ * enabled.
+ */
tmp = xs_read(ctx->xsh, XBT_NULL,
GCSPRINTF("%s/params", be_path), &len);
if (tmp && strchr(tmp, ':')) {
@@ -2630,6 +2649,49 @@ static int libxl__device_disk_from_xs_be(libxl__gc *gc,
disk->pdev_path = tmp;
}
+ tmp = xs_read(ctx->xsh, XBT_NULL,
+ GCSPRINTF("%s/colo-host", be_path), &len);
+ if (tmp) {
+ libxl_defbool_set(&disk->colo_enable, true);
+ disk->colo_host = tmp;
+ } else {
+ libxl_defbool_set(&disk->colo_enable, false);
+ }
+
+ if (libxl_defbool_val(disk->colo_enable)) {
+ tmp = xs_read(ctx->xsh, XBT_NULL,
+ GCSPRINTF("%s/colo-port", be_path), &len);
+ if (!tmp) {
+ LOG(ERROR, "Missing xenstore node %s/colo-port", be_path);
+ goto cleanup;
+ }
+ disk->colo_port = tmp;
+
+ tmp = xs_read(ctx->xsh, XBT_NULL,
+ GCSPRINTF("%s/colo-export", be_path), &len);
+ if (!tmp) {
+ LOG(ERROR, "Missing xenstore node %s/colo-export", be_path);
+ goto cleanup;
+ }
+ disk->colo_export = tmp;
+
+ tmp = xs_read(ctx->xsh, XBT_NULL,
+ GCSPRINTF("%s/active-disk", be_path), &len);
+ if (!tmp) {
+ LOG(ERROR, "Missing xenstore node %s/active-disk", be_path);
+ goto cleanup;
+ }
+ disk->active_disk = tmp;
+
+ tmp = xs_read(ctx->xsh, XBT_NULL,
+ GCSPRINTF("%s/hidden-disk", be_path), &len);
+ if (!tmp) {
+ LOG(ERROR, "Missing xenstore node %s/hidden-disk", be_path);
+ goto cleanup;
+ }
+ disk->hidden_disk = tmp;
+ }
+
tmp = libxl__xs_read(gc, XBT_NULL,
GCSPRINTF("%s/type", be_path));
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 3610a39..bff08b0 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1800,12 +1800,29 @@ static void domain_create_cb(libxl__egc *egc,
libxl__ao_complete(egc, ao, rc);
}
-
+
+static void set_disk_colo_restore(libxl_domain_config *d_config)
+{
+ int i;
+
+ for (i = 0; i < d_config->num_disks; i++)
+ libxl_defbool_set(&d_config->disks[i].colo_restore_enable, true);
+}
+
+static void unset_disk_colo_restore(libxl_domain_config *d_config)
+{
+ int i;
+
+ for (i = 0; i < d_config->num_disks; i++)
+ libxl_defbool_set(&d_config->disks[i].colo_restore_enable, false);
+}
+
int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
+ unset_disk_colo_restore(d_config);
return do_domain_create(ctx, d_config, domid, -1, -1, NULL,
ao_how, aop_console_how);
}
@@ -1817,6 +1834,12 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
+ if (params->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
+ set_disk_colo_restore(d_config);
+ } else {
+ unset_disk_colo_restore(d_config);
+ }
+
return do_domain_create(ctx, d_config, domid, restore_fd, send_back_fd,
params, ao_how, aop_console_how);
}
diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
index 8bb5e93..039afc6 100644
--- a/tools/libxl/libxl_device.c
+++ b/tools/libxl/libxl_device.c
@@ -196,6 +196,10 @@ static int disk_try_backend(disk_try_backend_args *a,
goto bad_format;
}
+ if (libxl_defbool_val(a->disk->colo_enable) ||
+ a->disk->active_disk || a->disk->hidden_disk)
+ goto bad_colo;
+
if (a->disk->backend_domid != LIBXL_TOOLSTACK_DOMID) {
LOG(DEBUG, "Disk vdev=%s, is using a storage driver domain, "
"skipping physical device check", a->disk->vdev);
@@ -218,6 +222,10 @@ static int disk_try_backend(disk_try_backend_args *a,
case LIBXL_DISK_BACKEND_TAP:
if (a->disk->script) goto bad_script;
+ if (libxl_defbool_val(a->disk->colo_enable) ||
+ a->disk->active_disk || a->disk->hidden_disk)
+ goto bad_colo;
+
if (a->disk->is_cdrom) {
LOG(DEBUG, "Disk vdev=%s, backend tap unsuitable for cdroms",
a->disk->vdev);
@@ -236,6 +244,22 @@ static int disk_try_backend(disk_try_backend_args *a,
case LIBXL_DISK_BACKEND_QDISK:
if (a->disk->script) goto bad_script;
+ if (libxl_defbool_val(a->disk->colo_enable)) {
+ if (!a->disk->colo_host)
+ goto bad_colo_host;
+
+ if (!a->disk->colo_port)
+ goto bad_colo_port;
+
+ if (!a->disk->colo_export)
+ goto bad_colo_export;
+
+ if (!a->disk->active_disk)
+ goto bad_active_disk;
+
+ if (!a->disk->hidden_disk)
+ goto bad_hidden_disk;
+ }
return backend;
default:
@@ -256,6 +280,36 @@ static int disk_try_backend(disk_try_backend_args *a,
LOG(DEBUG, "Disk vdev=%s, backend %s not compatible with script=...",
a->disk->vdev, libxl_disk_backend_to_string(backend));
return 0;
+
+ bad_colo:
+ LOG(DEBUG, "Disk vdev=%s, backend %s not compatible with colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
+
+ bad_colo_host:
+ LOG(DEBUG, "Disk vdev=%s, backend %s needs colo-host=... for colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
+
+ bad_colo_port:
+ LOG(DEBUG, "Disk vdev=%s, backend %s needs colo-port=... for colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
+
+ bad_colo_export:
+ LOG(DEBUG, "Disk vdev=%s, backend %s needs colo-export=... for colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
+
+ bad_active_disk:
+ LOG(DEBUG, "Disk vdev=%s, backend %s needs active-disk=... for colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
+
+ bad_hidden_disk:
+ LOG(DEBUG, "Disk vdev=%s, backend %s needs hidden-disk=... for colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
}
int libxl__device_disk_set_backend(libxl__gc *gc, libxl_device_disk *disk) {
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index 4aca38e..ba17251 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -751,6 +751,139 @@ static int libxl__dm_runas_helper(libxl__gc *gc, const char *username)
}
}
+/* colo mode */
+enum {
+ LIBXL__COLO_NONE = 0,
+ LIBXL__COLO_PRIMARY,
+ LIBXL__COLO_SECONDARY,
+};
+
+static char *qemu_disk_scsi_drive_string(libxl__gc *gc, const char *pdev_path,
+ int unit, const char *format,
+ const libxl_device_disk *disk,
+ int colo_mode)
+{
+ char *drive = NULL;
+ const char *exportname = disk->colo_export;
+ const char *active_disk = disk->active_disk;
+ const char *hidden_disk = disk->hidden_disk;
+
+ switch (colo_mode) {
+ case LIBXL__COLO_NONE:
+ drive = libxl__sprintf
+ (gc, "file=%s,if=scsi,bus=0,unit=%d,format=%s,cache=writeback",
+ pdev_path, unit, format);
+ break;
+ case LIBXL__COLO_PRIMARY:
+ /*
+ * primary:
+ * -dirve if=scsi,bus=0,unit=x,cache=writeback,driver=quorum,\
+ * id=exportname,\
+ * children.0.file.filename=pdev_path,\
+ * children.0.driver=format,\
+ * read-pattern=fifo,\
+ * vote-threshold=1
+ */
+ drive = GCSPRINTF(
+ "if=scsi,bus=0,unit=%d,cache=writeback,driver=quorum,"
+ "id=%s,"
+ "children.0.file.filename=%s,"
+ "children.0.driver=%s,"
+ "read-pattern=fifo,"
+ "vote-threshold=1",
+ unit, exportname, pdev_path, format);
+ break;
+ case LIBXL__COLO_SECONDARY:
+ /*
+ * secondary:
+ * -drive if=scsi,bus=0,unit=x,cache=writeback,driver=replication,\
+ * mode=secondary,\
+ * file.driver=qcow2,\
+ * file.file.filename=active_disk,\
+ * file.backing.driver=qcow2,\
+ * file.backing.file.filename=hidden_disk,\
+ * file.backing.backing=exportname,
+ */
+ drive = GCSPRINTF(
+ "if=scsi,bus=0,unit=%d,cache=writeback,driver=replication,"
+ "mode=secondary,"
+ "file.driver=qcow2,"
+ "file.file.filename=%s,"
+ "file.backing.driver=qcow2,"
+ "file.backing.file.filename=%s,"
+ "file.backing.backing=%s",
+ unit, active_disk, hidden_disk, exportname);
+ break;
+ default:
+ abort();
+ }
+
+ return drive;
+}
+
+static char *qemu_disk_ide_drive_string(libxl__gc *gc, const char *pdev_path,
+ int unit, const char *format,
+ const libxl_device_disk *disk,
+ int colo_mode)
+{
+ char *drive = NULL;
+ const char *exportname = disk->colo_export;
+ const char *active_disk = disk->active_disk;
+ const char *hidden_disk = disk->hidden_disk;
+
+ switch (colo_mode) {
+ case LIBXL__COLO_NONE:
+ drive = GCSPRINTF
+ ("file=%s,if=ide,index=%d,media=disk,format=%s,cache=writeback",
+ pdev_path, unit, format);
+ break;
+ case LIBXL__COLO_PRIMARY:
+ /*
+ * primary:
+ * -dirve if=ide,index=x,media=disk,cache=writeback,driver=quorum,\
+ * id=exportname,\
+ * children.0.file.filename=pdev_path,\
+ * children.0.driver=format,\
+ * read-pattern=fifo,\
+ * vote-threshold=1
+ */
+ drive = GCSPRINTF(
+ "if=ide,index=%d,media=disk,cache=writeback,driver=quorum,"
+ "id=%s,"
+ "children.0.file.filename=%s,"
+ "children.0.driver=%s,"
+ "read-pattern=fifo,"
+ "vote-threshold=1",
+ unit, exportname, pdev_path, format);
+ break;
+ case LIBXL__COLO_SECONDARY:
+ /*
+ * secondary:
+ * -drive if=ide,index=x,media=disk,cache=writeback,driver=replication,\
+ * mode=secondary,\
+ * file.driver=qcow2,\
+ * file.file.filename=active_disk,\
+ * file.backing.driver=qcow2,\
+ * file.backing.file.filename=hidden_disk,\
+ * file.backing.backing=exportname,
+ */
+ drive = GCSPRINTF(
+ "if=ide,index=%d,media=disk,cache=writeback,driver=replication,"
+ "mode=secondary,"
+ "file.driver=qcow2,"
+ "file.file.filename=%s,"
+ "file.backing.driver=qcow2,"
+ "file.backing.file.filename=%s,"
+ "file.backing.backing=%s",
+ unit, active_disk, hidden_disk, exportname);
+ break;
+ default:
+ abort();
+ }
+
+ return drive;
+}
+
static int libxl__build_device_model_args_new(libxl__gc *gc,
const char *dm, int guest_domid,
const libxl_domain_config *guest_config,
@@ -1164,6 +1297,7 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
const char *format = qemu_disk_format_string(disks[i].format);
char *drive;
const char *pdev_path;
+ int colo_mode;
if (dev_number == -1) {
LOG(WARN, "unable to determine"" disk number for %s",
@@ -1208,10 +1342,32 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
* For other disks we translate devices 0..3 into
* hd[a-d] and ignore the rest.
*/
+ if (libxl_defbool_val(disks[i].colo_enable)) {
+ if (libxl_defbool_val(disks[i].colo_restore_enable))
+ colo_mode = LIBXL__COLO_SECONDARY;
+ else
+ colo_mode = LIBXL__COLO_PRIMARY;
+ } else {
+ colo_mode = LIBXL__COLO_NONE;
+ }
+
if (strncmp(disks[i].vdev, "sd", 2) == 0) {
- drive = libxl__sprintf
- (gc, "file=%s,if=scsi,bus=0,unit=%d,format=%s,readonly=%s,cache=writeback",
- pdev_path, disk, format, disks[i].readwrite ? "off" : "on");
+ if (colo_mode == LIBXL__COLO_SECONDARY) {
+ /*
+ * -drive if=none,driver=format,file=pdev_path,\
+ * id=exportname
+ */
+ drive = libxl__sprintf
+ (gc, "if=none,driver=%s,file=%s,id=%s",
+ format, pdev_path, disks[i].colo_export);
+
+ flexarray_append(dm_args, "-drive");
+ flexarray_append(dm_args, drive);
+ }
+ drive = qemu_disk_scsi_drive_string(gc, pdev_path, disk,
+ format,
+ &disks[i],
+ colo_mode);
} else if (strncmp(disks[i].vdev, "xvd", 3) == 0) {
/*
* Do not add any emulated disk when PV disk are
@@ -1234,12 +1390,28 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
LOG(ERROR, "qemu-xen doesn't support read-only IDE disk drivers");
return ERROR_INVAL;
}
- drive = libxl__sprintf
- (gc, "file=%s,if=ide,index=%d,media=disk,format=%s,cache=writeback",
- pdev_path, disk, format);
+ if (colo_mode == LIBXL__COLO_SECONDARY) {
+ /*
+ * -drive if=none,driver=format,file=pdev_path,\
+ * id=exportname
+ */
+ drive = libxl__sprintf
+ (gc, "if=none,driver=%s,file=%s,id=%s",
+ format, pdev_path, disks[i].colo_export);
+
+ flexarray_append(dm_args, "-drive");
+ flexarray_append(dm_args, drive);
+ }
+ drive = qemu_disk_ide_drive_string(gc, pdev_path, disk,
+ format,
+ &disks[i],
+ colo_mode);
} else {
continue; /* Do not emulate this disk */
}
+
+ if (!drive)
+ continue;
}
flexarray_append(dm_args, "-drive");
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 9b0a537..a2078d1 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -575,6 +575,13 @@ libxl_device_disk = Struct("device_disk", [
("is_cdrom", integer),
("direct_io_safe", bool),
("discard_enable", libxl_defbool),
+ ("colo_enable", libxl_defbool),
+ ("colo_restore_enable", libxl_defbool),
+ ("colo_host", string),
+ ("colo_port", string),
+ ("colo_export", string),
+ ("active_disk", string),
+ ("hidden_disk", string)
])
libxl_device_nic = Struct("device_nic", [
diff --git a/tools/libxl/libxlu_disk_l.l b/tools/libxl/libxlu_disk_l.l
index 1a5deb5..58da943 100644
--- a/tools/libxl/libxlu_disk_l.l
+++ b/tools/libxl/libxlu_disk_l.l
@@ -176,6 +176,13 @@ script=[^,]*,? { STRIP(','); SAVESTRING("script", script, FROMEQUALS); }
direct-io-safe,? { DPC->disk->direct_io_safe = 1; }
discard,? { libxl_defbool_set(&DPC->disk->discard_enable, true); }
no-discard,? { libxl_defbool_set(&DPC->disk->discard_enable, false); }
+colo,? { libxl_defbool_set(&DPC->disk->colo_enable, true); }
+no-colo,? { libxl_defbool_set(&DPC->disk->colo_enable, false); }
+colo-host=[^,]*,? { STRIP(','); SAVESTRING("colo-host", colo_host, FROMEQUALS); }
+colo-port=[^,]*,? { STRIP(','); SAVESTRING("colo-port", colo_port, FROMEQUALS); }
+colo-export=[^,]*,? { STRIP(','); SAVESTRING("colo-export", colo_export, FROMEQUALS); }
+active-disk=[^,]*,? { STRIP(','); SAVESTRING("active-disk", active_disk, FROMEQUALS); }
+hidden-disk=[^,]*,? { STRIP(','); SAVESTRING("hidden-disk", hidden_disk, FROMEQUALS); }
/* the target magic parameter, eats the rest of the string */
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-04 8:41 ` [PATCH v11 20/27] Support colo mode for qemu disk Changlong Xie
@ 2016-03-04 17:44 ` Ian Jackson
2016-03-07 2:06 ` Wen Congyang
2016-03-04 17:52 ` Ian Jackson
1 sibling, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:44 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 20/27] Support colo mode for qemu disk"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> Usage: disk = ['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
> For QEMU block replication details:
> http://wiki.qemu.org/Features/BlockReplication
So now I am slightly confused by the design, I think.
When you replicate a VM with COLO using xl, its memory state is
transferred over ssh. But its disk replication is done unencrypted
and unauthenticated ?
And the disk replication is, out of band, and needs to be configured
separately ? This is rather awkward, although maybe not a
showstopper. (Maybe we can have a plan to fix it in the future...)
And, how does the disk replication, which doesn't depend on there
being xl running, relate to the vm state replication, which does ? I
think at the very least I'd like to see some information about the
principles of operation - either explained, or referred to, in the
user manual.
Is it possible to use COLO with an existing full-service disk
replication service such as DRBD ?
> +(a) An example for COLO replication's configuration: disk =['...,colo,colo-host
> +=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
> +
> +=item B<colo-host> :Secondary host's ip address.
> +
> +=item B<colo-port> :Secondary host's port, we will run a nbd server on
> +secondary host, and the nbd server will listen this port.
> +
> +=item B<colo-export> :Nbd server's disk export name of secondary host.
> +
> +=item B<active-disk> :Secondary's guest write will be buffered in this disk,
> +and it's used by secondary.
> +
> +=item B<hidden-disk> :Primary's modified contents will be buffered in this
> +disk, and it's used by secondary.
What would a typical configuration look like ? I don't understand the
relationship between active-disk and hidden-disk, etc.
> +colo-host
> +---------
> +
> +Description: Secondary host's address
> +Mandatory: Yes when COLO enabled
Is it permitted to specify a host DNS name ?
> + if (libxl_defbool_val(disk->colo_enable)) {
> + tmp = xs_read(ctx->xsh, XBT_NULL,
> + GCSPRINTF("%s/colo-port", be_path), &len);
> + if (!tmp) {
> + LOG(ERROR, "Missing xenstore node %s/colo-port", be_path);
> + goto cleanup;
> + }
> + disk->colo_port = tmp;
This is quite repetitive code.
> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
> index 3610a39..bff08b0 100644
> --- a/tools/libxl/libxl_create.c
> +++ b/tools/libxl/libxl_create.c
> @@ -1800,12 +1800,29 @@ static void domain_create_cb(libxl__egc *egc,
...
> @@ -256,6 +280,36 @@ static int disk_try_backend(disk_try_backend_args *a,
> LOG(DEBUG, "Disk vdev=%s, backend %s not compatible with script=...",
> a->disk->vdev, libxl_disk_backend_to_string(backend));
> return 0;
> +
> + bad_colo:
> + LOG(DEBUG, "Disk vdev=%s, backend %s not compatible with colo",
> + a->disk->vdev, libxl_disk_backend_to_string(backend));
> + return 0;
This is correct here, I think.
> + bad_colo_host:
> + LOG(DEBUG, "Disk vdev=%s, backend %s needs colo-host=... for colo",
> + a->disk->vdev, libxl_disk_backend_to_string(backend));
> + return 0;
I think these options should be checked later. disk_try_backend isn't
the place for general parameter checking; it is searching for which
backend to try.
> int libxl__device_disk_set_backend(libxl__gc *gc, libxl_device_disk *disk) {
> diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
> index 4aca38e..ba17251 100644
> --- a/tools/libxl/libxl_dm.c
> +++ b/tools/libxl/libxl_dm.c
> @@ -751,6 +751,139 @@ static int libxl__dm_runas_helper(libxl__gc *gc, const char *username)
...
> +static char *qemu_disk_scsi_drive_string(libxl__gc *gc, const char *pdev_path,
> + int unit, const char *format,
> + const libxl_device_disk *disk,
> + int colo_mode)
> +{
> + char *drive = NULL;
> + const char *exportname = disk->colo_export;
> + const char *active_disk = disk->active_disk;
> + const char *hidden_disk = disk->hidden_disk;
> +
> + switch (colo_mode) {
> + case LIBXL__COLO_NONE:
> + drive = libxl__sprintf
> + (gc, "file=%s,if=scsi,bus=0,unit=%d,format=%s,cache=writeback",
> + pdev_path, unit, format);
I think this would be a lot clearer if the refactoring was split into
a seperate patch.
> if (strncmp(disks[i].vdev, "sd", 2) == 0) {
> - drive = libxl__sprintf
> - (gc, "file=%s,if=scsi,bus=0,unit=%d,format=%s,readonly=%s,cache=writeback",
> - pdev_path, disk, format, disks[i].readwrite ? "off" : "on");
> + if (colo_mode == LIBXL__COLO_SECONDARY) {
> + /*
> + * -drive if=none,driver=format,file=pdev_path,\
> + * id=exportname
> + */
I think this comment adds nothing to the code and could be profitably
omitted.
> + drive = libxl__sprintf
> + (gc, "if=none,driver=%s,file=%s,id=%s",
> + format, pdev_path, disks[i].colo_export);
I don't understand how this works here. COLO_SECONDARY seems to
suppress the rest of the disk specification.
Also, this same logic seems to appear many times. Maybe it could be
centralised. Perhaps I would be able to advise more clearly if I
understood how this was put together.
> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
> index 9b0a537..a2078d1 100644
> --- a/tools/libxl/libxl_types.idl
> +++ b/tools/libxl/libxl_types.idl
> @@ -575,6 +575,13 @@ libxl_device_disk = Struct("device_disk", [
> ("is_cdrom", integer),
> ("direct_io_safe", bool),
> ("discard_enable", libxl_defbool),
> + ("colo_enable", libxl_defbool),
> + ("colo_restore_enable", libxl_defbool),
> + ("colo_host", string),
> + ("colo_port", string),
> + ("colo_export", string),
> + ("active_disk", string),
> + ("hidden_disk", string)
In general, many these should probably not be strings. Certainly the
port should be an integer. I don't quite understand the semantics of
the others.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-04 17:44 ` Ian Jackson
@ 2016-03-07 2:06 ` Wen Congyang
2016-03-17 17:18 ` Ian Jackson
0 siblings, 1 reply; 100+ messages in thread
From: Wen Congyang @ 2016-03-07 2:06 UTC (permalink / raw)
To: Ian Jackson, Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Andrew Cooper, Jiang Yunhong,
Dong Eddie, xen devel, Anthony Perard, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:44 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 20/27] Support colo mode for qemu disk"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> Usage: disk = ['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
>> For QEMU block replication details:
>> http://wiki.qemu.org/Features/BlockReplication
>
> So now I am slightly confused by the design, I think.
>
> When you replicate a VM with COLO using xl, its memory state is
> transferred over ssh. But its disk replication is done unencrypted
> and unauthenticated ?
Yes, it is a problem. I will think how to improve it.
>
> And the disk replication is, out of band, and needs to be configured
> separately ? This is rather awkward, although maybe not a
> showstopper. (Maybe we can have a plan to fix it in the future...)
colo-host,colo-port should be the global configuration. And colo-export,
active-disk,hidden-disk must be configured separately, because each
disk should have a different configuration.
>
> And, how does the disk replication, which doesn't depend on there
> being xl running, relate to the vm state replication, which does ? I
> think at the very least I'd like to see some information about the
> principles of operation - either explained, or referred to, in the
> user manual.
OK. The disk replication doesn't depend on xl. We only can operate it
via qemu monitor command:
1. stop the vm
2. do the checkpoint
3. start the vm
1/3 is suspend/resume the guest. We only need to do 2 when both vm are
in the consistent state.
>
> Is it possible to use COLO with an existing full-service disk
> replication service such as DRBD ?
DRBD doesn's support the case like COLO. Because both primary guest
and secondary guest need to write to the disk.
>
>> +(a) An example for COLO replication's configuration: disk =['...,colo,colo-host
>> +=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
>> +
>> +=item B<colo-host> :Secondary host's ip address.
>> +
>> +=item B<colo-port> :Secondary host's port, we will run a nbd server on
>> +secondary host, and the nbd server will listen this port.
>> +
>> +=item B<colo-export> :Nbd server's disk export name of secondary host.
>> +
>> +=item B<active-disk> :Secondary's guest write will be buffered in this disk,
>> +and it's used by secondary.
>> +
>> +=item B<hidden-disk> :Primary's modified contents will be buffered in this
>> +disk, and it's used by secondary.
>
> What would a typical configuration look like ? I don't understand the
> relationship between active-disk and hidden-disk, etc.
QEMU has a feature: backing file
For example: A's backing file is B
1. If we read from A, but the sector is not allocated in A. We wil return a zero
sector to the guest. If A has a backing file, we will read the sector from B
instead of returning a zero sector.
2. The backing file doesn't affect the write operation.
QEMU has another feature: backup block job
Backup job has two file: one is source and another is the target. It has some running
mode. For block replication, we use the mode "sync=none". In this mode, we will read
the data from the source disk before we modify it, and write it to the target disk.
We keep a bitmap to remeber which sector is backuped from the source disk to the
target disk. If the target disk is an empty disk, and empty disk's backing file is
the source disk, we can read from the target disk to get the source disk's originnal data.
How does block replication work:
A. primary qemu:
1. use the block driver quorum: it will read from all children and write to all children.
child 0: real disk
child 1: nbd client
reading from child 1 will fail, but we use the fifo mode. In this mode, we read from
child 0 will success and we don't read from child 0
write to child 1: because child 1 is nbd client, it will forward the write request to
nbd server
B. secondary qemu:
We have 3 disks: active disk(called it A), hidden disk(called it H), and secondary disk
(real disk, called it S).
A's backing file is H, and H's backing file is S.
We also start a backup job: the source disk is S, and the target disk is H.
we run nbd server in secondary qemu. And the nbd server will write to S.
Before resuming both primary vm and secondary vm: the state is:
1. primary disk and secondary disk are in the consistent state(contain the same data)
2. active disk and hidden disk are the empty disk
When the guest is running:
1. NBD server receives the primary write operation and writes the data to S
2. Before we write data to S, the backup job will read the original data and backup it
to H
3. The secondary vm will write data to A.
4. If secondary vm will read data from A:
I. If the sector is allocated in A, read it from A.
II. Otherwise, the secondary vm doesn't modify this sector after the latest is resumed.
III. In this case, we read it from H. We can read S's original data from H(See the explanation
In backup job).
If we have more than 1 real disk, we can use exportname to tag each disk. Each pair of primary disk and
secondary disk should have the same export name.
>
>> +colo-host
>> +---------
>> +
>> +Description: Secondary host's address
>> +Mandatory: Yes when COLO enabled
>
> Is it permitted to specify a host DNS name ?
IIRC, I think it is OK.
>
>> + if (libxl_defbool_val(disk->colo_enable)) {
>> + tmp = xs_read(ctx->xsh, XBT_NULL,
>> + GCSPRINTF("%s/colo-port", be_path), &len);
>> + if (!tmp) {
>> + LOG(ERROR, "Missing xenstore node %s/colo-port", be_path);
>> + goto cleanup;
>> + }
>> + disk->colo_port = tmp;
>
> This is quite repetitive code.
Yes. Will introduce a new function to avoid it in the next version.
>
>
>> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
>> index 3610a39..bff08b0 100644
>> --- a/tools/libxl/libxl_create.c
>> +++ b/tools/libxl/libxl_create.c
>> @@ -1800,12 +1800,29 @@ static void domain_create_cb(libxl__egc *egc,
> ...
>> @@ -256,6 +280,36 @@ static int disk_try_backend(disk_try_backend_args *a,
>> LOG(DEBUG, "Disk vdev=%s, backend %s not compatible with script=...",
>> a->disk->vdev, libxl_disk_backend_to_string(backend));
>> return 0;
>> +
>> + bad_colo:
>> + LOG(DEBUG, "Disk vdev=%s, backend %s not compatible with colo",
>> + a->disk->vdev, libxl_disk_backend_to_string(backend));
>> + return 0;
>
> This is correct here, I think.
>
>> + bad_colo_host:
>> + LOG(DEBUG, "Disk vdev=%s, backend %s needs colo-host=... for colo",
>> + a->disk->vdev, libxl_disk_backend_to_string(backend));
>> + return 0;
>
> I think these options should be checked later. disk_try_backend isn't
> the place for general parameter checking; it is searching for which
> backend to try.
Hmm, do you mean we check it when we need to use COLO?
>
>> int libxl__device_disk_set_backend(libxl__gc *gc, libxl_device_disk *disk) {
>> diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
>> index 4aca38e..ba17251 100644
>> --- a/tools/libxl/libxl_dm.c
>> +++ b/tools/libxl/libxl_dm.c
>> @@ -751,6 +751,139 @@ static int libxl__dm_runas_helper(libxl__gc *gc, const char *username)
> ...
>> +static char *qemu_disk_scsi_drive_string(libxl__gc *gc, const char *pdev_path,
>> + int unit, const char *format,
>> + const libxl_device_disk *disk,
>> + int colo_mode)
>> +{
>> + char *drive = NULL;
>> + const char *exportname = disk->colo_export;
>> + const char *active_disk = disk->active_disk;
>> + const char *hidden_disk = disk->hidden_disk;
>> +
>> + switch (colo_mode) {
>> + case LIBXL__COLO_NONE:
>> + drive = libxl__sprintf
>> + (gc, "file=%s,if=scsi,bus=0,unit=%d,format=%s,cache=writeback",
>> + pdev_path, unit, format);
>
> I think this would be a lot clearer if the refactoring was split into
> a seperate patch.
OK.
>
>> if (strncmp(disks[i].vdev, "sd", 2) == 0) {
>> - drive = libxl__sprintf
>> - (gc, "file=%s,if=scsi,bus=0,unit=%d,format=%s,readonly=%s,cache=writeback",
>> - pdev_path, disk, format, disks[i].readwrite ? "off" : "on");
>> + if (colo_mode == LIBXL__COLO_SECONDARY) {
>> + /*
>> + * -drive if=none,driver=format,file=pdev_path,\
>> + * id=exportname
>> + */
>
> I think this comment adds nothing to the code and could be profitably
> omitted.
OK.
>
>> + drive = libxl__sprintf
>> + (gc, "if=none,driver=%s,file=%s,id=%s",
>> + format, pdev_path, disks[i].colo_export);
>
> I don't understand how this works here. COLO_SECONDARY seems to
> suppress the rest of the disk specification.
COLO_SECONDAY will use two disks: one is for S, and one is for A.
H is sepecified in A. This line is for S, and the codes for A is
in the function qemu_disk_scsi_drive_string().
>
> Also, this same logic seems to appear many times. Maybe it could be
> centralised. Perhaps I would be able to advise more clearly if I
> understood how this was put together.
>
>> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
>> index 9b0a537..a2078d1 100644
>> --- a/tools/libxl/libxl_types.idl
>> +++ b/tools/libxl/libxl_types.idl
>> @@ -575,6 +575,13 @@ libxl_device_disk = Struct("device_disk", [
>> ("is_cdrom", integer),
>> ("direct_io_safe", bool),
>> ("discard_enable", libxl_defbool),
>> + ("colo_enable", libxl_defbool),
>> + ("colo_restore_enable", libxl_defbool),
>> + ("colo_host", string),
>> + ("colo_port", string),
>> + ("colo_export", string),
>> + ("active_disk", string),
>> + ("hidden_disk", string)
>
> In general, many these should probably not be strings. Certainly the
> port should be an integer. I don't quite understand the semantics of
> the others.
Yes, the port should be an integer. Will fix it in the next version.
Thanks
Wen Congyang
>
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-07 2:06 ` Wen Congyang
@ 2016-03-17 17:18 ` Ian Jackson
2016-03-18 5:42 ` Wen Congyang
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-17 17:18 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Wen Congyang writes ("Re: [PATCH v11 20/27] Support colo mode for qemu disk"):
> How does block replication work:
Thanks for this explanation, which is reallt helpful.
I would like to repeat back to you what I think I have understood:
Between resynchs, you allow each VM to run in parallel and to generate
possibly-divergent disk writes.
So on host B you retain both the A and B disk writes. They are
stored as differences (qcow2) for performance reasons.
If A fails and it becomes necessary to resume the VM only on B, you
use B's version of the VM (both disk and memory).
If B fails then A can use the A version (disk and memory).
If the two are still up, but they diverge in network traffic, you
resynch the memory from A to B, and drop B's disk and replace it with
a copy of A's.
Have I understood correctly ?
If so, what software, where, arranges for the management of the
different qcow2 `layers' ? Ie, what creates the layers; what resynchs
them, etc. ?
The reason I started asking all these questions is because of these
parameters in your disk config:
+ ("colo_enable", libxl_defbool),
+ ("colo_restore_enable", libxl_defbool),
+ ("colo_host", string),
+ ("colo_port", string),
+ ("colo_export", string),
+ ("active_disk", string),
+ ("hidden_disk", string)
For COLO to work properly it is necessary that the `active_disk' and
`hidden_disk' to relate in a specific way to the main disk: they must
be related snapshots (qcow2, currently).
Would it be possible for these disk names to have formulaic,
predicatable, names, so that they wouldn't need to be specified
separately ?
Is there any value in being able to specify them separately ?
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-17 17:18 ` Ian Jackson
@ 2016-03-18 5:42 ` Wen Congyang
0 siblings, 0 replies; 100+ messages in thread
From: Wen Congyang @ 2016-03-18 5:42 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/18/2016 01:18 AM, Ian Jackson wrote:
> Wen Congyang writes ("Re: [PATCH v11 20/27] Support colo mode for qemu disk"):
>> How does block replication work:
>
> Thanks for this explanation, which is reallt helpful.
>
> I would like to repeat back to you what I think I have understood:
>
> Between resynchs, you allow each VM to run in parallel and to generate
> possibly-divergent disk writes.
>
> So on host B you retain both the A and B disk writes. They are
> stored as differences (qcow2) for performance reasons.
>
> If A fails and it becomes necessary to resume the VM only on B, you
> use B's version of the VM (both disk and memory).
>
> If B fails then A can use the A version (disk and memory).
>
> If the two are still up, but they diverge in network traffic, you
> resynch the memory from A to B, and drop B's disk and replace it with
> a copy of A's.
>
> Have I understood correctly ?
Yes.
>
>
> If so, what software, where, arranges for the management of the
> different qcow2 `layers' ? Ie, what creates the layers; what resynchs
> them, etc. ?
active disk and hidden disk are seperate disk. The management application
can create an empty qcow disk before running COLO. These two disks are
empty disk, and have the same size with the secondary disk.
Thanks
Wen Congyang
>
> The reason I started asking all these questions is because of these
> parameters in your disk config:
>
> + ("colo_enable", libxl_defbool),
> + ("colo_restore_enable", libxl_defbool),
> + ("colo_host", string),
> + ("colo_port", string),
> + ("colo_export", string),
> + ("active_disk", string),
> + ("hidden_disk", string)
>
> For COLO to work properly it is necessary that the `active_disk' and
> `hidden_disk' to relate in a specific way to the main disk: they must
> be related snapshots (qcow2, currently).
>
> Would it be possible for these disk names to have formulaic,
> predicatable, names, so that they wouldn't need to be specified
> separately ?
>
> Is there any value in being able to specify them separately ?
>
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-04 8:41 ` [PATCH v11 20/27] Support colo mode for qemu disk Changlong Xie
2016-03-04 17:44 ` Ian Jackson
@ 2016-03-04 17:52 ` Ian Jackson
2016-03-04 20:30 ` Konrad Rzeszutek Wilk
1 sibling, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:52 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 20/27] Support colo mode for qemu disk"):
> +Enable COLO HA for disk. For better understanding block replication on
> +QEMU, please refer to:
> +http://wiki.qemu.org/Features/BlockReplication
Sorry, I missed this link on my first pass. I still think that at the
very least this needs something more user-facing (ie, how should one
set this up).
But, I'm kind of worried that qemu is the wrong place to be doing
this.
How can this be made to work with PV guests ?
What if an HVM guest has PV-on-HVM drivers ? In this case there might
be two relevant qemus, one for the qdisk Xen PV block backend, and one
for the emulated IDE.
I don't understand how discrepant writes are detected. Surely they
might occur and should trigger a resynch ?
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-04 17:52 ` Ian Jackson
@ 2016-03-04 20:30 ` Konrad Rzeszutek Wilk
2016-03-07 2:10 ` Wen Congyang
2016-03-17 17:10 ` Ian Jackson
0 siblings, 2 replies; 100+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-03-04 20:30 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Anthony Perard, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On Fri, Mar 04, 2016 at 05:52:09PM +0000, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 20/27] Support colo mode for qemu disk"):
> > +Enable COLO HA for disk. For better understanding block replication on
> > +QEMU, please refer to:
> > +http://wiki.qemu.org/Features/BlockReplication
>
> Sorry, I missed this link on my first pass. I still think that at the
> very least this needs something more user-facing (ie, how should one
> set this up).
>
> But, I'm kind of worried that qemu is the wrong place to be doing
> this.
>
> How can this be made to work with PV guests ?
QEMU can also serve PV guests (qdisk).
I think your question is more of - what about making this work with
PV block backend?
>
> What if an HVM guest has PV-on-HVM drivers ? In this case there might
> be two relevant qemus, one for the qdisk Xen PV block backend, and one
> for the emulated IDE.
In both cases QEMU would use the same underlaying API to actually write/read
out the blocks. That API would then use NBD, etc to replicate writes.
Maybe a little ASCII art?
qdisk ide
\ /
\ /
block API
|
QCOW2
|
NBD
Or such?
>
> I don't understand how discrepant writes are detected. Surely they
> might occur and should trigger a resynch ?
>
> Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-04 20:30 ` Konrad Rzeszutek Wilk
@ 2016-03-07 2:10 ` Wen Congyang
2016-03-08 17:22 ` Wei Liu
2016-03-17 17:10 ` Ian Jackson
1 sibling, 1 reply; 100+ messages in thread
From: Wen Congyang @ 2016-03-07 2:10 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk, Ian Jackson
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 04:30 AM, Konrad Rzeszutek Wilk wrote:
> On Fri, Mar 04, 2016 at 05:52:09PM +0000, Ian Jackson wrote:
>> Changlong Xie writes ("[PATCH v11 20/27] Support colo mode for qemu disk"):
>>> +Enable COLO HA for disk. For better understanding block replication on
>>> +QEMU, please refer to:
>>> +http://wiki.qemu.org/Features/BlockReplication
>>
>> Sorry, I missed this link on my first pass. I still think that at the
>> very least this needs something more user-facing (ie, how should one
>> set this up).
>>
>> But, I'm kind of worried that qemu is the wrong place to be doing
>> this.
>>
>> How can this be made to work with PV guests ?
>
> QEMU can also serve PV guests (qdisk).
>
> I think your question is more of - what about making this work with
> PV block backend?
I don't know how to work with PV block backend. It is one reason that
why we only support pure HVM now.
For PV block backend, there is also other problem. For exampe resuming
it in the secondary side is very slow, because we need to disconnect and
reconnect.
Thanks
Wen Congyang
>>
>> What if an HVM guest has PV-on-HVM drivers ? In this case there might
>> be two relevant qemus, one for the qdisk Xen PV block backend, and one
>> for the emulated IDE.
>
> In both cases QEMU would use the same underlaying API to actually write/read
> out the blocks. That API would then use NBD, etc to replicate writes.
>
> Maybe a little ASCII art?
>
> qdisk ide
> \ /
> \ /
> block API
> |
> QCOW2
> |
> NBD
>
> Or such?
>
>>
>> I don't understand how discrepant writes are detected. Surely they
>> might occur and should trigger a resynch ?
>>
>> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-07 2:10 ` Wen Congyang
@ 2016-03-08 17:22 ` Wei Liu
2016-03-09 2:09 ` Konrad Rzeszutek Wilk
2016-03-17 17:09 ` Ian Jackson
0 siblings, 2 replies; 100+ messages in thread
From: Wei Liu @ 2016-03-08 17:22 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Anthony Perard,
Dong Eddie, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On Mon, Mar 07, 2016 at 10:10:07AM +0800, Wen Congyang wrote:
> On 03/05/2016 04:30 AM, Konrad Rzeszutek Wilk wrote:
> > On Fri, Mar 04, 2016 at 05:52:09PM +0000, Ian Jackson wrote:
> >> Changlong Xie writes ("[PATCH v11 20/27] Support colo mode for qemu disk"):
> >>> +Enable COLO HA for disk. For better understanding block replication on
> >>> +QEMU, please refer to:
> >>> +http://wiki.qemu.org/Features/BlockReplication
> >>
> >> Sorry, I missed this link on my first pass. I still think that at the
> >> very least this needs something more user-facing (ie, how should one
> >> set this up).
> >>
> >> But, I'm kind of worried that qemu is the wrong place to be doing
> >> this.
> >>
> >> How can this be made to work with PV guests ?
> >
> > QEMU can also serve PV guests (qdisk).
> >
> > I think your question is more of - what about making this work with
> > PV block backend?
>
> I don't know how to work with PV block backend. It is one reason that
> why we only support pure HVM now.
> For PV block backend, there is also other problem. For exampe resuming
> it in the secondary side is very slow, because we need to disconnect and
> reconnect.
>
Supporting PV guest is certainly going to be non-trivial. And I don't
think we would ever ask you to actually implement that.
The point is to have a story that when other people want to implement
COLO for PV-aware guests (PVHVM, PV and PVH), they are not crippled by
existing interfaces.
Currently the disk spec seems to be designed exclusively for QEMU. This
is not very desirable, but at least it wouldn't stop people from either
reusing them or inventing new parameters.
Furthermore, I think coming up with a story for PV-aware guests (PVHVM,
PV and PVH) is also non-trivial. For one the disk replication logic is
not implemented in PV block backend, we're not sure how feasible to
replicate thing in QEMU into kernel, but we're quite sure it is not
going to be trivial technically and politically. The uncertainty is too
big to come up with a clear idea what it would look like.
Wei.
> Thanks
> Wen Congyang
>
> >>
> >> What if an HVM guest has PV-on-HVM drivers ? In this case there might
> >> be two relevant qemus, one for the qdisk Xen PV block backend, and one
> >> for the emulated IDE.
> >
> > In both cases QEMU would use the same underlaying API to actually write/read
> > out the blocks. That API would then use NBD, etc to replicate writes.
> >
> > Maybe a little ASCII art?
> >
> > qdisk ide
> > \ /
> > \ /
> > block API
> > |
> > QCOW2
> > |
> > NBD
> >
> > Or such?
> >
> >>
> >> I don't understand how discrepant writes are detected. Surely they
> >> might occur and should trigger a resynch ?
> >>
> >> Ian.
> >
> >
> > .
> >
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-08 17:22 ` Wei Liu
@ 2016-03-09 2:09 ` Konrad Rzeszutek Wilk
2016-03-09 16:55 ` Wei Liu
2016-03-17 17:09 ` Ian Jackson
1 sibling, 1 reply; 100+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-03-09 2:09 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Ian Jackson, xen devel,
Anthony Perard, Dong Eddie, Gui Jianfeng, Shriram Rajagopalan,
Yang Hongyang
> Furthermore, I think coming up with a story for PV-aware guests (PVHVM,
> PV and PVH) is also non-trivial. For one the disk replication logic is
Pls keep in mind that PV guests can use QEMU qdisk if they wish.
This is more of 'phy' vs 'file' question.
> not implemented in PV block backend, we're not sure how feasible to
> replicate thing in QEMU into kernel, but we're quite sure it is not
> going to be trivial technically and politically. The uncertainty is too
.. and I think doing it in the kernel is not a good idea. With PVH
coming as the initial domain - we will now have solved the @$@#@@
syscall bounce to hypervisor slowpath - which means that any libaio
operations QEMU does on a file - which ends up going to the kernel
with a syscall - will happen without incuring a huge context switch!
In other words the speed difference between qdisk and xen-blkback
will now be just the syscall + aio code + fs overhead.
With proper batching this can be made faster*.
*: I am ignoring the other issues such as grant unmap in PVH guests
and qdisk being quite spartan in features compared to xen-blkback.
> big to come up with a clear idea what it would look like.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-09 2:09 ` Konrad Rzeszutek Wilk
@ 2016-03-09 16:55 ` Wei Liu
0 siblings, 0 replies; 100+ messages in thread
From: Wei Liu @ 2016-03-09 16:55 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Ian Jackson, xen devel,
Anthony Perard, Dong Eddie, Gui Jianfeng, Shriram Rajagopalan,
Yang Hongyang
On Tue, Mar 08, 2016 at 09:09:20PM -0500, Konrad Rzeszutek Wilk wrote:
> > Furthermore, I think coming up with a story for PV-aware guests (PVHVM,
> > PV and PVH) is also non-trivial. For one the disk replication logic is
>
> Pls keep in mind that PV guests can use QEMU qdisk if they wish.
>
Oh right, I somehow talked myself into thinking blkback only.
> This is more of 'phy' vs 'file' question.
> > not implemented in PV block backend, we're not sure how feasible to
> > replicate thing in QEMU into kernel, but we're quite sure it is not
> > going to be trivial technically and politically. The uncertainty is too
>
> .. and I think doing it in the kernel is not a good idea. With PVH
> coming as the initial domain - we will now have solved the @$@#@@
> syscall bounce to hypervisor slowpath - which means that any libaio
> operations QEMU does on a file - which ends up going to the kernel
> with a syscall - will happen without incuring a huge context switch!
>
> In other words the speed difference between qdisk and xen-blkback
> will now be just the syscall + aio code + fs overhead.
>
> With proper batching this can be made faster*.
>
> *: I am ignoring the other issues such as grant unmap in PVH guests
> and qdisk being quite spartan in features compared to xen-blkback.
>
So now we do have a tangible story for PV-aware guest for disk
replication (note, other hindrance such as ckeckpointing the kernel in a
fast enough pace still needs to be solved) -- to use qdisk backend. That
means the current interfaces should be sufficient.
Wei.
> > big to come up with a clear idea what it would look like.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-08 17:22 ` Wei Liu
2016-03-09 2:09 ` Konrad Rzeszutek Wilk
@ 2016-03-17 17:09 ` Ian Jackson
1 sibling, 0 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-17 17:09 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Anthony Perard, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Wei Liu writes ("Re: [PATCH v11 20/27] Support colo mode for qemu disk"):
> Supporting PV guest is certainly going to be non-trivial. And I don't
> think we would ever ask you to actually implement that.
Indeed.
> The point is to have a story that when other people want to implement
> COLO for PV-aware guests (PVHVM, PV and PVH), they are not crippled by
> existing interfaces.
>
> Currently the disk spec seems to be designed exclusively for QEMU. This
> is not very desirable, but at least it wouldn't stop people from either
> reusing them or inventing new parameters.
I think in fact (following some in-person conversations) that I am
comfortable with this implementation requiring qemu right now.
Future PV[H] COLO arrangements might well use qdisk anyway. I think
it is OK for libxl to decide that qemu is needed in this case. All
that's needed now is to arrange that: if someone, in the future, wants
to make a version of COLO that works without qemu somehow then they
can do that without having to significantly change the libxl API.
So all that's needed is for the interface to libxl not to imply that
qemu is in use. I don't think the interface proposed here implies
that qemu is in use.
The proposed interface seems to mostly speak about things which are
not qemu-specific, so it's probably OK.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH v11 20/27] Support colo mode for qemu disk
2016-03-04 20:30 ` Konrad Rzeszutek Wilk
2016-03-07 2:10 ` Wen Congyang
@ 2016-03-17 17:10 ` Ian Jackson
1 sibling, 0 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-17 17:10 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Anthony Perard, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Konrad Rzeszutek Wilk writes ("Re: [PATCH v11 20/27] Support colo mode for qemu disk"):
> On Fri, Mar 04, 2016 at 05:52:09PM +0000, Ian Jackson wrote:
> > How can this be made to work with PV guests ?
>
> QEMU can also serve PV guests (qdisk).
Yes.
> I think your question is more of - what about making this work with
> PV block backend?
Yes.
> > What if an HVM guest has PV-on-HVM drivers ? In this case there might
> > be two relevant qemus, one for the qdisk Xen PV block backend, and one
> > for the emulated IDE.
>
> In both cases QEMU would use the same underlaying API to actually write/read
> out the blocks. That API would then use NBD, etc to replicate writes.
>
> Maybe a little ASCII art?
>
> qdisk ide
> \ /
> \ /
> block API
> |
> QCOW2
> |
> NBD
Except that currently libxl may launch two qemus: one to serve
emulated ide requests, and one to service PV qdisk.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 21/27] COLO: use qemu block replication
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (19 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 20/27] Support colo mode for qemu disk Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 8:41 ` [PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module Changlong Xie
` (6 subsequent siblings)
27 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Use qemu block replication as our block replication solution.
Note that guest must be paused before starting COLO, otherwise,
the disk won't be consistent between primary and secondary.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxl/Makefile | 1 +
tools/libxl/libxl_colo.h | 15 +++
tools/libxl/libxl_colo_qdisk.c | 222 +++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_colo_restore.c | 42 +++++++-
tools/libxl/libxl_colo_save.c | 54 +++++++++-
tools/libxl/libxl_internal.h | 5 +
6 files changed, 334 insertions(+), 5 deletions(-)
create mode 100644 tools/libxl/libxl_colo_qdisk.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 2016393..28d54d0 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -66,6 +66,7 @@ endif
LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
+LIBXL_OBJS-y += libxl_colo_qdisk.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index feec7f1..90345f4 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -19,6 +19,7 @@
struct libxl__ao;
struct libxl__egc;
struct libxl__colo_save_state;
+struct libxl__checkpoint_devices_state;
enum {
LIBXL_COLO_SETUPED,
@@ -26,6 +27,10 @@ enum {
LIBXL_COLO_RESUMED,
};
+typedef struct libxl__colo_qdisk {
+ bool setuped;
+} libxl__colo_qdisk;
+
typedef struct libxl__domain_create_state libxl__domain_create_state;
typedef void libxl__domain_create_cb(struct libxl__egc *egc,
libxl__domain_create_state *dcs,
@@ -47,8 +52,18 @@ struct libxl__colo_restore_state {
/* private, colo restore checkpoint state */
libxl__domain_create_cb *saved_cb;
void *crcs;
+
+ /* private, used by qdisk block replication */
+ bool qdisk_used;
+ bool qdisk_setuped;
+ const char *host;
+ const char *port;
};
+int init_subkind_qdisk(struct libxl__checkpoint_devices_state *cds);
+
+void cleanup_subkind_qdisk(struct libxl__checkpoint_devices_state *cds);
+
extern void libxl__colo_restore_setup(struct libxl__egc *egc,
libxl__colo_restore_state *crs);
extern void libxl__colo_restore_teardown(struct libxl__egc *egc, void *dcs_void,
diff --git a/tools/libxl/libxl_colo_qdisk.c b/tools/libxl/libxl_colo_qdisk.c
new file mode 100644
index 0000000..7af7ac1
--- /dev/null
+++ b/tools/libxl/libxl_colo_qdisk.c
@@ -0,0 +1,222 @@
+/*
+ * Copyright (C) 2015 FUJITSU LIMITED
+ * Author: Wen Congyang <wency@cn.fujitsu.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/* ========== init() and cleanup() ========== */
+int init_subkind_qdisk(libxl__checkpoint_devices_state *cds)
+{
+ /*
+ * We don't know if we use qemu block replication, so
+ * we cannot start block replication here.
+ */
+ return 0;
+}
+
+void cleanup_subkind_qdisk(libxl__checkpoint_devices_state *cds)
+{
+}
+
+/* ========== setup() and teardown() ========== */
+static void colo_qdisk_setup(libxl__egc *egc, libxl__checkpoint_device *dev,
+ bool primary)
+{
+ const libxl_device_disk *disk = dev->backend_dev;
+ int ret, rc = 0;
+ libxl__colo_qdisk *colo_qdisk = NULL;
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *const cds = dev->cds;
+ const char *host = disk->colo_host;
+ const char *port = disk->colo_port;
+ const char *export_name = disk->colo_export;
+ const int domid = cds->domid;
+
+ STATE_AO_GC(dev->cds->ao);
+
+ if (disk->backend != LIBXL_DISK_BACKEND_QDISK ||
+ !libxl_defbool_val(disk->colo_enable)) {
+ rc = ERROR_CHECKPOINT_DEVOPS_DOES_NOT_MATCH;
+ goto out;
+ }
+
+ dev->matched = true;
+
+ GCNEW(colo_qdisk);
+ dev->concrete_data = colo_qdisk;
+
+ if (primary) {
+ libxl__colo_save_state *css = cds->concrete_data;
+
+ css->qdisk_used = true;
+ /* NBD server is not ready, so we cannot start block replication now */
+ goto out;
+ } else {
+ libxl__colo_restore_state *crs = cds->concrete_data;
+
+ if (!crs->qdisk_used) {
+ /* start nbd server */
+ ret = libxl__qmp_nbd_server_start(gc, domid, host, port);
+ if (ret) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ crs->host = host;
+ crs->port = port;
+ } else {
+ if (strcmp(crs->host, host) || strcmp(crs->port, port)) {
+ LOG(ERROR, "The host and port of all disks must be the same");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ }
+
+ crs->qdisk_used = true;
+
+ ret = libxl__qmp_nbd_server_add(gc, domid, export_name);
+ if (ret)
+ rc = ERROR_FAIL;
+
+ colo_qdisk->setuped = true;
+ }
+
+out:
+ dev->aodev.rc = rc;
+ dev->aodev.callback(egc, &dev->aodev);
+}
+
+static void colo_qdisk_teardown(libxl__egc *egc, libxl__checkpoint_device *dev,
+ bool primary)
+{
+ int ret, rc = 0;
+ const libxl__colo_qdisk *colo_qdisk = dev->concrete_data;
+ const libxl_device_disk *disk = dev->backend_dev;
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *const cds = dev->cds;
+ const int domid = cds->domid;
+ const char *export_name = disk->colo_export;
+
+ EGC_GC;
+
+ if (primary) {
+ if (!colo_qdisk->setuped)
+ goto out;
+
+ /*
+ * There is no way to get the child name, but we know it is children.1
+ */
+ ret = libxl__qmp_x_blockdev_change(gc, domid, export_name,
+ "children.1", NULL);
+ if (ret)
+ rc = ERROR_FAIL;
+ } else {
+ libxl__colo_restore_state *crs = cds->concrete_data;
+
+ if (crs->qdisk_used) {
+ ret = libxl__qmp_nbd_server_stop(gc, domid);
+ if (ret)
+ rc = ERROR_FAIL;
+ }
+ }
+
+out:
+ dev->aodev.rc = rc;
+ dev->aodev.callback(egc, &dev->aodev);
+}
+
+/* ========== checkpointing APIs ========== */
+static void colo_qdisk_save_preresume(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ libxl__colo_qdisk *colo_qdisk = dev->concrete_data;
+ const libxl_device_disk *disk = dev->backend_dev;
+ int ret, rc = 0;
+ char *node = NULL;
+ char *cmd = NULL;
+
+ /* Convenience aliases */
+ const int domid = dev->cds->domid;
+ const char *host = disk->colo_host;
+ const char *port = disk->colo_port;
+ const char *export_name = disk->colo_export;
+
+ EGC_GC;
+
+ if (colo_qdisk->setuped)
+ goto out;
+
+ /* qmp command doesn't support the driver "nbd" */
+ node = GCSPRINTF("colo_node%d",
+ libxl__device_disk_dev_number(disk->vdev, NULL, NULL));
+ cmd = GCSPRINTF("drive_add buddy driver=replication,mode=primary,"
+ "file.driver=nbd,file.host=%s,file.port=%s,"
+ "file.export=%s,node-name=%s,if=none",
+ host, port, export_name, node);
+ ret = libxl__qmp_hmp(gc, domid, cmd);
+ if (ret)
+ rc = ERROR_FAIL;
+
+ ret = libxl__qmp_x_blockdev_change(gc, domid, export_name, NULL, node);
+ if (ret)
+ rc = ERROR_FAIL;
+
+ colo_qdisk->setuped = true;
+
+out:
+ dev->aodev.rc = rc;
+ dev->aodev.callback(egc, &dev->aodev);
+}
+
+/* ======== primary ======== */
+static void colo_qdisk_save_setup(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ colo_qdisk_setup(egc, dev, true);
+}
+
+static void colo_qdisk_save_teardown(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ colo_qdisk_teardown(egc, dev, true);
+}
+
+const libxl__checkpoint_device_instance_ops colo_save_device_qdisk = {
+ .kind = LIBXL__DEVICE_KIND_VBD,
+ .setup = colo_qdisk_save_setup,
+ .teardown = colo_qdisk_save_teardown,
+ .preresume = colo_qdisk_save_preresume,
+};
+
+/* ======== secondary ======== */
+static void colo_qdisk_restore_setup(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ colo_qdisk_setup(egc, dev, false);
+}
+
+static void colo_qdisk_restore_teardown(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ colo_qdisk_teardown(egc, dev, false);
+}
+
+const libxl__checkpoint_device_instance_ops colo_restore_device_qdisk = {
+ .kind = LIBXL__DEVICE_KIND_VBD,
+ .setup = colo_qdisk_restore_setup,
+ .teardown = colo_qdisk_restore_teardown,
+};
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index b0dd52b..8b343c0 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -37,7 +37,10 @@ struct libxl__colo_restore_checkpoint_state {
int);
};
+extern const libxl__checkpoint_device_instance_ops colo_restore_device_qdisk;
+
static const libxl__checkpoint_device_instance_ops *colo_restore_ops[] = {
+ &colo_restore_device_qdisk,
NULL,
};
@@ -136,7 +139,11 @@ static int init_device_subkind(libxl__checkpoint_devices_state *cds)
int rc;
STATE_AO_GC(cds->ao);
+ rc = init_subkind_qdisk(cds);
+ if (rc) goto out;
+
rc = 0;
+out:
return rc;
}
@@ -144,6 +151,8 @@ static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
{
/* cleanup device subkind-specific state in the libxl ctx */
STATE_AO_GC(cds->ao);
+
+ cleanup_subkind_qdisk(cds);
}
@@ -212,6 +221,8 @@ void libxl__colo_restore_setup(libxl__egc *egc,
GCNEW(crcs);
crs->crcs = crcs;
crcs->crs = crs;
+ crs->qdisk_setuped = false;
+ crs->qdisk_used = false;
/* setup dsps */
crcs->dsps.ao = ao;
@@ -299,6 +310,11 @@ void libxl__colo_restore_teardown(libxl__egc *egc, void *dcs_void,
}
libxl__xc_domain_restore_done(egc, dcs, ret, retval, errnoval);
+ if (crs->qdisk_setuped) {
+ libxl__qmp_stop_replication(gc, crs->domid, false);
+ crs->qdisk_setuped = false;
+ }
+
crcs->saved_rc = rc;
if (!crcs->teardown_devices) {
colo_restore_teardown_devices_done(egc, &dcs->cds, 0);
@@ -569,6 +585,13 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
goto out;
}
+ if (crs->qdisk_setuped) {
+ if (libxl__qmp_do_checkpoint(gc, crs->domid)) {
+ LOG(ERROR, "doing checkpoint fails");
+ goto out;
+ }
+ }
+
colo_restore_resume_vm(egc, crcs);
return;
@@ -726,8 +749,8 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
STATE_AO_GC(crs->ao);
- /* TODO: disk/nic support */
- cds->device_kind_flags = 0;
+ /* TODO: nic support */
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
cds->callback = colo_restore_setup_cds_done;
cds->ao = ao;
cds->domid = crs->domid;
@@ -764,6 +787,14 @@ static void colo_restore_setup_cds_done(libxl__egc *egc,
goto out;
}
+ if (crs->qdisk_used && !crs->qdisk_setuped) {
+ if (libxl__qmp_start_replication(gc, crs->domid, false)) {
+ LOG(ERROR, "starting replication fails");
+ goto out;
+ }
+ crs->qdisk_setuped = true;
+ }
+
colo_send_svm_ready(egc, crcs);
return;
@@ -918,13 +949,18 @@ static void colo_suspend_vm_done(libxl__egc *egc,
crcs->status = LIBXL_COLO_SUSPENDED;
+ if (libxl__qmp_get_replication_error(gc, crs->domid)) {
+ LOG(ERROR, "replication error occurs when secondary vm is running");
+ goto out;
+ }
+
cds->callback = colo_restore_postsuspend_cb;
libxl__checkpoint_devices_postsuspend(egc, cds);
return;
out:
- libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, !rc);
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, 0);
}
static void colo_restore_postsuspend_cb(libxl__egc *egc,
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index e318ea2..97efc8d 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -18,7 +18,10 @@
#include "libxl_internal.h"
+extern const libxl__checkpoint_device_instance_ops colo_save_device_qdisk;
+
static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
+ &colo_save_device_qdisk,
NULL,
};
@@ -29,7 +32,11 @@ static int init_device_subkind(libxl__checkpoint_devices_state *cds)
int rc;
STATE_AO_GC(cds->ao);
+ rc = init_subkind_qdisk(cds);
+ if (rc) goto out;
+
rc = 0;
+out:
return rc;
}
@@ -37,6 +44,8 @@ static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
{
/* cleanup device subkind-specific state in the libxl ctx */
STATE_AO_GC(cds->ao);
+
+ cleanup_subkind_qdisk(cds);
}
/* ================= colo: setup save environment ================= */
@@ -77,9 +86,12 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->send_fd = dss->fd;
css->recv_fd = dss->recv_fd;
css->svm_running = false;
+ css->paused = true;
+ css->qdisk_setuped = false;
+ css->qdisk_used = false;
- /* TODO: disk/nic support */
- cds->device_kind_flags = 0;
+ /* TODO: nic support */
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
cds->ops = colo_ops;
cds->callback = colo_save_setup_done;
cds->ao = ao;
@@ -159,6 +171,11 @@ void libxl__colo_save_teardown(libxl__egc *egc,
libxl__stream_read_abort(egc, &css->srs, 1);
+ if (css->qdisk_setuped) {
+ libxl__qmp_stop_replication(gc, dss->domid, true);
+ css->qdisk_setuped = false;
+ }
+
dss->cds.callback = colo_teardown_done;
libxl__checkpoint_devices_teardown(egc, &dss->cds);
return;
@@ -286,6 +303,11 @@ static void colo_read_svm_suspended_done(libxl__egc *egc,
goto out;
}
+ if (!css->paused && libxl__qmp_get_replication_error(gc, dss->domid)) {
+ LOG(ERROR, "replication error occurs when primary vm is running");
+ goto out;
+ }
+
ok = 1;
out:
@@ -383,12 +405,40 @@ static void colo_preresume_cb(libxl__egc *egc,
goto out;
}
+ if (css->qdisk_used && !css->qdisk_setuped) {
+ if (libxl__qmp_start_replication(gc, dss->domid, true)) {
+ LOG(ERROR, "starting replication fails");
+ goto out;
+ }
+ css->qdisk_setuped = true;
+ }
+
+ if (!css->paused) {
+ if (libxl__qmp_do_checkpoint(gc, dss->domid)) {
+ LOG(ERROR, "doing checkpoint fails");
+ goto out;
+ }
+ }
+
/* Resumes the domain and the device model */
if (libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1)) {
LOG(ERROR, "cannot resume primary vm");
goto out;
}
+ /*
+ * The guest should be paused before doing colo because there is
+ * no disk migration.
+ */
+ if (css->paused) {
+ rc = libxl_domain_unpause(CTX, dss->domid);
+ if (rc) {
+ LOG(ERROR, "cannot unpause primary vm");
+ goto out;
+ }
+ css->paused = false;
+ }
+
/* read CHECKPOINT_SVM_RESUMED */
css->callback = colo_read_svm_resumed_done;
css->srs.checkpoint_callback = colo_common_read_stream_done;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 8cb9f19..0aeaf18 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3189,6 +3189,11 @@ struct libxl__colo_save_state {
libxl__stream_read_state srs;
void (*callback)(libxl__egc *, libxl__colo_save_state *, int);
bool svm_running;
+ bool paused;
+
+ /* private, used by qdisk block replication */
+ bool qdisk_used;
+ bool qdisk_setuped;
};
typedef struct libxl__logdirty_switch {
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* [PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (20 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 21/27] COLO: use qemu block replication Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 17:59 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 23/27] COLO proxy: preresume, postresume and checkpoint Changlong Xie
` (5 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
setup/teardown of COLO proxy module.
we use netlink to communicate with proxy module.
About colo-proxy module:
https://lkml.org/lkml/2015/6/18/32
How to use:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxl/Makefile | 1 +
tools/libxl/libxl_colo.h | 27 +++++
tools/libxl/libxl_colo_proxy.c | 218 +++++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_internal.h | 3 +
4 files changed, 249 insertions(+)
create mode 100644 tools/libxl/libxl_colo_proxy.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 28d54d0..6fea9e0 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -67,6 +67,7 @@ endif
LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
LIBXL_OBJS-y += libxl_colo_qdisk.o
+LIBXL_OBJS-y += libxl_colo_proxy.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 90345f4..9e7f99c 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -16,21 +16,42 @@
#ifndef LIBXL_COLO_H
#define LIBXL_COLO_H
+#include <linux/netlink.h>
+
struct libxl__ao;
struct libxl__egc;
struct libxl__colo_save_state;
struct libxl__checkpoint_devices_state;
+#define NETLINK_COLO 28
+
enum {
LIBXL_COLO_SETUPED,
LIBXL_COLO_SUSPENDED,
LIBXL_COLO_RESUMED,
};
+enum colo_netlink_op {
+ COLO_QUERY_CHECKPOINT = (NLMSG_MIN_TYPE + 1),
+ COLO_CHECKPOINT,
+ COLO_FAILOVER,
+ COLO_PROXY_INIT,
+ COLO_PROXY_RESET, /* UNUSED, will be used for continuous FT */
+};
+
typedef struct libxl__colo_qdisk {
bool setuped;
} libxl__colo_qdisk;
+typedef struct libxl__colo_proxy_state libxl__colo_proxy_state;
+struct libxl__colo_proxy_state {
+ /* set by caller of colo_proxy_setup */
+ struct libxl__ao *ao;
+
+ int sock_fd;
+ int index;
+};
+
typedef struct libxl__domain_create_state libxl__domain_create_state;
typedef void libxl__domain_create_cb(struct libxl__egc *egc,
libxl__domain_create_state *dcs,
@@ -58,6 +79,9 @@ struct libxl__colo_restore_state {
bool qdisk_setuped;
const char *host;
const char *port;
+
+ /* private, used by colo-proxy */
+ libxl__colo_proxy_state cps;
};
int init_subkind_qdisk(struct libxl__checkpoint_devices_state *cds);
@@ -73,4 +97,7 @@ extern void libxl__colo_save_setup(struct libxl__egc *egc,
extern void libxl__colo_save_teardown(struct libxl__egc *egc,
struct libxl__colo_save_state *css,
int rc);
+extern int colo_proxy_setup(libxl__colo_proxy_state *cps);
+extern void colo_proxy_teardown(libxl__colo_proxy_state *cps);
+
#endif
diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
new file mode 100644
index 0000000..2b3baa3
--- /dev/null
+++ b/tools/libxl/libxl_colo_proxy.c
@@ -0,0 +1,218 @@
+/*
+ * Copyright (C) 2015 FUJITSU LIMITED
+ * Author: Yang Hongyang <hongyang.yang@easystack.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/* ========= colo-proxy: helper functions ========== */
+
+static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
+ uint64_t size, int type)
+{
+ struct sockaddr_nl sa;
+ struct nlmsghdr msg;
+ struct iovec iov;
+ struct msghdr mh;
+ int ret;
+
+ STATE_AO_GC(cps->ao);
+
+ memset(&sa, 0, sizeof(sa));
+ sa.nl_family = AF_NETLINK;
+ sa.nl_pid = 0;
+ sa.nl_groups = 0;
+
+ msg.nlmsg_len = NLMSG_SPACE(0);
+ msg.nlmsg_flags = NLM_F_REQUEST;
+ if (type == COLO_PROXY_INIT) {
+ msg.nlmsg_flags |= NLM_F_ACK;
+ }
+ msg.nlmsg_seq = 0;
+ /* This is untrusty */
+ msg.nlmsg_pid = cps->index;
+ msg.nlmsg_type = type;
+
+ iov.iov_base = &msg;
+ iov.iov_len = msg.nlmsg_len;
+
+ mh.msg_name = &sa;
+ mh.msg_namelen = sizeof(sa);
+ mh.msg_iov = &iov;
+ mh.msg_iovlen = 1;
+ mh.msg_control = NULL;
+ mh.msg_controllen = 0;
+ mh.msg_flags = 0;
+
+ ret = sendmsg(cps->sock_fd, &mh, 0);
+ if (ret <= 0) {
+ LOG(ERROR, "can't send msg to kernel by netlink: %s",
+ strerror(errno));
+ }
+
+ return ret;
+}
+
+/* error: return -1, otherwise return 0 */
+static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
+ unsigned int timeout_us)
+{
+ struct sockaddr_nl sa;
+ struct iovec iov;
+ struct msghdr mh = {
+ .msg_name = &sa,
+ .msg_namelen = sizeof(sa),
+ .msg_iov = &iov,
+ .msg_iovlen = 1,
+ };
+ struct timeval tv;
+ uint32_t size = 16384;
+ int64_t len = 0;
+ int ret;
+
+ STATE_AO_GC(cps->ao);
+ uint8_t *tmp = libxl__malloc(NOGC, size);
+
+ if (timeout_us) {
+ tv.tv_sec = timeout_us / 1000000;
+ tv.tv_usec = timeout_us % 1000000;
+ setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
+ }
+
+ iov.iov_base = tmp;
+ iov.iov_len = size;
+next:
+ ret = recvmsg(cps->sock_fd, &mh, 0);
+ if (ret <= 0) {
+ if (errno != EAGAIN && errno != EWOULDBLOCK)
+ LOGE(ERROR, "can't recv msg from kernel by netlink");
+ goto err;
+ }
+
+ len += ret;
+ if (mh.msg_flags & MSG_TRUNC) {
+ size += 16384;
+ tmp = libxl__realloc(NOGC, tmp, size);
+ iov.iov_base = tmp + len;
+ iov.iov_len = size - len;
+ goto next;
+ }
+
+ *buff = tmp;
+ ret = len;
+ goto out;
+
+err:
+ free(tmp);
+ *buff = NULL;
+
+out:
+ if (timeout_us) {
+ tv.tv_sec = 0;
+ tv.tv_usec = 0;
+ setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
+ }
+ return ret;
+}
+
+/* ========= colo-proxy: setup and teardown ========== */
+
+int colo_proxy_setup(libxl__colo_proxy_state *cps)
+{
+ int skfd = 0;
+ struct sockaddr_nl sa;
+ struct nlmsghdr *h;
+ int i = 1;
+ int ret = ERROR_FAIL;
+ uint8_t *buff = NULL;
+ int64_t size;
+
+ STATE_AO_GC(cps->ao);
+
+ skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
+ if (skfd < 0) {
+ LOG(ERROR, "can not create a netlink socket: %s", strerror(errno));
+ goto out;
+ }
+ cps->sock_fd = skfd;
+ memset(&sa, 0, sizeof(sa));
+ sa.nl_family = AF_NETLINK;
+ sa.nl_groups = 0;
+retry:
+ sa.nl_pid = i++;
+
+ if (i > 10) {
+ LOG(ERROR, "netlink bind error");
+ goto out;
+ }
+
+ ret = bind(skfd, (struct sockaddr *)&sa, sizeof(sa));
+ if (ret < 0 && errno == EADDRINUSE) {
+ LOG(ERROR, "colo index %d has already in used", sa.nl_pid);
+ goto retry;
+ } else if (ret < 0) {
+ LOG(ERROR, "netlink bind error");
+ goto out;
+ }
+
+ cps->index = sa.nl_pid;
+ ret = colo_proxy_send(cps, NULL, 0, COLO_PROXY_INIT);
+ if (ret < 0) {
+ goto out;
+ }
+ /* receive ack */
+ size = colo_proxy_recv(cps, &buff, 500000);
+ if (size < 0) {
+ LOG(ERROR, "Can't recv msg from kernel by netlink: %s",
+ strerror(errno));
+ goto out;
+ }
+
+ if (size) {
+ h = (struct nlmsghdr *)buff;
+ if (h->nlmsg_type == NLMSG_ERROR) {
+ /* ack's type is NLMSG_ERROR */
+ struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(h);
+
+ if (size - sizeof(*h) < sizeof(*err)) {
+ LOG(ERROR, "NLMSG_LENGTH is too short");
+ goto out;
+ }
+
+ if (err->error) {
+ LOG(ERROR, "NLMSG_ERROR contains error %d", err->error);
+ goto out;
+ }
+ }
+ }
+
+ ret = 0;
+
+out:
+ free(buff);
+ if (ret) {
+ close(cps->sock_fd);
+ cps->sock_fd = -1;
+ }
+ return ret;
+}
+
+void colo_proxy_teardown(libxl__colo_proxy_state *cps)
+{
+ if (cps->sock_fd >= 0) {
+ close(cps->sock_fd);
+ cps->sock_fd = -1;
+ }
+}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 0aeaf18..4940b59 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3194,6 +3194,9 @@ struct libxl__colo_save_state {
/* private, used by qdisk block replication */
bool qdisk_used;
bool qdisk_setuped;
+
+ /* private, used by colo-proxy */
+ libxl__colo_proxy_state cps;
};
typedef struct libxl__logdirty_switch {
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module
2016-03-04 8:41 ` [PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module Changlong Xie
@ 2016-03-04 17:59 ` Ian Jackson
2016-03-18 8:22 ` Changlong Xie
2016-03-22 5:44 ` Changlong Xie
0 siblings, 2 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 17:59 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> setup/teardown of COLO proxy module.
> we use netlink to communicate with proxy module.
> About colo-proxy module:
> https://lkml.org/lkml/2015/6/18/32
> How to use:
> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
This seems pleasingly self-contained, and to be done in the right
place, but:
I don't understand how discrepant packets are detected, in order to
cause a resync. I mean, I don't see any code in this patch which
receives a notification of a discrepancy and causes a resync. What am
I missing ?
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module
2016-03-04 17:59 ` Ian Jackson
@ 2016-03-18 8:22 ` Changlong Xie
2016-03-22 5:44 ` Changlong Xie
1 sibling, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-18 8:22 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:59 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> setup/teardown of COLO proxy module.
>> we use netlink to communicate with proxy module.
>> About colo-proxy module:
>> https://lkml.org/lkml/2015/6/18/32
>> How to use:
>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>
> This seems pleasingly self-contained, and to be done in the right
> place, but:
>
> I don't understand how discrepant packets are detected, in order to
> cause a resync. I mean, I don't see any code in this patch which
> receives a notification of a discrepancy and causes a resync. What am
> I missing ?
>
I'm not familar with this part, Wen will answer colo-proxy revelant
questions later.
Thanks
-Xie
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module
2016-03-04 17:59 ` Ian Jackson
2016-03-18 8:22 ` Changlong Xie
@ 2016-03-22 5:44 ` Changlong Xie
2016-03-22 5:55 ` Changlong Xie
1 sibling, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-22 5:44 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 01:59 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> setup/teardown of COLO proxy module.
>> we use netlink to communicate with proxy module.
>> About colo-proxy module:
>> https://lkml.org/lkml/2015/6/18/32
I just notice that, the url on lkml is out of operation.
>> How to use:
>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>
> This seems pleasingly self-contained, and to be done in the right
> place, but:
>
> I don't understand how discrepant packets are detected, in order to
> cause a resync. I mean, I don't see any code in this patch which
> receives a notification of a discrepancy and causes a resync. What am
> I missing ?
Please refer:
http://www.spinics.net/lists/netdev/msg333520.html
I think it's detail enough.
Thanks
-Xie
>
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module
2016-03-22 5:44 ` Changlong Xie
@ 2016-03-22 5:55 ` Changlong Xie
0 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-22 5:55 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Li Zhijian, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Shriram Rajagopalan, Gui Jianfeng, Anthony Perard, Yang Hongyang
CC: lizhijian@cn.fujitsu.com
On 03/22/2016 01:44 PM, Changlong Xie wrote:
> On 03/05/2016 01:59 AM, Ian Jackson wrote:
>> Changlong Xie writes ("[PATCH v11 22/27] COLO proxy: implement
>> setup/teardown of COLO proxy module"):
>>> From: Wen Congyang <wency@cn.fujitsu.com>
>>>
>>> setup/teardown of COLO proxy module.
>>> we use netlink to communicate with proxy module.
>>> About colo-proxy module:
>>> https://lkml.org/lkml/2015/6/18/32
>
> I just notice that, the url on lkml is out of operation.
>
>>> How to use:
>>> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>>
>> This seems pleasingly self-contained, and to be done in the right
>> place, but:
>>
>> I don't understand how discrepant packets are detected, in order to
>> cause a resync. I mean, I don't see any code in this patch which
>> receives a notification of a discrepancy and causes a resync. What am
>> I missing ?
>
> Please refer:
>
> http://www.spinics.net/lists/netdev/msg333520.html
>
> I think it's detail enough.
>
> Thanks
> -Xie
>
>>
>> Thanks,
>> Ian.
>>
>>
>> .
>>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 23/27] COLO proxy: preresume, postresume and checkpoint
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (21 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 22/27] COLO proxy: implement setup/teardown of COLO proxy module Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 18:01 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 24/27] COLO nic: implement COLO nic subkind Changlong Xie
` (4 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
preresume, postresume and checkpoint
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_colo.h | 4 +++
tools/libxl/libxl_colo_proxy.c | 62 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 66 insertions(+)
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 9e7f99c..736927e 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -99,5 +99,9 @@ extern void libxl__colo_save_teardown(struct libxl__egc *egc,
int rc);
extern int colo_proxy_setup(libxl__colo_proxy_state *cps);
extern void colo_proxy_teardown(libxl__colo_proxy_state *cps);
+extern void colo_proxy_preresume(libxl__colo_proxy_state *cps);
+extern void colo_proxy_postresume(libxl__colo_proxy_state *cps);
+extern int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
+ unsigned int timeout_us);
#endif
diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
index 2b3baa3..1b1dd40 100644
--- a/tools/libxl/libxl_colo_proxy.c
+++ b/tools/libxl/libxl_colo_proxy.c
@@ -216,3 +216,65 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
cps->sock_fd = -1;
}
}
+
+/* ========= colo-proxy: preresume, postresume and checkpoint ========== */
+
+void colo_proxy_preresume(libxl__colo_proxy_state *cps)
+{
+ colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
+ /* TODO: need to handle if the call fails... */
+}
+
+void colo_proxy_postresume(libxl__colo_proxy_state *cps)
+{
+ /* nothing to do... */
+}
+
+typedef struct colo_msg {
+ bool is_checkpoint;
+} colo_msg;
+
+/*
+ * Return value:
+ * -1: error
+ * 0: no checkpoint event is received before timeout
+ * 1: do checkpoint
+ */
+int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
+ unsigned int timeout_us)
+{
+ uint8_t *buff;
+ int64_t size;
+ struct nlmsghdr *h;
+ struct colo_msg *m;
+ int ret = -1;
+
+ STATE_AO_GC(cps->ao);
+
+ size = colo_proxy_recv(cps, &buff, timeout_us);
+
+ /* timeout, return no checkpoint message. */
+ if (size <= 0) {
+ return 0;
+ }
+
+ h = (struct nlmsghdr *) buff;
+
+ if (h->nlmsg_type == NLMSG_ERROR) {
+ LOG(ERROR, "receive NLMSG_ERROR");
+ goto out;
+ }
+
+ if (h->nlmsg_len < NLMSG_LENGTH(sizeof(*m))) {
+ LOG(ERROR, "NLMSG_LENGTH is too short");
+ goto out;
+ }
+
+ m = NLMSG_DATA(h);
+
+ ret = m->is_checkpoint ? 1 : 0;
+
+out:
+ free(buff);
+ return ret;
+}
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 23/27] COLO proxy: preresume, postresume and checkpoint
2016-03-04 8:41 ` [PATCH v11 23/27] COLO proxy: preresume, postresume and checkpoint Changlong Xie
@ 2016-03-04 18:01 ` Ian Jackson
2016-03-18 8:20 ` Changlong Xie
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 18:01 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 23/27] COLO proxy: preresume, postresume and checkpoint"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> preresume, postresume and checkpoint
I think maybe this needs to be combined with the previous patch ?
I don't think I quite understand the structure of the patch series at
this point.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 23/27] COLO proxy: preresume, postresume and checkpoint
2016-03-04 18:01 ` Ian Jackson
@ 2016-03-18 8:20 ` Changlong Xie
0 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-18 8:20 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 02:01 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 23/27] COLO proxy: preresume, postresume and checkpoint"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> preresume, postresume and checkpoint
>
> I think maybe this needs to be combined with the previous patch ?
>
Surely
Thanks
-Xie
> I don't think I quite understand the structure of the patch series at
> this point.
>
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 24/27] COLO nic: implement COLO nic subkind
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (22 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 23/27] COLO proxy: preresume, postresume and checkpoint Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 18:02 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 25/27] setup and control colo proxy on primary side Changlong Xie
` (3 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
implement COLO nic subkind.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/hotplug/Linux/Makefile | 1 +
tools/hotplug/Linux/colo-proxy-setup | 135 +++++++++++++++
tools/libxl/Makefile | 1 +
tools/libxl/libxl_colo.h | 10 ++
tools/libxl/libxl_colo_nic.c | 315 +++++++++++++++++++++++++++++++++++
tools/libxl/libxl_internal.h | 2 +
tools/libxl/libxl_types.idl | 1 +
7 files changed, 465 insertions(+)
create mode 100755 tools/hotplug/Linux/colo-proxy-setup
create mode 100644 tools/libxl/libxl_colo_nic.c
diff --git a/tools/hotplug/Linux/Makefile b/tools/hotplug/Linux/Makefile
index 6e10118..9bb852b 100644
--- a/tools/hotplug/Linux/Makefile
+++ b/tools/hotplug/Linux/Makefile
@@ -28,6 +28,7 @@ XEN_SCRIPTS += block-iscsi
XEN_SCRIPTS += block-tap
XEN_SCRIPTS += block-drbd-probe
XEN_SCRIPTS += $(XEN_SCRIPTS-y)
+XEN_SCRIPTS += colo-proxy-setup
SUBDIRS-$(CONFIG_SYSTEMD) += systemd
diff --git a/tools/hotplug/Linux/colo-proxy-setup b/tools/hotplug/Linux/colo-proxy-setup
new file mode 100755
index 0000000..94e2034
--- /dev/null
+++ b/tools/hotplug/Linux/colo-proxy-setup
@@ -0,0 +1,135 @@
+#! /bin/bash
+
+dir=$(dirname "$0")
+. "$dir/xen-hotplug-common.sh"
+. "$dir/hotplugpath.sh"
+
+findCommand "$@"
+
+if [ "$command" != "setup" -a "$command" != "teardown" ]
+then
+ echo "Invalid command: $command"
+ log err "Invalid command: $command"
+ exit 1
+fi
+
+evalVariables "$@"
+
+: ${vifname:?}
+: ${forwarddev:?}
+: ${mode:?}
+: ${index:?}
+: ${bridge:?}
+
+forwardbr="colobr0"
+
+if [ "$mode" != "primary" -a "$mode" != "secondary" ]
+then
+ echo "Invalid mode: $mode"
+ log err "Invalid mode: $mode"
+ exit 1
+fi
+
+if [ $index -lt 0 ] || [ $index -gt 100 ]; then
+ echo "index overflow"
+ exit 1
+fi
+
+function setup_primary()
+{
+ do_without_error tc qdisc add dev $vifname root handle 1: prio
+ do_without_error tc filter add dev $vifname parent 1: protocol ip prio 10 \
+ u32 match u32 0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+ do_without_error tc filter add dev $vifname parent 1: protocol arp prio 11 \
+ u32 match u32 0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+ do_without_error tc filter add dev $vifname parent 1: protocol ipv6 prio \
+ 12 u32 match u32 0 0 flowid 1:2 action mirred egress mirror \
+ dev $forwarddev
+
+ do_without_error modprobe nf_conntrack_ipv4
+ do_without_error modprobe xt_PMYCOLO sec_dev=$forwarddev
+
+ iptables -t mangle -I PREROUTING -m physdev --physdev-in \
+ $vifname -j PMYCOLO --index $index
+ ip6tables -t mangle -I PREROUTING -m physdev --physdev-in \
+ $vifname -j PMYCOLO --index $index
+ do_without_error arptables -I INPUT -i $forwarddev -j MARK --set-mark $index
+}
+
+function teardown_primary()
+{
+ do_without_error tc filter del dev $vifname parent 1: protocol ip prio 10 u32 match u32 \
+ 0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+ do_without_error tc filter del dev $vifname parent 1: protocol arp prio 11 u32 match u32 \
+ 0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+ do_without_error tc filter del dev $vifname parent 1: protocol ipv6 prio 12 u32 match u32 \
+ 0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+ do_without_error tc qdisc del dev $vifname root handle 1: prio
+
+ do_without_error iptables -t mangle -D PREROUTING -m physdev --physdev-in \
+ $vifname -j PMYCOLO --index $index
+ do_without_error ip6tables -t mangle -D PREROUTING -m physdev --physdev-in \
+ $vifname -j PMYCOLO --index $index
+ do_without_error arptables -F
+ do_without_error rmmod xt_PMYCOLO
+}
+
+function setup_secondary()
+{
+ do_without_error brctl delif $bridge $vifname
+ do_without_error brctl addbr $forwardbr
+ do_without_error brctl addif $forwardbr $vifname
+ do_without_error brctl addif $forwardbr $forwarddev
+ do_without_error ip link set dev $forwardbr up
+ do_without_error modprobe xt_SECCOLO
+
+ iptables -t mangle -I PREROUTING -m physdev --physdev-in \
+ $vifname -j SECCOLO --index $index
+ ip6tables -t mangle -I PREROUTING -m physdev --physdev-in \
+ $vifname -j SECCOLO --index $index
+}
+
+function teardown_secondary()
+{
+ do_without_error brctl delif $forwardbr $forwarddev
+ do_without_error brctl delif $forwardbr $vifname
+ do_without_error brctl delbr $forwardbr
+ do_without_error brctl addif $bridge $vifname
+
+ do_without_error iptables -t mangle -D PREROUTING -m physdev --physdev-in \
+ $vifname -j SECCOLO --index $index
+ do_without_error ip6tables -t mangle -D PREROUTING -m physdev --physdev-in \
+ $vifname -j SECCOLO --index $index
+ do_without_error rmmod xt_SECCOLO
+}
+
+case "$command" in
+ setup)
+ if [ "$mode" = "primary" ]
+ then
+ setup_primary
+ else
+ setup_secondary
+ fi
+
+ success
+ ;;
+ teardown)
+ if [ "$mode" = "primary" ]
+ then
+ teardown_primary
+ else
+ teardown_secondary
+ fi
+ ;;
+esac
+
+if [ "$mode" = "primary" ]
+then
+ log debug "Successful colo-proxy-setup $command for $vifname." \
+ " vifname: $vifname, index: $index, forwarddev: $forwarddev."
+else
+ log debug "Successful colo-proxy-setup $command for $vifname." \
+ " vifname: $vifname, index: $index, forwarddev: $forwarddev,"\
+ " forwardbr: $forwardbr."
+fi
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 6fea9e0..aefc378 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -68,6 +68,7 @@ LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
LIBXL_OBJS-y += libxl_colo_qdisk.o
LIBXL_OBJS-y += libxl_colo_proxy.o
+LIBXL_OBJS-y += libxl_colo_nic.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 736927e..0947953 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -39,6 +39,11 @@ enum colo_netlink_op {
COLO_PROXY_RESET, /* UNUSED, will be used for continuous FT */
};
+typedef struct libxl__colo_device_nic {
+ int devid;
+ const char *vif;
+} libxl__colo_device_nic;
+
typedef struct libxl__colo_qdisk {
bool setuped;
} libxl__colo_qdisk;
@@ -69,6 +74,7 @@ struct libxl__colo_restore_state {
int recv_fd;
int hvm;
libxl__colo_callback *callback;
+ char *colo_proxy_script;
/* private, colo restore checkpoint state */
libxl__domain_create_cb *saved_cb;
@@ -88,6 +94,10 @@ int init_subkind_qdisk(struct libxl__checkpoint_devices_state *cds);
void cleanup_subkind_qdisk(struct libxl__checkpoint_devices_state *cds);
+int init_subkind_colo_nic(struct libxl__checkpoint_devices_state *cds);
+
+void cleanup_subkind_colo_nic(struct libxl__checkpoint_devices_state *cds);
+
extern void libxl__colo_restore_setup(struct libxl__egc *egc,
libxl__colo_restore_state *crs);
extern void libxl__colo_restore_teardown(struct libxl__egc *egc, void *dcs_void,
diff --git a/tools/libxl/libxl_colo_nic.c b/tools/libxl/libxl_colo_nic.c
new file mode 100644
index 0000000..cbb4163
--- /dev/null
+++ b/tools/libxl/libxl_colo_nic.c
@@ -0,0 +1,315 @@
+/*
+ * Copyright (C) 2014 FUJITSU LIMITED
+ * Author: Wen Congyang <wency@cn.fujitsu.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+enum {
+ primary,
+ secondary,
+};
+
+/* ========== init() and cleanup() ========== */
+int init_subkind_colo_nic(libxl__checkpoint_devices_state *cds)
+{
+ return 0;
+}
+
+void cleanup_subkind_colo_nic(libxl__checkpoint_devices_state *cds)
+{
+}
+
+/* ========== helper functions ========== */
+static void colo_save_setup_script_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int rc, int status);
+static void colo_save_teardown_script_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int rc, int status);
+
+/*
+ * If the device has a vifname, then use that instead of
+ * the vifX.Y format.
+ * it must ONLY be used for remus because if driver domains
+ * were in use it would constitute a security vulnerability.
+ */
+static const char *get_vifname(libxl__checkpoint_device *dev,
+ const libxl_device_nic *nic)
+{
+ const char *vifname = NULL;
+ const char *path;
+ int rc;
+
+ STATE_AO_GC(dev->cds->ao);
+
+ /* Convenience aliases */
+ const uint32_t domid = dev->cds->domid;
+
+ path = GCSPRINTF("%s/backend/vif/%d/%d/vifname",
+ libxl__xs_get_dompath(gc, 0), domid, nic->devid);
+ rc = libxl__xs_read_checked(gc, XBT_NULL, path, &vifname);
+ if (!rc && !vifname) {
+ vifname = libxl__device_nic_devname(gc, domid,
+ nic->devid,
+ nic->nictype);
+ }
+
+ return vifname;
+}
+
+/*
+ * the script needs the following env & args
+ * $vifname
+ * $forwarddev
+ * $mode(primary/secondary)
+ * $index
+ * $bridge
+ * setup/teardown as command line arg.
+ */
+static void setup_async_exec(libxl__checkpoint_device *dev, char *op,
+ libxl__colo_proxy_state *cps, int side,
+ char *colo_proxy_script)
+{
+ int arraysize, nr = 0;
+ char **env = NULL, **args = NULL;
+ libxl__colo_device_nic *colo_nic = dev->concrete_data;
+ libxl__checkpoint_devices_state *cds = dev->cds;
+ libxl__async_exec_state *aes = &dev->aodev.aes;
+ const libxl_device_nic *nic = dev->backend_dev;
+
+ STATE_AO_GC(cds->ao);
+
+ /* Convenience aliases */
+ const char *const vif = colo_nic->vif;
+
+ arraysize = 11;
+ GCNEW_ARRAY(env, arraysize);
+ env[nr++] = "vifname";
+ env[nr++] = libxl__strdup(gc, vif);
+ env[nr++] = "forwarddev";
+ env[nr++] = libxl__strdup(gc, nic->forwarddev);
+ env[nr++] = "mode";
+ if (side == primary)
+ env[nr++] = "primary";
+ else
+ env[nr++] = "secondary";
+ env[nr++] = "index";
+ env[nr++] = GCSPRINTF("%d", cps->index);
+ env[nr++] = "bridge";
+ env[nr++] = libxl__strdup(gc, nic->bridge);
+ env[nr++] = NULL;
+ assert(nr == arraysize);
+
+ arraysize = 3; nr = 0;
+ GCNEW_ARRAY(args, arraysize);
+ args[nr++] = colo_proxy_script;
+ args[nr++] = op;
+ args[nr++] = NULL;
+ assert(nr == arraysize);
+
+ aes->ao = dev->cds->ao;
+ aes->what = GCSPRINTF("%s %s", args[0], args[1]);
+ aes->env = env;
+ aes->args = args;
+ aes->timeout_ms = LIBXL_HOTPLUG_TIMEOUT * 1000;
+ aes->stdfds[0] = -1;
+ aes->stdfds[1] = -1;
+ aes->stdfds[2] = -1;
+
+ if (!strcmp(op, "teardown"))
+ aes->callback = colo_save_teardown_script_cb;
+ else
+ aes->callback = colo_save_setup_script_cb;
+}
+
+/* ========== setup() and teardown() ========== */
+static void colo_nic_setup(libxl__egc *egc, libxl__checkpoint_device *dev,
+ libxl__colo_proxy_state *cps, int side,
+ char *colo_proxy_script)
+{
+ int rc;
+ libxl__colo_device_nic *colo_nic;
+ const libxl_device_nic *nic = dev->backend_dev;
+
+ STATE_AO_GC(dev->cds->ao);
+
+ /*
+ * thers's no subkind of nic devices, so nic ops is always matched
+ * with nic devices, we begin to setup the nic device
+ */
+ dev->matched = 1;
+
+ if (!nic->forwarddev) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ GCNEW(colo_nic);
+ dev->concrete_data = colo_nic;
+ colo_nic->devid = nic->devid;
+ colo_nic->vif = get_vifname(dev, nic);
+ if (!colo_nic->vif) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ setup_async_exec(dev, "setup", cps, side, colo_proxy_script);
+ rc = libxl__async_exec_start(&dev->aodev.aes);
+ if (rc)
+ goto out;
+
+ return;
+
+out:
+ dev->aodev.rc = rc;
+ dev->aodev.callback(egc, &dev->aodev);
+}
+
+static void colo_save_setup_script_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int rc, int status)
+{
+ libxl__ao_device *aodev = CONTAINER_OF(aes, *aodev, aes);
+ libxl__checkpoint_device *dev = CONTAINER_OF(aodev, *dev, aodev);
+ libxl__colo_device_nic *colo_nic = dev->concrete_data;
+ libxl__checkpoint_devices_state *cds = dev->cds;
+ const char *out_path_base, *hotplug_error = NULL;
+
+ EGC_GC;
+
+ /* Convenience aliases */
+ const uint32_t domid = cds->domid;
+ const int devid = colo_nic->devid;
+ const char *const vif = colo_nic->vif;
+
+ if (status && !rc)
+ rc = ERROR_FAIL;
+ if (rc)
+ goto out;
+
+ out_path_base = GCSPRINTF("%s/colo_proxy/%d",
+ libxl__xs_libxl_path(gc, domid), devid);
+
+ rc = libxl__xs_read_checked(gc, XBT_NULL,
+ GCSPRINTF("%s/hotplug-error", out_path_base),
+ &hotplug_error);
+ if (rc)
+ goto out;
+
+ if (hotplug_error) {
+ LOG(ERROR, "colo_proxy script %s setup failed for vif %s: %s",
+ aes->args[0], vif, hotplug_error);
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ if (status) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ rc = 0;
+
+out:
+ aodev->rc = rc;
+ aodev->callback(egc, aodev);
+}
+
+static void colo_nic_teardown(libxl__egc *egc, libxl__checkpoint_device *dev,
+ libxl__colo_proxy_state *cps, int side,
+ char *colo_proxy_script)
+{
+ int rc;
+ libxl__colo_device_nic *colo_nic = dev->concrete_data;
+
+ if (!colo_nic || !colo_nic->vif) {
+ /* colo nic has not yet been set up, just return */
+ rc = 0;
+ goto out;
+ }
+
+ setup_async_exec(dev, "teardown", cps, side, colo_proxy_script);
+
+ rc = libxl__async_exec_start(&dev->aodev.aes);
+ if (rc)
+ goto out;
+
+ return;
+
+out:
+ dev->aodev.rc = rc;
+ dev->aodev.callback(egc, &dev->aodev);
+}
+
+static void colo_save_teardown_script_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int rc, int status)
+{
+ libxl__ao_device *aodev = CONTAINER_OF(aes, *aodev, aes);
+
+ if (status && !rc)
+ rc = ERROR_FAIL;
+ else
+ rc = 0;
+
+ aodev->rc = rc;
+ aodev->callback(egc, aodev);
+}
+
+/* ======== primary ======== */
+static void colo_nic_save_setup(libxl__egc *egc, libxl__checkpoint_device *dev)
+{
+ libxl__colo_save_state *css = dev->cds->concrete_data;
+
+ colo_nic_setup(egc, dev, &css->cps, primary, css->colo_proxy_script);
+}
+
+static void colo_nic_save_teardown(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ libxl__colo_save_state *css = dev->cds->concrete_data;
+
+ colo_nic_teardown(egc, dev, &css->cps, primary, css->colo_proxy_script);
+}
+
+const libxl__checkpoint_device_instance_ops colo_save_device_nic = {
+ .kind = LIBXL__DEVICE_KIND_VIF,
+ .setup = colo_nic_save_setup,
+ .teardown = colo_nic_save_teardown,
+};
+
+/* ======== secondary ======== */
+static void colo_nic_restore_setup(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ libxl__colo_restore_state *crs = dev->cds->concrete_data;
+
+ colo_nic_setup(egc, dev, &crs->cps, secondary, crs->colo_proxy_script);
+}
+
+static void colo_nic_restore_teardown(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ libxl__colo_restore_state *crs = dev->cds->concrete_data;
+
+ colo_nic_teardown(egc, dev, &crs->cps, secondary, crs->colo_proxy_script);
+}
+
+const libxl__checkpoint_device_instance_ops colo_restore_device_nic = {
+ .kind = LIBXL__DEVICE_KIND_VIF,
+ .setup = colo_nic_restore_setup,
+ .teardown = colo_nic_restore_teardown,
+};
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 4940b59..ddaafc6 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3184,6 +3184,7 @@ typedef struct libxl__colo_save_state libxl__colo_save_state;
struct libxl__colo_save_state {
int send_fd;
int recv_fd;
+ char *colo_proxy_script;
/* private */
libxl__stream_read_state srs;
@@ -3523,6 +3524,7 @@ struct libxl__domain_create_state {
libxl_asyncprogress_how aop_console_how;
/* private to domain_create */
int guest_domid;
+ const char *colo_proxy_script;
libxl__domain_build_state build_state;
libxl__colo_restore_state crs;
libxl__checkpoint_devices_state cds;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a2078d1..6b57aba 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -599,6 +599,7 @@ libxl_device_nic = Struct("device_nic", [
("rate_bytes_per_interval", uint64),
("rate_interval_usecs", uint32),
("gatewaydev", string),
+ ("forwarddev", string)
])
libxl_device_pci = Struct("device_pci", [
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 24/27] COLO nic: implement COLO nic subkind
2016-03-04 8:41 ` [PATCH v11 24/27] COLO nic: implement COLO nic subkind Changlong Xie
@ 2016-03-04 18:02 ` Ian Jackson
2016-03-18 8:20 ` Changlong Xie
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 18:02 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
> index a2078d1..6b57aba 100644
> --- a/tools/libxl/libxl_types.idl
> +++ b/tools/libxl/libxl_types.idl
> @@ -599,6 +599,7 @@ libxl_device_nic = Struct("device_nic", [
> ("rate_bytes_per_interval", uint64),
> ("rate_interval_usecs", uint32),
> ("gatewaydev", string),
> + ("forwarddev", string)
> ])
The documentation for this seems missing... maybe it's in the next
patches...
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 24/27] COLO nic: implement COLO nic subkind
2016-03-04 18:02 ` Ian Jackson
@ 2016-03-18 8:20 ` Changlong Xie
0 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-18 8:20 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 02:02 AM, Ian Jackson wrote:
>> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
>> index a2078d1..6b57aba 100644
>> --- a/tools/libxl/libxl_types.idl
>> +++ b/tools/libxl/libxl_types.idl
>> @@ -599,6 +599,7 @@ libxl_device_nic = Struct("device_nic", [
>> ("rate_bytes_per_interval", uint64),
>> ("rate_interval_usecs", uint32),
>> ("gatewaydev", string),
>> + ("forwarddev", string)
>> ])
>
Will add some comments here
Thank
-Xie
> The documentation for this seems missing... maybe it's in the next
> patches...
>
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 25/27] setup and control colo proxy on primary side
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (23 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 24/27] COLO nic: implement COLO nic subkind Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 18:05 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 26/27] setup and control colo proxy on secondary side Changlong Xie
` (2 subsequent siblings)
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_colo_save.c | 124 +++++++++++++++++++++++++++++++++++++++---
tools/libxl/libxl_internal.h | 1 +
2 files changed, 117 insertions(+), 8 deletions(-)
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index 97efc8d..bb44e66 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -18,9 +18,11 @@
#include "libxl_internal.h"
+extern const libxl__checkpoint_device_instance_ops colo_save_device_nic;
extern const libxl__checkpoint_device_instance_ops colo_save_device_qdisk;
static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
+ &colo_save_device_nic,
&colo_save_device_qdisk,
NULL,
};
@@ -32,9 +34,15 @@ static int init_device_subkind(libxl__checkpoint_devices_state *cds)
int rc;
STATE_AO_GC(cds->ao);
- rc = init_subkind_qdisk(cds);
+ rc = init_subkind_colo_nic(cds);
if (rc) goto out;
+ rc = init_subkind_qdisk(cds);
+ if (rc) {
+ cleanup_subkind_colo_nic(cds);
+ goto out;
+ }
+
rc = 0;
out:
return rc;
@@ -45,6 +53,7 @@ static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
/* cleanup device subkind-specific state in the libxl ctx */
STATE_AO_GC(cds->ao);
+ cleanup_subkind_colo_nic(cds);
cleanup_subkind_qdisk(cds);
}
@@ -89,9 +98,16 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->paused = true;
css->qdisk_setuped = false;
css->qdisk_used = false;
+ libxl__ev_child_init(&css->child);
+
+ if (dss->remus->netbufscript)
+ css->colo_proxy_script = libxl__strdup(gc, dss->remus->netbufscript);
+ else
+ css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
+ libxl__xen_script_dir_path());
- /* TODO: nic support */
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
cds->ops = colo_ops;
cds->callback = colo_save_setup_done;
cds->ao = ao;
@@ -102,6 +118,12 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->srs.fd = css->recv_fd;
css->srs.back_channel = true;
libxl__stream_read_start(egc, &css->srs);
+ css->cps.ao = ao;
+ if (colo_proxy_setup(&css->cps)) {
+ LOG(ERROR, "COLO: failed to setup colo proxy for guest with domid %u",
+ cds->domid);
+ goto out;
+ }
if (init_device_subkind(cds))
goto out;
@@ -189,6 +211,7 @@ static void colo_teardown_done(libxl__egc *egc,
libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
cleanup_device_subkind(cds);
+ colo_proxy_teardown(&css->cps);
dss->callback(egc, dss, rc);
}
@@ -381,6 +404,8 @@ static void colo_read_svm_ready_done(libxl__egc *egc,
goto out;
}
+ colo_proxy_preresume(&css->cps);
+
css->svm_running = true;
dss->cds.callback = colo_preresume_cb;
libxl__checkpoint_devices_preresume(egc, &dss->cds);
@@ -465,6 +490,8 @@ static void colo_read_svm_resumed_done(libxl__egc *egc,
goto out;
}
+ colo_proxy_postresume(&css->cps);
+
ok = 1;
out:
@@ -473,6 +500,91 @@ out:
/* ===================== colo: wait new checkpoint ===================== */
+
+static void colo_start_new_checkpoint(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_proxy_async_wait_for_checkpoint(libxl__colo_save_state *css);
+static void colo_proxy_async_call_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ int pid,
+ int status);
+
+static void colo_proxy_async_call(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ void func(libxl__colo_save_state *),
+ libxl__ev_child_callback callback)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+ int pid = -1, rc;
+
+ STATE_AO_GC(dss->cds.ao);
+
+ /* Fork and call */
+ pid = libxl__ev_child_fork(gc, &css->child, callback);
+ if (pid == -1) {
+ LOG(ERROR, "unable to fork");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ if (!pid) {
+ /* child */
+ func(css);
+ /* notreached */
+ abort();
+ }
+
+ return;
+
+out:
+ callback(egc, &css->child, -1, 1);
+}
+
+static void colo_proxy_wait_for_checkpoint(libxl__egc *egc,
+ libxl__colo_save_state *css)
+{
+ colo_proxy_async_call(egc, css,
+ colo_proxy_async_wait_for_checkpoint,
+ colo_proxy_async_call_done);
+}
+
+static void colo_proxy_async_wait_for_checkpoint(libxl__colo_save_state *css)
+{
+ int req;
+
+ req = colo_proxy_checkpoint(&css->cps, 5000000);
+ if (req < 0) {
+ /* some error happens */
+ _exit(1);
+ } else if (!req) {
+ /* no checkpoint is needed, do a checkpint every 5s */
+ _exit(0);
+ } else {
+ /* net packets is not consistent, we need to start a checkpoint */
+ _exit(0);
+ }
+}
+
+static void colo_proxy_async_call_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ int pid,
+ int status)
+{
+ libxl__colo_save_state *css = CONTAINER_OF(child, *css, child);
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (status) {
+ LOG(ERROR, "failed to wait for new checkpoint");
+ colo_start_new_checkpoint(egc, &dss->cds, ERROR_FAIL);
+ return;
+ }
+
+ colo_start_new_checkpoint(egc, &dss->cds, 0);
+}
+
/*
* Do the following things:
* 1. do commit
@@ -482,9 +594,6 @@ out:
static void colo_device_commit_cb(libxl__egc *egc,
libxl__checkpoint_devices_state *cds,
int rc);
-static void colo_start_new_checkpoint(libxl__egc *egc,
- libxl__checkpoint_devices_state *cds,
- int rc);
static void libxl__colo_save_domain_wait_checkpoint_callback(void *data)
{
@@ -514,8 +623,7 @@ static void colo_device_commit_cb(libxl__egc *egc,
goto out;
}
- /* TODO: wait a new checkpoint */
- colo_start_new_checkpoint(egc, cds, 0);
+ colo_proxy_wait_for_checkpoint(egc, css);
return;
out:
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index ddaafc6..dd9d00f 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3198,6 +3198,7 @@ struct libxl__colo_save_state {
/* private, used by colo-proxy */
libxl__colo_proxy_state cps;
+ libxl__ev_child child;
};
typedef struct libxl__logdirty_switch {
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 25/27] setup and control colo proxy on primary side
2016-03-04 8:41 ` [PATCH v11 25/27] setup and control colo proxy on primary side Changlong Xie
@ 2016-03-04 18:05 ` Ian Jackson
2016-03-22 6:01 ` Changlong Xie
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 18:05 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
> +static void colo_proxy_async_call(libxl__egc *egc,
> + libxl__colo_save_state *css,
> + void func(libxl__colo_save_state *),
> + libxl__ev_child_callback callback)
> +{
This is a straight clone-and-hack of drbd_async_call. Please make a
common function, or something.
> +static void colo_proxy_async_wait_for_checkpoint(libxl__colo_save_state *css)
> +{
> + int req;
> +
> + req = colo_proxy_checkpoint(&css->cps, 5000000);
What is this magic number ? This seems quite odd...
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 25/27] setup and control colo proxy on primary side
2016-03-04 18:05 ` Ian Jackson
@ 2016-03-22 6:01 ` Changlong Xie
0 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-22 6:01 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Li Zhijian, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Anthony Perard, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 02:05 AM, Ian Jackson wrote:
>> +static void colo_proxy_async_call(libxl__egc *egc,
>> + libxl__colo_save_state *css,
>> + void func(libxl__colo_save_state *),
>> + libxl__ev_child_callback callback)
>> +{
>
> This is a straight clone-and-hack of drbd_async_call. Please make a
> common function, or something.
Ok
>
>> +static void colo_proxy_async_wait_for_checkpoint(libxl__colo_save_state *css)
>> +{
>> + int req;
>> +
>> + req = colo_proxy_checkpoint(&css->cps, 5000000);
>
> What is this magic number ? This seems quite odd...
>
It's an empirical timeout value(5s) for waiting a checkpoint.
Thanks
-Xie
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 26/27] setup and control colo proxy on secondary side
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (24 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 25/27] setup and control colo proxy on primary side Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 18:05 ` Ian Jackson
2016-03-04 8:41 ` [PATCH v11 27/27] cmdline switches and config vars to control colo-proxy Changlong Xie
2016-03-04 18:17 ` [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Ian Jackson
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_colo_restore.c | 28 +++++++++++++++++++++++++---
1 file changed, 25 insertions(+), 3 deletions(-)
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index 8b343c0..52e44a2 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -37,9 +37,11 @@ struct libxl__colo_restore_checkpoint_state {
int);
};
+extern const libxl__checkpoint_device_instance_ops colo_restore_device_nic;
extern const libxl__checkpoint_device_instance_ops colo_restore_device_qdisk;
static const libxl__checkpoint_device_instance_ops *colo_restore_ops[] = {
+ &colo_restore_device_nic,
&colo_restore_device_qdisk,
NULL,
};
@@ -139,8 +141,14 @@ static int init_device_subkind(libxl__checkpoint_devices_state *cds)
int rc;
STATE_AO_GC(cds->ao);
+ rc = init_subkind_colo_nic(cds);
+ if (rc) goto out;
+
rc = init_subkind_qdisk(cds);
- if (rc) goto out;
+ if (rc) {
+ cleanup_subkind_colo_nic(cds);
+ goto out;
+ }
rc = 0;
out:
@@ -152,6 +160,7 @@ static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
/* cleanup device subkind-specific state in the libxl ctx */
STATE_AO_GC(cds->ao);
+ cleanup_subkind_colo_nic(cds);
cleanup_subkind_qdisk(cds);
}
@@ -341,6 +350,8 @@ static void colo_restore_teardown_devices_done(libxl__egc *egc,
if (crcs->teardown_devices)
cleanup_device_subkind(cds);
+ colo_proxy_teardown(&crs->cps);
+
rc = crcs->saved_rc;
if (!rc) {
crcs->callback = do_failover_done;
@@ -592,6 +603,8 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
}
}
+ colo_proxy_preresume(&crs->cps);
+
colo_restore_resume_vm(egc, crcs);
return;
@@ -628,6 +641,8 @@ static void colo_resume_vm_done(libxl__egc *egc,
crcs->status = LIBXL_COLO_RESUMED;
+ colo_proxy_postresume(&crs->cps);
+
/* avoid calling stream->completion_callback() more than once */
if (crs->saved_cb) {
dcs->callback = crs->saved_cb;
@@ -749,13 +764,20 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
STATE_AO_GC(crs->ao);
- /* TODO: nic support */
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
cds->callback = colo_restore_setup_cds_done;
cds->ao = ao;
cds->domid = crs->domid;
cds->ops = colo_restore_ops;
+ crs->cps.ao = ao;
+ if (colo_proxy_setup(&crs->cps)) {
+ LOG(ERROR, "COLO: failed to setup colo proxy for guest with domid %u",
+ cds->domid);
+ goto out;
+ }
+
if (init_device_subkind(cds))
goto out;
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 26/27] setup and control colo proxy on secondary side
2016-03-04 8:41 ` [PATCH v11 26/27] setup and control colo proxy on secondary side Changlong Xie
@ 2016-03-04 18:05 ` Ian Jackson
0 siblings, 0 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 18:05 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 26/27] setup and control colo proxy on secondary side"):
> From: Wen Congyang <wency@cn.fujitsu.com>
I don't see anything here I have questions about. I don't intend to
review it in detail...
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* [PATCH v11 27/27] cmdline switches and config vars to control colo-proxy
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (25 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 26/27] setup and control colo proxy on secondary side Changlong Xie
@ 2016-03-04 8:41 ` Changlong Xie
2016-03-04 18:09 ` Ian Jackson
2016-03-04 18:17 ` [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Ian Jackson
27 siblings, 1 reply; 100+ messages in thread
From: Changlong Xie @ 2016-03-04 8:41 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
From: Wen Congyang <wency@cn.fujitsu.com>
Add cmdline switches to 'xl migrate-receive' command to specify
a domain-specific hotplug script to setup COLO proxy.
Add a new config var 'colo.default.agentscript' to xl.conf, that
allows the user to override the default global script used to
setup COLO proxy.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Changlong Xie <xiecl.fnst@cn.fujitsu.com>
---
docs/man/xl.conf.pod.5 | 6 ++++++
docs/man/xl.pod.1 | 6 +++++-
tools/libxl/libxl.c | 6 ++++++
tools/libxl/libxl_create.c | 14 ++++++++++++--
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl.c | 3 +++
tools/libxl/xl.h | 1 +
tools/libxl/xl_cmdimpl.c | 47 ++++++++++++++++++++++++++++++++++-----------
8 files changed, 70 insertions(+), 14 deletions(-)
diff --git a/docs/man/xl.conf.pod.5 b/docs/man/xl.conf.pod.5
index 8ae19bb..8f7fd28 100644
--- a/docs/man/xl.conf.pod.5
+++ b/docs/man/xl.conf.pod.5
@@ -111,6 +111,12 @@ Configures the default script used by Remus to setup network buffering.
Default: C</etc/xen/scripts/remus-netbuf-setup>
+=item B<colo.default.proxyscript="PATH">
+
+Configures the default script used by COLO to setup colo-proxy.
+
+Default: C</etc/xen/scripts/colo-proxy-setup>
+
=item B<output_format="json|sxp">
Configures the default output format used by xl when printing "machine
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 36b093b..3977d4b 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -458,7 +458,6 @@ Remus support in xl is still in experimental (proof-of-concept) phase.
Disk replication support is limited to DRBD disks.
COLO support in xl is still in experimental (proof-of-concept) phase.
-There is no support for network at the moment.
=back
@@ -482,6 +481,11 @@ and it's used by secondary.
=item B<hidden-disk> :Primary's modified contents will be buffered in this
disk, and it's used by secondary.
+(b) An example for COLO network configuration: vif =[ '...,forwarddev=xxx,...']
+
+=item B<forwarddev> :Forward devices for primary and secondary, there are
+directly connected.
+
=back
B<OPTIONS>
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index eb1b277..4699641 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -3392,6 +3392,11 @@ void libxl__device_nic_add(libxl__egc *egc, uint32_t domid,
flexarray_append(back, nic->ifname);
}
+ if (nic->forwarddev) {
+ flexarray_append(back, "forwarddev");
+ flexarray_append(back, nic->forwarddev);
+ }
+
flexarray_append(back, "mac");
flexarray_append(back,GCSPRINTF(LIBXL_MAC_FMT, LIBXL_MAC_BYTES(nic->mac)));
if (nic->ip) {
@@ -3514,6 +3519,7 @@ static int libxl__device_nic_from_xs_be(libxl__gc *gc,
nic->ip = READ_BACKEND(NOGC, "ip");
nic->bridge = READ_BACKEND(NOGC, "bridge");
nic->script = READ_BACKEND(NOGC, "script");
+ nic->forwarddev = READ_BACKEND(NOGC, "forwarddev");
/* vif_ioemu nics use the same xenstore entries as vif interfaces */
tmp = READ_BACKEND(gc, "type");
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index bff08b0..63e1a0d 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1072,6 +1072,11 @@ static void domcreate_bootloader_done(libxl__egc *egc,
crs->recv_fd = restore_fd;
crs->hvm = (info->type == LIBXL_DOMAIN_TYPE_HVM);
crs->callback = libxl__colo_restore_setup_done;
+ if (dcs->colo_proxy_script)
+ crs->colo_proxy_script = libxl__strdup(gc, dcs->colo_proxy_script);
+ else
+ crs->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
+ libxl__xen_script_dir_path());
libxl__colo_restore_setup(egc, crs);
break;
case LIBXL_CHECKPOINTED_STREAM_REMUS:
@@ -1614,6 +1619,7 @@ static void domain_create_cb(libxl__egc *egc,
static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid, int restore_fd, int send_back_fd,
const libxl_domain_restore_params *params,
+ const char *colo_proxy_script,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
@@ -1637,6 +1643,7 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
}
cdcs->dcs.callback = domain_create_cb;
cdcs->dcs.domid_soft_reset = INVALID_DOMID;
+ cdcs->dcs.colo_proxy_script = colo_proxy_script;
libxl__ao_progress_gethow(&cdcs->dcs.aop_console_how, aop_console_how);
cdcs->domid_out = domid;
@@ -1823,7 +1830,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncprogress_how *aop_console_how)
{
unset_disk_colo_restore(d_config);
- return do_domain_create(ctx, d_config, domid, -1, -1, NULL,
+ return do_domain_create(ctx, d_config, domid, -1, -1, NULL, NULL,
ao_how, aop_console_how);
}
@@ -1834,14 +1841,17 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
+ char *colo_proxy_script = NULL;
+
if (params->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
+ colo_proxy_script = params->colo_proxy_script;
set_disk_colo_restore(d_config);
} else {
unset_disk_colo_restore(d_config);
}
return do_domain_create(ctx, d_config, domid, restore_fd, send_back_fd,
- params, ao_how, aop_console_how);
+ params, colo_proxy_script, ao_how, aop_console_how);
}
int libxl_domain_soft_reset(libxl_ctx *ctx,
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 6b57aba..883d3a8 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -385,6 +385,7 @@ libxl_domain_create_info = Struct("domain_create_info",[
libxl_domain_restore_params = Struct("domain_restore_params", [
("checkpointed_stream", integer),
("stream_version", uint32, {'init_val': '1'}),
+ ("colo_proxy_script", string),
])
libxl_domain_sched_params = Struct("domain_sched_params",[
diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index dfae84a..a272258 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -45,6 +45,7 @@ char *default_bridge = NULL;
char *default_gatewaydev = NULL;
char *default_vifbackend = NULL;
char *default_remus_netbufscript = NULL;
+char *default_colo_proxy_script = NULL;
enum output_format default_output_format = OUTPUT_FORMAT_JSON;
int claim_mode = 1;
bool progress_use_cr = 0;
@@ -179,6 +180,8 @@ static void parse_global_config(const char *configfile,
xlu_cfg_replace_string (config, "remus.default.netbufscript",
&default_remus_netbufscript, 0);
+ xlu_cfg_replace_string (config, "colo.default.proxyscript",
+ &default_colo_proxy_script, 0);
xlu_cfg_destroy(config);
}
diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index bdab125..709c3fd 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -189,6 +189,7 @@ extern char *default_bridge;
extern char *default_gatewaydev;
extern char *default_vifbackend;
extern char *default_remus_netbufscript;
+extern char *default_colo_proxy_script;
extern char *blkdev_start;
enum output_format {
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 30d26bc..bec13e0 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -158,6 +158,7 @@ struct domain_create {
const char *config_file;
char *extra_config; /* extra config string */
const char *restore_file;
+ char *colo_proxy_script;
int migrate_fd; /* -1 means none */
int send_back_fd; /* -1 means none */
char **migration_domname_r; /* from malloc */
@@ -1053,6 +1054,8 @@ static int parse_nic_config(libxl_device_nic *nic, XLU_Config **config, char *to
replace_string(&nic->model, oparg);
} else if (MATCH_OPTION("rate", token, oparg)) {
parse_vif_rate(config, oparg, nic);
+ } else if (MATCH_OPTION("forwarddev", token, oparg)) {
+ replace_string(&nic->forwarddev, oparg);
} else if (MATCH_OPTION("accel", token, oparg)) {
fprintf(stderr, "the accel parameter for vifs is currently not supported\n");
} else {
@@ -2896,6 +2899,7 @@ start:
params.checkpointed_stream = dom_info->checkpointed_stream;
params.stream_version =
(hdr.mandatory_flags & XL_MANDATORY_FLAG_STREAMv2) ? 2 : 1;
+ params.colo_proxy_script = dom_info->colo_proxy_script;
ret = libxl_domain_create_restore(ctx, &d_config,
&domid, restore_fd,
@@ -4438,7 +4442,8 @@ static void migrate_domain(uint32_t domid, const char *rune, int debug,
static void migrate_receive(int debug, int daemonize, int monitor,
int send_fd, int recv_fd,
- libxl_checkpointed_stream checkpointed)
+ libxl_checkpointed_stream checkpointed,
+ char *colo_proxy_script)
{
uint32_t domid;
int rc, rc2;
@@ -4467,6 +4472,7 @@ static void migrate_receive(int debug, int daemonize, int monitor,
dom_info.send_back_fd = send_fd;
dom_info.migration_domname_r = &migration_domname;
dom_info.checkpointed_stream = checkpointed;
+ dom_info.colo_proxy_script = colo_proxy_script;
rc = create_domain(&dom_info);
if (rc < 0) {
@@ -4660,8 +4666,9 @@ int main_migrate_receive(int argc, char **argv)
int debug = 0, daemonize = 1, monitor = 1;
libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
int opt;
+ char *script = NULL;
- SWITCH_FOREACH_OPT(opt, "Fedrc", NULL, "migrate-receive", 0) {
+ SWITCH_FOREACH_OPT(opt, "Fedrcn:", NULL, "migrate-receive", 0) {
case 'F':
daemonize = 0;
break;
@@ -4678,6 +4685,9 @@ int main_migrate_receive(int argc, char **argv)
case 'c':
checkpointed = LIBXL_CHECKPOINTED_STREAM_COLO;
break;
+ case 'n':
+ script = optarg;
+ break;
}
if (argc-optind != 0) {
@@ -4686,7 +4696,7 @@ int main_migrate_receive(int argc, char **argv)
}
migrate_receive(debug, daemonize, monitor,
STDOUT_FILENO, STDIN_FILENO,
- checkpointed);
+ checkpointed, script);
return 0;
}
@@ -8096,8 +8106,10 @@ int main_remus(int argc, char **argv)
r_info.interval = 200;
if (libxl_defbool_val(r_info.colo)) {
- if (r_info.interval || libxl_defbool_val(r_info.blackhole)) {
- perror("Option -c conflicts with -i or -b");
+ if (r_info.interval || libxl_defbool_val(r_info.blackhole) ||
+ !libxl_defbool_is_default(r_info.netbuf) ||
+ !libxl_defbool_is_default(r_info.diskbuf)) {
+ perror("option -c is conflict with -i, -d, -n or -b");
exit(-1);
}
@@ -8108,8 +8120,12 @@ int main_remus(int argc, char **argv)
}
}
- if (!r_info.netbufscript)
- r_info.netbufscript = default_remus_netbufscript;
+ if (!r_info.netbufscript) {
+ if (libxl_defbool_val(r_info.colo))
+ r_info.netbufscript = default_colo_proxy_script;
+ else
+ r_info.netbufscript = default_remus_netbufscript;
+ }
if (libxl_defbool_val(r_info.blackhole)) {
send_fd = open("/dev/null", O_RDWR, 0644);
@@ -8122,10 +8138,19 @@ int main_remus(int argc, char **argv)
if (!ssh_command[0]) {
rune = host;
} else {
- xasprintf(&rune, "exec %s %s xl migrate-receive %s %s",
- ssh_command, host,
- libxl_defbool_val(r_info.colo) ? "-c" : "-r",
- daemonize ? "" : " -e");
+ if (!libxl_defbool_val(r_info.colo)) {
+ xasprintf(&rune, "exec %s %s xl migrate-receive %s %s",
+ ssh_command, host,
+ "-r",
+ daemonize ? "" : " -e");
+ } else {
+ xasprintf(&rune, "exec %s %s xl migrate-receive %s %s %s %s",
+ ssh_command, host,
+ "-c",
+ r_info.netbufscript ? "-n" : "",
+ r_info.netbufscript ? r_info.netbufscript : "",
+ daemonize ? "" : " -e");
+ }
}
save_domain_core_begin(domid, NULL, &config_data, &config_len);
--
1.9.3
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 100+ messages in thread* Re: [PATCH v11 27/27] cmdline switches and config vars to control colo-proxy
2016-03-04 8:41 ` [PATCH v11 27/27] cmdline switches and config vars to control colo-proxy Changlong Xie
@ 2016-03-04 18:09 ` Ian Jackson
2016-03-22 4:13 ` Changlong Xie
0 siblings, 1 reply; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 18:09 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 27/27] cmdline switches and config vars to control colo-proxy"):
> From: Wen Congyang <wency@cn.fujitsu.com>
>
> Add cmdline switches to 'xl migrate-receive' command to specify
> a domain-specific hotplug script to setup COLO proxy.
>
> Add a new config var 'colo.default.agentscript' to xl.conf, that
> allows the user to override the default global script used to
> setup COLO proxy.
...
> +(b) An example for COLO network configuration: vif =[ '...,forwarddev=xxx,...']
> +
> +=item B<forwarddev> :Forward devices for primary and secondary, there are
> +directly connected.
What should the user specify as a `forwarddev' ?
> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
> index bff08b0..63e1a0d 100644
> --- a/tools/libxl/libxl_create.c
> +++ b/tools/libxl/libxl_create.c
> @@ -1072,6 +1072,11 @@ static void domcreate_bootloader_done(libxl__egc *egc,
> crs->recv_fd = restore_fd;
> crs->hvm = (info->type == LIBXL_DOMAIN_TYPE_HVM);
> crs->callback = libxl__colo_restore_setup_done;
> + if (dcs->colo_proxy_script)
> + crs->colo_proxy_script = libxl__strdup(gc, dcs->colo_proxy_script);
> + else
> + crs->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
> + libxl__xen_script_dir_path());
Would it be possible to separate out this setting and plubming of
colo_proxy_script into its own patch ? It is quite formulaic and
makes the substance of the patch harder to read.
> @@ -4660,8 +4666,9 @@ int main_migrate_receive(int argc, char **argv)
> int debug = 0, daemonize = 1, monitor = 1;
> libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
> int opt;
> + char *script = NULL;
>
> - SWITCH_FOREACH_OPT(opt, "Fedrc", NULL, "migrate-receive", 0) {
> + SWITCH_FOREACH_OPT(opt, "Fedrcn:", NULL, "migrate-receive", 0) {
I don't think we want to allocate `-n' for this. I think needing to
specify this will be rare. Can we use a long option instead ?
Thanks,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 27/27] cmdline switches and config vars to control colo-proxy
2016-03-04 18:09 ` Ian Jackson
@ 2016-03-22 4:13 ` Changlong Xie
0 siblings, 0 replies; 100+ messages in thread
From: Changlong Xie @ 2016-03-22 4:13 UTC (permalink / raw)
To: Ian Jackson, Li Zhijian
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On 03/05/2016 02:09 AM, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 27/27] cmdline switches and config vars to control colo-proxy"):
>> From: Wen Congyang <wency@cn.fujitsu.com>
>>
>> Add cmdline switches to 'xl migrate-receive' command to specify
>> a domain-specific hotplug script to setup COLO proxy.
>>
>> Add a new config var 'colo.default.agentscript' to xl.conf, that
>> allows the user to override the default global script used to
>> setup COLO proxy.
> ...
>
>> +(b) An example for COLO network configuration: vif =[ '...,forwarddev=xxx,...']
>> +
>> +=item B<forwarddev> :Forward devices for primary and secondary, there are
>> +directly connected.
>
> What should the user specify as a `forwarddev' ?
a physical NIC(forwarddev=eth0) or a virtual NIC(forwarddev=vlan0) could
be fine.
>
>> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
>> index bff08b0..63e1a0d 100644
>> --- a/tools/libxl/libxl_create.c
>> +++ b/tools/libxl/libxl_create.c
>> @@ -1072,6 +1072,11 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>> crs->recv_fd = restore_fd;
>> crs->hvm = (info->type == LIBXL_DOMAIN_TYPE_HVM);
>> crs->callback = libxl__colo_restore_setup_done;
>> + if (dcs->colo_proxy_script)
>> + crs->colo_proxy_script = libxl__strdup(gc, dcs->colo_proxy_script);
>> + else
>> + crs->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
>> + libxl__xen_script_dir_path());
>
> Would it be possible to separate out this setting and plubming of
> colo_proxy_script into its own patch ? It is quite formulaic and
> makes the substance of the patch harder to read.
Will try that.
>
>> @@ -4660,8 +4666,9 @@ int main_migrate_receive(int argc, char **argv)
>> int debug = 0, daemonize = 1, monitor = 1;
>> libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
>> int opt;
>> + char *script = NULL;
>>
>> - SWITCH_FOREACH_OPT(opt, "Fedrc", NULL, "migrate-receive", 0) {
>> + SWITCH_FOREACH_OPT(opt, "Fedrcn:", NULL, "migrate-receive", 0) {
>
> I don't think we want to allocate `-n' for this. I think needing to
> specify this will be rare. Can we use a long option instead ?
>
Ok
Thanks
-Xie
> Thanks,
> Ian.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service
2016-03-04 8:41 [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Changlong Xie
` (26 preceding siblings ...)
2016-03-04 8:41 ` [PATCH v11 27/27] cmdline switches and config vars to control colo-proxy Changlong Xie
@ 2016-03-04 18:17 ` Ian Jackson
2016-03-04 20:35 ` Konrad Rzeszutek Wilk
2016-03-17 17:41 ` Ian Jackson
27 siblings, 2 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-04 18:17 UTC (permalink / raw)
To: Changlong Xie
Cc: Lars Kurth, Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Changlong Xie writes ("[PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service"):
> This patchset implemented the COLO feature for Xen.
> For detail/install/use of COLO feature, refer to:
> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
Thanks for this series. I've now gone and at least looked at each of
these patches. I think this is an important feature which I want to
see in Xen 4.7.
Most of the patches seem to do roughly sensible things in roughly
sensible ways. I have reviewed some of the areas where they touch
common code, and some of the API and protocol design. (I don't think
it very valuable to review the implementation in detail.)
Overall I think most of this is in good shape and on track.
But as you see from my mails I have some serious questions about the
disk checkpointing/plumbing architecture. I have reservations about
the use of qemu for this. I think this would perhaps be better done
as a devmapper module.
But I don't feel I understand it well enough yet. I have read the
docs which have been provided. The QEMU implementation doc was
helpful but I still feel confused. I think I should go away and read
it again more closely. In the meantime, answers to my questions would
be helpful.
If after discussion and further thought I do still think that doing
this in qemu is the wrong place, that doesn't mean that we need to
block this series in the hope of it being rearchitected. It just
means that I want to make sure that the _interface_ to libxl, as seen
from the outside and especially as seen from the user's point of view,
does not preclude future design changes.
In particular, what I care about in this context is that the libxl API
(and the xl config syntax) 1. doesn't preclude an implementation of
the same functionality elsewhere, and 2. doesn't preclude COLO for PV
guests (or hvm-lite-ng guests).
Thanks for your attention. I'm afraid I'm going to be away out of the
office for all of next week. So I will pick this up again when I get
back.
Regards,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service
2016-03-04 18:17 ` [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Ian Jackson
@ 2016-03-04 20:35 ` Konrad Rzeszutek Wilk
2016-03-17 17:19 ` Ian Jackson
2016-03-17 17:41 ` Ian Jackson
1 sibling, 1 reply; 100+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-03-04 20:35 UTC (permalink / raw)
To: Ian Jackson
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Anthony Perard, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On Fri, Mar 04, 2016 at 06:17:13PM +0000, Ian Jackson wrote:
> Changlong Xie writes ("[PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service"):
> > This patchset implemented the COLO feature for Xen.
> > For detail/install/use of COLO feature, refer to:
> > http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>
> Thanks for this series. I've now gone and at least looked at each of
> these patches. I think this is an important feature which I want to
> see in Xen 4.7.
>
> Most of the patches seem to do roughly sensible things in roughly
> sensible ways. I have reviewed some of the areas where they touch
> common code, and some of the API and protocol design. (I don't think
> it very valuable to review the implementation in detail.)
>
> Overall I think most of this is in good shape and on track.
>
> But as you see from my mails I have some serious questions about the
> disk checkpointing/plumbing architecture. I have reservations about
> the use of qemu for this. I think this would perhaps be better done
> as a devmapper module.
There has been a lot of development in the snapshotting implementation
in QEMU that simply isn't done in the kernel module. And in many
cases you do not want to have in the kernel b/c of so many pieces
it has to deal with (network, locking, checkpointing, etc) - which
can be done in user-space.
>
> But I don't feel I understand it well enough yet. I have read the
> docs which have been provided. The QEMU implementation doc was
> helpful but I still feel confused. I think I should go away and read
> it again more closely. In the meantime, answers to my questions would
> be helpful.
>
> If after discussion and further thought I do still think that doing
> this in qemu is the wrong place, that doesn't mean that we need to
> block this series in the hope of it being rearchitected. It just
> means that I want to make sure that the _interface_ to libxl, as seen
> from the outside and especially as seen from the user's point of view,
> does not preclude future design changes.
>
> In particular, what I care about in this context is that the libxl API
> (and the xl config syntax) 1. doesn't preclude an implementation of
> the same functionality elsewhere, and 2. doesn't preclude COLO for PV
> guests (or hvm-lite-ng guests).
>
> Thanks for your attention. I'm afraid I'm going to be away out of the
> office for all of next week. So I will pick this up again when I get
> back.
>
> Regards,
> Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread* Re: [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service
2016-03-04 20:35 ` Konrad Rzeszutek Wilk
@ 2016-03-17 17:19 ` Ian Jackson
0 siblings, 0 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-17 17:19 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Wen Congyang,
Andrew Cooper, Jiang Yunhong, Dong Eddie, xen devel,
Anthony Perard, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
Konrad Rzeszutek Wilk writes ("Re: [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service"):
> On Fri, Mar 04, 2016 at 06:17:13PM +0000, Ian Jackson wrote:
> > But as you see from my mails I have some serious questions about the
> > disk checkpointing/plumbing architecture. I have reservations about
> > the use of qemu for this. I think this would perhaps be better done
> > as a devmapper module.
>
> There has been a lot of development in the snapshotting implementation
> in QEMU that simply isn't done in the kernel module. And in many
> cases you do not want to have in the kernel b/c of so many pieces
> it has to deal with (network, locking, checkpointing, etc) - which
> can be done in user-space.
Right. I think I'm now convinced on this question.
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread
* Re: [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service
2016-03-04 18:17 ` [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Ian Jackson
2016-03-04 20:35 ` Konrad Rzeszutek Wilk
@ 2016-03-17 17:41 ` Ian Jackson
1 sibling, 0 replies; 100+ messages in thread
From: Ian Jackson @ 2016-03-17 17:41 UTC (permalink / raw)
To: Changlong Xie, xen devel, Konrad Rzeszutek Wilk, Andrew Cooper,
Ian Campbell, Wei Liu, Dong Eddie, Jiang Yunhong, Wen Congyang,
Shriram Rajagopalan, Gui Jianfeng, Yang Hongyang, Lars Kurth,
Anthony Perard
Ian Jackson writes ("Re: [PATCH v11 00/27] COarse-grain LOck-stepping Virtual Machines for Non-stop Service"):
> Thanks for your attention. I'm afraid I'm going to be away out of the
> office for all of next week. So I will pick this up again when I get
> back.
So, as you see, you have in fact answered many of my questions. I do
have some more questions but I don't think we want to block your next
version of this series until they are answered.
So (having consulted Wei) I think it would probably be best if you
addressed the comments you've had, so far as possible, and send your
next version of the series ASAP.
Thanks.
Regards,
Ian.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 100+ messages in thread