* [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service
@ 2016-02-22 2:52 Wen Congyang
2016-02-22 2:52 ` [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state Wen Congyang
` (31 more replies)
0 siblings, 32 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
This patchset implemented the COLO feature for Xen.
For detail/install/use of COLO feature, refer to:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
This patchset is based on:
1. http://lists.xenproject.org/archives/html/xen-devel/2016-02/msg02515.html
You can get the codes from here:
https://github.com/wencongyang/xen/tree/colo_v10
Changlog from v9 to v10
1. Rebased to the upstream xen
2. Fix one bug found in the test
3. Merge some patches from prepare series
4. Split patch 5 to two patches(patch 4 and 5) according to the comments from
Wei Liu
Changlog from v8 to v9:
1. Rebased to the upstream xen
2. Fix some bugs found in the test
Changelog from v7 to v8:
1. Rebased to the latest libxl migration v2.
Changelog from v6 to v7:
1. Ported to Libxl migration v2
2. Send dirty bitmap from secondary to primary on libxc side
3. Address review comments
Changelog from v5 to v6:
1. based on migration v2(libxc)
2. split the patchset into prerequisite patchset and this main patchset.
Changelog from v4 to v5:
1. rebase to the latest xen upstream
2. disk replication: blktap2->qdisk
3. nic replication: colo-agent->colo-proxy
Changelog from v3 to v4:
1. rebase to newest xen
2. bug fix
Changlog from v2 to v3:
1. rebase to newest remus
2. add nic replication support
Changlog from v1 to v2:
1. rebase to newest remus
2. add disk replication support
Wen Congyang (31):
tools/libxl: introduce libxl__domain_restore_device_model to load qemu
state
tools/libxl: introduce libxl__domain_common_switch_qemu_logdirty()
tools/libxl: Add back channel to allow migration target send data back
tools/libxl: Introduce new helper function dup_fd_helper()
tools/libx{l,c}: add back channel to libxc
docs: add colo readme
docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo
streams
libxc/migration: Specification update for DIRTY_PFN_LIST records
libxc/migration: export read_record for common use
tools/libxl: add back channel support to write stream
tools/libxl: write checkpoint_state records into the stream
tools/libxl: add back channel support to read stream
tools/libxl: handle checkpoint_state records in a libxl migration v2
read stream
tools/libx{l,c}: introduce wait_checkpoint callback
tools/libx{l,c}: add postcopy/suspend callback to restore side
secondary vm suspend/resume/checkpoint code
primary vm suspend/resume/checkpoint code
libxc/restore: support COLO restore
libxc/restore: send dirty pfn list to primary when checkpoint under
colo
send store gfn and console gfn to xl before resuming secondary vm
libxc/save: support COLO save
implement the cmdline for COLO
COLO: introduce new API to prepare/start/do/get_error/stop replication
Support colo mode for qemu disk
COLO: use qemu block replication
COLO proxy: implement setup/teardown of COLO proxy module
COLO proxy: preresume, postresume and checkpoint
COLO nic: implement COLO nic subkind
setup and control colo proxy on primary side
setup and control colo proxy on secondary side
cmdline switches and config vars to control colo-proxy
docs/README.colo | 9 +
docs/man/xl.conf.pod.5 | 6 +
docs/man/xl.pod.1 | 11 +-
docs/misc/xl-disk-configuration.txt | 50 ++
docs/specs/libxc-migration-stream.pandoc | 24 +-
docs/specs/libxl-migration-stream.pandoc | 25 +-
tools/hotplug/Linux/Makefile | 1 +
tools/hotplug/Linux/colo-proxy-setup | 135 ++++
tools/libxc/include/xenguest.h | 40 +-
tools/libxc/xc_nomigrate.c | 4 +-
tools/libxc/xc_sr_common.c | 80 ++-
tools/libxc/xc_sr_common.h | 26 +-
tools/libxc/xc_sr_restore.c | 246 +++++--
tools/libxc/xc_sr_save.c | 109 ++-
tools/libxc/xc_sr_stream_format.h | 31 +-
tools/libxl/Makefile | 4 +
tools/libxl/libxl.c | 100 ++-
tools/libxl/libxl.h | 29 +-
tools/libxl/libxl_colo.h | 36 +
tools/libxl/libxl_colo_nic.c | 321 +++++++++
tools/libxl/libxl_colo_proxy.c | 292 ++++++++
tools/libxl/libxl_colo_qdisk.c | 226 +++++++
tools/libxl/libxl_colo_restore.c | 1091 ++++++++++++++++++++++++++++++
tools/libxl/libxl_colo_save.c | 721 ++++++++++++++++++++
tools/libxl/libxl_create.c | 86 ++-
tools/libxl/libxl_device.c | 54 ++
tools/libxl/libxl_dm.c | 184 ++++-
tools/libxl/libxl_dom_save.c | 124 ++--
tools/libxl/libxl_internal.h | 247 +++++--
tools/libxl/libxl_qmp.c | 106 +++
tools/libxl/libxl_save_callout.c | 52 +-
tools/libxl/libxl_save_helper.c | 8 +-
tools/libxl/libxl_save_msgs_gen.pl | 13 +-
tools/libxl/libxl_sr_stream_format.h | 11 +
tools/libxl/libxl_stream_read.c | 96 ++-
tools/libxl/libxl_stream_write.c | 86 ++-
tools/libxl/libxl_types.idl | 11 +
tools/libxl/libxlu_disk_l.l | 7 +
tools/libxl/xl.c | 3 +
tools/libxl/xl.h | 1 +
tools/libxl/xl_cmdimpl.c | 107 ++-
tools/libxl/xl_cmdtable.c | 4 +-
tools/ocaml/libs/xl/xenlight_stubs.c | 2 +-
tools/python/xen/migration/libxc.py | 68 +-
tools/python/xen/migration/libxl.py | 9 +
45 files changed, 4567 insertions(+), 329 deletions(-)
create mode 100644 docs/README.colo
create mode 100755 tools/hotplug/Linux/colo-proxy-setup
create mode 100644 tools/libxl/libxl_colo.h
create mode 100644 tools/libxl/libxl_colo_nic.c
create mode 100644 tools/libxl/libxl_colo_proxy.c
create mode 100644 tools/libxl/libxl_colo_qdisk.c
create mode 100644 tools/libxl/libxl_colo_restore.c
create mode 100644 tools/libxl/libxl_colo_save.c
--
2.5.0
^ permalink raw reply [flat|nested] 70+ messages in thread
* [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:53 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 02/31] tools/libxl: introduce libxl__domain_common_switch_qemu_logdirty() Wen Congyang
` (30 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Anthony Perard, Shriram Rajagopalan,
Yang Hongyang
In normal migration, the qemu state is passed to qemu as a parameter.
With COLO, secondary vm is running. So we will do the following steps
at every checkpoint:
1. suspend both primary vm and secondary vm
2. sync the state
3. resume both primary vm and secondary vm
Primary will send qemu's state in step2, and secondary's qemu should
read it and restore the state before it is resumed. We can not pass
the state to qemu as a parameter because secondary QEMU already started
at this point, so we introduce libxl__domain_restore_device_model() to
do it. This API MUST be called before resuming secondary vm.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Cc: Anthony Perard <anthony.perard@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
tools/libxl/libxl_dom_save.c | 20 ++++++++++++++++++++
tools/libxl/libxl_internal.h | 4 ++++
tools/libxl/libxl_qmp.c | 10 ++++++++++
3 files changed, 34 insertions(+)
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index 4eb7960..7d345f9 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -512,6 +512,26 @@ int libxl__restore_emulator_xenstore_data(libxl__domain_create_state *dcs,
return rc;
}
+int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
+ const char *restore_file)
+{
+ int rc;
+
+ switch (libxl__device_model_version_running(gc, domid)) {
+ case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
+ /* Will never be supported. */
+ rc = ERROR_INVAL;
+ break;
+ case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
+ rc = libxl__qmp_restore(gc, domid, restore_file);
+ break;
+ default:
+ rc = ERROR_INVAL;
+ }
+
+ return rc;
+}
+
/*
* Local variables:
* mode: C
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index a1aae97..5320e5e 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1119,6 +1119,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, uint32_t domid,
const char *old_name, const char *new_name,
xs_transaction_t trans);
+_hidden int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
+ const char *restore_file);
_hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid);
_hidden const char *libxl__userdata_path(libxl__gc *gc, uint32_t domid,
@@ -1762,6 +1764,8 @@ _hidden int libxl__qmp_stop(libxl__gc *gc, int domid);
_hidden int libxl__qmp_resume(libxl__gc *gc, int domid);
/* Save current QEMU state into fd. */
_hidden int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename);
+/* Load current QEMU state from file. */
+_hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char *filename);
/* Set dirty bitmap logging status */
_hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool enable);
_hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const libxl_device_disk *disk);
diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
index 714038b..eec8a44 100644
--- a/tools/libxl/libxl_qmp.c
+++ b/tools/libxl/libxl_qmp.c
@@ -905,6 +905,16 @@ int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename)
NULL, NULL);
}
+int libxl__qmp_restore(libxl__gc *gc, int domid, const char *state_file)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_string(gc, &args, "filename", state_file);
+
+ return qmp_run_command(gc, domid, "xen-load-devices-state", args,
+ NULL, NULL);
+}
+
static int qmp_change(libxl__gc *gc, libxl__qmp_handler *qmp,
char *device, char *target, char *arg)
{
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 02/31] tools/libxl: introduce libxl__domain_common_switch_qemu_logdirty()
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
2016-02-22 2:52 ` [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 03/31] tools/libxl: Add back channel to allow migration target send data back Wen Congyang
` (29 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Secondary vm is running in COLO mode, we need to send secondary
vm's dirty page information to primary host at checkpoint, so we
have to enable qemu logdirty on secondary.
libxl__domain_suspend_common_switch_qemu_logdirty() is to enable
qemu logdirty. But it uses libxl__domain_save_state, and calls
libxl__xc_domain_saverestore_async_callback_done() before exits.
This can not be used for secondary vm.
Update libxl__domain_suspend_common_switch_qemu_logdirty() to
introduce a new API libxl__domain_common_switch_qemu_logdirty().
This API only uses libxl__logdirty_switch, and calls
lds->callback before exits. This new API will be used by the patch:
secondary vm suspend/resume/checkpoint code
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl_dom_save.c | 95 ++++++++++++++++++++++++--------------------
tools/libxl/libxl_internal.h | 8 ++++
2 files changed, 60 insertions(+), 43 deletions(-)
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index 7d345f9..72b61c7 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -42,7 +42,7 @@ static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev,
static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch*,
const char *watch_path, const char *event_path);
static void switch_logdirty_done(libxl__egc *egc,
- libxl__domain_save_state *dss, int rc);
+ libxl__logdirty_switch *lds, int rc);
void libxl__logdirty_init(libxl__logdirty_switch *lds)
{
@@ -52,13 +52,10 @@ void libxl__logdirty_init(libxl__logdirty_switch *lds)
}
static void domain_suspend_switch_qemu_xen_traditional_logdirty
- (int domid, unsigned enable,
- libxl__save_helper_state *shs)
+ (libxl__egc *egc, int domid, unsigned enable,
+ libxl__logdirty_switch *lds)
{
- libxl__egc *egc = shs->egc;
- libxl__domain_save_state *dss = shs->caller_state;
- libxl__logdirty_switch *lds = &dss->logdirty;
- STATE_AO_GC(dss->ao);
+ STATE_AO_GC(lds->ao);
int rc;
xs_transaction_t t = 0;
const char *got;
@@ -120,26 +117,34 @@ static void domain_suspend_switch_qemu_xen_traditional_logdirty
out:
LOG(ERROR,"logdirty switch failed (rc=%d), abandoning suspend",rc);
libxl__xs_transaction_abort(gc, &t);
- switch_logdirty_done(egc,dss,rc);
+ switch_logdirty_done(egc,lds,rc);
}
static void domain_suspend_switch_qemu_xen_logdirty
- (int domid, unsigned enable,
- libxl__save_helper_state *shs)
+ (libxl__egc *egc, int domid, unsigned enable,
+ libxl__logdirty_switch *lds)
{
- libxl__egc *egc = shs->egc;
- libxl__domain_save_state *dss = shs->caller_state;
- STATE_AO_GC(dss->ao);
+ STATE_AO_GC(lds->ao);
int rc;
rc = libxl__qmp_set_global_dirty_log(gc, domid, enable);
- if (!rc) {
- libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
- } else {
+ if (rc)
LOG(ERROR,"logdirty switch failed (rc=%d), abandoning suspend",rc);
+
+ lds->callback(egc, lds, rc);
+}
+
+static void domain_suspend_switch_qemu_logdirty_done
+ (libxl__egc *egc, libxl__logdirty_switch *lds, int rc)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(lds, *dss, logdirty);
+
+ if (rc) {
dss->rc = rc;
- libxl__xc_domain_saverestore_async_callback_done(egc, shs, -1);
- }
+ libxl__xc_domain_saverestore_async_callback_done(egc,
+ &dss->sws.shs, -1);
+ } else
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
}
void libxl__domain_suspend_common_switch_qemu_logdirty
@@ -148,42 +153,52 @@ void libxl__domain_suspend_common_switch_qemu_logdirty
libxl__save_helper_state *shs = user;
libxl__egc *egc = shs->egc;
libxl__domain_save_state *dss = shs->caller_state;
- STATE_AO_GC(dss->ao);
+
+ /* Convenience aliases. */
+ libxl__logdirty_switch *const lds = &dss->logdirty;
+
+ lds->callback = domain_suspend_switch_qemu_logdirty_done;
+ libxl__domain_common_switch_qemu_logdirty(egc, domid, enable, lds);
+}
+
+void libxl__domain_common_switch_qemu_logdirty(libxl__egc *egc,
+ int domid, unsigned enable,
+ libxl__logdirty_switch *lds)
+{
+ STATE_AO_GC(lds->ao);
switch (libxl__device_model_version_running(gc, domid)) {
case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
- domain_suspend_switch_qemu_xen_traditional_logdirty(domid, enable, shs);
+ domain_suspend_switch_qemu_xen_traditional_logdirty(egc, domid, enable,
+ lds);
break;
case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
- domain_suspend_switch_qemu_xen_logdirty(domid, enable, shs);
+ domain_suspend_switch_qemu_xen_logdirty(egc, domid, enable, lds);
break;
case LIBXL_DEVICE_MODEL_VERSION_NONE:
- libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+ lds->callback(egc, lds, 0);
break;
default:
LOG(ERROR,"logdirty switch failed"
", no valid device model version found, abandoning suspend");
- dss->rc = ERROR_FAIL;
- libxl__xc_domain_saverestore_async_callback_done(egc, shs, -1);
+ lds->callback(egc, lds, ERROR_FAIL);
}
}
static void switch_logdirty_timeout(libxl__egc *egc, libxl__ev_time *ev,
const struct timeval *requested_abs,
int rc)
{
- libxl__domain_save_state *dss = CONTAINER_OF(ev, *dss, logdirty.timeout);
- STATE_AO_GC(dss->ao);
+ libxl__logdirty_switch *lds = CONTAINER_OF(ev, *lds, timeout);
+ STATE_AO_GC(lds->ao);
LOG(ERROR,"logdirty switch: wait for device model timed out");
- switch_logdirty_done(egc,dss,ERROR_FAIL);
+ switch_logdirty_done(egc,lds,ERROR_FAIL);
}
static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch *watch,
const char *watch_path, const char *event_path)
{
- libxl__domain_save_state *dss =
- CONTAINER_OF(watch, *dss, logdirty.watch);
- libxl__logdirty_switch *lds = &dss->logdirty;
- STATE_AO_GC(dss->ao);
+ libxl__logdirty_switch *lds = CONTAINER_OF(watch, *lds, watch);
+ STATE_AO_GC(lds->ao);
const char *got;
xs_transaction_t t = 0;
int rc;
@@ -229,28 +244,20 @@ static void switch_logdirty_xswatch(libxl__egc *egc, libxl__ev_xswatch *watch,
if (rc <= 0) {
if (rc < 0)
LOG(ERROR,"logdirty switch: failed (rc=%d)",rc);
- switch_logdirty_done(egc,dss,rc);
+ switch_logdirty_done(egc,lds,rc);
}
}
static void switch_logdirty_done(libxl__egc *egc,
- libxl__domain_save_state *dss,
+ libxl__logdirty_switch *lds,
int rc)
{
- STATE_AO_GC(dss->ao);
- libxl__logdirty_switch *lds = &dss->logdirty;
+ STATE_AO_GC(lds->ao);
libxl__ev_xswatch_deregister(gc, &lds->watch);
libxl__ev_time_deregister(gc, &lds->timeout);
- int broke;
- if (rc) {
- broke = -1;
- dss->rc = rc;
- } else {
- broke = 0;
- }
- libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, broke);
+ lds->callback(egc, lds, rc);
}
/*----- callbacks, called by xc_domain_save -----*/
@@ -346,6 +353,8 @@ void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
dss->rc = 0;
libxl__logdirty_init(&dss->logdirty);
+ dss->logdirty.ao = ao;
+
dsps->ao = ao;
dsps->domid = domid;
rc = libxl__domain_suspend_init(egc, dsps, type);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 5320e5e..53a6388 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3088,6 +3088,11 @@ libxl__stream_write_inuse(const libxl__stream_write_state *stream)
}
typedef struct libxl__logdirty_switch {
+ /* Set by caller of libxl__domain_common_switch_qemu_logdirty */
+ libxl__ao *ao;
+ void (*callback)(libxl__egc *egc, struct libxl__logdirty_switch *lds,
+ int rc);
+
const char *cmd;
const char *cmd_path;
const char *ret_path;
@@ -3506,6 +3511,9 @@ void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
_hidden void libxl__domain_suspend_common_switch_qemu_logdirty
(int domid, unsigned int enable, void *data);
+_hidden void libxl__domain_common_switch_qemu_logdirty(libxl__egc *egc,
+ int domid, unsigned enable,
+ libxl__logdirty_switch *lds);
_hidden int libxl__save_emulator_xenstore_data(libxl__domain_save_state *dss,
char **buf, uint32_t *len);
_hidden int libxl__restore_emulator_xenstore_data
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 03/31] tools/libxl: Add back channel to allow migration target send data back
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
2016-02-22 2:52 ` [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state Wen Congyang
2016-02-22 2:52 ` [PATCH v10 02/31] tools/libxl: introduce libxl__domain_common_switch_qemu_logdirty() Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 04/31] tools/libxl: Introduce new helper function dup_fd_helper() Wen Congyang
` (28 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
In COLO mode, secondary needs to send the following data to primary:
1. In libxl
Secondary sends the following CHECKPOINT_CONTEXT to primary:
CHECKPOINT_SVM_SUSPENDED, CHECKPOINT_SVM_READY and CHECKPOINT_SVM_RESUMED
2. In libxc
Secondary sends the dirty pfn list to primary
But the io_fd only can be written in primary, and only can be read in
secondary. Save recv_fd in domain_suspend_state, and send_fd in
domain_create_state. Extend libxl_domain_create_restore API, add a
send_fd param to it. Add LIBXL_HAVE_CREATE_RESTORE_SEND_FD to indicate
the API change.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Acked-by: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl.c | 3 +--
tools/libxl/libxl.h | 29 ++++++++++++++++++++++++++++-
tools/libxl/libxl_create.c | 10 ++++++----
tools/libxl/libxl_internal.h | 2 ++
tools/libxl/xl_cmdimpl.c | 8 +++++++-
tools/ocaml/libs/xl/xenlight_stubs.c | 2 +-
6 files changed, 45 insertions(+), 9 deletions(-)
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 4cdc169..e9ab78c 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -834,7 +834,6 @@ out:
static void remus_failover_cb(libxl__egc *egc,
libxl__domain_save_state *dss, int rc);
-/* TODO: Explicit Checkpoint acknowledgements via recv_fd. */
int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
uint32_t domid, int send_fd, int recv_fd,
const libxl_asyncop_how *ao_how)
@@ -871,7 +870,7 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
dss->callback = remus_failover_cb;
dss->domid = domid;
dss->fd = send_fd;
- /* TODO do something with recv_fd */
+ dss->recv_fd = recv_fd;
dss->type = type;
dss->live = 1;
dss->debug = 0;
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index f9e3ef5..5a286ec 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -639,6 +639,15 @@ typedef struct libxl__ctx libxl_ctx;
#define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_PARAMS 1
/*
+ * LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
+ *
+ * If this is defined, libxl_domain_create_restore()'s API has changed to
+ * include a send_back_fd param which used for libxl migration back channel
+ * during COLO.
+ */
+#define LIBXL_HAVE_DOMAIN_CREATE_RESTORE_SEND_BACK_FD 1
+
+/*
* LIBXL_HAVE_CREATEINFO_PVH
* If this is defined, then libxl supports creation of a PVH guest.
*/
@@ -1165,6 +1174,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
LIBXL_EXTERNAL_CALLERS_ONLY;
int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid, int restore_fd,
+ int send_back_fd,
const libxl_domain_restore_params *params,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
@@ -1185,7 +1195,7 @@ int static inline libxl_domain_create_restore_0x040200(
libxl_domain_restore_params_init(¶ms);
ret = libxl_domain_create_restore(
- ctx, d_config, domid, restore_fd, ¶ms, ao_how, aop_console_how);
+ ctx, d_config, domid, restore_fd, -1, ¶ms, ao_how, aop_console_how);
libxl_domain_restore_params_dispose(¶ms);
return ret;
@@ -1193,6 +1203,23 @@ int static inline libxl_domain_create_restore_0x040200(
#define libxl_domain_create_restore libxl_domain_create_restore_0x040200
+#elif defined(LIBXL_API_VERSION) && LIBXL_API_VERSION >= 0x040400 \
+ && LIBXL_API_VERSION < 0x040700
+
+int static inline libxl_domain_create_restore_0x040400(
+ libxl_ctx *ctx, libxl_domain_config *d_config,
+ uint32_t *domid, int restore_fd,
+ const libxl_domain_restore_params *params,
+ const libxl_asyncop_how *ao_how,
+ const libxl_asyncprogress_how *aop_console_how)
+ LIBXL_EXTERNAL_CALLERS_ONLY
+{
+ return libxl_domain_create_restore(ctx, d_config, domid, restore_fd,
+ -1, params, ao_how, aop_console_how);
+}
+
+#define libxl_domain_create_restore libxl_domain_create_restore_0x040400
+
#endif
int libxl_domain_soft_reset(libxl_ctx *ctx,
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index f1028bc..525bf85 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1572,7 +1572,7 @@ static void domain_create_cb(libxl__egc *egc,
int rc, uint32_t domid);
static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
- uint32_t *domid, int restore_fd,
+ uint32_t *domid, int restore_fd, int send_back_fd,
const libxl_domain_restore_params *params,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
@@ -1587,6 +1587,7 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
libxl_domain_config_init(&cdcs->dcs.guest_config_saved);
libxl_domain_config_copy(ctx, &cdcs->dcs.guest_config_saved, d_config);
cdcs->dcs.restore_fd = cdcs->dcs.libxc_fd = restore_fd;
+ cdcs->dcs.send_back_fd = send_back_fd;
if (restore_fd > -1) {
cdcs->dcs.restore_params = *params;
rc = libxl__fd_flags_modify_save(gc, cdcs->dcs.restore_fd,
@@ -1765,18 +1766,19 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
- return do_domain_create(ctx, d_config, domid, -1, NULL,
+ return do_domain_create(ctx, d_config, domid, -1, -1, NULL,
ao_how, aop_console_how);
}
int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid, int restore_fd,
+ int send_back_fd,
const libxl_domain_restore_params *params,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
- return do_domain_create(ctx, d_config, domid, restore_fd, params,
- ao_how, aop_console_how);
+ return do_domain_create(ctx, d_config, domid, restore_fd, send_back_fd,
+ params, ao_how, aop_console_how);
}
int libxl_domain_soft_reset(libxl_ctx *ctx,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 53a6388..3d3e8e8 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3134,6 +3134,7 @@ struct libxl__domain_save_state {
uint32_t domid;
int fd;
int fdfl; /* original flags on fd */
+ int recv_fd;
libxl_domain_type type;
int live;
int debug;
@@ -3467,6 +3468,7 @@ struct libxl__domain_create_state {
libxl_domain_config guest_config_saved; /* vanilla config */
int restore_fd, libxc_fd;
int restore_fdfl; /* original flags of restore_fd */
+ int send_back_fd;
libxl_domain_restore_params restore_params;
uint32_t domid_soft_reset;
libxl__domain_create_cb *callback;
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 09037ca..df7268b 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -159,6 +159,7 @@ struct domain_create {
char *extra_config; /* extra config string */
const char *restore_file;
int migrate_fd; /* -1 means none */
+ int send_back_fd; /* -1 means none */
char **migration_domname_r; /* from malloc */
};
@@ -2686,6 +2687,7 @@ static uint32_t create_domain(struct domain_create *dom_info)
int config_len = 0;
int restore_fd = -1;
int restore_fd_to_close = -1;
+ int send_back_fd = -1;
const libxl_asyncprogress_how *autoconnect_console_how;
struct save_file_header hdr;
uint32_t domid_soft_reset = INVALID_DOMID;
@@ -2703,6 +2705,7 @@ static uint32_t create_domain(struct domain_create *dom_info)
if (migrate_fd >= 0) {
restore_source = "<incoming migration stream>";
restore_fd = migrate_fd;
+ send_back_fd = dom_info->send_back_fd;
} else {
restore_source = restore_file;
restore_fd = open(restore_file, O_RDONLY);
@@ -2891,7 +2894,7 @@ start:
ret = libxl_domain_create_restore(ctx, &d_config,
&domid, restore_fd,
- ¶ms,
+ send_back_fd, ¶ms,
0, autoconnect_console_how);
libxl_domain_restore_params_dispose(¶ms);
@@ -4454,6 +4457,7 @@ static void migrate_receive(int debug, int daemonize, int monitor,
dom_info.monitor = monitor;
dom_info.paused = 1;
dom_info.migrate_fd = recv_fd;
+ dom_info.send_back_fd = send_fd;
dom_info.migration_domname_r = &migration_domname;
dom_info.checkpointed_stream = checkpointed;
@@ -4627,6 +4631,7 @@ int main_restore(int argc, char **argv)
dom_info.config_file = config_file;
dom_info.restore_file = checkpoint_file;
dom_info.migrate_fd = -1;
+ dom_info.send_back_fd = -1;
dom_info.vnc = vnc;
dom_info.vncautopass = vncautopass;
dom_info.console_autoconnect = console_autoconnect;
@@ -5094,6 +5099,7 @@ int main_create(int argc, char **argv)
dom_info.quiet = quiet;
dom_info.config_file = filename;
dom_info.migrate_fd = -1;
+ dom_info.send_back_fd = -1;
dom_info.vnc = vnc;
dom_info.vncautopass = vncautopass;
dom_info.console_autoconnect = console_autoconnect;
diff --git a/tools/ocaml/libs/xl/xenlight_stubs.c b/tools/ocaml/libs/xl/xenlight_stubs.c
index 4133527..98b52b9 100644
--- a/tools/ocaml/libs/xl/xenlight_stubs.c
+++ b/tools/ocaml/libs/xl/xenlight_stubs.c
@@ -538,7 +538,7 @@ value stub_libxl_domain_create_restore(value ctx, value domain_config, value par
caml_enter_blocking_section();
ret = libxl_domain_create_restore(CTX, &c_dconfig, &c_domid, restore_fd,
- &c_params, ao_how, NULL);
+ -1, &c_params, ao_how, NULL);
caml_leave_blocking_section();
free(ao_how);
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 04/31] tools/libxl: Introduce new helper function dup_fd_helper()
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (2 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 03/31] tools/libxl: Add back channel to allow migration target send data back Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:53 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 05/31] tools/libx{l, c}: add back channel to libxc Wen Congyang
` (27 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
It is pure refactoring and no functional changes.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxl/libxl_save_callout.c | 24 ++++++++++++++++--------
1 file changed, 16 insertions(+), 8 deletions(-)
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 7f1f5d4..a885504 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -118,6 +118,21 @@ void libxl__save_helper_init(libxl__save_helper_state *shs)
}
/*----- helper execution -----*/
+static int dup_fd_helper(libxl__gc *gc, int fd, const char *what)
+{
+ int dup_fd = fd;
+
+ if (fd <= 2) {
+ dup_fd = dup(fd);
+ if (dup_fd < 0) {
+ LOGE(ERROR,"dup %s", what);
+ exit(-1);
+ }
+ }
+ libxl_fd_set_cloexec(CTX, dup_fd, 0);
+
+ return dup_fd;
+}
/*
* Both save and restore share four parameters:
@@ -186,14 +201,7 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
pid_t pid = libxl__ev_child_fork(gc, &shs->child, helper_exited);
if (!pid) {
- if (stream_fd <= 2) {
- stream_fd = dup(stream_fd);
- if (stream_fd < 0) {
- LOGE(ERROR,"dup migration stream fd");
- exit(-1);
- }
- }
- libxl_fd_set_cloexec(CTX, stream_fd, 0);
+ stream_fd = dup_fd_helper(gc, stream_fd, "migration stream fd");
*stream_fd_arg = GCSPRINTF("%d", stream_fd);
for (i=0; i<num_preserve_fds; i++)
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 05/31] tools/libx{l, c}: add back channel to libxc
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (3 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 04/31] tools/libxl: Introduce new helper function dup_fd_helper() Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:54 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 06/31] docs: add colo readme Wen Congyang
` (26 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Wen Congyang,
Gui Jianfeng, Jiang Yunhong, Dong Eddie, Shriram Rajagopalan,
Ian Jackson, Yang Hongyang
In COLO mode, both VMs are running, and are considered in sync if the
visible network traffic is identical. After some time, they fall out of
sync.
At this point, the two VMs have definitely diverged. Lets call the
primary dirty bitmap set A, while the secondary dirty bitmap set B.
Sets A and B are different.
Under normal migration, the page data for set A will be sent from the
primary to the secondary.
However, the set difference B - A (the one in B but not in A, lets
call this C) is out-of-date on the secondary (with respect to the
primary) and will not be sent by the primary (to secondary), as it
was not memory dirtied by the primary. The secondary needs C page data
to reconstruct an exact copy of the primary at the checkpoint.
The secondary cannot calculate C as it doesn't know A. Instead, the
secondary must send B to the primary, at which point the primary
calculates the union of A and B (lets call this D) which is all the
pages dirtied by both the primary and the secondary, and sends all page
data covered by D.
In the general case, D is a superset of both A and B. Without the
backchannel dirty bitmap, a COLO checkpoint can't reconstruct a valid
copy of the primary.
We transfer the dirty bitmap on libxc side, so we need to introduce back
channel to libxc.
Note: it is different from the paper. We change the original design to
the current one, according to our following concerns:
1. The original design needs extra memory on Secondary host. When there's
multiple backups on one host, the memory cost is high.
2. The memory cache code will be another 1k+, it will make the review
more time consuming.
Note: the back channel will be used in the patch
libxc/restore: send dirty pfn list to primary when checkpoint under COLO
to send dirty pfn list from secondary to primary. The patch is posted in
another series.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
---
tools/libxc/include/xenguest.h | 4 ++--
tools/libxc/xc_nomigrate.c | 4 ++--
tools/libxc/xc_sr_restore.c | 2 +-
tools/libxc/xc_sr_save.c | 2 +-
tools/libxl/libxl_save_callout.c | 21 +++++++++++++++------
tools/libxl/libxl_save_helper.c | 8 ++++++--
6 files changed, 27 insertions(+), 14 deletions(-)
diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index affc42b..8040ac7 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -88,7 +88,7 @@ struct save_callbacks {
int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iters,
uint32_t max_factor, uint32_t flags /* XCFLAGS_xxx */,
struct save_callbacks* callbacks, int hvm,
- int checkpointed_stream);
+ int checkpointed_stream, int recv_fd);
/* callbacks provided by xc_domain_restore */
struct restore_callbacks {
@@ -127,7 +127,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
unsigned long *console_mfn, domid_t console_domid,
unsigned int hvm, unsigned int pae, int superpages,
int checkpointed_stream,
- struct restore_callbacks *callbacks);
+ struct restore_callbacks *callbacks, int send_back_fd);
/**
* This function will create a domain for a paravirtualized Linux
diff --git a/tools/libxc/xc_nomigrate.c b/tools/libxc/xc_nomigrate.c
index c9124df..3634743 100644
--- a/tools/libxc/xc_nomigrate.c
+++ b/tools/libxc/xc_nomigrate.c
@@ -23,7 +23,7 @@
int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iters,
uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
- int checkpointed_stream)
+ int checkpointed_stream, int recv_fd)
{
errno = ENOSYS;
return -1;
@@ -35,7 +35,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
unsigned long *console_mfn, domid_t console_domid,
unsigned int hvm, unsigned int pae, int superpages,
int checkpointed_stream,
- struct restore_callbacks *callbacks)
+ struct restore_callbacks *callbacks, int send_back_fd)
{
errno = ENOSYS;
return -1;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index d4d33fd..e24560e 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -726,7 +726,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
unsigned long *console_gfn, domid_t console_domid,
unsigned int hvm, unsigned int pae, int superpages,
int checkpointed_stream,
- struct restore_callbacks *callbacks)
+ struct restore_callbacks *callbacks, int send_back_fd)
{
struct xc_sr_context ctx =
{
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index e258b7c..fe210cc 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -830,7 +830,7 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type)
int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
uint32_t max_iters, uint32_t max_factor, uint32_t flags,
struct save_callbacks* callbacks, int hvm,
- int checkpointed_stream)
+ int checkpointed_stream, int recv_fd)
{
struct xc_sr_context ctx =
{
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index a885504..0d6949a 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -27,7 +27,7 @@
*/
static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
const char *mode_arg,
- int stream_fd,
+ int stream_fd, int back_channel_fd,
const int *preserve_fds, int num_preserve_fds,
const unsigned long *argnums, int num_argnums);
@@ -50,6 +50,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
/* Convenience aliases */
const uint32_t domid = dcs->guest_domid;
const int restore_fd = dcs->libxc_fd;
+ const int send_back_fd = dcs->send_back_fd;
libxl__domain_build_state *const state = &dcs->build_state;
unsigned cbflags =
@@ -71,7 +72,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
shs->caller_state = dcs;
shs->need_results = 1;
- run_helper(egc, shs, "--restore-domain", restore_fd, 0, 0,
+ run_helper(egc, shs, "--restore-domain", restore_fd, send_back_fd, 0, 0,
argnums, ARRAY_SIZE(argnums));
}
@@ -95,7 +96,7 @@ void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_save_state *dss,
shs->caller_state = dss;
shs->need_results = 0;
- run_helper(egc, shs, "--save-domain", dss->fd,
+ run_helper(egc, shs, "--save-domain", dss->fd, dss->recv_fd,
NULL, 0,
argnums, ARRAY_SIZE(argnums));
return;
@@ -139,12 +140,14 @@ static int dup_fd_helper(libxl__gc *gc, int fd, const char *what)
* 1) Path to libxl-save-helper.
* 2) --[restore|save]-domain.
* 3) stream file descriptor.
+ * 4) back channel file descriptor.
* n) save/restore specific parameters.
- * 4) A \0 at the end.
+ * 5) A \0 at the end.
*/
-#define HELPER_NR_ARGS 4
+#define HELPER_NR_ARGS 5
static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
- const char *mode_arg, int stream_fd,
+ const char *mode_arg,
+ int stream_fd, int back_channel_fd,
const int *preserve_fds, int num_preserve_fds,
const unsigned long *argnums, int num_argnums)
{
@@ -177,6 +180,7 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
*arg++ = getenv("LIBXL_SAVE_HELPER") ?: LIBEXEC_BIN "/" "libxl-save-helper";
*arg++ = mode_arg;
const char **stream_fd_arg = arg++;
+ const char **back_channel_fd_arg = arg++;
for (i=0; i<num_argnums; i++)
*arg++ = GCSPRINTF("%lu", argnums[i]);
*arg++ = 0;
@@ -204,6 +208,11 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
stream_fd = dup_fd_helper(gc, stream_fd, "migration stream fd");
*stream_fd_arg = GCSPRINTF("%d", stream_fd);
+ if (back_channel_fd >= 0)
+ back_channel_fd = dup_fd_helper(gc, back_channel_fd,
+ "migration back channel fd");
+ *back_channel_fd_arg = GCSPRINTF("%d", back_channel_fd);
+
for (i=0; i<num_preserve_fds; i++)
if (preserve_fds[i] >= 0) {
assert(preserve_fds[i] > 2);
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index 6bdcf13..2ff4c6e 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -238,6 +238,7 @@ static struct restore_callbacks helper_restore_callbacks;
int main(int argc, char **argv)
{
int r;
+ int send_back_fd, recv_fd;
#define NEXTARG (++argv, assert(*argv), *argv)
@@ -247,6 +248,7 @@ int main(int argc, char **argv)
if (!strcmp(mode,"--save-domain")) {
io_fd = atoi(NEXTARG);
+ recv_fd = atoi(NEXTARG);
uint32_t dom = strtoul(NEXTARG,0,10);
uint32_t max_iters = strtoul(NEXTARG,0,10);
uint32_t max_factor = strtoul(NEXTARG,0,10);
@@ -262,12 +264,14 @@ int main(int argc, char **argv)
setup_signals(save_signal_handler);
r = xc_domain_save(xch, io_fd, dom, max_iters, max_factor, flags,
- &helper_save_callbacks, hvm, checkpointed_stream);
+ &helper_save_callbacks, hvm, checkpointed_stream,
+ recv_fd);
complete(r);
} else if (!strcmp(mode,"--restore-domain")) {
io_fd = atoi(NEXTARG);
+ send_back_fd = atoi(NEXTARG);
uint32_t dom = strtoul(NEXTARG,0,10);
unsigned store_evtchn = strtoul(NEXTARG,0,10);
domid_t store_domid = strtoul(NEXTARG,0,10);
@@ -292,7 +296,7 @@ int main(int argc, char **argv)
store_domid, console_evtchn, &console_mfn,
console_domid, hvm, pae, superpages,
checkpointed,
- &helper_restore_callbacks);
+ &helper_restore_callbacks, send_back_fd);
helper_stub_restore_results(store_mfn,console_mfn,0);
complete(r);
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 06/31] docs: add colo readme
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (4 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 05/31] tools/libx{l, c}: add back channel to libxc Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:54 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 07/31] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams Wen Congyang
` (25 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
add colo readme, refer to
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
docs/README.colo | 9 +++++++++
1 file changed, 9 insertions(+)
create mode 100644 docs/README.colo
diff --git a/docs/README.colo b/docs/README.colo
new file mode 100644
index 0000000..466eb72
--- /dev/null
+++ b/docs/README.colo
@@ -0,0 +1,9 @@
+COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
+project is a high availability solution. Both primary VM (PVM) and secondary VM
+(SVM) run in parallel. They receive the same request from client, and generate
+response in parallel too. If the response packets from PVM and SVM are
+identical, they are released immediately. Otherwise, a VM checkpoint (on demand)
+is conducted.
+
+See the website at http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
+for details.
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 07/31] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (5 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 06/31] docs: add colo readme Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:54 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 08/31] libxc/migration: Specification update for DIRTY_PFN_LIST records Wen Congyang
` (24 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
It is the negotiation record for COLO.
Primary->Secondary:
control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
Secondary->Primary:
0x00000001: Secondary VM is suspended
0x00000002: Secondary VM is ready
0x00000003: Secondary VM is resumed
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
---
docs/specs/libxl-migration-stream.pandoc | 25 +++++++++++++++++++++++--
tools/libxl/libxl_sr_stream_format.h | 11 +++++++++++
tools/python/xen/migration/libxl.py | 9 +++++++++
3 files changed, 43 insertions(+), 2 deletions(-)
diff --git a/docs/specs/libxl-migration-stream.pandoc b/docs/specs/libxl-migration-stream.pandoc
index 2c97d86..5166d66 100644
--- a/docs/specs/libxl-migration-stream.pandoc
+++ b/docs/specs/libxl-migration-stream.pandoc
@@ -1,6 +1,6 @@
% LibXenLight Domain Image Format
% Andrew Cooper <<andrew.cooper3@citrix.com>>
-% Revision 1
+% Revision 2
Introduction
============
@@ -119,7 +119,9 @@ type 0x00000000: END
0x00000004: CHECKPOINT_END
- 0x00000005 - 0x7FFFFFFF: Reserved for future _mandatory_
+ 0x00000005: CHECKPOINT_STATE
+
+ 0x00000006 - 0x7FFFFFFF: Reserved for future _mandatory_
records.
0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -249,6 +251,25 @@ A checkpoint end record marks the end of a checkpoint in the image.
The end record contains no fields; its body_length is 0.
+CHECKPOINT\_STATE
+--------------
+
+A checkpoint state record contains the control information for checkpoint.
+
+ 0 1 2 3 4 5 6 7 octet
+ +------------------------+------------------------+
+ | control_id | padding |
+ +------------------------+------------------------+
+
+--------------------------------------------------------------------
+Field Description
+------------ ---------------------------------------------------
+control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
+ 0x00000001: Secondary VM is suspended
+ 0x00000002: Secondary VM is ready
+ 0x00000003: Secondary VM is resumed
+--------------------------------------------------------------------
+
Future Extensions
=================
diff --git a/tools/libxl/libxl_sr_stream_format.h b/tools/libxl/libxl_sr_stream_format.h
index 54da360..75f5190 100644
--- a/tools/libxl/libxl_sr_stream_format.h
+++ b/tools/libxl/libxl_sr_stream_format.h
@@ -36,6 +36,7 @@ typedef struct libxl__sr_rec_hdr
#define REC_TYPE_EMULATOR_XENSTORE_DATA 0x00000002U
#define REC_TYPE_EMULATOR_CONTEXT 0x00000003U
#define REC_TYPE_CHECKPOINT_END 0x00000004U
+#define REC_TYPE_CHECKPOINT_STATE 0x00000005U
typedef struct libxl__sr_emulator_hdr
{
@@ -47,6 +48,16 @@ typedef struct libxl__sr_emulator_hdr
#define EMULATOR_QEMU_TRADITIONAL 0x00000001U
#define EMULATOR_QEMU_UPSTREAM 0x00000002U
+typedef struct libxl_sr_checkpoint_state
+{
+ uint32_t id;
+} libxl_sr_checkpoint_state;
+
+#define CHECKPOINT_NEW 0x00000000U
+#define CHECKPOINT_SVM_SUSPENDED 0x00000001U
+#define CHECKPOINT_SVM_READY 0x00000002U
+#define CHECKPOINT_SVM_RESUMED 0x00000003U
+
#endif /* LIBXL__SR_STREAM_FORMAT_H */
/*
diff --git a/tools/python/xen/migration/libxl.py b/tools/python/xen/migration/libxl.py
index fc0acf6..d5f54dc 100644
--- a/tools/python/xen/migration/libxl.py
+++ b/tools/python/xen/migration/libxl.py
@@ -37,6 +37,7 @@ REC_TYPE_libxc_context = 0x00000001
REC_TYPE_emulator_xenstore_data = 0x00000002
REC_TYPE_emulator_context = 0x00000003
REC_TYPE_checkpoint_end = 0x00000004
+REC_TYPE_checkpoint_state = 0x00000005
rec_type_to_str = {
REC_TYPE_end : "End",
@@ -44,6 +45,7 @@ rec_type_to_str = {
REC_TYPE_emulator_xenstore_data : "Emulator xenstore data",
REC_TYPE_emulator_context : "Emulator context",
REC_TYPE_checkpoint_end : "Checkpoint end",
+ REC_TYPE_checkpoint_state : "Checkpoint state"
}
# emulator_* header
@@ -212,6 +214,11 @@ class VerifyLibxl(VerifyBase):
if len(content) != 0:
raise RecordError("Checkpoint end record with non-zero length")
+ def verify_record_checkpoint_state(self, content):
+ """ Checkpoint state """
+ if len(content) == 0:
+ raise RecordError("Checkpoint state record with zero length")
+
record_verifiers = {
REC_TYPE_end:
@@ -224,4 +231,6 @@ record_verifiers = {
VerifyLibxl.verify_record_emulator_context,
REC_TYPE_checkpoint_end:
VerifyLibxl.verify_record_checkpoint_end,
+ REC_TYPE_checkpoint_state:
+ VerifyLibxl.verify_record_checkpoint_state,
}
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 08/31] libxc/migration: Specification update for DIRTY_PFN_LIST records
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (6 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 07/31] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 09/31] libxc/migration: export read_record for common use Wen Congyang
` (23 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Used by secondary to send it's dirty bitmap to primary under COLO.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
docs/specs/libxc-migration-stream.pandoc | 24 ++++++++++-
tools/libxc/xc_sr_common.c | 31 ++++++++-------
tools/libxc/xc_sr_stream_format.h | 31 ++++++++-------
tools/python/xen/migration/libxc.py | 68 ++++++++++++++++++--------------
4 files changed, 93 insertions(+), 61 deletions(-)
diff --git a/docs/specs/libxc-migration-stream.pandoc b/docs/specs/libxc-migration-stream.pandoc
index 8cd678f..36b880d 100644
--- a/docs/specs/libxc-migration-stream.pandoc
+++ b/docs/specs/libxc-migration-stream.pandoc
@@ -227,7 +227,9 @@ type 0x00000000: END
0x0000000E: CHECKPOINT
- 0x0000000F - 0x7FFFFFFF: Reserved for future _mandatory_
+ 0x0000000F: CHECKPOINT_DIRTY_PFN_LIST
+
+ 0x00000010 - 0x7FFFFFFF: Reserved for future _mandatory_
records.
0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -599,6 +601,26 @@ CHECKPOINT record or an END record.
\clearpage
+CHECKPOINT_DIRTY_PFN_LIST
+------------
+
+A checkpoint dirty pfn list record is used to convey information about
+dirty memory in the VM. It is an unordered list of PFNs. Currently only
+applicable in the backchannel of a checkpointed stream.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+ | pfn[0] |
+ +-------------------------------------------------+
+ ...
+ +-------------------------------------------------+
+ | pfn[C-1] |
+ +-------------------------------------------------+
+
+The count of pfns is: record->length/sizeof(uint64_t).
+
+\clearpage
+
Layout
======
diff --git a/tools/libxc/xc_sr_common.c b/tools/libxc/xc_sr_common.c
index 945cfa6..3313a90 100644
--- a/tools/libxc/xc_sr_common.c
+++ b/tools/libxc/xc_sr_common.c
@@ -20,21 +20,22 @@ const char *dhdr_type_to_str(uint32_t type)
static const char *mandatory_rec_types[] =
{
- [REC_TYPE_END] = "End",
- [REC_TYPE_PAGE_DATA] = "Page data",
- [REC_TYPE_X86_PV_INFO] = "x86 PV info",
- [REC_TYPE_X86_PV_P2M_FRAMES] = "x86 PV P2M frames",
- [REC_TYPE_X86_PV_VCPU_BASIC] = "x86 PV vcpu basic",
- [REC_TYPE_X86_PV_VCPU_EXTENDED] = "x86 PV vcpu extended",
- [REC_TYPE_X86_PV_VCPU_XSAVE] = "x86 PV vcpu xsave",
- [REC_TYPE_SHARED_INFO] = "Shared info",
- [REC_TYPE_TSC_INFO] = "TSC info",
- [REC_TYPE_HVM_CONTEXT] = "HVM context",
- [REC_TYPE_HVM_PARAMS] = "HVM params",
- [REC_TYPE_TOOLSTACK] = "Toolstack",
- [REC_TYPE_X86_PV_VCPU_MSRS] = "x86 PV vcpu msrs",
- [REC_TYPE_VERIFY] = "Verify",
- [REC_TYPE_CHECKPOINT] = "Checkpoint",
+ [REC_TYPE_END] = "End",
+ [REC_TYPE_PAGE_DATA] = "Page data",
+ [REC_TYPE_X86_PV_INFO] = "x86 PV info",
+ [REC_TYPE_X86_PV_P2M_FRAMES] = "x86 PV P2M frames",
+ [REC_TYPE_X86_PV_VCPU_BASIC] = "x86 PV vcpu basic",
+ [REC_TYPE_X86_PV_VCPU_EXTENDED] = "x86 PV vcpu extended",
+ [REC_TYPE_X86_PV_VCPU_XSAVE] = "x86 PV vcpu xsave",
+ [REC_TYPE_SHARED_INFO] = "Shared info",
+ [REC_TYPE_TSC_INFO] = "TSC info",
+ [REC_TYPE_HVM_CONTEXT] = "HVM context",
+ [REC_TYPE_HVM_PARAMS] = "HVM params",
+ [REC_TYPE_TOOLSTACK] = "Toolstack",
+ [REC_TYPE_X86_PV_VCPU_MSRS] = "x86 PV vcpu msrs",
+ [REC_TYPE_VERIFY] = "Verify",
+ [REC_TYPE_CHECKPOINT] = "Checkpoint",
+ [REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST] = "Checkpoint dirty pfn list",
};
const char *rec_type_to_str(uint32_t type)
diff --git a/tools/libxc/xc_sr_stream_format.h b/tools/libxc/xc_sr_stream_format.h
index 6d0f8fd..3291b25 100644
--- a/tools/libxc/xc_sr_stream_format.h
+++ b/tools/libxc/xc_sr_stream_format.h
@@ -60,21 +60,22 @@ struct xc_sr_rhdr
/* Somewhat arbitrary - 8MB */
#define REC_LENGTH_MAX (8U << 20)
-#define REC_TYPE_END 0x00000000U
-#define REC_TYPE_PAGE_DATA 0x00000001U
-#define REC_TYPE_X86_PV_INFO 0x00000002U
-#define REC_TYPE_X86_PV_P2M_FRAMES 0x00000003U
-#define REC_TYPE_X86_PV_VCPU_BASIC 0x00000004U
-#define REC_TYPE_X86_PV_VCPU_EXTENDED 0x00000005U
-#define REC_TYPE_X86_PV_VCPU_XSAVE 0x00000006U
-#define REC_TYPE_SHARED_INFO 0x00000007U
-#define REC_TYPE_TSC_INFO 0x00000008U
-#define REC_TYPE_HVM_CONTEXT 0x00000009U
-#define REC_TYPE_HVM_PARAMS 0x0000000aU
-#define REC_TYPE_TOOLSTACK 0x0000000bU
-#define REC_TYPE_X86_PV_VCPU_MSRS 0x0000000cU
-#define REC_TYPE_VERIFY 0x0000000dU
-#define REC_TYPE_CHECKPOINT 0x0000000eU
+#define REC_TYPE_END 0x00000000U
+#define REC_TYPE_PAGE_DATA 0x00000001U
+#define REC_TYPE_X86_PV_INFO 0x00000002U
+#define REC_TYPE_X86_PV_P2M_FRAMES 0x00000003U
+#define REC_TYPE_X86_PV_VCPU_BASIC 0x00000004U
+#define REC_TYPE_X86_PV_VCPU_EXTENDED 0x00000005U
+#define REC_TYPE_X86_PV_VCPU_XSAVE 0x00000006U
+#define REC_TYPE_SHARED_INFO 0x00000007U
+#define REC_TYPE_TSC_INFO 0x00000008U
+#define REC_TYPE_HVM_CONTEXT 0x00000009U
+#define REC_TYPE_HVM_PARAMS 0x0000000aU
+#define REC_TYPE_TOOLSTACK 0x0000000bU
+#define REC_TYPE_X86_PV_VCPU_MSRS 0x0000000cU
+#define REC_TYPE_VERIFY 0x0000000dU
+#define REC_TYPE_CHECKPOINT 0x0000000eU
+#define REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST 0x0000000fU
#define REC_TYPE_OPTIONAL 0x80000000U
diff --git a/tools/python/xen/migration/libxc.py b/tools/python/xen/migration/libxc.py
index b0255ac..85a78f4 100644
--- a/tools/python/xen/migration/libxc.py
+++ b/tools/python/xen/migration/libxc.py
@@ -45,38 +45,40 @@ dhdr_type_to_str = {
# Records
RH_FORMAT = "II"
-REC_TYPE_end = 0x00000000
-REC_TYPE_page_data = 0x00000001
-REC_TYPE_x86_pv_info = 0x00000002
-REC_TYPE_x86_pv_p2m_frames = 0x00000003
-REC_TYPE_x86_pv_vcpu_basic = 0x00000004
-REC_TYPE_x86_pv_vcpu_extended = 0x00000005
-REC_TYPE_x86_pv_vcpu_xsave = 0x00000006
-REC_TYPE_shared_info = 0x00000007
-REC_TYPE_tsc_info = 0x00000008
-REC_TYPE_hvm_context = 0x00000009
-REC_TYPE_hvm_params = 0x0000000a
-REC_TYPE_toolstack = 0x0000000b
-REC_TYPE_x86_pv_vcpu_msrs = 0x0000000c
-REC_TYPE_verify = 0x0000000d
-REC_TYPE_checkpoint = 0x0000000e
+REC_TYPE_end = 0x00000000
+REC_TYPE_page_data = 0x00000001
+REC_TYPE_x86_pv_info = 0x00000002
+REC_TYPE_x86_pv_p2m_frames = 0x00000003
+REC_TYPE_x86_pv_vcpu_basic = 0x00000004
+REC_TYPE_x86_pv_vcpu_extended = 0x00000005
+REC_TYPE_x86_pv_vcpu_xsave = 0x00000006
+REC_TYPE_shared_info = 0x00000007
+REC_TYPE_tsc_info = 0x00000008
+REC_TYPE_hvm_context = 0x00000009
+REC_TYPE_hvm_params = 0x0000000a
+REC_TYPE_toolstack = 0x0000000b
+REC_TYPE_x86_pv_vcpu_msrs = 0x0000000c
+REC_TYPE_verify = 0x0000000d
+REC_TYPE_checkpoint = 0x0000000e
+REC_TYPE_checkpoint_dirty_pfn_list = 0x0000000f
rec_type_to_str = {
- REC_TYPE_end : "End",
- REC_TYPE_page_data : "Page data",
- REC_TYPE_x86_pv_info : "x86 PV info",
- REC_TYPE_x86_pv_p2m_frames : "x86 PV P2M frames",
- REC_TYPE_x86_pv_vcpu_basic : "x86 PV vcpu basic",
- REC_TYPE_x86_pv_vcpu_extended : "x86 PV vcpu extended",
- REC_TYPE_x86_pv_vcpu_xsave : "x86 PV vcpu xsave",
- REC_TYPE_shared_info : "Shared info",
- REC_TYPE_tsc_info : "TSC info",
- REC_TYPE_hvm_context : "HVM context",
- REC_TYPE_hvm_params : "HVM params",
- REC_TYPE_toolstack : "Toolstack",
- REC_TYPE_x86_pv_vcpu_msrs : "x86 PV vcpu msrs",
- REC_TYPE_verify : "Verify",
- REC_TYPE_checkpoint : "Checkpoint",
+ REC_TYPE_end : "End",
+ REC_TYPE_page_data : "Page data",
+ REC_TYPE_x86_pv_info : "x86 PV info",
+ REC_TYPE_x86_pv_p2m_frames : "x86 PV P2M frames",
+ REC_TYPE_x86_pv_vcpu_basic : "x86 PV vcpu basic",
+ REC_TYPE_x86_pv_vcpu_extended : "x86 PV vcpu extended",
+ REC_TYPE_x86_pv_vcpu_xsave : "x86 PV vcpu xsave",
+ REC_TYPE_shared_info : "Shared info",
+ REC_TYPE_tsc_info : "TSC info",
+ REC_TYPE_hvm_context : "HVM context",
+ REC_TYPE_hvm_params : "HVM params",
+ REC_TYPE_toolstack : "Toolstack",
+ REC_TYPE_x86_pv_vcpu_msrs : "x86 PV vcpu msrs",
+ REC_TYPE_verify : "Verify",
+ REC_TYPE_checkpoint : "Checkpoint",
+ REC_TYPE_checkpoint_dirty_pfn_list : "Checkpoint dirty pfn list"
}
# page_data
@@ -403,6 +405,10 @@ class VerifyLibxc(VerifyBase):
if len(content) != 0:
raise RecordError("Checkpoint record with non-zero length")
+ def verify_record_checkpoint_dirty_pfn_list(self, content):
+ """ checkpoint dirty pfn list """
+ raise RecordError("Found checkpoint dirty pfn list record in stream")
+
record_verifiers = {
REC_TYPE_end:
@@ -443,4 +449,6 @@ record_verifiers = {
VerifyLibxc.verify_record_verify,
REC_TYPE_checkpoint:
VerifyLibxc.verify_record_checkpoint,
+ REC_TYPE_checkpoint_dirty_pfn_list:
+ VerifyLibxc.verify_record_checkpoint_dirty_pfn_list,
}
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 09/31] libxc/migration: export read_record for common use
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (7 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 08/31] libxc/migration: Specification update for DIRTY_PFN_LIST records Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 10/31] tools/libxl: add back channel support to write stream Wen Congyang
` (22 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
read_record() could be used by primary to read dirty bitmap
record sent by secondary under COLO.
When used by save side, we need to pass the backchannel fd
instead of ctx->fd to read_record(), so we added a fd param to
it.
No functional changes.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxc/xc_sr_common.c | 49 +++++++++++++++++++++++++++++++++++
tools/libxc/xc_sr_common.h | 14 ++++++++++
tools/libxc/xc_sr_restore.c | 63 +--------------------------------------------
3 files changed, 64 insertions(+), 62 deletions(-)
diff --git a/tools/libxc/xc_sr_common.c b/tools/libxc/xc_sr_common.c
index 3313a90..b228a15 100644
--- a/tools/libxc/xc_sr_common.c
+++ b/tools/libxc/xc_sr_common.c
@@ -89,6 +89,55 @@ int write_split_record(struct xc_sr_context *ctx, struct xc_sr_record *rec,
return -1;
}
+int read_record(struct xc_sr_context *ctx, int fd, struct xc_sr_record *rec)
+{
+ xc_interface *xch = ctx->xch;
+ struct xc_sr_rhdr rhdr;
+ size_t datasz;
+
+ if ( read_exact(fd, &rhdr, sizeof(rhdr)) )
+ {
+ PERROR("Failed to read Record Header from stream");
+ return -1;
+ }
+ else if ( rhdr.length > REC_LENGTH_MAX )
+ {
+ ERROR("Record (0x%08x, %s) length %#x exceeds max (%#x)", rhdr.type,
+ rec_type_to_str(rhdr.type), rhdr.length, REC_LENGTH_MAX);
+ return -1;
+ }
+
+ datasz = ROUNDUP(rhdr.length, REC_ALIGN_ORDER);
+
+ if ( datasz )
+ {
+ rec->data = malloc(datasz);
+
+ if ( !rec->data )
+ {
+ ERROR("Unable to allocate %zu bytes for record data (0x%08x, %s)",
+ datasz, rhdr.type, rec_type_to_str(rhdr.type));
+ return -1;
+ }
+
+ if ( read_exact(fd, rec->data, datasz) )
+ {
+ free(rec->data);
+ rec->data = NULL;
+ PERROR("Failed to read %zu bytes of data for record (0x%08x, %s)",
+ datasz, rhdr.type, rec_type_to_str(rhdr.type));
+ return -1;
+ }
+ }
+ else
+ rec->data = NULL;
+
+ rec->type = rhdr.type;
+ rec->length = rhdr.length;
+
+ return 0;
+};
+
static void __attribute__((unused)) build_assertions(void)
{
XC_BUILD_BUG_ON(sizeof(struct xc_sr_ihdr) != 24);
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 66f595f..5d9f497 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -382,6 +382,20 @@ static inline int write_record(struct xc_sr_context *ctx,
}
/*
+ * Reads a record from the stream, and fills in the record structure.
+ *
+ * Returns 0 on success and non-0 on failure.
+ *
+ * On success, the records type and size shall be valid.
+ * - If size is 0, data shall be NULL.
+ * - If size is non-0, data shall be a buffer allocated by malloc() which must
+ * be passed to free() by the caller.
+ *
+ * On failure, the contents of the record structure are undefined.
+ */
+int read_record(struct xc_sr_context *ctx, int fd, struct xc_sr_record *rec);
+
+/*
* This would ideally be private in restore.c, but is needed by
* x86_pv_localise_page() if we receive pagetables frames ahead of the
* contents of the frames they point at.
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index e24560e..aef9bca 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -69,67 +69,6 @@ static int read_headers(struct xc_sr_context *ctx)
}
/*
- * Reads a record from the stream, and fills in the record structure.
- *
- * Returns 0 on success and non-0 on failure.
- *
- * On success, the records type and size shall be valid.
- * - If size is 0, data shall be NULL.
- * - If size is non-0, data shall be a buffer allocated by malloc() which must
- * be passed to free() by the caller.
- *
- * On failure, the contents of the record structure are undefined.
- */
-static int read_record(struct xc_sr_context *ctx, struct xc_sr_record *rec)
-{
- xc_interface *xch = ctx->xch;
- struct xc_sr_rhdr rhdr;
- size_t datasz;
-
- if ( read_exact(ctx->fd, &rhdr, sizeof(rhdr)) )
- {
- PERROR("Failed to read Record Header from stream");
- return -1;
- }
- else if ( rhdr.length > REC_LENGTH_MAX )
- {
- ERROR("Record (0x%08x, %s) length %#x exceeds max (%#x)", rhdr.type,
- rec_type_to_str(rhdr.type), rhdr.length, REC_LENGTH_MAX);
- return -1;
- }
-
- datasz = ROUNDUP(rhdr.length, REC_ALIGN_ORDER);
-
- if ( datasz )
- {
- rec->data = malloc(datasz);
-
- if ( !rec->data )
- {
- ERROR("Unable to allocate %zu bytes for record data (0x%08x, %s)",
- datasz, rhdr.type, rec_type_to_str(rhdr.type));
- return -1;
- }
-
- if ( read_exact(ctx->fd, rec->data, datasz) )
- {
- free(rec->data);
- rec->data = NULL;
- PERROR("Failed to read %zu bytes of data for record (0x%08x, %s)",
- datasz, rhdr.type, rec_type_to_str(rhdr.type));
- return -1;
- }
- }
- else
- rec->data = NULL;
-
- rec->type = rhdr.type;
- rec->length = rhdr.length;
-
- return 0;
-};
-
-/*
* Is a pfn populated?
*/
static bool pfn_is_populated(const struct xc_sr_context *ctx, xen_pfn_t pfn)
@@ -650,7 +589,7 @@ static int restore(struct xc_sr_context *ctx)
do
{
- rc = read_record(ctx, &rec);
+ rc = read_record(ctx, ctx->fd, &rec);
if ( rc )
{
if ( ctx->restore.buffer_all_records )
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 10/31] tools/libxl: add back channel support to write stream
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (8 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 09/31] libxc/migration: export read_record for common use Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:54 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 11/31] tools/libxl: write checkpoint_state records into the stream Wen Congyang
` (21 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Add back channel support to write stream. If the write stream is
a back channel stream, this means the write stream is used by
Secondary to send some records back.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxl/libxl_dom_save.c | 1 +
tools/libxl/libxl_internal.h | 1 +
tools/libxl/libxl_stream_write.c | 26 ++++++++++++++++++++------
3 files changed, 22 insertions(+), 6 deletions(-)
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index 72b61c7..18946c5 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -404,6 +404,7 @@ void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
dss->sws.ao = dss->ao;
dss->sws.dss = dss;
dss->sws.fd = dss->fd;
+ dss->sws.back_channel = false;
dss->sws.completion_callback = stream_done;
libxl__stream_write_start(egc, &dss->sws);
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 3d3e8e8..e02e554 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3044,6 +3044,7 @@ struct libxl__stream_write_state {
libxl__ao *ao;
libxl__domain_save_state *dss;
int fd;
+ bool back_channel;
void (*completion_callback)(libxl__egc *egc,
libxl__stream_write_state *sws,
int rc);
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
index f6ea55d..5379126 100644
--- a/tools/libxl/libxl_stream_write.c
+++ b/tools/libxl/libxl_stream_write.c
@@ -49,6 +49,13 @@
* - if (hvm)
* - Emulator context record
* - Checkpoint end record
+ *
+ * For back channel stream:
+ * - libxl__stream_write_start()
+ * - Set up the stream to running state
+ *
+ * - Add a new API to write the record. When the record is written
+ * out, call stream->checkpoint_callback() to return.
*/
/* Success/error/cleanup handling. */
@@ -225,6 +232,15 @@ void libxl__stream_write_start(libxl__egc *egc,
stream->running = true;
+ dc->ao = ao;
+ dc->readfd = -1;
+ dc->copywhat = "save v2 stream";
+ dc->writefd = stream->fd;
+ dc->maxsz = -1;
+
+ if (stream->back_channel)
+ return;
+
if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
stream->device_model_version =
libxl__device_model_version_running(gc, dss->domid);
@@ -249,12 +265,7 @@ void libxl__stream_write_start(libxl__egc *egc,
stream->emu_sub_hdr.index = 0;
}
- dc->ao = ao;
- dc->readfd = -1;
dc->writewhat = "stream header";
- dc->copywhat = "save v2 stream";
- dc->writefd = stream->fd;
- dc->maxsz = -1;
dc->callback = stream_header_done;
rc = libxl__datacopier_start(dc);
@@ -279,6 +290,7 @@ void libxl__stream_write_start_checkpoint(libxl__egc *egc,
{
assert(stream->running);
assert(!stream->in_checkpoint);
+ assert(!stream->back_channel);
stream->in_checkpoint = true;
write_emulator_xenstore_record(egc, stream);
@@ -590,7 +602,9 @@ static void stream_done(libxl__egc *egc,
libxl__carefd_close(stream->emu_carefd);
free(stream->emu_body);
- check_all_finished(egc, stream, rc);
+ if (!stream->back_channel)
+ /* back channel stream doesn't have save helper */
+ check_all_finished(egc, stream, rc);
}
static void checkpoint_done(libxl__egc *egc,
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 11/31] tools/libxl: write checkpoint_state records into the stream
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (9 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 10/31] tools/libxl: add back channel support to write stream Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 12/31] tools/libxl: add back channel support to read stream Wen Congyang
` (20 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
write checkpoint_state records into the stream, used by both
primary and secondary to send colo context.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxl/libxl_internal.h | 5 ++++
tools/libxl/libxl_stream_write.c | 60 +++++++++++++++++++++++++++++++++++++++-
2 files changed, 64 insertions(+), 1 deletion(-)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index e02e554..cdf6751 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3056,6 +3056,7 @@ struct libxl__stream_write_state {
bool running;
bool in_checkpoint;
bool sync_teardown; /* Only used to coordinate shutdown on error path. */
+ bool in_checkpoint_state;
libxl__save_helper_state shs;
/* Main stream-writing data. */
@@ -3079,6 +3080,10 @@ _hidden void libxl__stream_write_start(libxl__egc *egc,
_hidden void
libxl__stream_write_start_checkpoint(libxl__egc *egc,
libxl__stream_write_state *stream);
+_hidden void
+libxl__stream_write_checkpoint_state(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ libxl_sr_checkpoint_state *srcs);
_hidden void libxl__stream_write_abort(libxl__egc *egc,
libxl__stream_write_state *stream,
int rc);
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
index 5379126..8e0e57e 100644
--- a/tools/libxl/libxl_stream_write.c
+++ b/tools/libxl/libxl_stream_write.c
@@ -98,6 +98,12 @@ static void write_checkpoint_end_record(libxl__egc *egc,
static void checkpoint_end_record_done(libxl__egc *egc,
libxl__stream_write_state *stream);
+/* checkpoint state */
+static void write_checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_write_state *stream, int rc);
+
/*----- Helpers -----*/
static void write_done(libxl__egc *egc,
@@ -589,6 +595,21 @@ static void stream_complete(libxl__egc *egc,
return;
}
+ if (stream->in_checkpoint_state) {
+ assert(rc);
+
+ /*
+ * If an error is encountered while in a checkpoint, pass it
+ * back to libxc. The failure will come back around to us via
+ * 1. normal stream
+ * libxl__xc_domain_save_done()
+ * 2. back_channel stream
+ * libxl__stream_write_abort()
+ */
+ checkpoint_state_done(egc, stream, rc);
+ return;
+ }
+
stream_done(egc, stream, rc);
}
@@ -596,6 +617,7 @@ static void stream_done(libxl__egc *egc,
libxl__stream_write_state *stream, int rc)
{
assert(stream->running);
+ assert(!stream->in_checkpoint_state);
stream->running = false;
if (stream->emu_carefd)
@@ -656,7 +678,43 @@ static void check_all_finished(libxl__egc *egc,
libxl__save_helper_inuse(&stream->shs))
return;
- stream->completion_callback(egc, stream, stream->rc);
+ if (stream->completion_callback)
+ /* back channel stream doesn't have completion_callback() */
+ stream->completion_callback(egc, stream, stream->rc);
+}
+
+/*----- checkpoint state -----*/
+void libxl__stream_write_checkpoint_state(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ libxl_sr_checkpoint_state *srcs)
+{
+ struct libxl__sr_rec_hdr rec;
+
+ assert(stream->running);
+ assert(!stream->in_checkpoint);
+ assert(!stream->in_checkpoint_state);
+ stream->in_checkpoint_state = true;
+
+ FILLZERO(rec);
+ rec.type = REC_TYPE_CHECKPOINT_STATE;
+ rec.length = sizeof(*srcs);
+
+ setup_write(egc, stream, "checkpoint state", &rec,
+ srcs, write_checkpoint_state_done);
+}
+
+static void write_checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ checkpoint_state_done(egc, stream, 0);
+}
+
+static void checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_write_state *stream, int rc)
+{
+ assert(stream->in_checkpoint_state);
+ stream->in_checkpoint_state = false;
+ stream->checkpoint_callback(egc, stream, rc);
}
/*
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 12/31] tools/libxl: add back channel support to read stream
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (10 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 11/31] tools/libxl: write checkpoint_state records into the stream Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:54 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 13/31] tools/libxl: handle checkpoint_state records in a libxl migration v2 " Wen Congyang
` (19 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
This is used by primay to read records sent by secondary.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxl/libxl_create.c | 1 +
tools/libxl/libxl_internal.h | 1 +
tools/libxl/libxl_stream_read.c | 27 +++++++++++++++++++++++----
3 files changed, 25 insertions(+), 4 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 525bf85..fc746fb 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1030,6 +1030,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
dcs->srs.dcs = dcs;
dcs->srs.fd = restore_fd;
dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
+ dcs->srs.back_channel = false;
dcs->srs.completion_callback = domcreate_stream_done;
if (restore_fd >= 0) {
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index cdf6751..32957ca 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3420,6 +3420,7 @@ struct libxl__stream_read_state {
libxl__domain_create_state *dcs;
int fd;
bool legacy;
+ bool back_channel;
void (*completion_callback)(libxl__egc *egc,
libxl__stream_read_state *srs,
int rc);
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index f4781eb..02a2c46 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -118,6 +118,15 @@
* record, and therefore the buffered state is inconsistent. In
* libxl__xc_domain_restore_done(), we just complete the stream and
* stream->completion_callback() will be called to resume the guest
+ *
+ * For back channel stream:
+ * - libxl__stream_read_start()
+ * - Set up the stream to running state
+ *
+ * - libxl__stream_read_continue()
+ * - Set up reading the next record from a started stream.
+ * Add some codes to process_record() to handle the record.
+ * Then call stream->checkpoint_callback() to return.
*/
/* Success/error/cleanup handling. */
@@ -221,6 +230,17 @@ void libxl__stream_read_start(libxl__egc *egc,
stream->running = true;
stream->phase = SRS_PHASE_NORMAL;
+ dc->ao = stream->ao;
+ dc->copywhat = "restore v2 stream";
+ dc->writefd = -1;
+
+ if (stream->back_channel) {
+ assert(!stream->legacy);
+
+ dc->readfd = stream->fd;
+ return;
+ }
+
if (stream->legacy) {
/* Convert the legacy stream. */
libxl__conversion_helper_state *chs = &stream->chs;
@@ -243,10 +263,7 @@ void libxl__stream_read_start(libxl__egc *egc,
}
/* stream->fd is now a v2 stream. */
- dc->ao = stream->ao;
- dc->copywhat = "restore v2 stream";
dc->readfd = stream->fd;
- dc->writefd = -1;
/* Start reading the stream header. */
rc = setup_read(stream, "stream header",
@@ -762,7 +779,9 @@ static void stream_done(libxl__egc *egc,
LIBXL_STAILQ_FOREACH_SAFE(rec, &stream->record_queue, entry, trec)
free_record(rec);
- check_all_finished(egc, stream, rc);
+ if (!stream->back_channel)
+ /* back channel stream doesn't have restore helper */
+ check_all_finished(egc, stream, rc);
}
void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 13/31] tools/libxl: handle checkpoint_state records in a libxl migration v2 read stream
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (11 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 12/31] tools/libxl: add back channel support to read stream Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 14/31] tools/libx{l, c}: introduce wait_checkpoint callback Wen Congyang
` (18 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Read a checkpoint_state and call stream->checkpoint_callback to handle it.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxl/libxl_internal.h | 3 +++
tools/libxl/libxl_stream_read.c | 57 ++++++++++++++++++++++++++++++++++++++++-
2 files changed, 59 insertions(+), 1 deletion(-)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 32957ca..33b658d 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3432,6 +3432,7 @@ struct libxl__stream_read_state {
bool running;
bool in_checkpoint;
bool sync_teardown; /* Only used to coordinate shutdown on error path. */
+ bool in_checkpoint_state;
libxl__save_helper_state shs;
libxl__conversion_helper_state chs;
@@ -3459,6 +3460,8 @@ _hidden void libxl__stream_read_start(libxl__egc *egc,
libxl__stream_read_state *stream);
_hidden void libxl__stream_read_start_checkpoint(libxl__egc *egc,
libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_checkpoint_state(libxl__egc *egc,
+ libxl__stream_read_state *stream);
_hidden void libxl__stream_read_abort(libxl__egc *egc,
libxl__stream_read_state *stream, int rc);
static inline bool
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index 02a2c46..5d980d9 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -166,6 +166,10 @@ static void write_emulator_done(libxl__egc *egc,
libxl__datacopier_state *dc,
int rc, int onwrite, int errnoval);
+/* Handlers for checkpoint state mini-loop */
+static void checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
+
/*----- Helpers -----*/
/* Helper to set up reading some data from the stream. */
@@ -549,6 +553,7 @@ static bool process_record(libxl__egc *egc,
STATE_AO_GC(stream->ao);
libxl__domain_create_state *dcs = stream->dcs;
libxl__sr_record_buf *rec;
+ libxl_sr_checkpoint_state *srcs;
bool further_action_needed = false;
int rc = 0;
@@ -619,6 +624,17 @@ static bool process_record(libxl__egc *egc,
checkpoint_done(egc, stream, 0);
break;
+ case REC_TYPE_CHECKPOINT_STATE:
+ if (!stream->in_checkpoint_state) {
+ LOG(ERROR, "Unexpected CHECKPOINT_STATE record in stream");
+ rc = ERROR_FAIL;
+ goto err;
+ }
+
+ srcs = rec->body;
+ checkpoint_state_done(egc, stream, srcs->id);
+ break;
+
default:
LOG(ERROR, "Unrecognised record 0x%08x", rec->hdr.type);
rc = ERROR_FAIL;
@@ -730,6 +746,21 @@ static void stream_complete(libxl__egc *egc,
return;
}
+ if (stream->in_checkpoint_state) {
+ assert(rc);
+
+ /*
+ * If an error is encountered while in a checkpoint, pass it
+ * back to libxc. The failure will come back around to us via
+ * 1. normal stream
+ * libxl__xc_domain_restore_done()
+ * 2. back_channel stream
+ * libxl__stream_read_abort()
+ */
+ checkpoint_state_done(egc, stream, rc);
+ return;
+ }
+
stream_done(egc, stream, rc);
}
@@ -760,6 +791,7 @@ static void stream_done(libxl__egc *egc,
assert(stream->running);
assert(!stream->in_checkpoint);
+ assert(!stream->in_checkpoint_state);
stream->running = false;
if (stream->incoming_record)
@@ -881,7 +913,30 @@ static void check_all_finished(libxl__egc *egc,
libxl__conversion_helper_inuse(&stream->chs))
return;
- stream->completion_callback(egc, stream, stream->rc);
+ if (stream->completion_callback)
+ /* back channel stream doesn't have completion_callback() */
+ stream->completion_callback(egc, stream, stream->rc);
+}
+
+/*----- Checkpoint state handlers -----*/
+
+void libxl__stream_read_checkpoint_state(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ assert(stream->running);
+ assert(!stream->in_checkpoint);
+ assert(!stream->in_checkpoint_state);
+ stream->in_checkpoint_state = true;
+
+ setup_read_record(egc, stream);
+}
+
+static void checkpoint_state_done(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc)
+{
+ assert(stream->in_checkpoint_state);
+ stream->in_checkpoint_state = false;
+ stream->checkpoint_callback(egc, stream, rc);
}
/*
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 14/31] tools/libx{l, c}: introduce wait_checkpoint callback
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (12 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 13/31] tools/libxl: handle checkpoint_state records in a libxl migration v2 " Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 15/31] tools/libx{l, c}: add postcopy/suspend callback to restore side Wen Congyang
` (17 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Under COLO, we are doing checkpoint on demand, if this
callback returns 1, we will take another checkpoint.
0 indicates unexpected error.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxc/include/xenguest.h | 18 ++++++++++++++++++
tools/libxl/libxl_save_msgs_gen.pl | 7 ++++---
2 files changed, 22 insertions(+), 3 deletions(-)
diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index 8040ac7..f598bec 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -68,6 +68,15 @@ struct save_callbacks {
* 1: take another checkpoint */
int (*checkpoint)(void* data);
+ /*
+ * Called after the checkpoint callback.
+ *
+ * returns:
+ * 0: terminate checkpointing gracefully
+ * 1: take another checkpoint
+ */
+ int (*wait_checkpoint)(void* data);
+
/* Enable qemu-dm logging dirty pages to xen */
int (*switch_qemu_logdirty)(int domid, unsigned enable, void *data); /* HVM only */
@@ -99,6 +108,15 @@ struct restore_callbacks {
#define XGR_CHECKPOINT_FAILOVER 2 /* Failover and resume VM */
int (*checkpoint)(void* data);
+ /*
+ * Called after the checkpoint callback.
+ *
+ * returns:
+ * 0: terminate checkpointing gracefully
+ * 1: take another checkpoint
+ */
+ int (*wait_checkpoint)(void* data);
+
/* to be provided as the last argument to each callback function */
void* data;
};
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index d6d2967..51d004d 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -26,11 +26,12 @@ our @msgs = (
[ 3, 'scxA', "suspend", [] ],
[ 4, 'scxA', "postcopy", [] ],
[ 5, 'srcxA', "checkpoint", [] ],
- [ 6, 'scxA', "switch_qemu_logdirty", [qw(int domid
+ [ 6, 'srcxA', "wait_checkpoint", [] ],
+ [ 7, 'scxA', "switch_qemu_logdirty", [qw(int domid
unsigned enable)] ],
- [ 7, 'r', "restore_results", ['unsigned long', 'store_mfn',
+ [ 8, 'r', "restore_results", ['unsigned long', 'store_mfn',
'unsigned long', 'console_mfn'] ],
- [ 8, 'srW', "complete", [qw(int retval
+ [ 9, 'srW', "complete", [qw(int retval
int errnoval)] ],
);
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 15/31] tools/libx{l, c}: add postcopy/suspend callback to restore side
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (13 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 14/31] tools/libx{l, c}: introduce wait_checkpoint callback Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code Wen Congyang
` (16 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Secondary(restore side) is running under COLO, we also need
postcopy/suspend callbacks.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxc/include/xenguest.h | 10 ++++++++++
tools/libxl/libxl_save_msgs_gen.pl | 4 ++--
2 files changed, 12 insertions(+), 2 deletions(-)
diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index f598bec..cb0bbcf 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -101,6 +101,16 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom, uint32_t max_iter
/* callbacks provided by xc_domain_restore */
struct restore_callbacks {
+ /* Called after a new checkpoint to suspend the guest.
+ */
+ int (*suspend)(void* data);
+
+ /* Called after the secondary vm is ready to resume.
+ * Callback function resumes the guest & the device model,
+ * returns to xc_domain_restore.
+ */
+ int (*postcopy)(void* data);
+
/* A checkpoint record has been found in the stream.
* returns: */
#define XGR_CHECKPOINT_ERROR 0 /* Terminate processing */
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index 51d004d..6016706 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -23,8 +23,8 @@ our @msgs = (
STRING doing_what),
'unsigned long', 'done',
'unsigned long', 'total'] ],
- [ 3, 'scxA', "suspend", [] ],
- [ 4, 'scxA', "postcopy", [] ],
+ [ 3, 'srcxA', "suspend", [] ],
+ [ 4, 'srcxA', "postcopy", [] ],
[ 5, 'srcxA', "checkpoint", [] ],
[ 6, 'srcxA', "wait_checkpoint", [] ],
[ 7, 'scxA', "switch_qemu_logdirty", [qw(int domid
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (14 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 15/31] tools/libx{l, c}: add postcopy/suspend callback to restore side Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:56 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 17/31] primary " Wen Congyang
` (15 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Secondary vm is running in colo mode. So we will do
the following things again and again:
1. Resume secondary vm
a. Send CHECKPOINT_SVM_READY to master.
b. If it is not the first resume, call libxl__checkpoint_devices_preresume().
c. If it is the first resume(resume right after live migration),
- call libxl__xc_domain_restore_done() to build the secondary vm.
- enable secondary vm's logdirty.
- call libxl__domain_resume() to resume secondary vm.
- call libxl__checkpoint_devices_setup() to setup checkpoint devices.
d. Send CHECKPOINT_SVM_RESUMED to master.
2. Wait a new checkpoint
a. Call libxl__checkpoint_devices_commit().
b. Read CHECKPOINT_NEW from master.
3. Suspend secondary vm
a. Suspend secondary vm.
b. Call libxl__checkpoint_devices_postsuspend().
c. Send CHECKPOINT_SVM_SUSPENDED to master.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
---
tools/libxc/xc_sr_common.h | 2 +
tools/libxc/xc_sr_save.c | 3 +-
tools/libxl/Makefile | 1 +
tools/libxl/libxl_colo.h | 24 +
tools/libxl/libxl_colo_restore.c | 1038 ++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_create.c | 37 ++
tools/libxl/libxl_internal.h | 19 +
tools/libxl/libxl_save_callout.c | 7 +-
tools/libxl/libxl_stream_read.c | 12 +
tools/libxl/libxl_types.idl | 1 +
10 files changed, 1142 insertions(+), 2 deletions(-)
create mode 100644 tools/libxl/libxl_colo.h
create mode 100644 tools/libxl/libxl_colo_restore.c
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 5d9f497..2bfed64 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -184,10 +184,12 @@ struct xc_sr_context
* migration stream
* 0: Plain VM
* 1: Remus
+ * 2: COLO
*/
enum {
MIG_STREAM_NONE, /* plain stream */
MIG_STREAM_REMUS,
+ MIG_STREAM_COLO,
} migration_stream;
union /* Common save or restore data. */
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index fe210cc..7393355 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -846,7 +846,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
/* If altering migration_stream update this assert too. */
assert(checkpointed_stream == MIG_STREAM_NONE ||
- checkpointed_stream == MIG_STREAM_REMUS);
+ checkpointed_stream == MIG_STREAM_REMUS ||
+ checkpointed_stream == MIG_STREAM_COLO);
/*
* TODO: Find some time to better tweak the live migration algorithm.
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 789a12e..d8612eb 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -65,6 +65,7 @@ LIBXL_OBJS-y += libxl_no_convert_callout.o
endif
LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
+LIBXL_OBJS-y += libxl_colo_restore.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
new file mode 100644
index 0000000..8bea1a2
--- /dev/null
+++ b/tools/libxl/libxl_colo.h
@@ -0,0 +1,24 @@
+/*
+ * Copyright (C) 2014 FUJITSU LIMITED
+ * Author: Wen Congyang <wency@cn.fujitsu.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#ifndef LIBXL_COLO_H
+#define LIBXL_COLO_H
+
+extern void libxl__colo_restore_setup(libxl__egc *egc,
+ libxl__colo_restore_state *crs);
+extern void libxl__colo_restore_teardown(libxl__egc *egc, void *dcs_void,
+ int ret, int retval, int errnoval);
+
+#endif
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
new file mode 100644
index 0000000..a417e6e
--- /dev/null
+++ b/tools/libxl/libxl_colo_restore.c
@@ -0,0 +1,1038 @@
+/*
+ * Copyright (C) 2014 FUJITSU LIMITED
+ * Author: Wen Congyang <wency@cn.fujitsu.com>
+ * Yang Hongyang <hongyang.yang@easystack.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+#include "libxl_colo.h"
+#include "libxl_sr_stream_format.h"
+
+enum {
+ LIBXL_COLO_SETUPED,
+ LIBXL_COLO_SUSPENDED,
+ LIBXL_COLO_RESUMED,
+};
+
+typedef struct libxl__colo_restore_checkpoint_state libxl__colo_restore_checkpoint_state;
+struct libxl__colo_restore_checkpoint_state {
+ libxl__domain_suspend_state dsps;
+ libxl__logdirty_switch lds;
+ libxl__colo_restore_state *crs;
+ libxl__stream_write_state sws;
+ int status;
+ bool preresume;
+ /* used for teardown */
+ int teardown_devices;
+ int saved_rc;
+ char *state_file;
+
+ void (*callback)(libxl__egc *,
+ libxl__colo_restore_checkpoint_state *,
+ int);
+};
+
+
+static void libxl__colo_restore_domain_resume_callback(void *data);
+static void libxl__colo_restore_domain_checkpoint_callback(void *data);
+static void libxl__colo_restore_domain_wait_checkpoint_callback(void *data);
+static void libxl__colo_restore_domain_suspend_callback(void *data);
+
+static const libxl__checkpoint_device_instance_ops *colo_restore_ops[] = {
+ NULL,
+};
+
+/* ===================== colo: common functions ===================== */
+static void colo_enable_logdirty(libxl__colo_restore_state *crs, libxl__egc *egc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ const uint32_t domid = crs->domid;
+ libxl__logdirty_switch *const lds = &crcs->lds;
+
+ EGC_GC;
+
+ /* we need to know which pages are dirty to restore the guest */
+ if (xc_shadow_control(CTX->xch, domid,
+ XEN_DOMCTL_SHADOW_OP_ENABLE_LOGDIRTY,
+ NULL, 0, NULL, 0, NULL) < 0) {
+ LOG(ERROR, "cannot enable secondary vm's logdirty");
+ lds->callback(egc, lds, ERROR_FAIL);
+ return;
+ }
+
+ if (crs->hvm) {
+ libxl__domain_common_switch_qemu_logdirty(egc, domid, 1, lds);
+ return;
+ }
+
+ lds->callback(egc, lds, 0);
+}
+
+static void colo_disable_logdirty(libxl__colo_restore_state *crs,
+ libxl__egc *egc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ const uint32_t domid = crs->domid;
+ libxl__logdirty_switch *const lds = &crcs->lds;
+
+ EGC_GC;
+
+ /* we need to know which pages are dirty to restore the guest */
+ if (xc_shadow_control(CTX->xch, domid, XEN_DOMCTL_SHADOW_OP_OFF,
+ NULL, 0, NULL, 0, NULL) < 0)
+ LOG(WARN, "cannot disable secondary vm's logdirty");
+
+ if (crs->hvm) {
+ libxl__domain_common_switch_qemu_logdirty(egc, domid, 0, lds);
+ return;
+ }
+
+ lds->callback(egc, lds, 0);
+}
+
+static void colo_resume_vm(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int restore_device_model)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+ int rc;
+
+ /* Convenience aliases */
+ libxl__colo_restore_state *const crs = crcs->crs;
+
+ EGC_GC;
+
+ if (!crs->saved_cb) {
+ /* TODO: sync mmu for hvm? */
+ if (restore_device_model) {
+ rc = libxl__domain_restore_device_model(gc, crs->domid,
+ crcs->state_file);
+ if (rc) {
+ LOG(ERROR, "cannot restore device model for secondary vm");
+ crcs->callback(egc, crcs, rc);
+ return;
+ }
+ }
+ rc = libxl__domain_resume(gc, crs->domid, 0);
+ if (rc)
+ LOG(ERROR, "cannot resume secondary vm");
+
+ crcs->callback(egc, crcs, rc);
+ return;
+ }
+
+ /*
+ * TODO: get store gfn and console gfn
+ * We should call the callback restore_results in
+ * xc_domain_restore() before resuming the guest.
+ */
+ libxl__xc_domain_restore_done(egc, dcs, 0, 0, 0);
+
+ return;
+}
+
+static int init_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+ /* init device subkind-specific state in the libxl ctx */
+ int rc;
+ STATE_AO_GC(cds->ao);
+
+ rc = 0;
+ return rc;
+}
+
+static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+ /* cleanup device subkind-specific state in the libxl ctx */
+ STATE_AO_GC(cds->ao);
+}
+
+
+/* ================ colo: setup restore environment ================ */
+static void libxl__colo_domain_create_cb(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int rc, uint32_t domid);
+
+static int init_dsps(libxl__domain_suspend_state *dsps)
+{
+ int rc = ERROR_FAIL;
+ libxl_domain_type type;
+
+ STATE_AO_GC(dsps->ao);
+
+ libxl__xswait_init(&dsps->pvcontrol);
+ libxl__ev_evtchn_init(&dsps->guest_evtchn);
+ libxl__ev_xswatch_init(&dsps->guest_watch);
+ libxl__ev_time_init(&dsps->guest_timeout);
+
+ type = libxl__domain_type(gc, dsps->domid);
+ if (type == LIBXL_DOMAIN_TYPE_INVALID)
+ goto out;
+
+ dsps->type = type;
+
+ dsps->guest_evtchn.port = -1;
+ dsps->guest_evtchn_lockfd = -1;
+ dsps->guest_responded = 0;
+ dsps->dm_savefile = libxl__device_model_savefile(gc, dsps->domid);
+
+ /* Secondary vm is not created, so we cannot get evtchn port */
+
+ rc = 0;
+
+out:
+ return rc;
+}
+
+void libxl__colo_restore_setup(libxl__egc *egc,
+ libxl__colo_restore_state *crs)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs;
+ int rc = ERROR_FAIL;
+
+ /* Convenience aliases */
+ libxl__srm_restore_autogen_callbacks *const callbacks =
+ &dcs->srs.shs.callbacks.restore.a;
+ const int domid = crs->domid;
+
+ STATE_AO_GC(crs->ao);
+
+ GCNEW(crcs);
+ crs->crcs = crcs;
+ crcs->crs = crs;
+
+ /* setup dsps */
+ crcs->dsps.ao = ao;
+ crcs->dsps.domid = domid;
+ if (init_dsps(&crcs->dsps))
+ goto err;
+
+ callbacks->suspend = libxl__colo_restore_domain_suspend_callback;
+ callbacks->postcopy = libxl__colo_restore_domain_resume_callback;
+ callbacks->checkpoint = libxl__colo_restore_domain_checkpoint_callback;
+ callbacks->wait_checkpoint = libxl__colo_restore_domain_wait_checkpoint_callback;
+
+ /*
+ * Secondary vm is running in colo mode, so we need to call
+ * libxl__xc_domain_restore_done() to create secondary vm.
+ * But we will exit in domain_create_cb(). So replace the
+ * callback here.
+ */
+ crs->saved_cb = dcs->callback;
+ dcs->callback = libxl__colo_domain_create_cb;
+ crcs->state_file = GCSPRINTF(LIBXL_DEVICE_MODEL_RESTORE_FILE".%d", domid);
+ crcs->status = LIBXL_COLO_SETUPED;
+
+ libxl__logdirty_init(&crcs->lds);
+ crcs->lds.ao = ao;
+
+ crcs->sws.fd = crs->send_back_fd;
+ crcs->sws.ao = ao;
+ crcs->sws.back_channel = true;
+
+ dcs->cds.concrete_data = crs;
+
+ libxl__stream_write_start(egc, &crcs->sws);
+
+ rc = 0;
+
+out:
+ crs->callback(egc, crs, rc);
+ return;
+
+err:
+ goto out;
+}
+
+static void libxl__colo_domain_create_cb(libxl__egc *egc,
+ libxl__domain_create_state *dcs,
+ int rc, uint32_t domid)
+{
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+
+ crcs->callback(egc, crcs, rc);
+}
+
+
+/* ================ colo: teardown restore environment ================ */
+static void colo_restore_teardown_devices_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds, int rc);
+static void do_failover(libxl__egc *egc, libxl__colo_restore_state *crs);
+static void do_failover_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state* crcs,
+ int rc);
+static void colo_disable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc);
+static void libxl__colo_restore_teardown_done(libxl__egc *egc,
+ libxl__colo_restore_state *crs,
+ int rc);
+
+void libxl__colo_restore_teardown(libxl__egc *egc, void *dcs_void,
+ int ret, int retval, int errnoval)
+{
+ libxl__domain_create_state *dcs = dcs_void;
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+ int rc = 1;
+
+ /* convenience aliases */
+ libxl__colo_restore_state *const crs = &dcs->crs;
+ EGC_GC;
+
+ if (ret == 0 && retval == 0)
+ rc = 0;
+
+ LOG(INFO, "%s", rc ? "colo fails" : "failover");
+
+ libxl__stream_write_abort(egc, &crcs->sws, 1);
+ if (crs->saved_cb) {
+ /* crcs->status is LIBXL_COLO_SETUPED */
+ dcs->srs.completion_callback = NULL;
+ }
+ libxl__xc_domain_restore_done(egc, dcs, ret, retval, errnoval);
+
+ crcs->saved_rc = rc;
+ if (!crcs->teardown_devices) {
+ colo_restore_teardown_devices_done(egc, &dcs->cds, 0);
+ return;
+ }
+
+ dcs->cds.callback = colo_restore_teardown_devices_done;
+ libxl__checkpoint_devices_teardown(egc, &dcs->cds);
+}
+
+static void colo_restore_teardown_devices_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds, int rc)
+{
+ libxl__colo_restore_state *crs = cds->concrete_data;
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+
+ EGC_GC;
+
+ if (rc)
+ LOG(ERROR, "COLO: failed to teardown device for guest with domid %u,"
+ " rc %d", cds->domid, rc);
+
+ if (crcs->teardown_devices)
+ cleanup_device_subkind(cds);
+
+ rc = crcs->saved_rc;
+ if (!rc) {
+ crcs->callback = do_failover_done;
+ do_failover(egc, crs);
+ return;
+ }
+
+ libxl__colo_restore_teardown_done(egc, crs, rc);
+}
+
+static void do_failover(libxl__egc *egc, libxl__colo_restore_state *crs)
+{
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ const int status = crcs->status;
+ libxl__logdirty_switch *const lds = &crcs->lds;
+
+ EGC_GC;
+
+ switch(status) {
+ case LIBXL_COLO_SETUPED:
+ /*
+ * We will come here only when reading emulator xenstore data or
+ * emulator context fails, and libxl__xc_domain_restore_done()
+ * is not called. In this case, the migration is not finished,
+ * so we cannot do failover.
+ */
+ LOG(ERROR, "migration fails");
+ crcs->callback(egc, crcs, ERROR_FAIL);
+ return;
+ case LIBXL_COLO_SUSPENDED:
+ case LIBXL_COLO_RESUMED:
+ /* disable logdirty first */
+ lds->callback = colo_disable_logdirty_done;
+ colo_disable_logdirty(crs, egc);
+ return;
+ default:
+ LOG(ERROR, "invalid status: %d", status);
+ crcs->callback(egc, crcs, ERROR_FAIL);
+ }
+}
+
+static void do_failover_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state* crcs,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__colo_restore_state *const crs = crcs->crs;
+
+ EGC_GC;
+
+ if (rc)
+ LOG(ERROR, "cannot do failover");
+
+ libxl__colo_restore_teardown_done(egc, crs, rc);
+}
+
+static void colo_disable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
+
+ EGC_GC;
+
+ if (rc)
+ LOG(WARN, "cannot disable logdirty");
+
+ if (crcs->status == LIBXL_COLO_SUSPENDED) {
+ /*
+ * failover when reading state from master, so no need to
+ * call libxl__domain_restore().
+ */
+ colo_resume_vm(egc, crcs, 0);
+ return;
+ }
+
+ /* If we cannot disable logdirty, we still can do failover */
+ crcs->callback(egc, crcs, 0);
+}
+
+static void libxl__colo_restore_teardown_done(libxl__egc *egc,
+ libxl__colo_restore_state *crs,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ EGC_GC;
+
+ /* convenience aliases */
+ const int domid = crs->domid;
+ const libxl_ctx *const ctx = libxl__gc_owner(gc);
+ xc_interface *const xch = ctx->xch;
+
+ if (!rc)
+ /* failover, no need to destroy the secondary vm */
+ goto out;
+
+ xc_domain_destroy(xch, domid);
+
+out:
+ if (crs->saved_cb) {
+ dcs->callback = crs->saved_cb;
+ crs->saved_cb = NULL;
+ }
+
+ dcs->callback(egc, dcs, rc, crs->domid);
+}
+
+/*
+ * checkpoint callbacks are called in the following order:
+ * 1. checkpoint
+ * 2. resume
+ * 3. wait checkpoint
+ * 4. suspend
+ */
+static void colo_common_write_stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc);
+static void colo_common_read_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ int rc);
+/* ======================== colo: checkpoint ======================= */
+/*
+ * Do the following things when resuming secondary vm:
+ * 1. read emulator xenstore data
+ * 2. read emulator context
+ * 3. REC_TYPE_CHECKPOINT_END
+ */
+static void libxl__colo_restore_domain_checkpoint_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_read_state *srs = CONTAINER_OF(shs, *srs, shs);
+ libxl__domain_create_state *dcs = CONTAINER_OF(srs, *dcs, srs);
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+
+ crcs->callback = NULL;
+ dcs->srs.checkpoint_callback = colo_common_read_stream_done;
+ libxl__stream_read_start_checkpoint(shs->egc, &dcs->srs);
+}
+
+
+/* ===================== colo: resume secondary vm ===================== */
+/*
+ * Do the following things when resuming secondary vm the first time:
+ * 1. resume secondary vm
+ * 2. enable log dirty
+ * 3. setup checkpoint devices
+ * 4. write CHECKPOINT_SVM_READY
+ * 5. unpause secondary vm
+ * 6. write CHECKPOINT_SVM_RESUMED
+ *
+ * Do the following things when resuming secondary vm:
+ * 1. write CHECKPOINT_SVM_READY
+ * 2. resume secondary vm
+ * 3. write CHECKPOINT_SVM_RESUMED
+ */
+static void colo_send_svm_ready(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs);
+static void colo_send_svm_ready_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int rc);
+static void colo_restore_preresume_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_restore_resume_vm(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs);
+static void colo_resume_vm_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int rc);
+static void colo_write_svm_resumed(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs);
+static void colo_enable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int retval);
+static void colo_reenable_logdirty(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc);
+static void colo_reenable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc);
+static void colo_setup_checkpoint_devices(libxl__egc *egc,
+ libxl__colo_restore_state *crs);
+static void colo_restore_setup_cds_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_unpause_svm(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs);
+
+static void libxl__colo_restore_domain_resume_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_read_state *srs = CONTAINER_OF(shs, *srs, shs);
+ libxl__domain_create_state *dcs = CONTAINER_OF(srs, *dcs, srs);
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+
+ if (crcs->teardown_devices)
+ colo_send_svm_ready(shs->egc, crcs);
+ else
+ colo_restore_resume_vm(shs->egc, crcs);
+}
+
+static void colo_send_svm_ready(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs)
+{
+ libxl_sr_checkpoint_state srcs = { .id = CHECKPOINT_SVM_READY };
+
+ crcs->callback = colo_send_svm_ready_done;
+ crcs->sws.checkpoint_callback = colo_common_write_stream_done;
+ libxl__stream_write_checkpoint_state(egc, &crcs->sws, &srcs);
+}
+
+static void colo_send_svm_ready_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *cds = &dcs->cds;
+
+ if (!crcs->preresume) {
+ crcs->preresume = true;
+ colo_unpause_svm(egc, crcs);
+ return;
+ }
+
+ cds->callback = colo_restore_preresume_cb;
+ libxl__checkpoint_devices_preresume(egc, cds);
+}
+
+static void colo_restore_preresume_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_restore_state *crs = cds->concrete_data;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "preresume fails");
+ goto out;
+ }
+
+ colo_restore_resume_vm(egc, crcs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+static void colo_restore_resume_vm(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs)
+{
+
+ crcs->callback = colo_resume_vm_done;
+ colo_resume_vm(egc, crcs, 1);
+}
+
+static void colo_resume_vm_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__colo_restore_state *const crs = crcs->crs;
+ libxl__logdirty_switch *const lds = &crcs->lds;
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "cannot resume secondary vm");
+ goto out;
+ }
+
+ crcs->status = LIBXL_COLO_RESUMED;
+
+ /* avoid calling stream->completion_callback() more than once */
+ if (crs->saved_cb) {
+ dcs->callback = crs->saved_cb;
+ crs->saved_cb = NULL;
+
+ dcs->srs.completion_callback = NULL;
+
+ lds->callback = colo_enable_logdirty_done;
+ colo_enable_logdirty(crs, egc);
+ return;
+ }
+
+ colo_write_svm_resumed(egc, crcs);
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+static void colo_write_svm_resumed(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs)
+{
+ libxl_sr_checkpoint_state srcs = { .id = CHECKPOINT_SVM_RESUMED };
+
+ crcs->callback = NULL;
+ crcs->sws.checkpoint_callback = colo_common_write_stream_done;
+ libxl__stream_write_checkpoint_state(egc, &crcs->sws, &srcs);
+}
+
+static void colo_enable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
+
+ /* Convenience aliases */
+ libxl__colo_restore_state *const crs = crcs->crs;
+
+ EGC_GC;
+
+ if (rc) {
+ /*
+ * log-dirty already enabled? There's no test op,
+ * so attempt to disable then reenable it
+ */
+ lds->callback = colo_reenable_logdirty;
+ colo_disable_logdirty(crs, egc);
+ return;
+ }
+
+ colo_setup_checkpoint_devices(egc, crs);
+}
+
+static void colo_reenable_logdirty(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__colo_restore_state *const crs = crcs->crs;
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "cannot enable logdirty");
+ goto out;
+ }
+
+ lds->callback = colo_reenable_logdirty_done;
+ colo_enable_logdirty(crs, egc);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+static void colo_reenable_logdirty_done(libxl__egc *egc,
+ libxl__logdirty_switch *lds,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "cannot enable logdirty");
+ goto out;
+ }
+
+ colo_setup_checkpoint_devices(egc, crcs->crs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+/*
+ * We cannot setup checkpoint devices in libxl__colo_restore_setup(),
+ * because the guest is not ready.
+ */
+static void colo_setup_checkpoint_devices(libxl__egc *egc,
+ libxl__colo_restore_state *crs)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *cds = &dcs->cds;
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ STATE_AO_GC(crs->ao);
+
+ /* TODO: disk/nic support */
+ cds->device_kind_flags = 0;
+ cds->callback = colo_restore_setup_cds_done;
+ cds->ao = ao;
+ cds->domid = crs->domid;
+ cds->ops = colo_restore_ops;
+
+ if (init_device_subkind(cds))
+ goto out;
+
+ crcs->teardown_devices = 1;
+
+ libxl__checkpoint_devices_setup(egc, cds);
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+static void colo_restore_setup_cds_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_restore_state *crs = cds->concrete_data;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ /* Convenience aliases */
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "COLO: failed to setup device for guest with domid %u",
+ cds->domid);
+ goto out;
+ }
+
+ colo_send_svm_ready(egc, crcs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+static void colo_unpause_svm(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+ int rc;
+
+ /* Convenience aliases */
+ const uint32_t domid = crcs->crs->domid;
+ libxl__save_helper_state *const shs = &dcs->srs.shs;
+
+ EGC_GC;
+
+ /* We have enabled secondary vm's logdirty, so we can unpause it now */
+ rc = libxl_domain_unpause(CTX, domid);
+ if (rc) {
+ LOG(ERROR, "cannot unpause secondary vm");
+ goto out;
+ }
+
+ colo_write_svm_resumed(egc, crcs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, shs, 0);
+}
+
+
+/* ===================== colo: wait new checkpoint ===================== */
+static void colo_restore_commit_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_stream_read_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int real_size);
+
+static void libxl__colo_restore_domain_wait_checkpoint_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_read_state *srs = CONTAINER_OF(shs, *srs, shs);
+ libxl__domain_create_state *dcs = CONTAINER_OF(srs, *dcs, srs);
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *cds = &dcs->cds;
+
+ cds->callback = colo_restore_commit_cb;
+ libxl__checkpoint_devices_commit(shs->egc, cds);
+}
+
+static void colo_restore_commit_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_restore_state *crs = cds->concrete_data;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "commit fails");
+ goto out;
+ }
+
+ crcs->callback = colo_stream_read_done;
+ dcs->srs.checkpoint_callback = colo_common_read_stream_done;
+ libxl__stream_read_checkpoint_state(egc, &dcs->srs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, 0);
+}
+
+static void colo_stream_read_done(libxl__egc *egc,
+ libxl__colo_restore_checkpoint_state *crcs,
+ int id)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+ int ok = 0;
+
+ EGC_GC;
+
+ if (id != CHECKPOINT_NEW) {
+ LOG(ERROR, "invalid section: %d", id);
+ goto out;
+ }
+
+ ok = 1;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, ok);
+}
+
+
+/* ===================== colo: suspend secondary vm ===================== */
+/*
+ * Do the following things when resuming secondary vm:
+ * 1. suspend secondary vm
+ * 2. send CHECKPOINT_SVM_SUSPENDED
+ */
+static void colo_suspend_vm_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dsps,
+ int ok);
+static void colo_restore_postsuspend_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+
+static void libxl__colo_restore_domain_suspend_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_read_state *srs = CONTAINER_OF(shs, *srs, shs);
+ libxl__domain_create_state *dcs = CONTAINER_OF(srs, *dcs, srs);
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+
+ STATE_AO_GC(dcs->ao);
+
+ /* Convenience aliases */
+ libxl__domain_suspend_state *const dsps = &crcs->dsps;
+
+ /* suspend secondary vm */
+ dsps->callback_common_done = colo_suspend_vm_done;
+
+ libxl__domain_suspend(shs->egc, dsps);
+}
+
+static void colo_suspend_vm_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dsps,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(dsps, *crcs, dsps);
+ libxl__colo_restore_state *crs = crcs->crs;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *cds = &dcs->cds;
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "cannot suspend secondary vm");
+ goto out;
+ }
+
+ crcs->status = LIBXL_COLO_SUSPENDED;
+
+ cds->callback = colo_restore_postsuspend_cb;
+ libxl__checkpoint_devices_postsuspend(egc, cds);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, !rc);
+}
+
+static void colo_restore_postsuspend_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_restore_state *crs = cds->concrete_data;
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+ libxl__colo_restore_checkpoint_state *crcs = crs->crcs;
+ libxl_sr_checkpoint_state srcs = { .id = CHECKPOINT_SVM_SUSPENDED };
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "postsuspend fails");
+ goto out;
+ }
+
+ crcs->callback = NULL;
+ crcs->sws.checkpoint_callback = colo_common_write_stream_done;
+ libxl__stream_write_checkpoint_state(egc, &crcs->sws, &srcs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, !rc);
+}
+
+
+/* ===================== colo: common callback ===================== */
+static void colo_common_write_stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc)
+{
+ libxl__colo_restore_checkpoint_state *crcs =
+ CONTAINER_OF(stream, *crcs, sws);
+ libxl__domain_create_state *dcs = CONTAINER_OF(crcs->crs, *dcs, crs);
+ int ok;
+
+ EGC_GC;
+
+ if (rc < 0) {
+ /* TODO: it may be a internal error, but we don't know */
+ LOG(ERROR, "sending data fails");
+ ok = 2;
+ goto out;
+ }
+
+ if (!crcs->callback) {
+ /* Everythins is OK */
+ ok = 1;
+ goto out;
+ }
+
+ crcs->callback(egc, crcs, 0);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, ok);
+}
+
+static void colo_common_read_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
+ libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
+ int ok;
+
+ EGC_GC;
+
+ if (rc < 0) {
+ /* TODO: it may be a internal error, but we don't know */
+ LOG(ERROR, "reading data fails");
+ ok = 2;
+ goto out;
+ }
+
+ if (!crcs->callback) {
+ /* Everythins is OK */
+ ok = 1;
+ goto out;
+ }
+
+ /* rc contains the id */
+ crcs->callback(egc, crcs, rc);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, ok);
+}
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index fc746fb..39458b7 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -19,6 +19,7 @@
#include "libxl_internal.h"
#include "libxl_arch.h"
+#include "libxl_colo.h"
#include <xc_dom.h>
#include <xenguest.h>
@@ -981,6 +982,23 @@ static void domcreate_console_available(libxl__egc *egc,
dcs->aop_console_how.for_event));
}
+static void libxl__colo_restore_setup_done(libxl__egc *egc,
+ libxl__colo_restore_state *crs,
+ int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "colo restore setup fails: %d", rc);
+ domcreate_stream_done(egc, &dcs->srs, rc);
+ return;
+ }
+
+ libxl__stream_read_start(egc, &dcs->srs);
+}
+
static void domcreate_bootloader_done(libxl__egc *egc,
libxl__bootloader_state *bl,
int rc)
@@ -994,6 +1012,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
+ libxl__colo_restore_state *const crs = &dcs->crs;
+ libxl_domain_build_info *const info = &d_config->b_info;
if (rc) {
domcreate_rebuild_done(egc, dcs, rc);
@@ -1022,6 +1042,13 @@ static void domcreate_bootloader_done(libxl__egc *egc,
/* Restore */
+ /* COLO only supports HVM now */
+ if (info->type != LIBXL_DOMAIN_TYPE_HVM &&
+ checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
rc = libxl__build_pre(gc, domid, d_config, state);
if (rc)
goto out;
@@ -1035,6 +1062,16 @@ static void domcreate_bootloader_done(libxl__egc *egc,
if (restore_fd >= 0) {
switch (checkpointed_stream) {
+ case LIBXL_CHECKPOINTED_STREAM_COLO:
+ /* colo restore setup */
+ crs->ao = ao;
+ crs->domid = domid;
+ crs->send_back_fd = dcs->send_back_fd;
+ crs->recv_fd = restore_fd;
+ crs->hvm = (info->type == LIBXL_DOMAIN_TYPE_HVM);
+ crs->callback = libxl__colo_restore_setup_done;
+ libxl__colo_restore_setup(egc, crs);
+ break;
case LIBXL_CHECKPOINTED_STREAM_REMUS:
libxl__remus_restore_setup(egc, dcs);
/* fall through */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 33b658d..b9ca81c 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3470,6 +3470,23 @@ libxl__stream_read_inuse(const libxl__stream_read_state *stream)
return stream->running;
}
+/* colo related structure */
+typedef struct libxl__colo_restore_state libxl__colo_restore_state;
+typedef void libxl__colo_callback(libxl__egc *,
+ libxl__colo_restore_state *, int rc);
+struct libxl__colo_restore_state {
+ /* must set by caller of libxl__colo_(setup|teardown) */
+ libxl__ao *ao;
+ uint32_t domid;
+ int send_back_fd;
+ int recv_fd;
+ int hvm;
+ libxl__colo_callback *callback;
+
+ /* private, colo restore checkpoint state */
+ libxl__domain_create_cb *saved_cb;
+ void *crcs;
+};
struct libxl__domain_create_state {
/* filled in by user */
@@ -3486,6 +3503,8 @@ struct libxl__domain_create_state {
/* private to domain_create */
int guest_domid;
libxl__domain_build_state build_state;
+ libxl__colo_restore_state crs;
+ libxl__checkpoint_devices_state cds;
libxl__bootloader_state bl;
libxl__stub_dm_spawn_state dmss;
/* If we're not doing stubdom, we use only dmss.dm,
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 0d6949a..b1810b2 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -15,6 +15,7 @@
#include "libxl_osdeps.h"
#include "libxl_internal.h"
+#include "libxl_colo.h"
/* stream_fd is as from the caller (eventually, the application).
* It may be 0, 1 or 2, in which case we need to dup it elsewhere.
@@ -68,7 +69,11 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
shs->ao = ao;
shs->domid = domid;
shs->recv_callback = libxl__srm_callout_received_restore;
- shs->completion_callback = libxl__xc_domain_restore_done;
+ if (dcs->restore_params.checkpointed_stream ==
+ LIBXL_CHECKPOINTED_STREAM_COLO)
+ shs->completion_callback = libxl__colo_restore_teardown;
+ else
+ shs->completion_callback = libxl__xc_domain_restore_done;
shs->caller_state = dcs;
shs->need_results = 1;
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index 5d980d9..d6bd2fe 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -846,6 +846,18 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
*/
if (libxl__stream_read_inuse(stream)) {
switch (checkpointed_stream) {
+ case LIBXL_CHECKPOINTED_STREAM_COLO:
+ if (stream->completion_callback) {
+ /*
+ * restore, just build the secondary vm, don't close
+ * the stream
+ */
+ stream->completion_callback(egc, stream, 0);
+ } else {
+ /* failover, just close the stream */
+ stream_complete(egc, stream, 0);
+ }
+ break;
case LIBXL_CHECKPOINTED_STREAM_REMUS:
/*
* Failover from primary. Domain state is currently at a
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 632c009..33f4a90 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -232,6 +232,7 @@ libxl_hdtype = Enumeration("hdtype", [
libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
(0, "NONE"),
(1, "REMUS"),
+ (2, "COLO"),
])
#
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 17/31] primary vm suspend/resume/checkpoint code
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (15 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:57 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 18/31] libxc/restore: support COLO restore Wen Congyang
` (14 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
We will do the following things again and again:
1. Suspend primary vm
a. Suspend primary vm
b. do postsuspend
c. Read CHECKPOINT_SVM_SUSPENDED sent by secondary
2. Resume primary vm
a. Read CHECKPOINT_SVM_READY from slave
b. Do presume
c. Resume primary vm
d. Read CHECKPOINT_SVM_RESUMED from slave
3. Wait a new checkpoint
a. Wait a new checkpoint(not implemented)
b. Send CHECKPOINT_NEW to slave
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
---
tools/libxl/Makefile | 2 +-
tools/libxl/libxl.c | 6 +-
tools/libxl/libxl_colo.h | 6 +
tools/libxl/libxl_colo_save.c | 563 ++++++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_dom_save.c | 8 +-
tools/libxl/libxl_internal.h | 168 +++++++------
tools/libxl/libxl_types.idl | 1 +
7 files changed, 674 insertions(+), 80 deletions(-)
create mode 100644 tools/libxl/libxl_colo_save.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index d8612eb..2016393 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -65,7 +65,7 @@ LIBXL_OBJS-y += libxl_no_convert_callout.o
endif
LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
-LIBXL_OBJS-y += libxl_colo_restore.o
+LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index e9ab78c..7236175 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -17,6 +17,7 @@
#include "libxl_osdeps.h"
#include "libxl_internal.h"
+#include "libxl_colo.h"
#define PAGE_TO_MEMKB(pages) ((pages) * 4)
#define BACKEND_STRING_SIZE 5
@@ -880,7 +881,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
assert(info);
/* Point of no return */
- libxl__remus_setup(egc, &dss->rs);
+ if (libxl_defbool_val(info->colo))
+ libxl__colo_save_setup(egc, &dss->css);
+ else
+ libxl__remus_setup(egc, &dss->rs);
return AO_INPROGRESS;
out:
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 8bea1a2..b08f672 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -21,4 +21,10 @@ extern void libxl__colo_restore_setup(libxl__egc *egc,
extern void libxl__colo_restore_teardown(libxl__egc *egc, void *dcs_void,
int ret, int retval, int errnoval);
+extern void libxl__colo_save_setup(libxl__egc *egc,
+ libxl__colo_save_state *css);
+extern void libxl__colo_save_teardown(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int rc);
+
#endif
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
new file mode 100644
index 0000000..8b18a91
--- /dev/null
+++ b/tools/libxl/libxl_colo_save.c
@@ -0,0 +1,563 @@
+/*
+ * Copyright (C) 2014 FUJITSU LIMITED
+ * Author: Wen Congyang <wency@cn.fujitsu.com>
+ * Yang Hongyang <hongyang.yang@easystack.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+#include "libxl_colo.h"
+
+static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
+ NULL,
+};
+
+/* ================= helper functions ================= */
+static int init_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+ /* init device subkind-specific state in the libxl ctx */
+ int rc;
+ STATE_AO_GC(cds->ao);
+
+ rc = 0;
+ return rc;
+}
+
+static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
+{
+ /* cleanup device subkind-specific state in the libxl ctx */
+ STATE_AO_GC(cds->ao);
+}
+
+/* ================= colo: setup save environment ================= */
+static void colo_save_setup_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_save_setup_failed(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void libxl__colo_save_domain_suspend_callback(void *data);
+static void libxl__colo_save_domain_checkpoint_callback(void *data);
+static void libxl__colo_save_domain_resume_callback(void *data);
+static void libxl__colo_save_domain_wait_checkpoint_callback(void *data);
+
+void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *const cds = &dss->cds;
+ libxl__srm_save_autogen_callbacks *const callbacks =
+ &dss->sws.shs.callbacks.save.a;
+
+ STATE_AO_GC(dss->ao);
+
+ if (dss->type != LIBXL_DOMAIN_TYPE_HVM) {
+ LOG(ERROR, "COLO only supports hvm now");
+ goto out;
+ }
+
+ css->send_fd = dss->fd;
+ css->recv_fd = dss->recv_fd;
+ css->svm_running = false;
+
+ /* TODO: disk/nic support */
+ cds->device_kind_flags = 0;
+ cds->ops = colo_ops;
+ cds->callback = colo_save_setup_done;
+ cds->ao = ao;
+ cds->domid = dss->domid;
+ cds->concrete_data = css;
+
+ css->srs.ao = ao;
+ css->srs.fd = css->recv_fd;
+ css->srs.back_channel = true;
+ libxl__stream_read_start(egc, &css->srs);
+
+ if (init_device_subkind(cds))
+ goto out;
+
+ callbacks->suspend = libxl__colo_save_domain_suspend_callback;
+ callbacks->postcopy = libxl__colo_save_domain_resume_callback;
+ callbacks->checkpoint = libxl__colo_save_domain_checkpoint_callback;
+ callbacks->wait_checkpoint = libxl__colo_save_domain_wait_checkpoint_callback;
+
+ libxl__checkpoint_devices_setup(egc, &dss->cds);
+
+ return;
+
+out:
+ libxl__ao_complete(egc, ao, ERROR_FAIL);
+}
+
+static void colo_save_setup_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+ EGC_GC;
+
+ if (!rc) {
+ libxl__domain_save(egc, dss);
+ return;
+ }
+
+ LOG(ERROR, "COLO: failed to setup device for guest with domid %u",
+ dss->domid);
+ cds->callback = colo_save_setup_failed;
+ libxl__checkpoint_devices_teardown(egc, cds);
+}
+
+static void colo_save_setup_failed(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ STATE_AO_GC(cds->ao);
+
+ if (rc)
+ LOG(ERROR, "COLO: failed to teardown device after setup failed"
+ " for guest with domid %u, rc %d", cds->domid, rc);
+
+ cleanup_device_subkind(cds);
+ libxl__ao_complete(egc, ao, rc);
+}
+
+
+/* ================= colo: teardown save environment ================= */
+static void colo_teardown_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+
+void libxl__colo_save_teardown(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int rc)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ LOG(WARN, "COLO: Domain suspend terminated with rc %d,"
+ " teardown COLO devices...", rc);
+
+ libxl__stream_read_abort(egc, &css->srs, 1);
+
+ dss->cds.callback = colo_teardown_done;
+ libxl__checkpoint_devices_teardown(egc, &dss->cds);
+ return;
+}
+
+static void colo_teardown_done(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ cleanup_device_subkind(cds);
+ dss->callback(egc, dss, rc);
+}
+
+/*
+ * checkpoint callbacks are called in the following order:
+ * 1. suspend
+ * 2. checkpoint
+ * 3. resume
+ * 4. wait checkpoint
+ */
+static void colo_common_write_stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc);
+static void colo_common_read_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ int rc);
+/* ===================== colo: suspend primary vm ===================== */
+
+static void colo_read_svm_suspended_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id);
+/*
+ * Do the following things when suspending primary vm:
+ * 1. suspend primary vm
+ * 2. do postsuspend
+ * 3. read CHECKPOINT_SVM_SUSPENDED
+ * 4. read secondary vm's dirty pages
+ */
+static void colo_suspend_primary_vm_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dsps,
+ int ok);
+static void colo_postsuspend_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+
+static void libxl__colo_save_domain_suspend_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__egc *egc = shs->egc;
+ libxl__stream_write_state *sws = CONTAINER_OF(shs, *sws, shs);
+ libxl__domain_save_state *dss = sws->dss;
+
+ /* Convenience aliases */
+ libxl__domain_suspend_state *dsps = &dss->dsps;
+
+ dsps->callback_common_done = colo_suspend_primary_vm_done;
+ libxl__domain_suspend(egc, dsps);
+}
+
+static void colo_suspend_primary_vm_done(libxl__egc *egc,
+ libxl__domain_suspend_state *dsps,
+ int rc)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(dsps, *dss, dsps);
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "cannot suspend primary vm");
+ goto out;
+ }
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *const cds = &dss->cds;
+
+ cds->callback = colo_postsuspend_cb;
+ libxl__checkpoint_devices_postsuspend(egc, cds);
+ return;
+
+out:
+ dss->rc = rc;
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, !rc);
+}
+
+static void colo_postsuspend_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "postsuspend fails");
+ goto out;
+ }
+
+ if (!css->svm_running) {
+ rc = 0;
+ goto out;
+ }
+
+ /*
+ * read CHECKPOINT_SVM_SUSPENDED
+ */
+ css->callback = colo_read_svm_suspended_done;
+ css->srs.checkpoint_callback = colo_common_read_stream_done;
+ libxl__stream_read_checkpoint_state(egc, &css->srs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, !rc);
+}
+
+static void colo_read_svm_suspended_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id)
+{
+ int ok = 0;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (id != CHECKPOINT_SVM_SUSPENDED) {
+ LOG(ERROR, "invalid section: %d, expected: %d", id,
+ CHECKPOINT_SVM_SUSPENDED);
+ goto out;
+ }
+
+ ok = 1;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, ok);
+}
+
+
+/* ===================== colo: send tailbuf ========================== */
+static void libxl__colo_save_domain_checkpoint_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_write_state *sws = CONTAINER_OF(shs, *sws, shs);
+ libxl__domain_save_state *dss = sws->dss;
+
+ /* Convenience aliases */
+ libxl__colo_save_state *const css = &dss->css;
+
+ /* write emulator xenstore data, emulator context, and checkpoint end */
+ css->callback = NULL;
+ dss->sws.checkpoint_callback = colo_common_write_stream_done;
+ libxl__stream_write_start_checkpoint(shs->egc, &dss->sws);
+}
+
+/* ===================== colo: resume primary vm ===================== */
+/*
+ * Do the following things when resuming primary vm:
+ * 1. read CHECKPOINT_SVM_READY
+ * 2. do preresume
+ * 3. resume primary vm
+ * 4. read CHECKPOINT_SVM_RESUMED
+ */
+static void colo_read_svm_ready_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id);
+static void colo_preresume_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_read_svm_resumed_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id);
+
+static void libxl__colo_save_domain_resume_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__egc *egc = shs->egc;
+ libxl__stream_write_state *sws = CONTAINER_OF(shs, *sws, shs);
+ libxl__domain_save_state *dss = sws->dss;
+
+ /* Convenience aliases */
+ libxl__colo_save_state *const css = &dss->css;
+
+ EGC_GC;
+
+ /* read CHECKPOINT_SVM_READY */
+ css->callback = colo_read_svm_ready_done;
+ css->srs.checkpoint_callback = colo_common_read_stream_done;
+ libxl__stream_read_checkpoint_state(egc, &css->srs);
+}
+
+static void colo_read_svm_ready_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (id != CHECKPOINT_SVM_READY) {
+ LOG(ERROR, "invalid section: %d, expected: %d", id,
+ CHECKPOINT_SVM_READY);
+ goto out;
+ }
+
+ css->svm_running = true;
+ dss->cds.callback = colo_preresume_cb;
+ libxl__checkpoint_devices_preresume(egc, &dss->cds);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
+}
+
+static void colo_preresume_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "preresume fails");
+ goto out;
+ }
+
+ /* Resumes the domain and the device model */
+ if (libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1)) {
+ LOG(ERROR, "cannot resume primary vm");
+ goto out;
+ }
+
+ /* read CHECKPOINT_SVM_RESUMED */
+ css->callback = colo_read_svm_resumed_done;
+ css->srs.checkpoint_callback = colo_common_read_stream_done;
+ libxl__stream_read_checkpoint_state(egc, &css->srs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
+}
+
+static void colo_read_svm_resumed_done(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ int id)
+{
+ int ok = 0;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (id != CHECKPOINT_SVM_RESUMED) {
+ LOG(ERROR, "invalid section: %d, expected: %d", id,
+ CHECKPOINT_SVM_RESUMED);
+ goto out;
+ }
+
+ ok = 1;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, ok);
+}
+
+
+/* ===================== colo: wait new checkpoint ===================== */
+/*
+ * Do the following things:
+ * 1. do commit
+ * 2. wait for a new checkpoint
+ * 3. write CHECKPOINT_NEW
+ */
+static void colo_device_commit_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_start_new_checkpoint(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+
+static void libxl__colo_save_domain_wait_checkpoint_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__stream_write_state *sws = CONTAINER_OF(shs, *sws, shs);
+ libxl__domain_save_state *dss = sws->dss;
+ libxl__egc *egc = dss->sws.shs.egc;
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *const cds = &dss->cds;
+
+ cds->callback = colo_device_commit_cb;
+ libxl__checkpoint_devices_commit(egc, cds);
+}
+
+static void colo_device_commit_cb(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (rc) {
+ LOG(ERROR, "commit fails");
+ goto out;
+ }
+
+ /* TODO: wait a new checkpoint */
+ colo_start_new_checkpoint(egc, cds, 0);
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
+}
+
+static void colo_start_new_checkpoint(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc)
+{
+ libxl__colo_save_state *css = cds->concrete_data;
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+ libxl_sr_checkpoint_state srcs = { .id = CHECKPOINT_NEW };
+
+ if (rc)
+ goto out;
+
+ /* write CHECKPOINT_NEW */
+ css->callback = NULL;
+ dss->sws.checkpoint_callback = colo_common_write_stream_done;
+ libxl__stream_write_checkpoint_state(egc, &dss->sws, &srcs);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
+}
+
+
+/* ===================== colo: common callback ===================== */
+static void colo_common_write_stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(stream, *dss, sws);
+ int ok;
+
+ /* Convenience aliases */
+ libxl__colo_save_state *const css = &dss->css;
+
+ EGC_GC;
+
+ if (rc < 0) {
+ /* TODO: it may be a internal error, but we don't know */
+ LOG(ERROR, "sending data fails");
+ ok = 0;
+ goto out;
+ }
+
+ if (!css->callback) {
+ /* Everythins is OK */
+ ok = 1;
+ goto out;
+ }
+
+ css->callback(egc, css, 0);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, ok);
+}
+
+static void colo_common_read_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ int rc)
+{
+ libxl__colo_save_state *css = CONTAINER_OF(stream, *css, srs);
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+ int ok;
+
+ EGC_GC;
+
+ if (rc < 0) {
+ /* TODO: it may be a internal error, but we don't know */
+ LOG(ERROR, "reading data fails");
+ ok = 0;
+ goto out;
+ }
+
+ if (!css->callback) {
+ /* Everythins is OK */
+ ok = 1;
+ goto out;
+ }
+
+ /* rc contains the id */
+ css->callback(egc, css, rc);
+
+ return;
+
+out:
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, ok);
+}
diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
index 18946c5..0f35afe 100644
--- a/tools/libxl/libxl_dom_save.c
+++ b/tools/libxl/libxl_dom_save.c
@@ -16,6 +16,7 @@
#include "libxl_osdeps.h" /* must come before any other headers */
#include "libxl_internal.h"
+#include "libxl_colo.h"
#include <xen/errno.h>
@@ -437,12 +438,15 @@ static void domain_save_done(libxl__egc *egc,
if (dss->remus) {
/*
- * With Remus, if we reach this point, it means either
+ * With Remus/COLO, if we reach this point, it means either
* backup died or some network error occurred preventing us
* from sending checkpoints. Teardown the network buffers and
* release netlink resources. This is an async op.
*/
- libxl__remus_teardown(egc, &dss->rs, rc);
+ if (libxl_defbool_val(dss->remus->colo))
+ libxl__colo_save_teardown(egc, &dss->css, rc);
+ else
+ libxl__remus_teardown(egc, &dss->rs, rc);
return;
}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index b9ca81c..d23956c 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2802,7 +2802,7 @@ typedef struct libxl__save_helper_state {
/*
* The abstract checkpoint device layer exposes a common
* set of API to [external] libxl for manipulating devices attached to
- * a guest protected by Remus. The device layer also exposes a set of
+ * a guest protected by Remus/COLO. The device layer also exposes a set of
* [internal] interfaces that every device type must implement.
*
* The following API are exposed to libxl:
@@ -2820,7 +2820,7 @@ typedef struct libxl__save_helper_state {
* +libxl__checkpoint_devices_commit
*
* Each device type needs to implement the interfaces specified in
- * the libxl__checkpoint_device_instance_ops if it wishes to support Remus.
+ * the libxl__checkpoint_device_instance_ops if it wishes to support Remus/COLO.
*
* The high-level control flow through the checkpoint device layer is shown
* below:
@@ -2840,7 +2840,7 @@ typedef struct libxl__checkpoint_device_instance_ops libxl__checkpoint_device_in
/*
* Interfaces to be implemented by every device subkind that wishes to
- * support Remus. Functions must be implemented unless otherwise
+ * support Remus/COLO. Functions must be implemented unless otherwise
* stated. Many of these functions are asynchronous. They call
* dev->aodev.callback when done. The actual implementations may be
* synchronous and call dev->aodev.callback directly (as the last
@@ -3017,6 +3017,89 @@ static inline bool libxl__conversion_helper_inuse
(const libxl__conversion_helper_state *chs)
{ return libxl__ev_child_inuse(&chs->child); }
+/* State for manipulating a libxl migration v2 stream */
+typedef struct libxl__domain_create_state libxl__domain_create_state;
+
+typedef void libxl__domain_create_cb(libxl__egc *egc,
+ libxl__domain_create_state*,
+ int rc, uint32_t domid);
+
+typedef struct libxl__stream_read_state libxl__stream_read_state;
+
+typedef struct libxl__sr_record_buf {
+ /* private to stream read helper */
+ LIBXL_STAILQ_ENTRY(struct libxl__sr_record_buf) entry;
+ libxl__sr_rec_hdr hdr;
+ void *body; /* iff hdr.length != 0 */
+} libxl__sr_record_buf;
+
+struct libxl__stream_read_state {
+ /* filled by the user */
+ libxl__ao *ao;
+ libxl__domain_create_state *dcs;
+ int fd;
+ bool legacy;
+ bool back_channel;
+ void (*completion_callback)(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int rc);
+ void (*checkpoint_callback)(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int rc);
+ /* Private */
+ int rc;
+ bool running;
+ bool in_checkpoint;
+ bool sync_teardown; /* Only used to coordinate shutdown on error path. */
+ bool in_checkpoint_state;
+ libxl__save_helper_state shs;
+ libxl__conversion_helper_state chs;
+
+ /* Main stream-reading data. */
+ libxl__datacopier_state dc; /* Only used when reading a record */
+ libxl__sr_hdr hdr;
+ LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue; /* NOGC */
+ enum {
+ SRS_PHASE_NORMAL,
+ SRS_PHASE_BUFFERING,
+ SRS_PHASE_UNBUFFERING,
+ } phase;
+ bool recursion_guard;
+
+ /* Only used while actively reading a record from the stream. */
+ libxl__sr_record_buf *incoming_record; /* NOGC */
+
+ /* Both only used when processing an EMULATOR record. */
+ libxl__datacopier_state emu_dc;
+ libxl__carefd *emu_carefd;
+};
+
+_hidden void libxl__stream_read_init(libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_start(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_start_checkpoint(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_checkpoint_state(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_abort(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
+static inline bool
+libxl__stream_read_inuse(const libxl__stream_read_state *stream)
+{
+ return stream->running;
+}
+
+/*----- colo related state structure -----*/
+typedef struct libxl__colo_save_state libxl__colo_save_state;
+struct libxl__colo_save_state {
+ int send_fd;
+ int recv_fd;
+
+ /* private */
+ libxl__stream_read_state srs;
+ void (*callback)(libxl__egc *, libxl__colo_save_state *, int);
+ bool svm_running;
+};
/*----- Domain suspend (save) state structure -----*/
/*
@@ -3151,7 +3234,12 @@ struct libxl__domain_save_state {
int hvm;
int xcflags;
libxl__domain_suspend_state dsps;
- libxl__remus_state rs;
+ union {
+ /* for Remus */
+ libxl__remus_state rs;
+ /* for COLO */
+ libxl__colo_save_state css;
+ };
libxl__checkpoint_devices_state cds;
libxl__stream_write_state sws;
libxl__logdirty_switch logdirty;
@@ -3398,78 +3486,6 @@ _hidden int libxl__destroy_qdisk_backend(libxl__gc *gc, uint32_t domid);
/*----- Domain creation -----*/
-typedef struct libxl__domain_create_state libxl__domain_create_state;
-
-typedef void libxl__domain_create_cb(libxl__egc *egc,
- libxl__domain_create_state*,
- int rc, uint32_t domid);
-
-/* State for manipulating a libxl migration v2 stream */
-typedef struct libxl__stream_read_state libxl__stream_read_state;
-
-typedef struct libxl__sr_record_buf {
- /* private to stream read helper */
- LIBXL_STAILQ_ENTRY(struct libxl__sr_record_buf) entry;
- libxl__sr_rec_hdr hdr;
- void *body; /* iff hdr.length != 0 */
-} libxl__sr_record_buf;
-
-struct libxl__stream_read_state {
- /* filled by the user */
- libxl__ao *ao;
- libxl__domain_create_state *dcs;
- int fd;
- bool legacy;
- bool back_channel;
- void (*completion_callback)(libxl__egc *egc,
- libxl__stream_read_state *srs,
- int rc);
- void (*checkpoint_callback)(libxl__egc *egc,
- libxl__stream_read_state *srs,
- int rc);
- /* Private */
- int rc;
- bool running;
- bool in_checkpoint;
- bool sync_teardown; /* Only used to coordinate shutdown on error path. */
- bool in_checkpoint_state;
- libxl__save_helper_state shs;
- libxl__conversion_helper_state chs;
-
- /* Main stream-reading data. */
- libxl__datacopier_state dc; /* Only used when reading a record */
- libxl__sr_hdr hdr;
- LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue; /* NOGC */
- enum {
- SRS_PHASE_NORMAL,
- SRS_PHASE_BUFFERING,
- SRS_PHASE_UNBUFFERING,
- } phase;
- bool recursion_guard;
-
- /* Only used while actively reading a record from the stream. */
- libxl__sr_record_buf *incoming_record; /* NOGC */
-
- /* Both only used when processing an EMULATOR record. */
- libxl__datacopier_state emu_dc;
- libxl__carefd *emu_carefd;
-};
-
-_hidden void libxl__stream_read_init(libxl__stream_read_state *stream);
-_hidden void libxl__stream_read_start(libxl__egc *egc,
- libxl__stream_read_state *stream);
-_hidden void libxl__stream_read_start_checkpoint(libxl__egc *egc,
- libxl__stream_read_state *stream);
-_hidden void libxl__stream_read_checkpoint_state(libxl__egc *egc,
- libxl__stream_read_state *stream);
-_hidden void libxl__stream_read_abort(libxl__egc *egc,
- libxl__stream_read_state *stream, int rc);
-static inline bool
-libxl__stream_read_inuse(const libxl__stream_read_state *stream)
-{
- return stream->running;
-}
-
/* colo related structure */
typedef struct libxl__colo_restore_state libxl__colo_restore_state;
typedef void libxl__colo_callback(libxl__egc *,
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 33f4a90..9b0a537 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -763,6 +763,7 @@ libxl_domain_remus_info = Struct("domain_remus_info",[
("netbuf", libxl_defbool),
("netbufscript", string),
("diskbuf", libxl_defbool),
+ ("colo", libxl_defbool)
])
libxl_event_type = Enumeration("event_type", [
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 18/31] libxc/restore: support COLO restore
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (16 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 17/31] primary " Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:57 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 19/31] libxc/restore: send dirty pfn list to primary when checkpoint under colo Wen Congyang
` (13 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
call the callbacks resume/checkpoint/suspend while secondary vm
status is consistent with primary.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
---
tools/libxc/xc_sr_common.h | 4 +--
tools/libxc/xc_sr_restore.c | 60 +++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 62 insertions(+), 2 deletions(-)
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 2bfed64..a24a9ad 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -234,13 +234,13 @@ struct xc_sr_context
uint32_t guest_page_size;
/* Plain VM, or checkpoints over time. */
- bool checkpointed;
+ int checkpointed;
/* Currently buffering records between a checkpoint */
bool buffer_all_records;
/*
- * With Remus, we buffer the records sent by the primary at checkpoint,
+ * With Remus/COLO, we buffer the records sent by the primary at checkpoint,
* in case the primary will fail, we can recover from the last
* checkpoint state.
* This should be enough for most of the cases because primary only send
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index aef9bca..2ae8154 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -460,6 +460,49 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
else
ctx->restore.buffer_all_records = true;
+ if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+ {
+#define HANDLE_CALLBACK_RETURN_VALUE(ret) \
+ do { \
+ if ( ret == 1 ) \
+ rc = 0; /* Success */ \
+ else \
+ { \
+ if ( ret == 2 ) \
+ rc = BROKEN_CHANNEL; \
+ else \
+ rc = -1; /* Some unspecified error */ \
+ goto err; \
+ } \
+ } while (0)
+
+ /* COLO */
+
+ /* We need to resume guest */
+ rc = ctx->restore.ops.stream_complete(ctx);
+ if ( rc )
+ goto err;
+
+ /* TODO: call restore_results */
+
+ /* Resume secondary vm */
+ ret = ctx->restore.callbacks->postcopy(ctx->restore.callbacks->data);
+ HANDLE_CALLBACK_RETURN_VALUE(ret);
+
+ /* Wait for a new checkpoint */
+ ret = ctx->restore.callbacks->wait_checkpoint(
+ ctx->restore.callbacks->data);
+ HANDLE_CALLBACK_RETURN_VALUE(ret);
+
+ /* suspend secondary vm */
+ ret = ctx->restore.callbacks->suspend(ctx->restore.callbacks->data);
+ HANDLE_CALLBACK_RETURN_VALUE(ret);
+
+#undef HANDLE_CALLBACK_RETURN_VALUE
+
+ /* TODO: send dirty pfn list to primary */
+ }
+
err:
return rc;
}
@@ -631,6 +674,15 @@ static int restore(struct xc_sr_context *ctx)
} while ( rec.type != REC_TYPE_END );
remus_failover:
+
+ if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+ {
+ /* With COLO, we have already called stream_complete */
+ rc = 0;
+ IPRINTF("COLO Failover");
+ goto done;
+ }
+
/*
* With Remus, if we reach here, there must be some error on primary,
* failover from the last checkpoint state.
@@ -685,6 +737,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
if ( checkpointed_stream )
assert(callbacks->checkpoint);
+ if ( ctx.restore.checkpointed == MIG_STREAM_COLO )
+ {
+ /* this is COLO restore */
+ assert(callbacks->suspend &&
+ callbacks->postcopy &&
+ callbacks->wait_checkpoint);
+ }
+
DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
", checkpointed_stream %d", io_fd, dom, hvm, pae,
superpages, checkpointed_stream);
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 19/31] libxc/restore: send dirty pfn list to primary when checkpoint under colo
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (17 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 18/31] libxc/restore: support COLO restore Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 20/31] send store gfn and console gfn to xl before resuming secondary vm Wen Congyang
` (12 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Send dirty pfn list to primary when checkpoint under colo.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxc/xc_sr_common.h | 4 ++
tools/libxc/xc_sr_restore.c | 120 +++++++++++++++++++++++++++++++++++++++++++-
2 files changed, 123 insertions(+), 1 deletion(-)
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index a24a9ad..29ab4eb 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -226,6 +226,10 @@ struct xc_sr_context
struct xc_sr_restore_ops ops;
struct restore_callbacks *callbacks;
+ int send_back_fd;
+ unsigned long p2m_size;
+ xc_hypercall_buffer_t dirty_bitmap_hbuf;
+
/* From Image Header. */
uint32_t format_version;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index 2ae8154..37c0416 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -411,6 +411,92 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec)
return rc;
}
+/*
+ * Send checkpoint dirty pfn list to primary.
+ */
+static int send_checkpoint_dirty_pfn_list(struct xc_sr_context *ctx)
+{
+ xc_interface *xch = ctx->xch;
+ int rc = -1;
+ unsigned count, written;
+ uint64_t i, *pfns = NULL;
+ struct iovec *iov = NULL;
+ xc_shadow_op_stats_t stats = { 0, ctx->restore.p2m_size };
+ struct xc_sr_record rec =
+ {
+ .type = REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST,
+ };
+ DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+ &ctx->restore.dirty_bitmap_hbuf);
+
+ if ( xc_shadow_control(
+ xch, ctx->domid, XEN_DOMCTL_SHADOW_OP_CLEAN,
+ HYPERCALL_BUFFER(dirty_bitmap), ctx->restore.p2m_size,
+ NULL, 0, &stats) != ctx->restore.p2m_size )
+ {
+ PERROR("Failed to retrieve logdirty bitmap");
+ goto err;
+ }
+
+ for ( i = 0, count = 0; i < ctx->restore.p2m_size; i++ )
+ {
+ if ( test_bit(i, dirty_bitmap) )
+ count++;
+ }
+
+
+ pfns = malloc(count * sizeof(*pfns));
+ if ( !pfns )
+ {
+ ERROR("Unable to allocate %zu bytes of memory for dirty pfn list",
+ count * sizeof(*pfns));
+ goto err;
+ }
+
+ for ( i = 0, written = 0; i < ctx->restore.p2m_size; ++i )
+ {
+ if ( !test_bit(i, dirty_bitmap) )
+ continue;
+
+ if ( written > count )
+ {
+ ERROR("Dirty pfn list exceed");
+ goto err;
+ }
+
+ pfns[written++] = i;
+ }
+
+ /* iovec[] for writev(). */
+ iov = malloc(3 * sizeof(*iov));
+ if ( !iov )
+ {
+ ERROR("Unable to allocate memory for sending dirty bitmap");
+ goto err;
+ }
+
+ rec.length = count * sizeof(*pfns);
+
+ iov[0].iov_base = &rec.type;
+ iov[0].iov_len = sizeof(rec.type);
+
+ iov[1].iov_base = &rec.length;
+ iov[1].iov_len = sizeof(rec.length);
+
+ iov[2].iov_base = pfns;
+ iov[2].iov_len = count * sizeof(*pfns);
+
+ if ( writev_exact(ctx->restore.send_back_fd, iov, 3) )
+ {
+ PERROR("Failed to write dirty bitmap to stream");
+ goto err;
+ }
+
+ rc = 0;
+ err:
+ return rc;
+}
+
static int process_record(struct xc_sr_context *ctx, struct xc_sr_record *rec);
static int handle_checkpoint(struct xc_sr_context *ctx)
{
@@ -500,7 +586,9 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
#undef HANDLE_CALLBACK_RETURN_VALUE
- /* TODO: send dirty pfn list to primary */
+ rc = send_checkpoint_dirty_pfn_list(ctx);
+ if ( rc )
+ goto err;
}
err:
@@ -572,6 +660,21 @@ static int setup(struct xc_sr_context *ctx)
{
xc_interface *xch = ctx->xch;
int rc;
+ DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+ &ctx->restore.dirty_bitmap_hbuf);
+
+ if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+ {
+ dirty_bitmap = xc_hypercall_buffer_alloc_pages(xch, dirty_bitmap,
+ NRPAGES(bitmap_size(ctx->restore.p2m_size)));
+
+ if ( !dirty_bitmap )
+ {
+ ERROR("Unable to allocate memory for dirty bitmap");
+ rc = -1;
+ goto err;
+ }
+ }
rc = ctx->restore.ops.setup(ctx);
if ( rc )
@@ -605,10 +708,15 @@ static void cleanup(struct xc_sr_context *ctx)
{
xc_interface *xch = ctx->xch;
unsigned i;
+ DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+ &ctx->restore.dirty_bitmap_hbuf);
for ( i = 0; i < ctx->restore.buffered_rec_num; i++ )
free(ctx->restore.buffered_records[i].data);
+ if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
+ xc_hypercall_buffer_free_pages(xch, dirty_bitmap,
+ NRPAGES(bitmap_size(ctx->restore.p2m_size)));
free(ctx->restore.buffered_records);
free(ctx->restore.populated_pfns);
if ( ctx->restore.ops.cleanup(ctx) )
@@ -719,6 +827,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
int checkpointed_stream,
struct restore_callbacks *callbacks, int send_back_fd)
{
+ xen_pfn_t nr_pfns;
struct xc_sr_context ctx =
{
.xch = xch,
@@ -732,6 +841,7 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
ctx.restore.xenstore_domid = store_domid;
ctx.restore.checkpointed = checkpointed_stream;
ctx.restore.callbacks = callbacks;
+ ctx.restore.send_back_fd = send_back_fd;
/* Sanity checks for callbacks. */
if ( checkpointed_stream )
@@ -766,6 +876,14 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
if ( read_headers(&ctx) )
return -1;
+ if ( xc_domain_nr_gpfns(xch, dom, &nr_pfns) < 0 )
+ {
+ PERROR("Unable to obtain the guest p2m size");
+ return -1;
+ }
+
+ ctx.restore.p2m_size = nr_pfns;
+
if ( ctx.dominfo.hvm )
{
ctx.restore.ops = restore_ops_x86_hvm;
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 20/31] send store gfn and console gfn to xl before resuming secondary vm
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (18 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 19/31] libxc/restore: send dirty pfn list to primary when checkpoint under colo Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 21/31] libxc/save: support COLO save Wen Congyang
` (11 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
We will call libxl__xc_domain_restore_done() to rebuild secondary vm. But
we need store gfn and console gfn when rebuilding secondary vm. So make
restore_results a function pointer in callback struct and struct
{save,restore}_callbacks, and use this callback to send store gfn and
console gfn to xl.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
---
tools/libxc/include/xenguest.h | 8 ++++++++
tools/libxc/xc_sr_restore.c | 7 +++++--
tools/libxl/libxl_colo_restore.c | 5 -----
tools/libxl/libxl_create.c | 3 +++
tools/libxl/libxl_save_msgs_gen.pl | 4 ++--
5 files changed, 18 insertions(+), 9 deletions(-)
diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index cb0bbcf..67cb540 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -127,6 +127,14 @@ struct restore_callbacks {
*/
int (*wait_checkpoint)(void* data);
+ /*
+ * callback to send store gfn and console gfn to xl
+ * if we want to resume vm before xc_domain_save()
+ * exits.
+ */
+ void (*restore_results)(xen_pfn_t store_gfn, xen_pfn_t console_gfn,
+ void *data);
+
/* to be provided as the last argument to each callback function */
void* data;
};
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index 37c0416..24c4719 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -569,7 +569,9 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
if ( rc )
goto err;
- /* TODO: call restore_results */
+ ctx->restore.callbacks->restore_results(ctx->restore.xenstore_gfn,
+ ctx->restore.console_gfn,
+ ctx->restore.callbacks->data);
/* Resume secondary vm */
ret = ctx->restore.callbacks->postcopy(ctx->restore.callbacks->data);
@@ -852,7 +854,8 @@ int xc_domain_restore(xc_interface *xch, int io_fd, uint32_t dom,
/* this is COLO restore */
assert(callbacks->suspend &&
callbacks->postcopy &&
- callbacks->wait_checkpoint);
+ callbacks->wait_checkpoint &&
+ callbacks->restore_results);
}
DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index a417e6e..0ef5b5f 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -139,11 +139,6 @@ static void colo_resume_vm(libxl__egc *egc,
return;
}
- /*
- * TODO: get store gfn and console gfn
- * We should call the callback restore_results in
- * xc_domain_restore() before resuming the guest.
- */
libxl__xc_domain_restore_done(egc, dcs, 0, 0, 0);
return;
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 39458b7..f32ca98 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1014,6 +1014,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
const int checkpointed_stream = dcs->restore_params.checkpointed_stream;
libxl__colo_restore_state *const crs = &dcs->crs;
libxl_domain_build_info *const info = &d_config->b_info;
+ libxl__srm_restore_autogen_callbacks *const callbacks =
+ &dcs->srs.shs.callbacks.restore.a;
if (rc) {
domcreate_rebuild_done(egc, dcs, rc);
@@ -1041,6 +1043,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
}
/* Restore */
+ callbacks->restore_results = libxl__srm_callout_callback_restore_results;
/* COLO only supports HVM now */
if (info->type != LIBXL_DOMAIN_TYPE_HVM &&
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index 6016706..c2243f2 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -29,8 +29,8 @@ our @msgs = (
[ 6, 'srcxA', "wait_checkpoint", [] ],
[ 7, 'scxA', "switch_qemu_logdirty", [qw(int domid
unsigned enable)] ],
- [ 8, 'r', "restore_results", ['unsigned long', 'store_mfn',
- 'unsigned long', 'console_mfn'] ],
+ [ 8, 'rcx', "restore_results", ['unsigned long', 'store_gfn',
+ 'unsigned long', 'console_gfn'] ],
[ 9, 'srW', "complete", [qw(int retval
int errnoval)] ],
);
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 21/31] libxc/save: support COLO save
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (19 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 20/31] send store gfn and console gfn to xl before resuming secondary vm Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-25 15:58 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 22/31] implement the cmdline for COLO Wen Congyang
` (10 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
After suspend primary vm, get dirty bitmap on secondary vm,
and send pages both dirty on primary/secondary to secondary.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
CC: Andrew Cooper <andrew.cooper3@citrix.com>
---
tools/libxc/xc_sr_common.h | 2 +
tools/libxc/xc_sr_save.c | 104 ++++++++++++++++++++++++++++++++++++++++++---
2 files changed, 101 insertions(+), 5 deletions(-)
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 29ab4eb..5301746 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -196,6 +196,8 @@ struct xc_sr_context
{
struct /* Save data. */
{
+ int recv_fd;
+
struct xc_sr_save_ops ops;
struct save_callbacks *callbacks;
diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
index 7393355..89a5eaf 100644
--- a/tools/libxc/xc_sr_save.c
+++ b/tools/libxc/xc_sr_save.c
@@ -517,6 +517,58 @@ static int send_memory_live(struct xc_sr_context *ctx)
return rc;
}
+static int merge_secondary_dirty_bitmap(struct xc_sr_context *ctx)
+{
+ xc_interface *xch = ctx->xch;
+ struct xc_sr_record rec;
+ uint64_t *pfns = NULL;
+ uint64_t pfn;
+ unsigned count, i;
+ int rc;
+ DECLARE_HYPERCALL_BUFFER_SHADOW(unsigned long, dirty_bitmap,
+ &ctx->save.dirty_bitmap_hbuf);
+
+ rc = read_record(ctx, ctx->save.recv_fd, &rec);
+ if ( rc )
+ goto err;
+
+ if ( rec.type != REC_TYPE_CHECKPOINT_DIRTY_PFN_LIST )
+ {
+ PERROR("Expect dirty bitmap record, but received %u", rec.type );
+ rc = -1;
+ goto err;
+ }
+
+ if ( rec.length % sizeof(*pfns) )
+ {
+ PERROR("Invalid dirty pfn list record length %u", rec.length );
+ rc = -1;
+ goto err;
+ }
+
+ count = rec.length / sizeof(*pfns);
+ pfns = rec.data;
+
+ for ( i = 0; i < count; i++ )
+ {
+ pfn = pfns[i];
+ if (pfn > ctx->save.p2m_size)
+ {
+ PERROR("Invalid pfn %#lx", pfn );
+ rc = -1;
+ goto err;
+ }
+
+ set_bit(pfn, dirty_bitmap);
+ }
+
+ rc = 0;
+
+ err:
+ free(rec.data);
+ return rc;
+}
+
/*
* Suspend the domain and send dirty memory.
* This is the last iteration of the live migration and the
@@ -558,6 +610,16 @@ static int suspend_and_send_dirty(struct xc_sr_context *ctx)
bitmap_or(dirty_bitmap, ctx->save.deferred_pages, ctx->save.p2m_size);
+ if ( !ctx->save.live && ctx->save.checkpointed == MIG_STREAM_COLO )
+ {
+ rc = merge_secondary_dirty_bitmap(ctx);
+ if ( rc )
+ {
+ PERROR("Failed to get secondary vm's dirty pages");
+ goto out;
+ }
+ }
+
rc = send_dirty_pages(ctx, stats.dirty_count + ctx->save.nr_deferred_pages);
if ( rc )
goto out;
@@ -791,13 +853,42 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type)
if ( rc )
goto err;
- rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
- if ( rc <= 0 )
- goto err;
+ if ( ctx->save.checkpointed == MIG_STREAM_COLO )
+ {
+ rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
+ if ( !rc )
+ {
+ rc = -1;
+ goto err;
+ }
+ }
- rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
- if ( rc <= 0 )
+ rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
+ if ( !rc )
+ {
+ rc = -1;
goto err;
+ }
+
+ if ( ctx->save.checkpointed == MIG_STREAM_COLO )
+ {
+ rc = ctx->save.callbacks->wait_checkpoint(
+ ctx->save.callbacks->data);
+ if ( rc <= 0 )
+ goto err;
+ }
+ else if ( ctx->save.checkpointed == MIG_STREAM_REMUS )
+ {
+ rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
+ if ( rc <= 0 )
+ goto err;
+ }
+ else
+ {
+ ERROR("Unknown checkpointed stream");
+ rc = -1;
+ goto err;
+ }
}
} while ( ctx->save.checkpointed != MIG_STREAM_NONE );
@@ -843,6 +934,7 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
ctx.save.live = !!(flags & XCFLAGS_LIVE);
ctx.save.debug = !!(flags & XCFLAGS_DEBUG);
ctx.save.checkpointed = checkpointed_stream;
+ ctx.save.recv_fd = recv_fd;
/* If altering migration_stream update this assert too. */
assert(checkpointed_stream == MIG_STREAM_NONE ||
@@ -863,6 +955,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
assert(callbacks->switch_qemu_logdirty);
if ( ctx.save.checkpointed )
assert(callbacks->checkpoint && callbacks->postcopy);
+ if ( ctx.save.checkpointed == MIG_STREAM_COLO )
+ assert(callbacks->wait_checkpoint);
DPRINTF("fd %d, dom %u, max_iters %u, max_factor %u, flags %u, hvm %d",
io_fd, dom, max_iters, max_factor, flags, hvm);
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 22/31] implement the cmdline for COLO
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (20 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 21/31] libxc/save: support COLO save Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-03-02 15:03 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 23/31] COLO: introduce new API to prepare/start/do/get_error/stop replication Wen Congyang
` (9 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Add a new option -c to the command 'xl remus'. If you want
to use COLO HA instead of Remus HA, please use -c option.
Update man pages to reflect the addition of a new option to
'xl remus' command.
Also add a new option -c to the internal command 'xl migrate-receive'.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
---
docs/man/xl.pod.1 | 12 +++++++--
tools/libxl/libxl.c | 23 ++++++++++++++++--
tools/libxl/xl_cmdimpl.c | 62 ++++++++++++++++++++++++++++++++++++-----------
tools/libxl/xl_cmdtable.c | 4 ++-
4 files changed, 82 insertions(+), 19 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 4279c7c..1c6dd87 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -447,12 +447,15 @@ Print huge (!) amount of debug during the migration process.
=item B<remus> [I<OPTIONS>] I<domain-id> I<host>
-Enable Remus HA for domain. By default B<xl> relies on ssh as a transport
-mechanism between the two hosts.
+Enable Remus HA or COLO HA for domain. By default B<xl> relies on ssh as a
+transport mechanism between the two hosts.
N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
Disk replication support is limited to DRBD disks.
+ COLO support in xl is still in experimental (proof-of-concept) phase.
+ There is no support for network or disk at the moment.
+
B<OPTIONS>
=over 4
@@ -498,6 +501,11 @@ Disable network output buffering. Requires enabling unsafe mode.
Disable disk replication. Requires enabling unsafe mode.
+=item B<-c>
+
+Enable COLO HA. This conflicts with B<-i> and B<-b>, and memory
+checkpoint compression must be disabled.
+
=back
=item B<pause> I<domain-id>
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 7236175..12df81a 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -849,12 +849,28 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
goto out;
}
+ /* The caller must set this defbool */
+ if (libxl_defbool_is_default(info->colo)) {
+ LOG(ERROR, "colo mode must be enabled/disabled");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
libxl_defbool_setdefault(&info->allow_unsafe, false);
libxl_defbool_setdefault(&info->blackhole, false);
- libxl_defbool_setdefault(&info->compression, true);
+ libxl_defbool_setdefault(&info->compression,
+ !libxl_defbool_val(info->colo));
libxl_defbool_setdefault(&info->netbuf, true);
libxl_defbool_setdefault(&info->diskbuf, true);
+ if (libxl_defbool_val(info->colo)) {
+ if (libxl_defbool_val(info->compression)) {
+ LOG(ERROR, "cannot use memory checkpoint compression in COLO mode");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ }
+
if (!libxl_defbool_val(info->allow_unsafe) &&
(libxl_defbool_val(info->blackhole) ||
!libxl_defbool_val(info->netbuf) ||
@@ -876,7 +892,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
dss->live = 1;
dss->debug = 0;
dss->remus = info;
- dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
+ if (libxl_defbool_val(info->colo))
+ dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
+ else
+ dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
assert(info);
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index df7268b..0dc7220 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -4440,6 +4440,8 @@ static void migrate_receive(int debug, int daemonize, int monitor,
char rc_buf;
char *migration_domname;
struct domain_create dom_info;
+ const char *ha = checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO ?
+ "COLO" : "Remus";
signal(SIGPIPE, SIG_IGN);
/* if we get SIGPIPE we'd rather just have it as an error */
@@ -4460,6 +4462,9 @@ static void migrate_receive(int debug, int daemonize, int monitor,
dom_info.send_back_fd = send_fd;
dom_info.migration_domname_r = &migration_domname;
dom_info.checkpointed_stream = checkpointed;
+ if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
+ /* COLO uses stdout to send control message to master */
+ dom_info.quiet = 1;
rc = create_domain(&dom_info);
if (rc < 0) {
@@ -4472,11 +4477,12 @@ static void migrate_receive(int debug, int daemonize, int monitor,
switch (checkpointed) {
case LIBXL_CHECKPOINTED_STREAM_REMUS:
+ case LIBXL_CHECKPOINTED_STREAM_COLO:
/* If we are here, it means that the sender (primary) has crashed.
* TODO: Split-Brain Check.
*/
- fprintf(stderr, "migration target: Remus Failover for domain %u\n",
- domid);
+ fprintf(stderr, "migration target: %s Failover for domain %u\n",
+ ha, domid);
/*
* If domain renaming fails, lets just continue (as we need the domain
@@ -4492,16 +4498,20 @@ static void migrate_receive(int debug, int daemonize, int monitor,
rc = libxl_domain_rename(ctx, domid, migration_domname,
common_domname);
if (rc)
- fprintf(stderr, "migration target (Remus): "
+ fprintf(stderr, "migration target (%s): "
"Failed to rename domain from %s to %s:%d\n",
- migration_domname, common_domname, rc);
+ ha, migration_domname, common_domname, rc);
}
+ if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
+ /* The guest is running after failover in COLO mode */
+ exit(rc ? -ERROR_FAIL: 0);
+
rc = libxl_domain_unpause(ctx, domid);
if (rc)
- fprintf(stderr, "migration target (Remus): "
+ fprintf(stderr, "migration target (%s): "
"Failed to unpause domain %s (id: %u):%d\n",
- common_domname, domid, rc);
+ ha, common_domname, domid, rc);
exit(rc ? -ERROR_FAIL: 0);
default:
@@ -4649,7 +4659,7 @@ int main_migrate_receive(int argc, char **argv)
libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
int opt;
- SWITCH_FOREACH_OPT(opt, "Fedr", NULL, "migrate-receive", 0) {
+ SWITCH_FOREACH_OPT(opt, "Fedrc", NULL, "migrate-receive", 0) {
case 'F':
daemonize = 0;
break;
@@ -4663,6 +4673,9 @@ int main_migrate_receive(int argc, char **argv)
case 'r':
checkpointed = LIBXL_CHECKPOINTED_STREAM_REMUS;
break;
+ case 'c':
+ checkpointed = LIBXL_CHECKPOINTED_STREAM_COLO;
+ break;
}
if (argc-optind != 0) {
@@ -8032,11 +8045,8 @@ int main_remus(int argc, char **argv)
int config_len;
memset(&r_info, 0, sizeof(libxl_domain_remus_info));
- /* Defaults */
- r_info.interval = 200;
- libxl_defbool_setdefault(&r_info.blackhole, false);
- SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:e", NULL, "remus", 2) {
+ SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:ec", NULL, "remus", 2) {
case 'i':
r_info.interval = atoi(optarg);
break;
@@ -8064,11 +8074,32 @@ int main_remus(int argc, char **argv)
case 'e':
daemonize = 0;
break;
+ case 'c':
+ libxl_defbool_set(&r_info.colo, true);
}
domid = find_domain(argv[optind]);
host = argv[optind + 1];
+ /* Defaults */
+ libxl_defbool_setdefault(&r_info.blackhole, false);
+ libxl_defbool_setdefault(&r_info.colo, false);
+ if (!libxl_defbool_val(r_info.colo) && !r_info.interval)
+ r_info.interval = 200;
+
+ if (libxl_defbool_val(r_info.colo)) {
+ if (r_info.interval || libxl_defbool_val(r_info.blackhole)) {
+ perror("Option -c conflicts with -i or -b");
+ exit(-1);
+ }
+
+ if (libxl_defbool_is_default(r_info.compression)) {
+ perror("COLO can't be used with memory compression. "
+ "Disable memory checkpoint compression now...");
+ libxl_defbool_set(&r_info.compression, false);
+ }
+ }
+
if (!r_info.netbufscript)
r_info.netbufscript = default_remus_netbufscript;
@@ -8083,8 +8114,9 @@ int main_remus(int argc, char **argv)
if (!ssh_command[0]) {
rune = host;
} else {
- xasprintf(&rune, "exec %s %s xl migrate-receive -r %s",
+ xasprintf(&rune, "exec %s %s xl migrate-receive %s %s",
ssh_command, host,
+ libxl_defbool_val(r_info.colo) ? "-c" : "-r",
daemonize ? "" : " -e");
}
@@ -8112,7 +8144,8 @@ int main_remus(int argc, char **argv)
* domain to force failover
*/
if (libxl_domain_info(ctx, 0, domid)) {
- fprintf(stderr, "Remus: Primary domain has been destroyed.\n");
+ fprintf(stderr, "%s: Primary domain has been destroyed.\n",
+ libxl_defbool_val(r_info.colo) ? "COLO" : "Remus");
close(send_fd);
return 0;
}
@@ -8124,7 +8157,8 @@ int main_remus(int argc, char **argv)
if (rc == ERROR_GUEST_TIMEDOUT)
fprintf(stderr, "Failed to suspend domain at primary.\n");
else {
- fprintf(stderr, "Remus: Backup failed? resuming domain at primary.\n");
+ fprintf(stderr, "%s: Backup failed? resuming domain at primary.\n",
+ libxl_defbool_val(r_info.colo) ? "COLO" : "Remus");
libxl_domain_resume(ctx, domid, 1, 0);
}
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index fdc1ac6..b6b630c 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -499,7 +499,9 @@ struct cmd_spec cmd_table[] = {
"-b Replicate memory checkpoints to /dev/null (blackhole).\n"
" Works only in unsafe mode.\n"
"-n Disable network output buffering. Works only in unsafe mode.\n"
- "-d Disable disk replication. Works only in unsafe mode."
+ "-d Disable disk replication. Works only in unsafe mode.\n"
+ "-c Enable COLO HA. It is conflict with -i and -b, and memory\n"
+ " checkpoint must be disabled"
},
#endif
{ "devd",
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 23/31] COLO: introduce new API to prepare/start/do/get_error/stop replication
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (21 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 22/31] implement the cmdline for COLO Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-03-02 15:03 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 24/31] Support colo mode for qemu disk Wen Congyang
` (8 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
We will use qemu block replication, and qemu provides some qmp commands
to prepare replication, start replication, get replication error, and
stop replication. Introduce new API to execute these qmp commands.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxl/libxl_internal.h | 20 +++++++++
tools/libxl/libxl_qmp.c | 96 ++++++++++++++++++++++++++++++++++++++++++++
2 files changed, 116 insertions(+)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index d23956c..227b1d3 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -1771,6 +1771,26 @@ _hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool enabl
_hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const libxl_device_disk *disk);
/* Add a virtual CPU */
_hidden int libxl__qmp_cpu_add(libxl__gc *gc, int domid, int index);
+/* Start NBD server */
+_hidden int libxl__qmp_nbd_server_start(libxl__gc *gc, int domid,
+ const char *host, const char *port);
+/* Add a disk to NBD server */
+_hidden int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char *disk);
+/* Start replication */
+_hidden int libxl__qmp_start_replication(libxl__gc *gc, int domid, bool primary);
+/* Get replication error that occurs when the vm is running */
+_hidden int libxl__qmp_get_replication_error(libxl__gc *gc, int domid);
+/* Do checkpoint */
+_hidden int libxl__qmp_do_checkpoint(libxl__gc *gc, int domid);
+/* Stop replication */
+_hidden int libxl__qmp_stop_replication(libxl__gc *gc, int domid, bool primary);
+/* Stop NBD server */
+_hidden int libxl__qmp_nbd_server_stop(libxl__gc *gc, int domid);
+/* Add or remove a child to/from quorum */
+_hidden int libxl__qmp_x_blockdev_change(libxl__gc *gc, int domid, const char *parant,
+ const char *child, const char *node);
+/* run a hmp command in qmp mode */
+_hidden int libxl__qmp_hmp(libxl__gc *gc, int domid, const char *command_line);
/* close and free the QMP handler */
_hidden void libxl__qmp_close(libxl__qmp_handler *qmp);
/* remove the socket file, if the file has already been removed,
diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
index eec8a44..53860fc 100644
--- a/tools/libxl/libxl_qmp.c
+++ b/tools/libxl/libxl_qmp.c
@@ -978,6 +978,102 @@ int libxl__qmp_cpu_add(libxl__gc *gc, int domid, int idx)
return qmp_run_command(gc, domid, "cpu-add", args, NULL, NULL);
}
+int libxl__qmp_nbd_server_start(libxl__gc *gc, int domid,
+ const char *host, const char *port)
+{
+ libxl__json_object *args = NULL;
+ libxl__json_object *addr = NULL;
+ libxl__json_object *data = NULL;
+
+ /* 'addr': {
+ * 'type': 'inet',
+ * 'data': {
+ * 'host': '$nbd_host',
+ * 'port': '$nbd_port'
+ * }
+ * }
+ */
+ qmp_parameters_add_string(gc, &data, "host", host);
+ qmp_parameters_add_string(gc, &data, "port", port);
+
+ qmp_parameters_add_string(gc, &addr, "type", "inet");
+ qmp_parameters_common_add(gc, &addr, "data", data);
+
+ qmp_parameters_common_add(gc, &args, "addr", addr);
+
+ return qmp_run_command(gc, domid, "nbd-server-start", args, NULL, NULL);
+}
+
+int libxl__qmp_nbd_server_add(libxl__gc *gc, int domid, const char *disk)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_string(gc, &args, "device", disk);
+ qmp_parameters_add_bool(gc, &args, "writable", true);
+
+ return qmp_run_command(gc, domid, "nbd-server-add", args, NULL, NULL);
+}
+
+int libxl__qmp_start_replication(libxl__gc *gc, int domid, bool primary)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_bool(gc, &args, "enable", true);
+ qmp_parameters_add_bool(gc, &args, "primary", primary);
+
+ return qmp_run_command(gc, domid, "xen-set-replication", args, NULL, NULL);
+}
+
+int libxl__qmp_get_replication_error(libxl__gc *gc, int domid)
+{
+ return qmp_run_command(gc, domid, "xen-get-replication-error", NULL,
+ NULL, NULL);
+}
+
+int libxl__qmp_do_checkpoint(libxl__gc *gc, int domid)
+{
+ return qmp_run_command(gc, domid, "xen-do-checkpoint", NULL, NULL, NULL);
+}
+
+int libxl__qmp_stop_replication(libxl__gc *gc, int domid, bool primary)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_bool(gc, &args, "enable", false);
+ qmp_parameters_add_bool(gc, &args, "primary", primary);
+
+ return qmp_run_command(gc, domid, "xen-set-replication", args, NULL, NULL);
+}
+
+int libxl__qmp_nbd_server_stop(libxl__gc *gc, int domid)
+{
+ return qmp_run_command(gc, domid, "nbd-server-stop", NULL, NULL, NULL);
+}
+
+int libxl__qmp_x_blockdev_change(libxl__gc *gc, int domid, const char *parent,
+ const char *child, const char *node)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_string(gc, &args, "parent", parent);
+ if (child)
+ qmp_parameters_add_string(gc, &args, "child", child);
+ if (node)
+ qmp_parameters_add_string(gc, &args, "node", node);
+
+ return qmp_run_command(gc, domid, "x-blockdev-change", args, NULL, NULL);
+}
+
+int libxl__qmp_hmp(libxl__gc *gc, int domid, const char *command_line)
+{
+ libxl__json_object *args = NULL;
+
+ qmp_parameters_add_string(gc, &args, "command-line", command_line);
+
+ return qmp_run_command(gc, domid, "human-monitor-command", args,
+ NULL, NULL);
+}
+
int libxl__qmp_initializations(libxl__gc *gc, uint32_t domid,
const libxl_domain_config *guest_config)
{
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 24/31] Support colo mode for qemu disk
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (22 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 23/31] COLO: introduce new API to prepare/start/do/get_error/stop replication Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-03-02 15:04 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 25/31] COLO: use qemu block replication Wen Congyang
` (7 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Usage: disk = ['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
For QEMU block replication details:
http://wiki.qemu.org/Features/BlockReplication
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
---
docs/man/xl.pod.1 | 2 +-
docs/misc/xl-disk-configuration.txt | 50 ++++++++++
tools/libxl/libxl.c | 62 +++++++++++-
tools/libxl/libxl_create.c | 25 ++++-
tools/libxl/libxl_device.c | 54 +++++++++++
tools/libxl/libxl_dm.c | 184 ++++++++++++++++++++++++++++++++++--
tools/libxl/libxl_types.idl | 7 ++
tools/libxl/libxlu_disk_l.l | 7 ++
8 files changed, 382 insertions(+), 9 deletions(-)
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 1c6dd87..4f1901d 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -454,7 +454,7 @@ N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
Disk replication support is limited to DRBD disks.
COLO support in xl is still in experimental (proof-of-concept) phase.
- There is no support for network or disk at the moment.
+ There is no support for network at the moment.
B<OPTIONS>
diff --git a/docs/misc/xl-disk-configuration.txt b/docs/misc/xl-disk-configuration.txt
index 29f6ddb..6f23c2d 100644
--- a/docs/misc/xl-disk-configuration.txt
+++ b/docs/misc/xl-disk-configuration.txt
@@ -234,6 +234,56 @@ were intentionally created non-sparse to avoid fragmentation of the
file.
+===============
+COLO PARAMETERS
+===============
+
+
+colo
+----
+
+Enable COLO HA for disk. For better understanding block replication on
+QEMU, please refer to:
+http://wiki.qemu.org/Features/BlockReplication
+
+
+colo-host
+---------
+Description: Secondary host's address
+Mandatory: Yes when COLO enabled
+
+
+colo-port
+---------
+Description: Secondary port
+ We will run a nbd server on secondary host,
+ and the nbd server will listen this port.
+Mandatory: Yes when COLO enabled
+
+
+colo-export
+---------
+Description: We will run a nbd server on secondary host,
+ exportname is the nbd server's disk export name.
+Mandatory: Yes when COLO enabled
+
+
+active-disk
+-----------
+
+Description: This is used by secondary. Secondary guest's write
+ will be buffered in this disk.
+Mandatory: Yes when COLO enabled
+
+
+hidden-disk
+-----------
+
+Description: This is used by secondary. It buffers the original
+ content that is modified by the primary VM.
+Mandatory: Yes when COLO enabled
+
+
============================================
DEPRECATED PARAMETERS, PREFIXES AND SYNTAXES
============================================
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index 12df81a..f691628 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -2309,6 +2309,8 @@ int libxl__device_disk_setdefault(libxl__gc *gc, libxl_device_disk *disk)
int rc;
libxl_defbool_setdefault(&disk->discard_enable, !!disk->readwrite);
+ libxl_defbool_setdefault(&disk->colo_enable, false);
+ libxl_defbool_setdefault(&disk->colo_restore_enable, false);
rc = libxl__resolve_domid(gc, disk->backend_domname, &disk->backend_domid);
if (rc < 0) return rc;
@@ -2507,6 +2509,18 @@ static void device_disk_add(libxl__egc *egc, uint32_t domid,
flexarray_append(back, "params");
flexarray_append(back, GCSPRINTF("%s:%s",
libxl__device_disk_string_of_format(disk->format), disk->pdev_path));
+ if (libxl_defbool_val(disk->colo_enable)) {
+ flexarray_append(back, "colo-host");
+ flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_host));
+ flexarray_append(back, "colo-port");
+ flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_port));
+ flexarray_append(back, "colo-export");
+ flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_export));
+ flexarray_append(back, "active-disk");
+ flexarray_append(back, libxl__sprintf(gc, "%s", disk->active_disk));
+ flexarray_append(back, "hidden-disk");
+ flexarray_append(back, libxl__sprintf(gc, "%s", disk->hidden_disk));
+ }
assert(device->backend_kind == LIBXL__DEVICE_KIND_QDISK);
break;
default:
@@ -2622,7 +2636,10 @@ static int libxl__device_disk_from_xs_be(libxl__gc *gc,
goto cleanup;
}
- /* "params" may not be present; but everything else must be. */
+ /*
+ * "params" and "colo-host" may not be present; but everything
+ * else must be.
+ */
tmp = xs_read(ctx->xsh, XBT_NULL,
GCSPRINTF("%s/params", be_path), &len);
if (tmp && strchr(tmp, ':')) {
@@ -2632,6 +2649,49 @@ static int libxl__device_disk_from_xs_be(libxl__gc *gc,
disk->pdev_path = tmp;
}
+ tmp = xs_read(ctx->xsh, XBT_NULL,
+ GCSPRINTF("%s/colo-host", be_path), &len);
+ if (tmp) {
+ libxl_defbool_set(&disk->colo_enable, true);
+ disk->colo_host = tmp;
+ } else {
+ libxl_defbool_set(&disk->colo_enable, false);
+ }
+
+ if (libxl_defbool_val(disk->colo_enable)) {
+ tmp = xs_read(ctx->xsh, XBT_NULL,
+ GCSPRINTF("%s/colo-port", be_path), &len);
+ if (!tmp) {
+ LOG(ERROR, "Missing xenstore node %s/colo-port", be_path);
+ goto cleanup;
+ }
+ disk->colo_port = tmp;
+
+ tmp = xs_read(ctx->xsh, XBT_NULL,
+ GCSPRINTF("%s/colo-export", be_path), &len);
+ if (!tmp) {
+ LOG(ERROR, "Missing xenstore node %s/colo-export", be_path);
+ goto cleanup;
+ }
+ disk->colo_export = tmp;
+
+ tmp = xs_read(ctx->xsh, XBT_NULL,
+ GCSPRINTF("%s/active-disk", be_path), &len);
+ if (!tmp) {
+ LOG(ERROR, "Missing xenstore node %s/active-disk", be_path);
+ goto cleanup;
+ }
+ disk->active_disk = tmp;
+
+ tmp = xs_read(ctx->xsh, XBT_NULL,
+ GCSPRINTF("%s/hidden-disk", be_path), &len);
+ if (!tmp) {
+ LOG(ERROR, "Missing xenstore node %s/hidden-disk", be_path);
+ goto cleanup;
+ }
+ disk->hidden_disk = tmp;
+ }
+
tmp = libxl__xs_read(gc, XBT_NULL,
GCSPRINTF("%s/type", be_path));
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index f32ca98..c68dd49 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1801,12 +1801,29 @@ static void domain_create_cb(libxl__egc *egc,
libxl__ao_complete(egc, ao, rc);
}
-
+
+static void set_disk_colo_restore(libxl_domain_config *d_config)
+{
+ int i;
+
+ for (i = 0; i < d_config->num_disks; i++)
+ libxl_defbool_set(&d_config->disks[i].colo_restore_enable, true);
+}
+
+static void unset_disk_colo_restore(libxl_domain_config *d_config)
+{
+ int i;
+
+ for (i = 0; i < d_config->num_disks; i++)
+ libxl_defbool_set(&d_config->disks[i].colo_restore_enable, false);
+}
+
int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
+ unset_disk_colo_restore(d_config);
return do_domain_create(ctx, d_config, domid, -1, -1, NULL,
ao_how, aop_console_how);
}
@@ -1818,6 +1835,12 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
+ if (params->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
+ set_disk_colo_restore(d_config);
+ } else {
+ unset_disk_colo_restore(d_config);
+ }
+
return do_domain_create(ctx, d_config, domid, restore_fd, send_back_fd,
params, ao_how, aop_console_how);
}
diff --git a/tools/libxl/libxl_device.c b/tools/libxl/libxl_device.c
index 8bb5e93..039afc6 100644
--- a/tools/libxl/libxl_device.c
+++ b/tools/libxl/libxl_device.c
@@ -196,6 +196,10 @@ static int disk_try_backend(disk_try_backend_args *a,
goto bad_format;
}
+ if (libxl_defbool_val(a->disk->colo_enable) ||
+ a->disk->active_disk || a->disk->hidden_disk)
+ goto bad_colo;
+
if (a->disk->backend_domid != LIBXL_TOOLSTACK_DOMID) {
LOG(DEBUG, "Disk vdev=%s, is using a storage driver domain, "
"skipping physical device check", a->disk->vdev);
@@ -218,6 +222,10 @@ static int disk_try_backend(disk_try_backend_args *a,
case LIBXL_DISK_BACKEND_TAP:
if (a->disk->script) goto bad_script;
+ if (libxl_defbool_val(a->disk->colo_enable) ||
+ a->disk->active_disk || a->disk->hidden_disk)
+ goto bad_colo;
+
if (a->disk->is_cdrom) {
LOG(DEBUG, "Disk vdev=%s, backend tap unsuitable for cdroms",
a->disk->vdev);
@@ -236,6 +244,22 @@ static int disk_try_backend(disk_try_backend_args *a,
case LIBXL_DISK_BACKEND_QDISK:
if (a->disk->script) goto bad_script;
+ if (libxl_defbool_val(a->disk->colo_enable)) {
+ if (!a->disk->colo_host)
+ goto bad_colo_host;
+
+ if (!a->disk->colo_port)
+ goto bad_colo_port;
+
+ if (!a->disk->colo_export)
+ goto bad_colo_export;
+
+ if (!a->disk->active_disk)
+ goto bad_active_disk;
+
+ if (!a->disk->hidden_disk)
+ goto bad_hidden_disk;
+ }
return backend;
default:
@@ -256,6 +280,36 @@ static int disk_try_backend(disk_try_backend_args *a,
LOG(DEBUG, "Disk vdev=%s, backend %s not compatible with script=...",
a->disk->vdev, libxl_disk_backend_to_string(backend));
return 0;
+
+ bad_colo:
+ LOG(DEBUG, "Disk vdev=%s, backend %s not compatible with colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
+
+ bad_colo_host:
+ LOG(DEBUG, "Disk vdev=%s, backend %s needs colo-host=... for colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
+
+ bad_colo_port:
+ LOG(DEBUG, "Disk vdev=%s, backend %s needs colo-port=... for colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
+
+ bad_colo_export:
+ LOG(DEBUG, "Disk vdev=%s, backend %s needs colo-export=... for colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
+
+ bad_active_disk:
+ LOG(DEBUG, "Disk vdev=%s, backend %s needs active-disk=... for colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
+
+ bad_hidden_disk:
+ LOG(DEBUG, "Disk vdev=%s, backend %s needs hidden-disk=... for colo",
+ a->disk->vdev, libxl_disk_backend_to_string(backend));
+ return 0;
}
int libxl__device_disk_set_backend(libxl__gc *gc, libxl_device_disk *disk) {
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index 4aca38e..ba17251 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -751,6 +751,139 @@ static int libxl__dm_runas_helper(libxl__gc *gc, const char *username)
}
}
+/* colo mode */
+enum {
+ LIBXL__COLO_NONE = 0,
+ LIBXL__COLO_PRIMARY,
+ LIBXL__COLO_SECONDARY,
+};
+
+static char *qemu_disk_scsi_drive_string(libxl__gc *gc, const char *pdev_path,
+ int unit, const char *format,
+ const libxl_device_disk *disk,
+ int colo_mode)
+{
+ char *drive = NULL;
+ const char *exportname = disk->colo_export;
+ const char *active_disk = disk->active_disk;
+ const char *hidden_disk = disk->hidden_disk;
+
+ switch (colo_mode) {
+ case LIBXL__COLO_NONE:
+ drive = libxl__sprintf
+ (gc, "file=%s,if=scsi,bus=0,unit=%d,format=%s,cache=writeback",
+ pdev_path, unit, format);
+ break;
+ case LIBXL__COLO_PRIMARY:
+ /*
+ * primary:
+ * -dirve if=scsi,bus=0,unit=x,cache=writeback,driver=quorum,\
+ * id=exportname,\
+ * children.0.file.filename=pdev_path,\
+ * children.0.driver=format,\
+ * read-pattern=fifo,\
+ * vote-threshold=1
+ */
+ drive = GCSPRINTF(
+ "if=scsi,bus=0,unit=%d,cache=writeback,driver=quorum,"
+ "id=%s,"
+ "children.0.file.filename=%s,"
+ "children.0.driver=%s,"
+ "read-pattern=fifo,"
+ "vote-threshold=1",
+ unit, exportname, pdev_path, format);
+ break;
+ case LIBXL__COLO_SECONDARY:
+ /*
+ * secondary:
+ * -drive if=scsi,bus=0,unit=x,cache=writeback,driver=replication,\
+ * mode=secondary,\
+ * file.driver=qcow2,\
+ * file.file.filename=active_disk,\
+ * file.backing.driver=qcow2,\
+ * file.backing.file.filename=hidden_disk,\
+ * file.backing.backing=exportname,
+ */
+ drive = GCSPRINTF(
+ "if=scsi,bus=0,unit=%d,cache=writeback,driver=replication,"
+ "mode=secondary,"
+ "file.driver=qcow2,"
+ "file.file.filename=%s,"
+ "file.backing.driver=qcow2,"
+ "file.backing.file.filename=%s,"
+ "file.backing.backing=%s",
+ unit, active_disk, hidden_disk, exportname);
+ break;
+ default:
+ abort();
+ }
+
+ return drive;
+}
+
+static char *qemu_disk_ide_drive_string(libxl__gc *gc, const char *pdev_path,
+ int unit, const char *format,
+ const libxl_device_disk *disk,
+ int colo_mode)
+{
+ char *drive = NULL;
+ const char *exportname = disk->colo_export;
+ const char *active_disk = disk->active_disk;
+ const char *hidden_disk = disk->hidden_disk;
+
+ switch (colo_mode) {
+ case LIBXL__COLO_NONE:
+ drive = GCSPRINTF
+ ("file=%s,if=ide,index=%d,media=disk,format=%s,cache=writeback",
+ pdev_path, unit, format);
+ break;
+ case LIBXL__COLO_PRIMARY:
+ /*
+ * primary:
+ * -dirve if=ide,index=x,media=disk,cache=writeback,driver=quorum,\
+ * id=exportname,\
+ * children.0.file.filename=pdev_path,\
+ * children.0.driver=format,\
+ * read-pattern=fifo,\
+ * vote-threshold=1
+ */
+ drive = GCSPRINTF(
+ "if=ide,index=%d,media=disk,cache=writeback,driver=quorum,"
+ "id=%s,"
+ "children.0.file.filename=%s,"
+ "children.0.driver=%s,"
+ "read-pattern=fifo,"
+ "vote-threshold=1",
+ unit, exportname, pdev_path, format);
+ break;
+ case LIBXL__COLO_SECONDARY:
+ /*
+ * secondary:
+ * -drive if=ide,index=x,media=disk,cache=writeback,driver=replication,\
+ * mode=secondary,\
+ * file.driver=qcow2,\
+ * file.file.filename=active_disk,\
+ * file.backing.driver=qcow2,\
+ * file.backing.file.filename=hidden_disk,\
+ * file.backing.backing=exportname,
+ */
+ drive = GCSPRINTF(
+ "if=ide,index=%d,media=disk,cache=writeback,driver=replication,"
+ "mode=secondary,"
+ "file.driver=qcow2,"
+ "file.file.filename=%s,"
+ "file.backing.driver=qcow2,"
+ "file.backing.file.filename=%s,"
+ "file.backing.backing=%s",
+ unit, active_disk, hidden_disk, exportname);
+ break;
+ default:
+ abort();
+ }
+
+ return drive;
+}
+
static int libxl__build_device_model_args_new(libxl__gc *gc,
const char *dm, int guest_domid,
const libxl_domain_config *guest_config,
@@ -1164,6 +1297,7 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
const char *format = qemu_disk_format_string(disks[i].format);
char *drive;
const char *pdev_path;
+ int colo_mode;
if (dev_number == -1) {
LOG(WARN, "unable to determine"" disk number for %s",
@@ -1208,10 +1342,32 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
* For other disks we translate devices 0..3 into
* hd[a-d] and ignore the rest.
*/
+ if (libxl_defbool_val(disks[i].colo_enable)) {
+ if (libxl_defbool_val(disks[i].colo_restore_enable))
+ colo_mode = LIBXL__COLO_SECONDARY;
+ else
+ colo_mode = LIBXL__COLO_PRIMARY;
+ } else {
+ colo_mode = LIBXL__COLO_NONE;
+ }
+
if (strncmp(disks[i].vdev, "sd", 2) == 0) {
- drive = libxl__sprintf
- (gc, "file=%s,if=scsi,bus=0,unit=%d,format=%s,readonly=%s,cache=writeback",
- pdev_path, disk, format, disks[i].readwrite ? "off" : "on");
+ if (colo_mode == LIBXL__COLO_SECONDARY) {
+ /*
+ * -drive if=none,driver=format,file=pdev_path,\
+ * id=exportname
+ */
+ drive = libxl__sprintf
+ (gc, "if=none,driver=%s,file=%s,id=%s",
+ format, pdev_path, disks[i].colo_export);
+
+ flexarray_append(dm_args, "-drive");
+ flexarray_append(dm_args, drive);
+ }
+ drive = qemu_disk_scsi_drive_string(gc, pdev_path, disk,
+ format,
+ &disks[i],
+ colo_mode);
} else if (strncmp(disks[i].vdev, "xvd", 3) == 0) {
/*
* Do not add any emulated disk when PV disk are
@@ -1234,12 +1390,28 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
LOG(ERROR, "qemu-xen doesn't support read-only IDE disk drivers");
return ERROR_INVAL;
}
- drive = libxl__sprintf
- (gc, "file=%s,if=ide,index=%d,media=disk,format=%s,cache=writeback",
- pdev_path, disk, format);
+ if (colo_mode == LIBXL__COLO_SECONDARY) {
+ /*
+ * -drive if=none,driver=format,file=pdev_path,\
+ * id=exportname
+ */
+ drive = libxl__sprintf
+ (gc, "if=none,driver=%s,file=%s,id=%s",
+ format, pdev_path, disks[i].colo_export);
+
+ flexarray_append(dm_args, "-drive");
+ flexarray_append(dm_args, drive);
+ }
+ drive = qemu_disk_ide_drive_string(gc, pdev_path, disk,
+ format,
+ &disks[i],
+ colo_mode);
} else {
continue; /* Do not emulate this disk */
}
+
+ if (!drive)
+ continue;
}
flexarray_append(dm_args, "-drive");
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 9b0a537..a2078d1 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -575,6 +575,13 @@ libxl_device_disk = Struct("device_disk", [
("is_cdrom", integer),
("direct_io_safe", bool),
("discard_enable", libxl_defbool),
+ ("colo_enable", libxl_defbool),
+ ("colo_restore_enable", libxl_defbool),
+ ("colo_host", string),
+ ("colo_port", string),
+ ("colo_export", string),
+ ("active_disk", string),
+ ("hidden_disk", string)
])
libxl_device_nic = Struct("device_nic", [
diff --git a/tools/libxl/libxlu_disk_l.l b/tools/libxl/libxlu_disk_l.l
index 1a5deb5..58da943 100644
--- a/tools/libxl/libxlu_disk_l.l
+++ b/tools/libxl/libxlu_disk_l.l
@@ -176,6 +176,13 @@ script=[^,]*,? { STRIP(','); SAVESTRING("script", script, FROMEQUALS); }
direct-io-safe,? { DPC->disk->direct_io_safe = 1; }
discard,? { libxl_defbool_set(&DPC->disk->discard_enable, true); }
no-discard,? { libxl_defbool_set(&DPC->disk->discard_enable, false); }
+colo,? { libxl_defbool_set(&DPC->disk->colo_enable, true); }
+no-colo,? { libxl_defbool_set(&DPC->disk->colo_enable, false); }
+colo-host=[^,]*,? { STRIP(','); SAVESTRING("colo-host", colo_host, FROMEQUALS); }
+colo-port=[^,]*,? { STRIP(','); SAVESTRING("colo-port", colo_port, FROMEQUALS); }
+colo-export=[^,]*,? { STRIP(','); SAVESTRING("colo-export", colo_export, FROMEQUALS); }
+active-disk=[^,]*,? { STRIP(','); SAVESTRING("active-disk", active_disk, FROMEQUALS); }
+hidden-disk=[^,]*,? { STRIP(','); SAVESTRING("hidden-disk", hidden_disk, FROMEQUALS); }
/* the target magic parameter, eats the rest of the string */
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 25/31] COLO: use qemu block replication
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (23 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 24/31] Support colo mode for qemu disk Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-03-02 15:03 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module Wen Congyang
` (6 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Use qemu block replication as our block replication solution.
Note that guest must be paused before starting COLO, otherwise,
the disk won't be consistent between primary and secondary.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
---
tools/libxl/Makefile | 1 +
tools/libxl/libxl_colo_qdisk.c | 226 +++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_colo_restore.c | 42 +++++++-
tools/libxl/libxl_colo_save.c | 54 +++++++++-
tools/libxl/libxl_internal.h | 13 +++
5 files changed, 331 insertions(+), 5 deletions(-)
create mode 100644 tools/libxl/libxl_colo_qdisk.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 2016393..28d54d0 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -66,6 +66,7 @@ endif
LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
+LIBXL_OBJS-y += libxl_colo_qdisk.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo_qdisk.c b/tools/libxl/libxl_colo_qdisk.c
new file mode 100644
index 0000000..6179947
--- /dev/null
+++ b/tools/libxl/libxl_colo_qdisk.c
@@ -0,0 +1,226 @@
+/*
+ * Copyright (C) 2015 FUJITSU LIMITED
+ * Author: Wen Congyang <wency@cn.fujitsu.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+typedef struct libxl__colo_qdisk {
+ bool setuped;
+} libxl__colo_qdisk;
+
+/* ========== init() and cleanup() ========== */
+int init_subkind_qdisk(libxl__checkpoint_devices_state *cds)
+{
+ /*
+ * We don't know if we use qemu block replication, so
+ * we cannot start block replication here.
+ */
+ return 0;
+}
+
+void cleanup_subkind_qdisk(libxl__checkpoint_devices_state *cds)
+{
+}
+
+/* ========== setup() and teardown() ========== */
+static void colo_qdisk_setup(libxl__egc *egc, libxl__checkpoint_device *dev,
+ bool primary)
+{
+ const libxl_device_disk *disk = dev->backend_dev;
+ int ret, rc = 0;
+ libxl__colo_qdisk *colo_qdisk = NULL;
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *const cds = dev->cds;
+ const char *host = disk->colo_host;
+ const char *port = disk->colo_port;
+ const char *export_name = disk->colo_export;
+ const int domid = cds->domid;
+
+ STATE_AO_GC(dev->cds->ao);
+
+ if (disk->backend != LIBXL_DISK_BACKEND_QDISK ||
+ !libxl_defbool_val(disk->colo_enable)) {
+ rc = ERROR_CHECKPOINT_DEVOPS_DOES_NOT_MATCH;
+ goto out;
+ }
+
+ dev->matched = true;
+
+ GCNEW(colo_qdisk);
+ dev->concrete_data = colo_qdisk;
+
+ if (primary) {
+ libxl__colo_save_state *css = cds->concrete_data;
+
+ css->qdisk_used = true;
+ /* NBD server is not ready, so we cannot start block replication now */
+ goto out;
+ } else {
+ libxl__colo_restore_state *crs = cds->concrete_data;
+
+ if (!crs->qdisk_used) {
+ /* start nbd server */
+ ret = libxl__qmp_nbd_server_start(gc, domid, host, port);
+ if (ret) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ crs->host = host;
+ crs->port = port;
+ } else {
+ if (strcmp(crs->host, host) || strcmp(crs->port, port)) {
+ LOG(ERROR, "The host and port of all disks must be the same");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+ }
+
+ crs->qdisk_used = true;
+
+ ret = libxl__qmp_nbd_server_add(gc, domid, export_name);
+ if (ret)
+ rc = ERROR_FAIL;
+
+ colo_qdisk->setuped = true;
+ }
+
+out:
+ dev->aodev.rc = rc;
+ dev->aodev.callback(egc, &dev->aodev);
+}
+
+static void colo_qdisk_teardown(libxl__egc *egc, libxl__checkpoint_device *dev,
+ bool primary)
+{
+ int ret, rc = 0;
+ const libxl__colo_qdisk *colo_qdisk = dev->concrete_data;
+ const libxl_device_disk *disk = dev->backend_dev;
+
+ /* Convenience aliases */
+ libxl__checkpoint_devices_state *const cds = dev->cds;
+ const int domid = cds->domid;
+ const char *export_name = disk->colo_export;
+
+ EGC_GC;
+
+ if (primary) {
+ if (!colo_qdisk->setuped)
+ goto out;
+
+ /*
+ * There is no way to get the child name, but we know it is children.1
+ */
+ ret = libxl__qmp_x_blockdev_change(gc, domid, export_name,
+ "children.1", NULL);
+ if (ret)
+ rc = ERROR_FAIL;
+ } else {
+ libxl__colo_restore_state *crs = cds->concrete_data;
+
+ if (crs->qdisk_used) {
+ ret = libxl__qmp_nbd_server_stop(gc, domid);
+ if (ret)
+ rc = ERROR_FAIL;
+ }
+ }
+
+out:
+ dev->aodev.rc = rc;
+ dev->aodev.callback(egc, &dev->aodev);
+}
+
+/* ========== checkpointing APIs ========== */
+static void colo_qdisk_save_preresume(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ libxl__colo_qdisk *colo_qdisk = dev->concrete_data;
+ const libxl_device_disk *disk = dev->backend_dev;
+ int ret, rc = 0;
+ char *node = NULL;
+ char *cmd = NULL;
+
+ /* Convenience aliases */
+ const int domid = dev->cds->domid;
+ const char *host = disk->colo_host;
+ const char *port = disk->colo_port;
+ const char *export_name = disk->colo_export;
+
+ EGC_GC;
+
+ if (colo_qdisk->setuped)
+ goto out;
+
+ /* qmp command doesn't support the driver "nbd" */
+ node = GCSPRINTF("colo_node%d",
+ libxl__device_disk_dev_number(disk->vdev, NULL, NULL));
+ cmd = GCSPRINTF("drive_add buddy driver=replication,mode=primary,"
+ "file.driver=nbd,file.host=%s,file.port=%s,"
+ "file.export=%s,node-name=%s,if=none",
+ host, port, export_name, node);
+ ret = libxl__qmp_hmp(gc, domid, cmd);
+ if (ret)
+ rc = ERROR_FAIL;
+
+ ret = libxl__qmp_x_blockdev_change(gc, domid, export_name, NULL, node);
+ if (ret)
+ rc = ERROR_FAIL;
+
+ colo_qdisk->setuped = true;
+
+out:
+ dev->aodev.rc = rc;
+ dev->aodev.callback(egc, &dev->aodev);
+}
+
+/* ======== primary ======== */
+static void colo_qdisk_save_setup(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ colo_qdisk_setup(egc, dev, true);
+}
+
+static void colo_qdisk_save_teardown(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ colo_qdisk_teardown(egc, dev, true);
+}
+
+const libxl__checkpoint_device_instance_ops colo_save_device_qdisk = {
+ .kind = LIBXL__DEVICE_KIND_VBD,
+ .setup = colo_qdisk_save_setup,
+ .teardown = colo_qdisk_save_teardown,
+ .preresume = colo_qdisk_save_preresume,
+};
+
+/* ======== secondary ======== */
+static void colo_qdisk_restore_setup(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ colo_qdisk_setup(egc, dev, false);
+}
+
+static void colo_qdisk_restore_teardown(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ colo_qdisk_teardown(egc, dev, false);
+}
+
+const libxl__checkpoint_device_instance_ops colo_restore_device_qdisk = {
+ .kind = LIBXL__DEVICE_KIND_VBD,
+ .setup = colo_qdisk_restore_setup,
+ .teardown = colo_qdisk_restore_teardown,
+};
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index 0ef5b5f..eff80ca 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -50,7 +50,10 @@ static void libxl__colo_restore_domain_checkpoint_callback(void *data);
static void libxl__colo_restore_domain_wait_checkpoint_callback(void *data);
static void libxl__colo_restore_domain_suspend_callback(void *data);
+extern const libxl__checkpoint_device_instance_ops colo_restore_device_qdisk;
+
static const libxl__checkpoint_device_instance_ops *colo_restore_ops[] = {
+ &colo_restore_device_qdisk,
NULL,
};
@@ -150,7 +153,11 @@ static int init_device_subkind(libxl__checkpoint_devices_state *cds)
int rc;
STATE_AO_GC(cds->ao);
+ rc = init_subkind_qdisk(cds);
+ if (rc) goto out;
+
rc = 0;
+out:
return rc;
}
@@ -158,6 +165,8 @@ static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
{
/* cleanup device subkind-specific state in the libxl ctx */
STATE_AO_GC(cds->ao);
+
+ cleanup_subkind_qdisk(cds);
}
@@ -214,6 +223,8 @@ void libxl__colo_restore_setup(libxl__egc *egc,
GCNEW(crcs);
crs->crcs = crcs;
crcs->crs = crs;
+ crs->qdisk_setuped = false;
+ crs->qdisk_used = false;
/* setup dsps */
crcs->dsps.ao = ao;
@@ -305,6 +316,11 @@ void libxl__colo_restore_teardown(libxl__egc *egc, void *dcs_void,
}
libxl__xc_domain_restore_done(egc, dcs, ret, retval, errnoval);
+ if (crs->qdisk_setuped) {
+ libxl__qmp_stop_replication(gc, crs->domid, false);
+ crs->qdisk_setuped = false;
+ }
+
crcs->saved_rc = rc;
if (!crcs->teardown_devices) {
colo_restore_teardown_devices_done(egc, &dcs->cds, 0);
@@ -582,6 +598,13 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
goto out;
}
+ if (crs->qdisk_setuped) {
+ if (libxl__qmp_do_checkpoint(gc, crs->domid)) {
+ LOG(ERROR, "doing checkpoint fails");
+ goto out;
+ }
+ }
+
colo_restore_resume_vm(egc, crcs);
return;
@@ -739,8 +762,8 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
STATE_AO_GC(crs->ao);
- /* TODO: disk/nic support */
- cds->device_kind_flags = 0;
+ /* TODO: nic support */
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
cds->callback = colo_restore_setup_cds_done;
cds->ao = ao;
cds->domid = crs->domid;
@@ -777,6 +800,14 @@ static void colo_restore_setup_cds_done(libxl__egc *egc,
goto out;
}
+ if (crs->qdisk_used && !crs->qdisk_setuped) {
+ if (libxl__qmp_start_replication(gc, crs->domid, false)) {
+ LOG(ERROR, "starting replication fails");
+ goto out;
+ }
+ crs->qdisk_setuped = true;
+ }
+
colo_send_svm_ready(egc, crcs);
return;
@@ -931,13 +962,18 @@ static void colo_suspend_vm_done(libxl__egc *egc,
crcs->status = LIBXL_COLO_SUSPENDED;
+ if (libxl__qmp_get_replication_error(gc, crs->domid)) {
+ LOG(ERROR, "replication error occurs when secondary vm is running");
+ goto out;
+ }
+
cds->callback = colo_restore_postsuspend_cb;
libxl__checkpoint_devices_postsuspend(egc, cds);
return;
out:
- libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, !rc);
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->srs.shs, 0);
}
static void colo_restore_postsuspend_cb(libxl__egc *egc,
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index 8b18a91..5f8456c 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -19,7 +19,10 @@
#include "libxl_internal.h"
#include "libxl_colo.h"
+extern const libxl__checkpoint_device_instance_ops colo_save_device_qdisk;
+
static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
+ &colo_save_device_qdisk,
NULL,
};
@@ -30,7 +33,11 @@ static int init_device_subkind(libxl__checkpoint_devices_state *cds)
int rc;
STATE_AO_GC(cds->ao);
+ rc = init_subkind_qdisk(cds);
+ if (rc) goto out;
+
rc = 0;
+out:
return rc;
}
@@ -38,6 +45,8 @@ static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
{
/* cleanup device subkind-specific state in the libxl ctx */
STATE_AO_GC(cds->ao);
+
+ cleanup_subkind_qdisk(cds);
}
/* ================= colo: setup save environment ================= */
@@ -71,9 +80,12 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->send_fd = dss->fd;
css->recv_fd = dss->recv_fd;
css->svm_running = false;
+ css->paused = true;
+ css->qdisk_setuped = false;
+ css->qdisk_used = false;
- /* TODO: disk/nic support */
- cds->device_kind_flags = 0;
+ /* TODO: nic support */
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
cds->ops = colo_ops;
cds->callback = colo_save_setup_done;
cds->ao = ao;
@@ -153,6 +165,11 @@ void libxl__colo_save_teardown(libxl__egc *egc,
libxl__stream_read_abort(egc, &css->srs, 1);
+ if (css->qdisk_setuped) {
+ libxl__qmp_stop_replication(gc, dss->domid, true);
+ css->qdisk_setuped = false;
+ }
+
dss->cds.callback = colo_teardown_done;
libxl__checkpoint_devices_teardown(egc, &dss->cds);
return;
@@ -287,6 +304,11 @@ static void colo_read_svm_suspended_done(libxl__egc *egc,
goto out;
}
+ if (!css->paused && libxl__qmp_get_replication_error(gc, dss->domid)) {
+ LOG(ERROR, "replication error occurs when primary vm is running");
+ goto out;
+ }
+
ok = 1;
out:
@@ -384,12 +406,40 @@ static void colo_preresume_cb(libxl__egc *egc,
goto out;
}
+ if (css->qdisk_used && !css->qdisk_setuped) {
+ if (libxl__qmp_start_replication(gc, dss->domid, true)) {
+ LOG(ERROR, "starting replication fails");
+ goto out;
+ }
+ css->qdisk_setuped = true;
+ }
+
+ if (!css->paused) {
+ if (libxl__qmp_do_checkpoint(gc, dss->domid)) {
+ LOG(ERROR, "doing checkpoint fails");
+ goto out;
+ }
+ }
+
/* Resumes the domain and the device model */
if (libxl__domain_resume(gc, dss->domid, /* Fast Suspend */1)) {
LOG(ERROR, "cannot resume primary vm");
goto out;
}
+ /*
+ * The guest should be paused before doing colo because there is
+ * no disk migration.
+ */
+ if (css->paused) {
+ rc = libxl_domain_unpause(CTX, dss->domid);
+ if (rc) {
+ LOG(ERROR, "cannot unpause primary vm");
+ goto out;
+ }
+ css->paused = false;
+ }
+
/* read CHECKPOINT_SVM_RESUMED */
css->callback = colo_read_svm_resumed_done;
css->srs.checkpoint_callback = colo_common_read_stream_done;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 227b1d3..3af5fdd 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2902,6 +2902,8 @@ int init_subkind_nic(libxl__checkpoint_devices_state *cds);
void cleanup_subkind_nic(libxl__checkpoint_devices_state *cds);
int init_subkind_drbd_disk(libxl__checkpoint_devices_state *cds);
void cleanup_subkind_drbd_disk(libxl__checkpoint_devices_state *cds);
+int init_subkind_qdisk(libxl__checkpoint_devices_state *cds);
+void cleanup_subkind_qdisk(libxl__checkpoint_devices_state *cds);
typedef void libxl__checkpoint_callback(libxl__egc *,
libxl__checkpoint_devices_state *,
@@ -3119,6 +3121,11 @@ struct libxl__colo_save_state {
libxl__stream_read_state srs;
void (*callback)(libxl__egc *, libxl__colo_save_state *, int);
bool svm_running;
+ bool paused;
+
+ /* private, used by qdisk block replication */
+ bool qdisk_used;
+ bool qdisk_setuped;
};
/*----- Domain suspend (save) state structure -----*/
@@ -3522,6 +3529,12 @@ struct libxl__colo_restore_state {
/* private, colo restore checkpoint state */
libxl__domain_create_cb *saved_cb;
void *crcs;
+
+ /* private, used by qdisk block replication */
+ bool qdisk_used;
+ bool qdisk_setuped;
+ const char *host;
+ const char *port;
};
struct libxl__domain_create_state {
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (24 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 25/31] COLO: use qemu block replication Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-03-02 15:04 ` Wei Liu
2016-03-11 22:25 ` Konrad Rzeszutek Wilk
2016-02-22 2:52 ` [PATCH v10 27/31] COLO proxy: preresume, postresume and checkpoint Wen Congyang
` (5 subsequent siblings)
31 siblings, 2 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
setup/teardown of COLO proxy module.
we use netlink to communicate with proxy module.
About colo-proxy module:
https://lkml.org/lkml/2015/6/18/32
How to use:
http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxl/Makefile | 1 +
tools/libxl/libxl_colo.h | 2 +
tools/libxl/libxl_colo_proxy.c | 230 +++++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_internal.h | 15 +++
4 files changed, 248 insertions(+)
create mode 100644 tools/libxl/libxl_colo_proxy.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 28d54d0..6fea9e0 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -67,6 +67,7 @@ endif
LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
LIBXL_OBJS-y += libxl_colo_qdisk.o
+LIBXL_OBJS-y += libxl_colo_proxy.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index b08f672..6d8cd4f 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -27,4 +27,6 @@ extern void libxl__colo_save_teardown(libxl__egc *egc,
libxl__colo_save_state *css,
int rc);
+extern int colo_proxy_setup(libxl__colo_proxy_state *cps);
+extern void colo_proxy_teardown(libxl__colo_proxy_state *cps);
#endif
diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
new file mode 100644
index 0000000..e07e640
--- /dev/null
+++ b/tools/libxl/libxl_colo_proxy.c
@@ -0,0 +1,230 @@
+/*
+ * Copyright (C) 2015 FUJITSU LIMITED
+ * Author: Yang Hongyang <hongyang.yang@easystack.cn>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+#include "libxl_colo.h"
+#include <linux/netlink.h>
+
+#define NETLINK_COLO 28
+
+enum colo_netlink_op {
+ COLO_QUERY_CHECKPOINT = (NLMSG_MIN_TYPE + 1),
+ COLO_CHECKPOINT,
+ COLO_FAILOVER,
+ COLO_PROXY_INIT,
+ COLO_PROXY_RESET, /* UNUSED, will be used for continuous FT */
+};
+
+/* ========= colo-proxy: helper functions ========== */
+
+static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
+ uint64_t size, int type)
+{
+ struct sockaddr_nl sa;
+ struct nlmsghdr msg;
+ struct iovec iov;
+ struct msghdr mh;
+ int ret;
+
+ STATE_AO_GC(cps->ao);
+
+ memset(&sa, 0, sizeof(sa));
+ sa.nl_family = AF_NETLINK;
+ sa.nl_pid = 0;
+ sa.nl_groups = 0;
+
+ msg.nlmsg_len = NLMSG_SPACE(0);
+ msg.nlmsg_flags = NLM_F_REQUEST;
+ if (type == COLO_PROXY_INIT) {
+ msg.nlmsg_flags |= NLM_F_ACK;
+ }
+ msg.nlmsg_seq = 0;
+ /* This is untrusty */
+ msg.nlmsg_pid = cps->index;
+ msg.nlmsg_type = type;
+
+ iov.iov_base = &msg;
+ iov.iov_len = msg.nlmsg_len;
+
+ mh.msg_name = &sa;
+ mh.msg_namelen = sizeof(sa);
+ mh.msg_iov = &iov;
+ mh.msg_iovlen = 1;
+ mh.msg_control = NULL;
+ mh.msg_controllen = 0;
+ mh.msg_flags = 0;
+
+ ret = sendmsg(cps->sock_fd, &mh, 0);
+ if (ret <= 0) {
+ LOG(ERROR, "can't send msg to kernel by netlink: %s",
+ strerror(errno));
+ }
+
+ return ret;
+}
+
+/* error: return -1, otherwise return 0 */
+static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
+ unsigned int timeout_us)
+{
+ struct sockaddr_nl sa;
+ struct iovec iov;
+ struct msghdr mh = {
+ .msg_name = &sa,
+ .msg_namelen = sizeof(sa),
+ .msg_iov = &iov,
+ .msg_iovlen = 1,
+ };
+ struct timeval tv;
+ uint32_t size = 16384;
+ int64_t len = 0;
+ int ret;
+
+ STATE_AO_GC(cps->ao);
+ uint8_t *tmp = libxl__malloc(NOGC, size);
+
+ if (timeout_us) {
+ tv.tv_sec = timeout_us / 1000000;
+ tv.tv_usec = timeout_us % 1000000;
+ setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
+ }
+
+ iov.iov_base = tmp;
+ iov.iov_len = size;
+next:
+ ret = recvmsg(cps->sock_fd, &mh, 0);
+ if (ret <= 0) {
+ if (errno != EAGAIN && errno != EWOULDBLOCK)
+ LOGE(ERROR, "can't recv msg from kernel by netlink");
+ goto err;
+ }
+
+ len += ret;
+ if (mh.msg_flags & MSG_TRUNC) {
+ size += 16384;
+ tmp = libxl__realloc(NOGC, tmp, size);
+ iov.iov_base = tmp + len;
+ iov.iov_len = size - len;
+ goto next;
+ }
+
+ *buff = tmp;
+ ret = len;
+ goto out;
+
+err:
+ free(tmp);
+ *buff = NULL;
+
+out:
+ if (timeout_us) {
+ tv.tv_sec = 0;
+ tv.tv_usec = 0;
+ setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
+ }
+ return ret;
+}
+
+/* ========= colo-proxy: setup and teardown ========== */
+
+int colo_proxy_setup(libxl__colo_proxy_state *cps)
+{
+ int skfd = 0;
+ struct sockaddr_nl sa;
+ struct nlmsghdr *h;
+ int i = 1;
+ int ret = ERROR_FAIL;
+ uint8_t *buff = NULL;
+ int64_t size;
+
+ STATE_AO_GC(cps->ao);
+
+ skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
+ if (skfd < 0) {
+ LOG(ERROR, "can not create a netlink socket: %s", strerror(errno));
+ goto out;
+ }
+ cps->sock_fd = skfd;
+ memset(&sa, 0, sizeof(sa));
+ sa.nl_family = AF_NETLINK;
+ sa.nl_groups = 0;
+retry:
+ sa.nl_pid = i++;
+
+ if (i > 10) {
+ LOG(ERROR, "netlink bind error");
+ goto out;
+ }
+
+ ret = bind(skfd, (struct sockaddr *)&sa, sizeof(sa));
+ if (ret < 0 && errno == EADDRINUSE) {
+ LOG(ERROR, "colo index %d has already in used", sa.nl_pid);
+ goto retry;
+ } else if (ret < 0) {
+ LOG(ERROR, "netlink bind error");
+ goto out;
+ }
+
+ cps->index = sa.nl_pid;
+ ret = colo_proxy_send(cps, NULL, 0, COLO_PROXY_INIT);
+ if (ret < 0) {
+ goto out;
+ }
+ /* receive ack */
+ size = colo_proxy_recv(cps, &buff, 500000);
+ if (size < 0) {
+ LOG(ERROR, "Can't recv msg from kernel by netlink: %s",
+ strerror(errno));
+ goto out;
+ }
+
+ if (size) {
+ h = (struct nlmsghdr *)buff;
+ if (h->nlmsg_type == NLMSG_ERROR) {
+ /* ack's type is NLMSG_ERROR */
+ struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(h);
+
+ if (size - sizeof(*h) < sizeof(*err)) {
+ LOG(ERROR, "NLMSG_LENGTH is too short");
+ goto out;
+ }
+
+ if (err->error) {
+ LOG(ERROR, "NLMSG_ERROR contains error %d", err->error);
+ goto out;
+ }
+ }
+ }
+
+ ret = 0;
+
+out:
+ free(buff);
+ if (ret) {
+ close(cps->sock_fd);
+ cps->sock_fd = -1;
+ }
+ return ret;
+}
+
+void colo_proxy_teardown(libxl__colo_proxy_state *cps)
+{
+ if (cps->sock_fd >= 0) {
+ close(cps->sock_fd);
+ cps->sock_fd = -1;
+ }
+}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 3af5fdd..3b44b09 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3112,6 +3112,15 @@ libxl__stream_read_inuse(const libxl__stream_read_state *stream)
}
/*----- colo related state structure -----*/
+typedef struct libxl__colo_proxy_state libxl__colo_proxy_state;
+struct libxl__colo_proxy_state {
+ /* set by caller of colo_proxy_setup */
+ libxl__ao *ao;
+
+ int sock_fd;
+ int index;
+};
+
typedef struct libxl__colo_save_state libxl__colo_save_state;
struct libxl__colo_save_state {
int send_fd;
@@ -3126,6 +3135,9 @@ struct libxl__colo_save_state {
/* private, used by qdisk block replication */
bool qdisk_used;
bool qdisk_setuped;
+
+ /* private, used by colo-proxy */
+ libxl__colo_proxy_state cps;
};
/*----- Domain suspend (save) state structure -----*/
@@ -3535,6 +3547,9 @@ struct libxl__colo_restore_state {
bool qdisk_setuped;
const char *host;
const char *port;
+
+ /* private, used by colo-proxy */
+ libxl__colo_proxy_state cps;
};
struct libxl__domain_create_state {
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 27/31] COLO proxy: preresume, postresume and checkpoint
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (25 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-03-02 15:04 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 28/31] COLO nic: implement COLO nic subkind Wen Congyang
` (4 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
preresume, postresume and checkpoint
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxl/libxl_colo.h | 4 +++
tools/libxl/libxl_colo_proxy.c | 62 ++++++++++++++++++++++++++++++++++++++++++
2 files changed, 66 insertions(+)
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 6d8cd4f..17dd07f 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -29,4 +29,8 @@ extern void libxl__colo_save_teardown(libxl__egc *egc,
extern int colo_proxy_setup(libxl__colo_proxy_state *cps);
extern void colo_proxy_teardown(libxl__colo_proxy_state *cps);
+extern void colo_proxy_preresume(libxl__colo_proxy_state *cps);
+extern void colo_proxy_postresume(libxl__colo_proxy_state *cps);
+extern int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
+ unsigned int timeout_us);
#endif
diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
index e07e640..c714b52 100644
--- a/tools/libxl/libxl_colo_proxy.c
+++ b/tools/libxl/libxl_colo_proxy.c
@@ -228,3 +228,65 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
cps->sock_fd = -1;
}
}
+
+/* ========= colo-proxy: preresume, postresume and checkpoint ========== */
+
+void colo_proxy_preresume(libxl__colo_proxy_state *cps)
+{
+ colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
+ /* TODO: need to handle if the call fails... */
+}
+
+void colo_proxy_postresume(libxl__colo_proxy_state *cps)
+{
+ /* nothing to do... */
+}
+
+typedef struct colo_msg {
+ bool is_checkpoint;
+} colo_msg;
+
+/*
+ * Return value:
+ * -1: error
+ * 0: no checkpoint event is received before timeout
+ * 1: do checkpoint
+ */
+int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
+ unsigned int timeout_us)
+{
+ uint8_t *buff;
+ int64_t size;
+ struct nlmsghdr *h;
+ struct colo_msg *m;
+ int ret = -1;
+
+ STATE_AO_GC(cps->ao);
+
+ size = colo_proxy_recv(cps, &buff, timeout_us);
+
+ /* timeout, return no checkpoint message. */
+ if (size <= 0) {
+ return 0;
+ }
+
+ h = (struct nlmsghdr *) buff;
+
+ if (h->nlmsg_type == NLMSG_ERROR) {
+ LOG(ERROR, "receive NLMSG_ERROR");
+ goto out;
+ }
+
+ if (h->nlmsg_len < NLMSG_LENGTH(sizeof(*m))) {
+ LOG(ERROR, "NLMSG_LENGTH is too short");
+ goto out;
+ }
+
+ m = NLMSG_DATA(h);
+
+ ret = m->is_checkpoint ? 1 : 0;
+
+out:
+ free(buff);
+ return ret;
+}
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 28/31] COLO nic: implement COLO nic subkind
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (26 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 27/31] COLO proxy: preresume, postresume and checkpoint Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-03-02 15:04 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 29/31] setup and control colo proxy on primary side Wen Congyang
` (3 subsequent siblings)
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
implement COLO nic subkind.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/hotplug/Linux/Makefile | 1 +
tools/hotplug/Linux/colo-proxy-setup | 135 +++++++++++++++
tools/libxl/Makefile | 1 +
tools/libxl/libxl_colo_nic.c | 321 +++++++++++++++++++++++++++++++++++
tools/libxl/libxl_internal.h | 5 +
tools/libxl/libxl_types.idl | 1 +
6 files changed, 464 insertions(+)
create mode 100755 tools/hotplug/Linux/colo-proxy-setup
create mode 100644 tools/libxl/libxl_colo_nic.c
diff --git a/tools/hotplug/Linux/Makefile b/tools/hotplug/Linux/Makefile
index 6e10118..9bb852b 100644
--- a/tools/hotplug/Linux/Makefile
+++ b/tools/hotplug/Linux/Makefile
@@ -28,6 +28,7 @@ XEN_SCRIPTS += block-iscsi
XEN_SCRIPTS += block-tap
XEN_SCRIPTS += block-drbd-probe
XEN_SCRIPTS += $(XEN_SCRIPTS-y)
+XEN_SCRIPTS += colo-proxy-setup
SUBDIRS-$(CONFIG_SYSTEMD) += systemd
diff --git a/tools/hotplug/Linux/colo-proxy-setup b/tools/hotplug/Linux/colo-proxy-setup
new file mode 100755
index 0000000..94e2034
--- /dev/null
+++ b/tools/hotplug/Linux/colo-proxy-setup
@@ -0,0 +1,135 @@
+#! /bin/bash
+
+dir=$(dirname "$0")
+. "$dir/xen-hotplug-common.sh"
+. "$dir/hotplugpath.sh"
+
+findCommand "$@"
+
+if [ "$command" != "setup" -a "$command" != "teardown" ]
+then
+ echo "Invalid command: $command"
+ log err "Invalid command: $command"
+ exit 1
+fi
+
+evalVariables "$@"
+
+: ${vifname:?}
+: ${forwarddev:?}
+: ${mode:?}
+: ${index:?}
+: ${bridge:?}
+
+forwardbr="colobr0"
+
+if [ "$mode" != "primary" -a "$mode" != "secondary" ]
+then
+ echo "Invalid mode: $mode"
+ log err "Invalid mode: $mode"
+ exit 1
+fi
+
+if [ $index -lt 0 ] || [ $index -gt 100 ]; then
+ echo "index overflow"
+ exit 1
+fi
+
+function setup_primary()
+{
+ do_without_error tc qdisc add dev $vifname root handle 1: prio
+ do_without_error tc filter add dev $vifname parent 1: protocol ip prio 10 \
+ u32 match u32 0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+ do_without_error tc filter add dev $vifname parent 1: protocol arp prio 11 \
+ u32 match u32 0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+ do_without_error tc filter add dev $vifname parent 1: protocol ipv6 prio \
+ 12 u32 match u32 0 0 flowid 1:2 action mirred egress mirror \
+ dev $forwarddev
+
+ do_without_error modprobe nf_conntrack_ipv4
+ do_without_error modprobe xt_PMYCOLO sec_dev=$forwarddev
+
+ iptables -t mangle -I PREROUTING -m physdev --physdev-in \
+ $vifname -j PMYCOLO --index $index
+ ip6tables -t mangle -I PREROUTING -m physdev --physdev-in \
+ $vifname -j PMYCOLO --index $index
+ do_without_error arptables -I INPUT -i $forwarddev -j MARK --set-mark $index
+}
+
+function teardown_primary()
+{
+ do_without_error tc filter del dev $vifname parent 1: protocol ip prio 10 u32 match u32 \
+ 0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+ do_without_error tc filter del dev $vifname parent 1: protocol arp prio 11 u32 match u32 \
+ 0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+ do_without_error tc filter del dev $vifname parent 1: protocol ipv6 prio 12 u32 match u32 \
+ 0 0 flowid 1:2 action mirred egress mirror dev $forwarddev
+ do_without_error tc qdisc del dev $vifname root handle 1: prio
+
+ do_without_error iptables -t mangle -D PREROUTING -m physdev --physdev-in \
+ $vifname -j PMYCOLO --index $index
+ do_without_error ip6tables -t mangle -D PREROUTING -m physdev --physdev-in \
+ $vifname -j PMYCOLO --index $index
+ do_without_error arptables -F
+ do_without_error rmmod xt_PMYCOLO
+}
+
+function setup_secondary()
+{
+ do_without_error brctl delif $bridge $vifname
+ do_without_error brctl addbr $forwardbr
+ do_without_error brctl addif $forwardbr $vifname
+ do_without_error brctl addif $forwardbr $forwarddev
+ do_without_error ip link set dev $forwardbr up
+ do_without_error modprobe xt_SECCOLO
+
+ iptables -t mangle -I PREROUTING -m physdev --physdev-in \
+ $vifname -j SECCOLO --index $index
+ ip6tables -t mangle -I PREROUTING -m physdev --physdev-in \
+ $vifname -j SECCOLO --index $index
+}
+
+function teardown_secondary()
+{
+ do_without_error brctl delif $forwardbr $forwarddev
+ do_without_error brctl delif $forwardbr $vifname
+ do_without_error brctl delbr $forwardbr
+ do_without_error brctl addif $bridge $vifname
+
+ do_without_error iptables -t mangle -D PREROUTING -m physdev --physdev-in \
+ $vifname -j SECCOLO --index $index
+ do_without_error ip6tables -t mangle -D PREROUTING -m physdev --physdev-in \
+ $vifname -j SECCOLO --index $index
+ do_without_error rmmod xt_SECCOLO
+}
+
+case "$command" in
+ setup)
+ if [ "$mode" = "primary" ]
+ then
+ setup_primary
+ else
+ setup_secondary
+ fi
+
+ success
+ ;;
+ teardown)
+ if [ "$mode" = "primary" ]
+ then
+ teardown_primary
+ else
+ teardown_secondary
+ fi
+ ;;
+esac
+
+if [ "$mode" = "primary" ]
+then
+ log debug "Successful colo-proxy-setup $command for $vifname." \
+ " vifname: $vifname, index: $index, forwarddev: $forwarddev."
+else
+ log debug "Successful colo-proxy-setup $command for $vifname." \
+ " vifname: $vifname, index: $index, forwarddev: $forwarddev,"\
+ " forwardbr: $forwardbr."
+fi
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 6fea9e0..aefc378 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -68,6 +68,7 @@ LIBXL_OBJS-y += libxl_remus.o libxl_checkpoint_device.o libxl_remus_disk_drbd.o
LIBXL_OBJS-y += libxl_colo_restore.o libxl_colo_save.o
LIBXL_OBJS-y += libxl_colo_qdisk.o
LIBXL_OBJS-y += libxl_colo_proxy.o
+LIBXL_OBJS-y += libxl_colo_nic.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
diff --git a/tools/libxl/libxl_colo_nic.c b/tools/libxl/libxl_colo_nic.c
new file mode 100644
index 0000000..998e09c
--- /dev/null
+++ b/tools/libxl/libxl_colo_nic.c
@@ -0,0 +1,321 @@
+/*
+ * Copyright (C) 2014 FUJITSU LIMITED
+ * Author: Wen Congyang <wency@cn.fujitsu.com>
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+typedef struct libxl__colo_device_nic {
+ int devid;
+ const char *vif;
+} libxl__colo_device_nic;
+
+enum {
+ primary,
+ secondary,
+};
+
+
+/* ========== init() and cleanup() ========== */
+int init_subkind_colo_nic(libxl__checkpoint_devices_state *cds)
+{
+ return 0;
+}
+
+void cleanup_subkind_colo_nic(libxl__checkpoint_devices_state *cds)
+{
+}
+
+/* ========== helper functions ========== */
+static void colo_save_setup_script_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int rc, int status);
+static void colo_save_teardown_script_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int rc, int status);
+
+/*
+ * If the device has a vifname, then use that instead of
+ * the vifX.Y format.
+ * it must ONLY be used for remus because if driver domains
+ * were in use it would constitute a security vulnerability.
+ */
+static const char *get_vifname(libxl__checkpoint_device *dev,
+ const libxl_device_nic *nic)
+{
+ const char *vifname = NULL;
+ const char *path;
+ int rc;
+
+ STATE_AO_GC(dev->cds->ao);
+
+ /* Convenience aliases */
+ const uint32_t domid = dev->cds->domid;
+
+ path = GCSPRINTF("%s/backend/vif/%d/%d/vifname",
+ libxl__xs_get_dompath(gc, 0), domid, nic->devid);
+ rc = libxl__xs_read_checked(gc, XBT_NULL, path, &vifname);
+ if (!rc && !vifname) {
+ vifname = libxl__device_nic_devname(gc, domid,
+ nic->devid,
+ nic->nictype);
+ }
+
+ return vifname;
+}
+
+/*
+ * the script needs the following env & args
+ * $vifname
+ * $forwarddev
+ * $mode(primary/secondary)
+ * $index
+ * $bridge
+ * setup/teardown as command line arg.
+ */
+static void setup_async_exec(libxl__checkpoint_device *dev, char *op,
+ libxl__colo_proxy_state *cps, int side,
+ char *colo_proxy_script)
+{
+ int arraysize, nr = 0;
+ char **env = NULL, **args = NULL;
+ libxl__colo_device_nic *colo_nic = dev->concrete_data;
+ libxl__checkpoint_devices_state *cds = dev->cds;
+ libxl__async_exec_state *aes = &dev->aodev.aes;
+ const libxl_device_nic *nic = dev->backend_dev;
+
+ STATE_AO_GC(cds->ao);
+
+ /* Convenience aliases */
+ const char *const vif = colo_nic->vif;
+
+ arraysize = 11;
+ GCNEW_ARRAY(env, arraysize);
+ env[nr++] = "vifname";
+ env[nr++] = libxl__strdup(gc, vif);
+ env[nr++] = "forwarddev";
+ env[nr++] = libxl__strdup(gc, nic->forwarddev);
+ env[nr++] = "mode";
+ if (side == primary)
+ env[nr++] = "primary";
+ else
+ env[nr++] = "secondary";
+ env[nr++] = "index";
+ env[nr++] = GCSPRINTF("%d", cps->index);
+ env[nr++] = "bridge";
+ env[nr++] = libxl__strdup(gc, nic->bridge);
+ env[nr++] = NULL;
+ assert(nr == arraysize);
+
+ arraysize = 3; nr = 0;
+ GCNEW_ARRAY(args, arraysize);
+ args[nr++] = colo_proxy_script;
+ args[nr++] = op;
+ args[nr++] = NULL;
+ assert(nr == arraysize);
+
+ aes->ao = dev->cds->ao;
+ aes->what = GCSPRINTF("%s %s", args[0], args[1]);
+ aes->env = env;
+ aes->args = args;
+ aes->timeout_ms = LIBXL_HOTPLUG_TIMEOUT * 1000;
+ aes->stdfds[0] = -1;
+ aes->stdfds[1] = -1;
+ aes->stdfds[2] = -1;
+
+ if (!strcmp(op, "teardown"))
+ aes->callback = colo_save_teardown_script_cb;
+ else
+ aes->callback = colo_save_setup_script_cb;
+}
+
+/* ========== setup() and teardown() ========== */
+static void colo_nic_setup(libxl__egc *egc, libxl__checkpoint_device *dev,
+ libxl__colo_proxy_state *cps, int side,
+ char *colo_proxy_script)
+{
+ int rc;
+ libxl__colo_device_nic *colo_nic;
+ const libxl_device_nic *nic = dev->backend_dev;
+
+ STATE_AO_GC(dev->cds->ao);
+
+ /*
+ * thers's no subkind of nic devices, so nic ops is always matched
+ * with nic devices, we begin to setup the nic device
+ */
+ dev->matched = 1;
+
+ if (!nic->forwarddev) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ GCNEW(colo_nic);
+ dev->concrete_data = colo_nic;
+ colo_nic->devid = nic->devid;
+ colo_nic->vif = get_vifname(dev, nic);
+ if (!colo_nic->vif) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ setup_async_exec(dev, "setup", cps, side, colo_proxy_script);
+ rc = libxl__async_exec_start(&dev->aodev.aes);
+ if (rc)
+ goto out;
+
+ return;
+
+out:
+ dev->aodev.rc = rc;
+ dev->aodev.callback(egc, &dev->aodev);
+}
+
+static void colo_save_setup_script_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int rc, int status)
+{
+ libxl__ao_device *aodev = CONTAINER_OF(aes, *aodev, aes);
+ libxl__checkpoint_device *dev = CONTAINER_OF(aodev, *dev, aodev);
+ libxl__colo_device_nic *colo_nic = dev->concrete_data;
+ libxl__checkpoint_devices_state *cds = dev->cds;
+ const char *out_path_base, *hotplug_error = NULL;
+
+ EGC_GC;
+
+ /* Convenience aliases */
+ const uint32_t domid = cds->domid;
+ const int devid = colo_nic->devid;
+ const char *const vif = colo_nic->vif;
+
+ if (status && !rc)
+ rc = ERROR_FAIL;
+ if (rc)
+ goto out;
+
+ out_path_base = GCSPRINTF("%s/colo_proxy/%d",
+ libxl__xs_libxl_path(gc, domid), devid);
+
+ rc = libxl__xs_read_checked(gc, XBT_NULL,
+ GCSPRINTF("%s/hotplug-error", out_path_base),
+ &hotplug_error);
+ if (rc)
+ goto out;
+
+ if (hotplug_error) {
+ LOG(ERROR, "colo_proxy script %s setup failed for vif %s: %s",
+ aes->args[0], vif, hotplug_error);
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ if (status) {
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ rc = 0;
+
+out:
+ aodev->rc = rc;
+ aodev->callback(egc, aodev);
+}
+
+static void colo_nic_teardown(libxl__egc *egc, libxl__checkpoint_device *dev,
+ libxl__colo_proxy_state *cps, int side,
+ char *colo_proxy_script)
+{
+ int rc;
+ libxl__colo_device_nic *colo_nic = dev->concrete_data;
+
+ if (!colo_nic || !colo_nic->vif) {
+ /* colo nic has not yet been set up, just return */
+ rc = 0;
+ goto out;
+ }
+
+ setup_async_exec(dev, "teardown", cps, side, colo_proxy_script);
+
+ rc = libxl__async_exec_start(&dev->aodev.aes);
+ if (rc)
+ goto out;
+
+ return;
+
+out:
+ dev->aodev.rc = rc;
+ dev->aodev.callback(egc, &dev->aodev);
+}
+
+static void colo_save_teardown_script_cb(libxl__egc *egc,
+ libxl__async_exec_state *aes,
+ int rc, int status)
+{
+ libxl__ao_device *aodev = CONTAINER_OF(aes, *aodev, aes);
+
+ if (status && !rc)
+ rc = ERROR_FAIL;
+ else
+ rc = 0;
+
+ aodev->rc = rc;
+ aodev->callback(egc, aodev);
+}
+
+/* ======== primary ======== */
+static void colo_nic_save_setup(libxl__egc *egc, libxl__checkpoint_device *dev)
+{
+ libxl__colo_save_state *css = dev->cds->concrete_data;
+
+ colo_nic_setup(egc, dev, &css->cps, primary, css->colo_proxy_script);
+}
+
+static void colo_nic_save_teardown(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ libxl__colo_save_state *css = dev->cds->concrete_data;
+
+ colo_nic_teardown(egc, dev, &css->cps, primary, css->colo_proxy_script);
+}
+
+const libxl__checkpoint_device_instance_ops colo_save_device_nic = {
+ .kind = LIBXL__DEVICE_KIND_VIF,
+ .setup = colo_nic_save_setup,
+ .teardown = colo_nic_save_teardown,
+};
+
+/* ======== secondary ======== */
+static void colo_nic_restore_setup(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ libxl__colo_restore_state *crs = dev->cds->concrete_data;
+
+ colo_nic_setup(egc, dev, &crs->cps, secondary, crs->colo_proxy_script);
+}
+
+static void colo_nic_restore_teardown(libxl__egc *egc,
+ libxl__checkpoint_device *dev)
+{
+ libxl__colo_restore_state *crs = dev->cds->concrete_data;
+
+ colo_nic_teardown(egc, dev, &crs->cps, secondary, crs->colo_proxy_script);
+}
+
+const libxl__checkpoint_device_instance_ops colo_restore_device_nic = {
+ .kind = LIBXL__DEVICE_KIND_VIF,
+ .setup = colo_nic_restore_setup,
+ .teardown = colo_nic_restore_teardown,
+};
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 3b44b09..3a8c268 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2904,6 +2904,8 @@ int init_subkind_drbd_disk(libxl__checkpoint_devices_state *cds);
void cleanup_subkind_drbd_disk(libxl__checkpoint_devices_state *cds);
int init_subkind_qdisk(libxl__checkpoint_devices_state *cds);
void cleanup_subkind_qdisk(libxl__checkpoint_devices_state *cds);
+int init_subkind_colo_nic(libxl__checkpoint_devices_state *cds);
+void cleanup_subkind_colo_nic(libxl__checkpoint_devices_state *cds);
typedef void libxl__checkpoint_callback(libxl__egc *,
libxl__checkpoint_devices_state *,
@@ -3125,6 +3127,7 @@ typedef struct libxl__colo_save_state libxl__colo_save_state;
struct libxl__colo_save_state {
int send_fd;
int recv_fd;
+ char *colo_proxy_script;
/* private */
libxl__stream_read_state srs;
@@ -3537,6 +3540,7 @@ struct libxl__colo_restore_state {
int recv_fd;
int hvm;
libxl__colo_callback *callback;
+ char *colo_proxy_script;
/* private, colo restore checkpoint state */
libxl__domain_create_cb *saved_cb;
@@ -3566,6 +3570,7 @@ struct libxl__domain_create_state {
libxl_asyncprogress_how aop_console_how;
/* private to domain_create */
int guest_domid;
+ const char *colo_proxy_script;
libxl__domain_build_state build_state;
libxl__colo_restore_state crs;
libxl__checkpoint_devices_state cds;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a2078d1..6b57aba 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -599,6 +599,7 @@ libxl_device_nic = Struct("device_nic", [
("rate_bytes_per_interval", uint64),
("rate_interval_usecs", uint32),
("gatewaydev", string),
+ ("forwarddev", string)
])
libxl_device_pci = Struct("device_pci", [
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 29/31] setup and control colo proxy on primary side
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (27 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 28/31] COLO nic: implement COLO nic subkind Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 30/31] setup and control colo proxy on secondary side Wen Congyang
` (2 subsequent siblings)
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
setup and control colo proxy on primary side
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxl/libxl_colo_save.c | 124 +++++++++++++++++++++++++++++++++++++++---
tools/libxl/libxl_internal.h | 1 +
2 files changed, 117 insertions(+), 8 deletions(-)
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index 5f8456c..ccfbb17 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -19,9 +19,11 @@
#include "libxl_internal.h"
#include "libxl_colo.h"
+extern const libxl__checkpoint_device_instance_ops colo_save_device_nic;
extern const libxl__checkpoint_device_instance_ops colo_save_device_qdisk;
static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
+ &colo_save_device_nic,
&colo_save_device_qdisk,
NULL,
};
@@ -33,9 +35,15 @@ static int init_device_subkind(libxl__checkpoint_devices_state *cds)
int rc;
STATE_AO_GC(cds->ao);
- rc = init_subkind_qdisk(cds);
+ rc = init_subkind_colo_nic(cds);
if (rc) goto out;
+ rc = init_subkind_qdisk(cds);
+ if (rc) {
+ cleanup_subkind_colo_nic(cds);
+ goto out;
+ }
+
rc = 0;
out:
return rc;
@@ -46,6 +54,7 @@ static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
/* cleanup device subkind-specific state in the libxl ctx */
STATE_AO_GC(cds->ao);
+ cleanup_subkind_colo_nic(cds);
cleanup_subkind_qdisk(cds);
}
@@ -83,9 +92,16 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->paused = true;
css->qdisk_setuped = false;
css->qdisk_used = false;
+ libxl__ev_child_init(&css->child);
- /* TODO: nic support */
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+ if (dss->remus->netbufscript)
+ css->colo_proxy_script = libxl__strdup(gc, dss->remus->netbufscript);
+ else
+ css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
+ libxl__xen_script_dir_path());
+
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
cds->ops = colo_ops;
cds->callback = colo_save_setup_done;
cds->ao = ao;
@@ -96,6 +112,12 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->srs.fd = css->recv_fd;
css->srs.back_channel = true;
libxl__stream_read_start(egc, &css->srs);
+ css->cps.ao = ao;
+ if (colo_proxy_setup(&css->cps)) {
+ LOG(ERROR, "COLO: failed to setup colo proxy for guest with domid %u",
+ cds->domid);
+ goto out;
+ }
if (init_device_subkind(cds))
goto out;
@@ -183,6 +205,7 @@ static void colo_teardown_done(libxl__egc *egc,
libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
cleanup_device_subkind(cds);
+ colo_proxy_teardown(&css->cps);
dss->callback(egc, dss, rc);
}
@@ -382,6 +405,8 @@ static void colo_read_svm_ready_done(libxl__egc *egc,
goto out;
}
+ colo_proxy_preresume(&css->cps);
+
css->svm_running = true;
dss->cds.callback = colo_preresume_cb;
libxl__checkpoint_devices_preresume(egc, &dss->cds);
@@ -466,6 +491,8 @@ static void colo_read_svm_resumed_done(libxl__egc *egc,
goto out;
}
+ colo_proxy_postresume(&css->cps);
+
ok = 1;
out:
@@ -474,6 +501,91 @@ out:
/* ===================== colo: wait new checkpoint ===================== */
+
+static void colo_start_new_checkpoint(libxl__egc *egc,
+ libxl__checkpoint_devices_state *cds,
+ int rc);
+static void colo_proxy_async_wait_for_checkpoint(libxl__colo_save_state *css);
+static void colo_proxy_async_call_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ int pid,
+ int status);
+
+static void colo_proxy_async_call(libxl__egc *egc,
+ libxl__colo_save_state *css,
+ void func(libxl__colo_save_state *),
+ libxl__ev_child_callback callback)
+{
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+ int pid = -1, rc;
+
+ STATE_AO_GC(dss->cds.ao);
+
+ /* Fork and call */
+ pid = libxl__ev_child_fork(gc, &css->child, callback);
+ if (pid == -1) {
+ LOG(ERROR, "unable to fork");
+ rc = ERROR_FAIL;
+ goto out;
+ }
+
+ if (!pid) {
+ /* child */
+ func(css);
+ /* notreached */
+ abort();
+ }
+
+ return;
+
+out:
+ callback(egc, &css->child, -1, 1);
+}
+
+static void colo_proxy_wait_for_checkpoint(libxl__egc *egc,
+ libxl__colo_save_state *css)
+{
+ colo_proxy_async_call(egc, css,
+ colo_proxy_async_wait_for_checkpoint,
+ colo_proxy_async_call_done);
+}
+
+static void colo_proxy_async_wait_for_checkpoint(libxl__colo_save_state *css)
+{
+ int req;
+
+ req = colo_proxy_checkpoint(&css->cps, 5000000);
+ if (req < 0) {
+ /* some error happens */
+ _exit(1);
+ } else if (!req) {
+ /* no checkpoint is needed, do a checkpint every 5s */
+ _exit(0);
+ } else {
+ /* net packets is not consistent, we need to start a checkpoint */
+ _exit(0);
+ }
+}
+
+static void colo_proxy_async_call_done(libxl__egc *egc,
+ libxl__ev_child *child,
+ int pid,
+ int status)
+{
+ libxl__colo_save_state *css = CONTAINER_OF(child, *css, child);
+ libxl__domain_save_state *dss = CONTAINER_OF(css, *dss, css);
+
+ EGC_GC;
+
+ if (status) {
+ LOG(ERROR, "failed to wait for new checkpoint");
+ colo_start_new_checkpoint(egc, &dss->cds, ERROR_FAIL);
+ return;
+ }
+
+ colo_start_new_checkpoint(egc, &dss->cds, 0);
+}
+
/*
* Do the following things:
* 1. do commit
@@ -483,9 +595,6 @@ out:
static void colo_device_commit_cb(libxl__egc *egc,
libxl__checkpoint_devices_state *cds,
int rc);
-static void colo_start_new_checkpoint(libxl__egc *egc,
- libxl__checkpoint_devices_state *cds,
- int rc);
static void libxl__colo_save_domain_wait_checkpoint_callback(void *data)
{
@@ -515,8 +624,7 @@ static void colo_device_commit_cb(libxl__egc *egc,
goto out;
}
- /* TODO: wait a new checkpoint */
- colo_start_new_checkpoint(egc, cds, 0);
+ colo_proxy_wait_for_checkpoint(egc, css);
return;
out:
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 3a8c268..77691a4 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3141,6 +3141,7 @@ struct libxl__colo_save_state {
/* private, used by colo-proxy */
libxl__colo_proxy_state cps;
+ libxl__ev_child child;
};
/*----- Domain suspend (save) state structure -----*/
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 30/31] setup and control colo proxy on secondary side
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (28 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 29/31] setup and control colo proxy on primary side Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 31/31] cmdline switches and config vars to control colo-proxy Wen Congyang
2016-02-25 16:05 ` [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wei Liu
31 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
setup and control colo proxy on secondary side
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
tools/libxl/libxl_colo_restore.c | 28 +++++++++++++++++++++++++---
1 file changed, 25 insertions(+), 3 deletions(-)
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index eff80ca..c4b1fb4 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -50,9 +50,11 @@ static void libxl__colo_restore_domain_checkpoint_callback(void *data);
static void libxl__colo_restore_domain_wait_checkpoint_callback(void *data);
static void libxl__colo_restore_domain_suspend_callback(void *data);
+extern const libxl__checkpoint_device_instance_ops colo_restore_device_nic;
extern const libxl__checkpoint_device_instance_ops colo_restore_device_qdisk;
static const libxl__checkpoint_device_instance_ops *colo_restore_ops[] = {
+ &colo_restore_device_nic,
&colo_restore_device_qdisk,
NULL,
};
@@ -153,8 +155,14 @@ static int init_device_subkind(libxl__checkpoint_devices_state *cds)
int rc;
STATE_AO_GC(cds->ao);
+ rc = init_subkind_colo_nic(cds);
+ if (rc) goto out;
+
rc = init_subkind_qdisk(cds);
- if (rc) goto out;
+ if (rc) {
+ cleanup_subkind_colo_nic(cds);
+ goto out;
+ }
rc = 0;
out:
@@ -166,6 +174,7 @@ static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
/* cleanup device subkind-specific state in the libxl ctx */
STATE_AO_GC(cds->ao);
+ cleanup_subkind_colo_nic(cds);
cleanup_subkind_qdisk(cds);
}
@@ -347,6 +356,8 @@ static void colo_restore_teardown_devices_done(libxl__egc *egc,
if (crcs->teardown_devices)
cleanup_device_subkind(cds);
+ colo_proxy_teardown(&crs->cps);
+
rc = crcs->saved_rc;
if (!rc) {
crcs->callback = do_failover_done;
@@ -605,6 +616,8 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
}
}
+ colo_proxy_preresume(&crs->cps);
+
colo_restore_resume_vm(egc, crcs);
return;
@@ -641,6 +654,8 @@ static void colo_resume_vm_done(libxl__egc *egc,
crcs->status = LIBXL_COLO_RESUMED;
+ colo_proxy_postresume(&crs->cps);
+
/* avoid calling stream->completion_callback() more than once */
if (crs->saved_cb) {
dcs->callback = crs->saved_cb;
@@ -762,13 +777,20 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
STATE_AO_GC(crs->ao);
- /* TODO: nic support */
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
cds->callback = colo_restore_setup_cds_done;
cds->ao = ao;
cds->domid = crs->domid;
cds->ops = colo_restore_ops;
+ crs->cps.ao = ao;
+ if (colo_proxy_setup(&crs->cps)) {
+ LOG(ERROR, "COLO: failed to setup colo proxy for guest with domid %u",
+ cds->domid);
+ goto out;
+ }
+
if (init_device_subkind(cds))
goto out;
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* [PATCH v10 31/31] cmdline switches and config vars to control colo-proxy
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (29 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 30/31] setup and control colo proxy on secondary side Wen Congyang
@ 2016-02-22 2:52 ` Wen Congyang
2016-03-02 15:05 ` Wei Liu
2016-02-25 16:05 ` [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wei Liu
31 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-22 2:52 UTC (permalink / raw)
To: xen devel, Konrad Rzeszutek Wilk, Andrew Cooper, Ian Campbell,
Ian Jackson, Wei Liu
Cc: Lars Kurth, Changlong Xie, Wen Congyang, Gui Jianfeng,
Jiang Yunhong, Dong Eddie, Shriram Rajagopalan, Yang Hongyang
Add cmdline switches to 'xl migrate-receive' command to specify
a domain-specific hotplug script to setup COLO proxy.
Add a new config var 'colo.default.agentscript' to xl.conf, that
allows the user to override the default global script used to
setup COLO proxy.
Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
---
docs/man/xl.conf.pod.5 | 6 ++++++
docs/man/xl.pod.1 | 1 -
tools/libxl/libxl.c | 6 ++++++
tools/libxl/libxl_create.c | 14 ++++++++++++--
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl.c | 3 +++
tools/libxl/xl.h | 1 +
tools/libxl/xl_cmdimpl.c | 47 ++++++++++++++++++++++++++++++++++-----------
8 files changed, 65 insertions(+), 14 deletions(-)
diff --git a/docs/man/xl.conf.pod.5 b/docs/man/xl.conf.pod.5
index 8ae19bb..8f7fd28 100644
--- a/docs/man/xl.conf.pod.5
+++ b/docs/man/xl.conf.pod.5
@@ -111,6 +111,12 @@ Configures the default script used by Remus to setup network buffering.
Default: C</etc/xen/scripts/remus-netbuf-setup>
+=item B<colo.default.proxyscript="PATH">
+
+Configures the default script used by COLO to setup colo-proxy.
+
+Default: C</etc/xen/scripts/colo-proxy-setup>
+
=item B<output_format="json|sxp">
Configures the default output format used by xl when printing "machine
diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
index 4f1901d..edeafcf 100644
--- a/docs/man/xl.pod.1
+++ b/docs/man/xl.pod.1
@@ -454,7 +454,6 @@ N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
Disk replication support is limited to DRBD disks.
COLO support in xl is still in experimental (proof-of-concept) phase.
- There is no support for network at the moment.
B<OPTIONS>
diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
index f691628..054d3b5 100644
--- a/tools/libxl/libxl.c
+++ b/tools/libxl/libxl.c
@@ -3392,6 +3392,11 @@ void libxl__device_nic_add(libxl__egc *egc, uint32_t domid,
flexarray_append(back, nic->ifname);
}
+ if (nic->forwarddev) {
+ flexarray_append(back, "forwarddev");
+ flexarray_append(back, nic->forwarddev);
+ }
+
flexarray_append(back, "mac");
flexarray_append(back,GCSPRINTF(LIBXL_MAC_FMT, LIBXL_MAC_BYTES(nic->mac)));
if (nic->ip) {
@@ -3514,6 +3519,7 @@ static int libxl__device_nic_from_xs_be(libxl__gc *gc,
nic->ip = READ_BACKEND(NOGC, "ip");
nic->bridge = READ_BACKEND(NOGC, "bridge");
nic->script = READ_BACKEND(NOGC, "script");
+ nic->forwarddev = READ_BACKEND(NOGC, "forwarddev");
/* vif_ioemu nics use the same xenstore entries as vif interfaces */
tmp = READ_BACKEND(gc, "type");
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index c68dd49..7baa95f 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1073,6 +1073,11 @@ static void domcreate_bootloader_done(libxl__egc *egc,
crs->recv_fd = restore_fd;
crs->hvm = (info->type == LIBXL_DOMAIN_TYPE_HVM);
crs->callback = libxl__colo_restore_setup_done;
+ if (dcs->colo_proxy_script)
+ crs->colo_proxy_script = libxl__strdup(gc, dcs->colo_proxy_script);
+ else
+ crs->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
+ libxl__xen_script_dir_path());
libxl__colo_restore_setup(egc, crs);
break;
case LIBXL_CHECKPOINTED_STREAM_REMUS:
@@ -1615,6 +1620,7 @@ static void domain_create_cb(libxl__egc *egc,
static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid, int restore_fd, int send_back_fd,
const libxl_domain_restore_params *params,
+ const char *colo_proxy_script,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
@@ -1638,6 +1644,7 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
}
cdcs->dcs.callback = domain_create_cb;
cdcs->dcs.domid_soft_reset = INVALID_DOMID;
+ cdcs->dcs.colo_proxy_script = colo_proxy_script;
libxl__ao_progress_gethow(&cdcs->dcs.aop_console_how, aop_console_how);
cdcs->domid_out = domid;
@@ -1824,7 +1831,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncprogress_how *aop_console_how)
{
unset_disk_colo_restore(d_config);
- return do_domain_create(ctx, d_config, domid, -1, -1, NULL,
+ return do_domain_create(ctx, d_config, domid, -1, -1, NULL, NULL,
ao_how, aop_console_how);
}
@@ -1835,14 +1842,17 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
+ char *colo_proxy_script = NULL;
+
if (params->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
+ colo_proxy_script = params->colo_proxy_script;
set_disk_colo_restore(d_config);
} else {
unset_disk_colo_restore(d_config);
}
return do_domain_create(ctx, d_config, domid, restore_fd, send_back_fd,
- params, ao_how, aop_console_how);
+ params, colo_proxy_script, ao_how, aop_console_how);
}
int libxl_domain_soft_reset(libxl_ctx *ctx,
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 6b57aba..883d3a8 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -385,6 +385,7 @@ libxl_domain_create_info = Struct("domain_create_info",[
libxl_domain_restore_params = Struct("domain_restore_params", [
("checkpointed_stream", integer),
("stream_version", uint32, {'init_val': '1'}),
+ ("colo_proxy_script", string),
])
libxl_domain_sched_params = Struct("domain_sched_params",[
diff --git a/tools/libxl/xl.c b/tools/libxl/xl.c
index dfae84a..a272258 100644
--- a/tools/libxl/xl.c
+++ b/tools/libxl/xl.c
@@ -45,6 +45,7 @@ char *default_bridge = NULL;
char *default_gatewaydev = NULL;
char *default_vifbackend = NULL;
char *default_remus_netbufscript = NULL;
+char *default_colo_proxy_script = NULL;
enum output_format default_output_format = OUTPUT_FORMAT_JSON;
int claim_mode = 1;
bool progress_use_cr = 0;
@@ -179,6 +180,8 @@ static void parse_global_config(const char *configfile,
xlu_cfg_replace_string (config, "remus.default.netbufscript",
&default_remus_netbufscript, 0);
+ xlu_cfg_replace_string (config, "colo.default.proxyscript",
+ &default_colo_proxy_script, 0);
xlu_cfg_destroy(config);
}
diff --git a/tools/libxl/xl.h b/tools/libxl/xl.h
index bdab125..709c3fd 100644
--- a/tools/libxl/xl.h
+++ b/tools/libxl/xl.h
@@ -189,6 +189,7 @@ extern char *default_bridge;
extern char *default_gatewaydev;
extern char *default_vifbackend;
extern char *default_remus_netbufscript;
+extern char *default_colo_proxy_script;
extern char *blkdev_start;
enum output_format {
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 0dc7220..6c2589c 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -158,6 +158,7 @@ struct domain_create {
const char *config_file;
char *extra_config; /* extra config string */
const char *restore_file;
+ char *colo_proxy_script;
int migrate_fd; /* -1 means none */
int send_back_fd; /* -1 means none */
char **migration_domname_r; /* from malloc */
@@ -1048,6 +1049,8 @@ static int parse_nic_config(libxl_device_nic *nic, XLU_Config **config, char *to
replace_string(&nic->model, oparg);
} else if (MATCH_OPTION("rate", token, oparg)) {
parse_vif_rate(config, oparg, nic);
+ } else if (MATCH_OPTION("forwarddev", token, oparg)) {
+ replace_string(&nic->forwarddev, oparg);
} else if (MATCH_OPTION("accel", token, oparg)) {
fprintf(stderr, "the accel parameter for vifs is currently not supported\n");
} else {
@@ -2891,6 +2894,7 @@ start:
params.checkpointed_stream = dom_info->checkpointed_stream;
params.stream_version =
(hdr.mandatory_flags & XL_MANDATORY_FLAG_STREAMv2) ? 2 : 1;
+ params.colo_proxy_script = dom_info->colo_proxy_script;
ret = libxl_domain_create_restore(ctx, &d_config,
&domid, restore_fd,
@@ -4433,7 +4437,8 @@ static void migrate_domain(uint32_t domid, const char *rune, int debug,
static void migrate_receive(int debug, int daemonize, int monitor,
int send_fd, int recv_fd,
- libxl_checkpointed_stream checkpointed)
+ libxl_checkpointed_stream checkpointed,
+ char *colo_proxy_script)
{
uint32_t domid;
int rc, rc2;
@@ -4462,6 +4467,7 @@ static void migrate_receive(int debug, int daemonize, int monitor,
dom_info.send_back_fd = send_fd;
dom_info.migration_domname_r = &migration_domname;
dom_info.checkpointed_stream = checkpointed;
+ dom_info.colo_proxy_script = colo_proxy_script;
if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
/* COLO uses stdout to send control message to master */
dom_info.quiet = 1;
@@ -4658,8 +4664,9 @@ int main_migrate_receive(int argc, char **argv)
int debug = 0, daemonize = 1, monitor = 1;
libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
int opt;
+ char *script = NULL;
- SWITCH_FOREACH_OPT(opt, "Fedrc", NULL, "migrate-receive", 0) {
+ SWITCH_FOREACH_OPT(opt, "Fedrcn:", NULL, "migrate-receive", 0) {
case 'F':
daemonize = 0;
break;
@@ -4676,6 +4683,9 @@ int main_migrate_receive(int argc, char **argv)
case 'c':
checkpointed = LIBXL_CHECKPOINTED_STREAM_COLO;
break;
+ case 'n':
+ script = optarg;
+ break;
}
if (argc-optind != 0) {
@@ -4684,7 +4694,7 @@ int main_migrate_receive(int argc, char **argv)
}
migrate_receive(debug, daemonize, monitor,
STDOUT_FILENO, STDIN_FILENO,
- checkpointed);
+ checkpointed, script);
return 0;
}
@@ -8088,8 +8098,10 @@ int main_remus(int argc, char **argv)
r_info.interval = 200;
if (libxl_defbool_val(r_info.colo)) {
- if (r_info.interval || libxl_defbool_val(r_info.blackhole)) {
- perror("Option -c conflicts with -i or -b");
+ if (r_info.interval || libxl_defbool_val(r_info.blackhole) ||
+ !libxl_defbool_is_default(r_info.netbuf) ||
+ !libxl_defbool_is_default(r_info.diskbuf)) {
+ perror("option -c is conflict with -i, -d, -n or -b");
exit(-1);
}
@@ -8100,8 +8112,12 @@ int main_remus(int argc, char **argv)
}
}
- if (!r_info.netbufscript)
- r_info.netbufscript = default_remus_netbufscript;
+ if (!r_info.netbufscript) {
+ if (libxl_defbool_val(r_info.colo))
+ r_info.netbufscript = default_colo_proxy_script;
+ else
+ r_info.netbufscript = default_remus_netbufscript;
+ }
if (libxl_defbool_val(r_info.blackhole)) {
send_fd = open("/dev/null", O_RDWR, 0644);
@@ -8114,10 +8130,19 @@ int main_remus(int argc, char **argv)
if (!ssh_command[0]) {
rune = host;
} else {
- xasprintf(&rune, "exec %s %s xl migrate-receive %s %s",
- ssh_command, host,
- libxl_defbool_val(r_info.colo) ? "-c" : "-r",
- daemonize ? "" : " -e");
+ if (!libxl_defbool_val(r_info.colo)) {
+ xasprintf(&rune, "exec %s %s xl migrate-receive %s %s",
+ ssh_command, host,
+ "-r",
+ daemonize ? "" : " -e");
+ } else {
+ xasprintf(&rune, "exec %s %s xl migrate-receive %s %s %s %s",
+ ssh_command, host,
+ "-c",
+ r_info.netbufscript ? "-n" : "",
+ r_info.netbufscript ? r_info.netbufscript : "",
+ daemonize ? "" : " -e");
+ }
}
save_domain_core_begin(domid, NULL, &config_data, &config_len);
--
2.5.0
^ permalink raw reply related [flat|nested] 70+ messages in thread
* Re: [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state
2016-02-22 2:52 ` [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state Wen Congyang
@ 2016-02-25 15:53 ` Wei Liu
2016-02-26 1:55 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:53 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Anthony Perard,
Dong Eddie, Gui Jianfeng, Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:05AM +0800, Wen Congyang wrote:
> In normal migration, the qemu state is passed to qemu as a parameter.
> With COLO, secondary vm is running. So we will do the following steps
> at every checkpoint:
> 1. suspend both primary vm and secondary vm
> 2. sync the state
> 3. resume both primary vm and secondary vm
> Primary will send qemu's state in step2, and secondary's qemu should
> read it and restore the state before it is resumed. We can not pass
> the state to qemu as a parameter because secondary QEMU already started
is already started.
> at this point, so we introduce libxl__domain_restore_device_model() to
> do it. This API MUST be called before resuming secondary vm.
>
The last sentence is of no relevancy of this function itself, so I would
just remove it if this patch will be resent in its current form.
And some comments below.
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Cc: Anthony Perard <anthony.perard@citrix.com>
> Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
> ---
> tools/libxl/libxl_dom_save.c | 20 ++++++++++++++++++++
> tools/libxl/libxl_internal.h | 4 ++++
> tools/libxl/libxl_qmp.c | 10 ++++++++++
> 3 files changed, 34 insertions(+)
>
> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
> index 4eb7960..7d345f9 100644
> --- a/tools/libxl/libxl_dom_save.c
> +++ b/tools/libxl/libxl_dom_save.c
> @@ -512,6 +512,26 @@ int libxl__restore_emulator_xenstore_data(libxl__domain_create_state *dcs,
> return rc;
> }
>
> +int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
> + const char *restore_file)
> +{
> + int rc;
> +
> + switch (libxl__device_model_version_running(gc, domid)) {
> + case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
> + /* Will never be supported. */
> + rc = ERROR_INVAL;
> + break;
> + case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
> + rc = libxl__qmp_restore(gc, domid, restore_file);
> + break;
> + default:
> + rc = ERROR_INVAL;
> + }
> +
> + return rc;
> +}
> +
The function name suggests that it should be universally applied to all
device model cases and all places that involves restoring device model.
This is not the case in this series. I search for this function in the
rest of this series, it seems to be only used by COLO (in patch #10).
It's also unclear where the other half libxl__domain_save_device_model
is. I don't think this is your problem, though.
Based on the two reasons above, I would say let's not have this function
at all. You can call libxl__qmp_restore in COLO code directly.
Does this sound plausible?
If you agree, this patch can be turned into introduction of
libxl__qmp_restore.
> /*
> * Local variables:
> * mode: C
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index a1aae97..5320e5e 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -1119,6 +1119,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, uint32_t domid,
> const char *old_name, const char *new_name,
> xs_transaction_t trans);
>
> +_hidden int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
> + const char *restore_file);
> _hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid);
>
> _hidden const char *libxl__userdata_path(libxl__gc *gc, uint32_t domid,
> @@ -1762,6 +1764,8 @@ _hidden int libxl__qmp_stop(libxl__gc *gc, int domid);
> _hidden int libxl__qmp_resume(libxl__gc *gc, int domid);
> /* Save current QEMU state into fd. */
> _hidden int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename);
> +/* Load current QEMU state from file. */
> +_hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char *filename);
> /* Set dirty bitmap logging status */
> _hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool enable);
> _hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const libxl_device_disk *disk);
> diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
> index 714038b..eec8a44 100644
> --- a/tools/libxl/libxl_qmp.c
> +++ b/tools/libxl/libxl_qmp.c
> @@ -905,6 +905,16 @@ int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename)
> NULL, NULL);
> }
>
> +int libxl__qmp_restore(libxl__gc *gc, int domid, const char *state_file)
> +{
> + libxl__json_object *args = NULL;
> +
> + qmp_parameters_add_string(gc, &args, "filename", state_file);
> +
> + return qmp_run_command(gc, domid, "xen-load-devices-state", args,
> + NULL, NULL);
> +}
> +
This looks OK.
> static int qmp_change(libxl__gc *gc, libxl__qmp_handler *qmp,
> char *device, char *target, char *arg)
> {
> --
> 2.5.0
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 04/31] tools/libxl: Introduce new helper function dup_fd_helper()
2016-02-22 2:52 ` [PATCH v10 04/31] tools/libxl: Introduce new helper function dup_fd_helper() Wen Congyang
@ 2016-02-25 15:53 ` Wei Liu
0 siblings, 0 replies; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:53 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:08AM +0800, Wen Congyang wrote:
> It is pure refactoring and no functional changes.
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Assuming this is going to be used later:
Acked-by: Wei Liu <wei.liu2@citrix.com>
> ---
> tools/libxl/libxl_save_callout.c | 24 ++++++++++++++++--------
> 1 file changed, 16 insertions(+), 8 deletions(-)
>
> diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
> index 7f1f5d4..a885504 100644
> --- a/tools/libxl/libxl_save_callout.c
> +++ b/tools/libxl/libxl_save_callout.c
> @@ -118,6 +118,21 @@ void libxl__save_helper_init(libxl__save_helper_state *shs)
> }
>
> /*----- helper execution -----*/
> +static int dup_fd_helper(libxl__gc *gc, int fd, const char *what)
> +{
> + int dup_fd = fd;
> +
> + if (fd <= 2) {
> + dup_fd = dup(fd);
> + if (dup_fd < 0) {
> + LOGE(ERROR,"dup %s", what);
> + exit(-1);
> + }
> + }
> + libxl_fd_set_cloexec(CTX, dup_fd, 0);
> +
> + return dup_fd;
> +}
>
> /*
> * Both save and restore share four parameters:
> @@ -186,14 +201,7 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
>
> pid_t pid = libxl__ev_child_fork(gc, &shs->child, helper_exited);
> if (!pid) {
> - if (stream_fd <= 2) {
> - stream_fd = dup(stream_fd);
> - if (stream_fd < 0) {
> - LOGE(ERROR,"dup migration stream fd");
> - exit(-1);
> - }
> - }
> - libxl_fd_set_cloexec(CTX, stream_fd, 0);
> + stream_fd = dup_fd_helper(gc, stream_fd, "migration stream fd");
> *stream_fd_arg = GCSPRINTF("%d", stream_fd);
>
> for (i=0; i<num_preserve_fds; i++)
> --
> 2.5.0
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 05/31] tools/libx{l, c}: add back channel to libxc
2016-02-22 2:52 ` [PATCH v10 05/31] tools/libx{l, c}: add back channel to libxc Wen Congyang
@ 2016-02-25 15:54 ` Wei Liu
0 siblings, 0 replies; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:54 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:09AM +0800, Wen Congyang wrote:
> In COLO mode, both VMs are running, and are considered in sync if the
> visible network traffic is identical. After some time, they fall out of
> sync.
>
> At this point, the two VMs have definitely diverged. Lets call the
> primary dirty bitmap set A, while the secondary dirty bitmap set B.
>
> Sets A and B are different.
>
> Under normal migration, the page data for set A will be sent from the
> primary to the secondary.
>
> However, the set difference B - A (the one in B but not in A, lets
> call this C) is out-of-date on the secondary (with respect to the
> primary) and will not be sent by the primary (to secondary), as it
> was not memory dirtied by the primary. The secondary needs C page data
> to reconstruct an exact copy of the primary at the checkpoint.
>
> The secondary cannot calculate C as it doesn't know A. Instead, the
> secondary must send B to the primary, at which point the primary
> calculates the union of A and B (lets call this D) which is all the
> pages dirtied by both the primary and the secondary, and sends all page
> data covered by D.
>
> In the general case, D is a superset of both A and B. Without the
> backchannel dirty bitmap, a COLO checkpoint can't reconstruct a valid
> copy of the primary.
>
> We transfer the dirty bitmap on libxc side, so we need to introduce back
> channel to libxc.
>
> Note: it is different from the paper. We change the original design to
> the current one, according to our following concerns:
> 1. The original design needs extra memory on Secondary host. When there's
> multiple backups on one host, the memory cost is high.
> 2. The memory cache code will be another 1k+, it will make the review
> more time consuming.
>
> Note: the back channel will be used in the patch
> libxc/restore: send dirty pfn list to primary when checkpoint under COLO
> to send dirty pfn list from secondary to primary. The patch is posted in
> another series.
I think you need to update the commit message a bit. I take the liberty
of writing one for you:
This patch merely adds new parameters to various prototypes and
functions. The new parameters are used in later patch called
"libxc/restore: send dirty pfn list to primary when checkpoint under
COLO".
I would normally ask the submitter to merge patch like this to the
actual patch that uses the parameter. It you can do that, that would be
great. But this patch series is already quite large, if this involves a
lot of conflict resolution on your side, just leave it as-is.
Either way, please confirm your intention before doing actual work.
Bottom line: patch like this needs to compile and not introduce any
regression. I think that's the case here.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 06/31] docs: add colo readme
2016-02-22 2:52 ` [PATCH v10 06/31] docs: add colo readme Wen Congyang
@ 2016-02-25 15:54 ` Wei Liu
0 siblings, 0 replies; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:54 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:10AM +0800, Wen Congyang wrote:
> add colo readme, refer to
> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
> ---
> docs/README.colo | 9 +++++++++
> 1 file changed, 9 insertions(+)
> create mode 100644 docs/README.colo
>
> diff --git a/docs/README.colo b/docs/README.colo
> new file mode 100644
> index 0000000..466eb72
> --- /dev/null
> +++ b/docs/README.colo
> @@ -0,0 +1,9 @@
> +COLO FT/HA (COarse-grain LOck-stepping Virtual Machines for Non-stop Service)
> +project is a high availability solution. Both primary VM (PVM) and secondary VM
> +(SVM) run in parallel. They receive the same request from client, and generate
> +response in parallel too. If the response packets from PVM and SVM are
> +identical, they are released immediately. Otherwise, a VM checkpoint (on demand)
> +is conducted.
> +
> +See the website at http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
> +for details.
> --
> 2.5.0
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 07/31] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams
2016-02-22 2:52 ` [PATCH v10 07/31] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams Wen Congyang
@ 2016-02-25 15:54 ` Wei Liu
2016-02-26 1:59 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:54 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:11AM +0800, Wen Congyang wrote:
> It is the negotiation record for COLO.
> Primary->Secondary:
> control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
> Secondary->Primary:
> 0x00000001: Secondary VM is suspended
> 0x00000002: Secondary VM is ready
> 0x00000003: Secondary VM is resumed
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> ---
> docs/specs/libxl-migration-stream.pandoc | 25 +++++++++++++++++++++++--
> tools/libxl/libxl_sr_stream_format.h | 11 +++++++++++
> tools/python/xen/migration/libxl.py | 9 +++++++++
> 3 files changed, 43 insertions(+), 2 deletions(-)
>
> diff --git a/docs/specs/libxl-migration-stream.pandoc b/docs/specs/libxl-migration-stream.pandoc
> index 2c97d86..5166d66 100644
> --- a/docs/specs/libxl-migration-stream.pandoc
> +++ b/docs/specs/libxl-migration-stream.pandoc
> @@ -1,6 +1,6 @@
> % LibXenLight Domain Image Format
> % Andrew Cooper <<andrew.cooper3@citrix.com>>
You can add yours and Hongyang's name and email here as well.
> -% Revision 1
> +% Revision 2
>
> Introduction
> ============
> @@ -119,7 +119,9 @@ type 0x00000000: END
>
> 0x00000004: CHECKPOINT_END
>
> - 0x00000005 - 0x7FFFFFFF: Reserved for future _mandatory_
> + 0x00000005: CHECKPOINT_STATE
> +
> + 0x00000006 - 0x7FFFFFFF: Reserved for future _mandatory_
> records.
>
> 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
> @@ -249,6 +251,25 @@ A checkpoint end record marks the end of a checkpoint in the image.
> The end record contains no fields; its body_length is 0.
>
>
> +CHECKPOINT\_STATE
> +--------------
> +
> +A checkpoint state record contains the control information for checkpoint.
> +
> + 0 1 2 3 4 5 6 7 octet
> + +------------------------+------------------------+
> + | control_id | padding |
> + +------------------------+------------------------+
> +
> +--------------------------------------------------------------------
> +Field Description
> +------------ ---------------------------------------------------
> +control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
> + 0x00000001: Secondary VM is suspended
> + 0x00000002: Secondary VM is ready
> + 0x00000003: Secondary VM is resumed
You commit message seems to suggest 0 can only go from primary to
secondary, while the other three only go from secondary to primary. You
probably need to spell out the direction here (and later enforce this
behaviour in code).
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 12/31] tools/libxl: add back channel support to read stream
2016-02-22 2:52 ` [PATCH v10 12/31] tools/libxl: add back channel support to read stream Wen Congyang
@ 2016-02-25 15:54 ` Wei Liu
2016-02-26 2:16 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:54 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:16AM +0800, Wen Congyang wrote:
> This is used by primay to read records sent by secondary.
>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> ---
> tools/libxl/libxl_create.c | 1 +
> tools/libxl/libxl_internal.h | 1 +
> tools/libxl/libxl_stream_read.c | 27 +++++++++++++++++++++++----
> 3 files changed, 25 insertions(+), 4 deletions(-)
>
> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
> index 525bf85..fc746fb 100644
> --- a/tools/libxl/libxl_create.c
> +++ b/tools/libxl/libxl_create.c
> @@ -1030,6 +1030,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
> dcs->srs.dcs = dcs;
> dcs->srs.fd = restore_fd;
> dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
> + dcs->srs.back_channel = false;
> dcs->srs.completion_callback = domcreate_stream_done;
>
> if (restore_fd >= 0) {
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index cdf6751..32957ca 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -3420,6 +3420,7 @@ struct libxl__stream_read_state {
> libxl__domain_create_state *dcs;
> int fd;
> bool legacy;
> + bool back_channel;
> void (*completion_callback)(libxl__egc *egc,
> libxl__stream_read_state *srs,
> int rc);
> diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
> index f4781eb..02a2c46 100644
> --- a/tools/libxl/libxl_stream_read.c
> +++ b/tools/libxl/libxl_stream_read.c
> @@ -118,6 +118,15 @@
> * record, and therefore the buffered state is inconsistent. In
> * libxl__xc_domain_restore_done(), we just complete the stream and
> * stream->completion_callback() will be called to resume the guest
> + *
> + * For back channel stream:
> + * - libxl__stream_read_start()
> + * - Set up the stream to running state
> + *
> + * - libxl__stream_read_continue()
> + * - Set up reading the next record from a started stream.
> + * Add some codes to process_record() to handle the record.
> + * Then call stream->checkpoint_callback() to return.
> */
>
> /* Success/error/cleanup handling. */
> @@ -221,6 +230,17 @@ void libxl__stream_read_start(libxl__egc *egc,
> stream->running = true;
> stream->phase = SRS_PHASE_NORMAL;
>
> + dc->ao = stream->ao;
> + dc->copywhat = "restore v2 stream";
> + dc->writefd = -1;
> +
> + if (stream->back_channel) {
> + assert(!stream->legacy);
> +
> + dc->readfd = stream->fd;
Why is this needed? dc->readfd is set to stream->fd no matter it is back
channel or not. This can be moved outside this if {}.
> + return;
> + }
> +
> if (stream->legacy) {
> /* Convert the legacy stream. */
> libxl__conversion_helper_state *chs = &stream->chs;
> @@ -243,10 +263,7 @@ void libxl__stream_read_start(libxl__egc *egc,
> }
> /* stream->fd is now a v2 stream. */
>
> - dc->ao = stream->ao;
> - dc->copywhat = "restore v2 stream";
> dc->readfd = stream->fd;
> - dc->writefd = -1;
>
> /* Start reading the stream header. */
> rc = setup_read(stream, "stream header",
> @@ -762,7 +779,9 @@ static void stream_done(libxl__egc *egc,
> LIBXL_STAILQ_FOREACH_SAFE(rec, &stream->record_queue, entry, trec)
> free_record(rec);
>
> - check_all_finished(egc, stream, rc);
> + if (!stream->back_channel)
> + /* back channel stream doesn't have restore helper */
> + check_all_finished(egc, stream, rc);
Even if it doesn't have restore helper, check_all_finished also checks
if the stream and the conversion helper are till in use. The
explanation in the comment doesn't seem to justify this change.
Wei.
> }
>
> void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
> --
> 2.5.0
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 10/31] tools/libxl: add back channel support to write stream
2016-02-22 2:52 ` [PATCH v10 10/31] tools/libxl: add back channel support to write stream Wen Congyang
@ 2016-02-25 15:54 ` Wei Liu
2016-02-26 2:11 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:54 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:14AM +0800, Wen Congyang wrote:
> Add back channel support to write stream. If the write stream is
> a back channel stream, this means the write stream is used by
> Secondary to send some records back.
>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> ---
> tools/libxl/libxl_dom_save.c | 1 +
> tools/libxl/libxl_internal.h | 1 +
> tools/libxl/libxl_stream_write.c | 26 ++++++++++++++++++++------
> 3 files changed, 22 insertions(+), 6 deletions(-)
>
> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
> index 72b61c7..18946c5 100644
> --- a/tools/libxl/libxl_dom_save.c
> +++ b/tools/libxl/libxl_dom_save.c
> @@ -404,6 +404,7 @@ void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
> dss->sws.ao = dss->ao;
> dss->sws.dss = dss;
> dss->sws.fd = dss->fd;
> + dss->sws.back_channel = false;
> dss->sws.completion_callback = stream_done;
>
> libxl__stream_write_start(egc, &dss->sws);
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 3d3e8e8..e02e554 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -3044,6 +3044,7 @@ struct libxl__stream_write_state {
> libxl__ao *ao;
> libxl__domain_save_state *dss;
> int fd;
> + bool back_channel;
> void (*completion_callback)(libxl__egc *egc,
> libxl__stream_write_state *sws,
> int rc);
> diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
> index f6ea55d..5379126 100644
> --- a/tools/libxl/libxl_stream_write.c
> +++ b/tools/libxl/libxl_stream_write.c
> @@ -49,6 +49,13 @@
> * - if (hvm)
> * - Emulator context record
> * - Checkpoint end record
> + *
> + * For back channel stream:
> + * - libxl__stream_write_start()
> + * - Set up the stream to running state
> + *
> + * - Add a new API to write the record. When the record is written
> + * out, call stream->checkpoint_callback() to return.
What does this mean? Which new API?
> */
>
> /* Success/error/cleanup handling. */
> @@ -225,6 +232,15 @@ void libxl__stream_write_start(libxl__egc *egc,
>
> stream->running = true;
>
> + dc->ao = ao;
> + dc->readfd = -1;
> + dc->copywhat = "save v2 stream";
> + dc->writefd = stream->fd;
> + dc->maxsz = -1;
> +
> + if (stream->back_channel)
> + return;
> +
There seems to be very subtle change of behaviour.
Before this patch, the dc->* are not set until the emulator check is
done. With this path, it is possible in the normal case some of the
fields are initialised in the error path.
I think this is OK given the callbacks in the upper layer and in
the writer don't rely on those fields to clean up. Just one thing to
note.
That said, I suggest you move all initialisation of dc->* in one place.
> if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
> stream->device_model_version =
> libxl__device_model_version_running(gc, dss->domid);
> @@ -249,12 +265,7 @@ void libxl__stream_write_start(libxl__egc *egc,
> stream->emu_sub_hdr.index = 0;
> }
>
> - dc->ao = ao;
> - dc->readfd = -1;
> dc->writewhat = "stream header";
> - dc->copywhat = "save v2 stream";
> - dc->writefd = stream->fd;
> - dc->maxsz = -1;
> dc->callback = stream_header_done;
>
> rc = libxl__datacopier_start(dc);
> @@ -279,6 +290,7 @@ void libxl__stream_write_start_checkpoint(libxl__egc *egc,
> {
> assert(stream->running);
> assert(!stream->in_checkpoint);
> + assert(!stream->back_channel);
> stream->in_checkpoint = true;
>
> write_emulator_xenstore_record(egc, stream);
> @@ -590,7 +602,9 @@ static void stream_done(libxl__egc *egc,
> libxl__carefd_close(stream->emu_carefd);
> free(stream->emu_body);
>
> - check_all_finished(egc, stream, rc);
> + if (!stream->back_channel)
> + /* back channel stream doesn't have save helper */
> + check_all_finished(egc, stream, rc);
Though it doesn't have helper, do you not need to check if the back
channel stream itself is OK? The comment itself doesn't seem to justify
this change.
Wei.
> }
>
> static void checkpoint_done(libxl__egc *egc,
> --
> 2.5.0
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code
2016-02-22 2:52 ` [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code Wen Congyang
@ 2016-02-25 15:56 ` Wei Liu
2016-02-26 2:30 ` Wen Congyang
2016-03-01 10:06 ` Wen Congyang
0 siblings, 2 replies; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:56 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:20AM +0800, Wen Congyang wrote:
> Secondary vm is running in colo mode. So we will do
> the following things again and again:
> 1. Resume secondary vm
> a. Send CHECKPOINT_SVM_READY to master.
> b. If it is not the first resume, call libxl__checkpoint_devices_preresume().
> c. If it is the first resume(resume right after live migration),
> - call libxl__xc_domain_restore_done() to build the secondary vm.
> - enable secondary vm's logdirty.
> - call libxl__domain_resume() to resume secondary vm.
> - call libxl__checkpoint_devices_setup() to setup checkpoint devices.
> d. Send CHECKPOINT_SVM_RESUMED to master.
> 2. Wait a new checkpoint
> a. Call libxl__checkpoint_devices_commit().
> b. Read CHECKPOINT_NEW from master.
> 3. Suspend secondary vm
> a. Suspend secondary vm.
> b. Call libxl__checkpoint_devices_postsuspend().
> c. Send CHECKPOINT_SVM_SUSPENDED to master.
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> ---
> tools/libxc/xc_sr_common.h | 2 +
> tools/libxc/xc_sr_save.c | 3 +-
> tools/libxl/Makefile | 1 +
> tools/libxl/libxl_colo.h | 24 +
> tools/libxl/libxl_colo_restore.c | 1038 ++++++++++++++++++++++++++++++++++++++
> tools/libxl/libxl_create.c | 37 ++
> tools/libxl/libxl_internal.h | 19 +
> tools/libxl/libxl_save_callout.c | 7 +-
> tools/libxl/libxl_stream_read.c | 12 +
> tools/libxl/libxl_types.idl | 1 +
There is a bunch of TODOs in libxl_colo.c but I don't think you're in a
better position to judge whether they should be blocker or not.
> 10 files changed, 1142 insertions(+), 2 deletions(-)
> create mode 100644 tools/libxl/libxl_colo.h
> create mode 100644 tools/libxl/libxl_colo_restore.c
>
> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
> index 5d9f497..2bfed64 100644
> --- a/tools/libxc/xc_sr_common.h
> +++ b/tools/libxc/xc_sr_common.h
> @@ -184,10 +184,12 @@ struct xc_sr_context
> * migration stream
> * 0: Plain VM
> * 1: Remus
> + * 2: COLO
> */
> enum {
> MIG_STREAM_NONE, /* plain stream */
> MIG_STREAM_REMUS,
> + MIG_STREAM_COLO,
> } migration_stream;
>
> union /* Common save or restore data. */
> diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
> index fe210cc..7393355 100644
> --- a/tools/libxc/xc_sr_save.c
> +++ b/tools/libxc/xc_sr_save.c
> @@ -846,7 +846,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
>
> /* If altering migration_stream update this assert too. */
> assert(checkpointed_stream == MIG_STREAM_NONE ||
> - checkpointed_stream == MIG_STREAM_REMUS);
> + checkpointed_stream == MIG_STREAM_REMUS ||
> + checkpointed_stream == MIG_STREAM_COLO);
>
> /*
> * TODO: Find some time to better tweak the live migration algorithm.
[...]
> +
> +#include "libxl_osdeps.h" /* must come before any other headers */
> +
> +#include "libxl_internal.h"
> +#include "libxl_colo.h"
> +#include "libxl_sr_stream_format.h"
> +
> +enum {
> + LIBXL_COLO_SETUPED,
> + LIBXL_COLO_SUSPENDED,
> + LIBXL_COLO_RESUMED,
> +};
> +
> +typedef struct libxl__colo_restore_checkpoint_state libxl__colo_restore_checkpoint_state;
> +struct libxl__colo_restore_checkpoint_state {
> + libxl__domain_suspend_state dsps;
> + libxl__logdirty_switch lds;
> + libxl__colo_restore_state *crs;
> + libxl__stream_write_state sws;
> + int status;
> + bool preresume;
> + /* used for teardown */
> + int teardown_devices;
> + int saved_rc;
> + char *state_file;
> +
> + void (*callback)(libxl__egc *,
> + libxl__colo_restore_checkpoint_state *,
> + int);
> +};
> +
Shouldn't the enum and struct belong to libxl_colo.h ?
> +
> +static void libxl__colo_restore_domain_resume_callback(void *data);
> +static void libxl__colo_restore_domain_checkpoint_callback(void *data);
> +static void libxl__colo_restore_domain_wait_checkpoint_callback(void *data);
> +static void libxl__colo_restore_domain_suspend_callback(void *data);
> +
> +static const libxl__checkpoint_device_instance_ops *colo_restore_ops[] = {
> + NULL,
> +};
> +
It would be helpful to list the callbacks at the beginning of the time
in the order they are supposed to occur.
See libxl_create.c for example. Search for "Event callbacks, in this
order".
I've tried to map the algorithm you described in commit message to all
the callbacks, but without some references it is just too time consuming
from my end.
I think what I'm going to do is to make sure the normal path that
doesn't use COLO is not broken and leave the internal to you and Ian (if
he fancies to dig into details).
[...]
> +
> +void libxl__colo_restore_setup(libxl__egc *egc,
> + libxl__colo_restore_state *crs)
> +{
> + libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
> + libxl__colo_restore_checkpoint_state *crcs;
> + int rc = ERROR_FAIL;
> +
> + /* Convenience aliases */
> + libxl__srm_restore_autogen_callbacks *const callbacks =
> + &dcs->srs.shs.callbacks.restore.a;
> + const int domid = crs->domid;
> +
> + STATE_AO_GC(crs->ao);
> +
> + GCNEW(crcs);
> + crs->crcs = crcs;
> + crcs->crs = crs;
> +
> + /* setup dsps */
> + crcs->dsps.ao = ao;
> + crcs->dsps.domid = domid;
> + if (init_dsps(&crcs->dsps))
> + goto err;
> +
> + callbacks->suspend = libxl__colo_restore_domain_suspend_callback;
> + callbacks->postcopy = libxl__colo_restore_domain_resume_callback;
> + callbacks->checkpoint = libxl__colo_restore_domain_checkpoint_callback;
> + callbacks->wait_checkpoint = libxl__colo_restore_domain_wait_checkpoint_callback;
> +
> + /*
> + * Secondary vm is running in colo mode, so we need to call
> + * libxl__xc_domain_restore_done() to create secondary vm.
> + * But we will exit in domain_create_cb(). So replace the
> + * callback here.
> + */
> + crs->saved_cb = dcs->callback;
> + dcs->callback = libxl__colo_domain_create_cb;
> + crcs->state_file = GCSPRINTF(LIBXL_DEVICE_MODEL_RESTORE_FILE".%d", domid);
Can you use a different name space from the normal one?
For example, you can put
#define LIBXL_COLO_DEVICE_MODEL_RESTORE_FILE XXXX
in libxl_colo.h and use it in all COLO code.
> + crcs->status = LIBXL_COLO_SETUPED;
> +
> + libxl__logdirty_init(&crcs->lds);
> + crcs->lds.ao = ao;
> +
> + crcs->sws.fd = crs->send_back_fd;
> + crcs->sws.ao = ao;
> + crcs->sws.back_channel = true;
> +
> + dcs->cds.concrete_data = crs;
> +
> + libxl__stream_write_start(egc, &crcs->sws);
> +
> + rc = 0;
> +
> +out:
> + crs->callback(egc, crs, rc);
> + return;
> +
> +err:
> + goto out;
> +}
> +
> +static void libxl__colo_domain_create_cb(libxl__egc *egc,
> + libxl__domain_create_state *dcs,
> + int rc, uint32_t domid)
> +{
> + libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
> +
> + crcs->callback(egc, crcs, rc);
> +}
> +
> +
[...]
> +
> +static void colo_disable_logdirty_done(libxl__egc *egc,
> + libxl__logdirty_switch *lds,
> + int rc)
> +{
> + libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
> +
> + EGC_GC;
> +
> + if (rc)
> + LOG(WARN, "cannot disable logdirty");
> +
> + if (crcs->status == LIBXL_COLO_SUSPENDED) {
> + /*
> + * failover when reading state from master, so no need to
> + * call libxl__domain_restore().
You need to update this comment to the right function name.
> + */
> + colo_resume_vm(egc, crcs, 0);
> + return;
> + }
> +
> + /* If we cannot disable logdirty, we still can do failover */
> + crcs->callback(egc, crcs, 0);
> +}
> +
[...]
>
> +/* colo related structure */
> +typedef struct libxl__colo_restore_state libxl__colo_restore_state;
> +typedef void libxl__colo_callback(libxl__egc *,
> + libxl__colo_restore_state *, int rc);
> +struct libxl__colo_restore_state {
> + /* must set by caller of libxl__colo_(setup|teardown) */
> + libxl__ao *ao;
> + uint32_t domid;
> + int send_back_fd;
> + int recv_fd;
> + int hvm;
> + libxl__colo_callback *callback;
> +
> + /* private, colo restore checkpoint state */
> + libxl__domain_create_cb *saved_cb;
> + void *crcs;
> +};
>
And this should go to libxl_colo.h, too? And libxl_internal.h includes
libxl_colo.h?
I just don't want to colo structures and functions scatter in
different places.
> struct libxl__domain_create_state {
> /* filled in by user */
> @@ -3486,6 +3503,8 @@ struct libxl__domain_create_state {
> /* private to domain_create */
> int guest_domid;
> libxl__domain_build_state build_state;
> + libxl__colo_restore_state crs;
> + libxl__checkpoint_devices_state cds;
> libxl__bootloader_state bl;
> libxl__stub_dm_spawn_state dmss;
> /* If we're not doing stubdom, we use only dmss.dm,
> diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
> index 0d6949a..b1810b2 100644
> --- a/tools/libxl/libxl_save_callout.c
> +++ b/tools/libxl/libxl_save_callout.c
> @@ -15,6 +15,7 @@
> #include "libxl_osdeps.h"
>
> #include "libxl_internal.h"
> +#include "libxl_colo.h"
>
> /* stream_fd is as from the caller (eventually, the application).
> * It may be 0, 1 or 2, in which case we need to dup it elsewhere.
> @@ -68,7 +69,11 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
> shs->ao = ao;
> shs->domid = domid;
> shs->recv_callback = libxl__srm_callout_received_restore;
> - shs->completion_callback = libxl__xc_domain_restore_done;
> + if (dcs->restore_params.checkpointed_stream ==
> + LIBXL_CHECKPOINTED_STREAM_COLO)
This is very strange line wrap.
> + shs->completion_callback = libxl__colo_restore_teardown;
> + else
> + shs->completion_callback = libxl__xc_domain_restore_done;
> shs->caller_state = dcs;
> shs->need_results = 1;
>
> diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
> index 5d980d9..d6bd2fe 100644
> --- a/tools/libxl/libxl_stream_read.c
> +++ b/tools/libxl/libxl_stream_read.c
> @@ -846,6 +846,18 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
> */
> if (libxl__stream_read_inuse(stream)) {
> switch (checkpointed_stream) {
> + case LIBXL_CHECKPOINTED_STREAM_COLO:
> + if (stream->completion_callback) {
> + /*
> + * restore, just build the secondary vm, don't close
> + * the stream
> + */
> + stream->completion_callback(egc, stream, 0);
> + } else {
> + /* failover, just close the stream */
> + stream_complete(egc, stream, 0);
> + }
> + break;
> case LIBXL_CHECKPOINTED_STREAM_REMUS:
> /*
> * Failover from primary. Domain state is currently at a
> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
> index 632c009..33f4a90 100644
> --- a/tools/libxl/libxl_types.idl
> +++ b/tools/libxl/libxl_types.idl
> @@ -232,6 +232,7 @@ libxl_hdtype = Enumeration("hdtype", [
> libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
> (0, "NONE"),
> (1, "REMUS"),
> + (2, "COLO"),
> ])
>
> #
> --
> 2.5.0
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 17/31] primary vm suspend/resume/checkpoint code
2016-02-22 2:52 ` [PATCH v10 17/31] primary " Wen Congyang
@ 2016-02-25 15:57 ` Wei Liu
2016-02-26 2:32 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:57 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:21AM +0800, Wen Congyang wrote:
[...]
> +#include "libxl_osdeps.h" /* must come before any other headers */
> +
> +#include "libxl_internal.h"
> +#include "libxl_colo.h"
> +
> +static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
> + NULL,
> +};
> +
> +/* ================= helper functions ================= */
> +static int init_device_subkind(libxl__checkpoint_devices_state *cds)
> +{
> + /* init device subkind-specific state in the libxl ctx */
> + int rc;
> + STATE_AO_GC(cds->ao);
> +
> + rc = 0;
> + return rc;
> +}
> +
> +static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
> +{
> + /* cleanup device subkind-specific state in the libxl ctx */
> + STATE_AO_GC(cds->ao);
> +}
> +
> +/* ================= colo: setup save environment ================= */
> +static void colo_save_setup_done(libxl__egc *egc,
> + libxl__checkpoint_devices_state *cds,
> + int rc);
> +static void colo_save_setup_failed(libxl__egc *egc,
> + libxl__checkpoint_devices_state *cds,
> + int rc);
> +static void libxl__colo_save_domain_suspend_callback(void *data);
> +static void libxl__colo_save_domain_checkpoint_callback(void *data);
> +static void libxl__colo_save_domain_resume_callback(void *data);
> +static void libxl__colo_save_domain_wait_checkpoint_callback(void *data);
> +
An ordered list of callbacks is appreciated. This doesn't seem to cover
all callbacks and I'm not sure if this is the order they are supposed to
fire either.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 18/31] libxc/restore: support COLO restore
2016-02-22 2:52 ` [PATCH v10 18/31] libxc/restore: support COLO restore Wen Congyang
@ 2016-02-25 15:57 ` Wei Liu
2016-02-26 2:33 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:57 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:22AM +0800, Wen Congyang wrote:
[...]
> - * With Remus, we buffer the records sent by the primary at checkpoint,
> + * With Remus/COLO, we buffer the records sent by the primary at checkpoint,
> * in case the primary will fail, we can recover from the last
> * checkpoint state.
> * This should be enough for most of the cases because primary only send
> diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
> index aef9bca..2ae8154 100644
> --- a/tools/libxc/xc_sr_restore.c
> +++ b/tools/libxc/xc_sr_restore.c
> @@ -460,6 +460,49 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
> else
> ctx->restore.buffer_all_records = true;
>
> + if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
> + {
> +#define HANDLE_CALLBACK_RETURN_VALUE(ret) \
> + do { \
> + if ( ret == 1 ) \
> + rc = 0; /* Success */ \
> + else \
> + { \
> + if ( ret == 2 ) \
> + rc = BROKEN_CHANNEL; \
> + else \
> + rc = -1; /* Some unspecified error */ \
> + goto err; \
> + } \
> + } while (0)
> +
> + /* COLO */
> +
> + /* We need to resume guest */
> + rc = ctx->restore.ops.stream_complete(ctx);
> + if ( rc )
> + goto err;
> +
> + /* TODO: call restore_results */
> +
> + /* Resume secondary vm */
> + ret = ctx->restore.callbacks->postcopy(ctx->restore.callbacks->data);
> + HANDLE_CALLBACK_RETURN_VALUE(ret);
> +
> + /* Wait for a new checkpoint */
> + ret = ctx->restore.callbacks->wait_checkpoint(
> + ctx->restore.callbacks->data);
> + HANDLE_CALLBACK_RETURN_VALUE(ret);
> +
> + /* suspend secondary vm */
> + ret = ctx->restore.callbacks->suspend(ctx->restore.callbacks->data);
> + HANDLE_CALLBACK_RETURN_VALUE(ret);
> +
> +#undef HANDLE_CALLBACK_RETURN_VALUE
> +
> + /* TODO: send dirty pfn list to primary */
You replace the TODOs with actual code in the next two patches.
You can rearrange them a bit so that you don't need to add TODOs at all.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 21/31] libxc/save: support COLO save
2016-02-22 2:52 ` [PATCH v10 21/31] libxc/save: support COLO save Wen Congyang
@ 2016-02-25 15:58 ` Wei Liu
2016-02-26 2:35 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-02-25 15:58 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:25AM +0800, Wen Congyang wrote:
[...]
> /*
> * Suspend the domain and send dirty memory.
> * This is the last iteration of the live migration and the
> @@ -558,6 +610,16 @@ static int suspend_and_send_dirty(struct xc_sr_context *ctx)
>
> bitmap_or(dirty_bitmap, ctx->save.deferred_pages, ctx->save.p2m_size);
>
> + if ( !ctx->save.live && ctx->save.checkpointed == MIG_STREAM_COLO )
> + {
> + rc = merge_secondary_dirty_bitmap(ctx);
> + if ( rc )
> + {
> + PERROR("Failed to get secondary vm's dirty pages");
> + goto out;
> + }
> + }
> +
> rc = send_dirty_pages(ctx, stats.dirty_count + ctx->save.nr_deferred_pages);
> if ( rc )
> goto out;
> @@ -791,13 +853,42 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type)
> if ( rc )
> goto err;
>
> - rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
> - if ( rc <= 0 )
> - goto err;
> + if ( ctx->save.checkpointed == MIG_STREAM_COLO )
> + {
> + rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
> + if ( !rc )
> + {
> + rc = -1;
> + goto err;
> + }
> + }
>
> - rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
> - if ( rc <= 0 )
> + rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
> + if ( !rc )
This original code for checking postcopy return value is if ( rc <= 0 ).
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
` (30 preceding siblings ...)
2016-02-22 2:52 ` [PATCH v10 31/31] cmdline switches and config vars to control colo-proxy Wen Congyang
@ 2016-02-25 16:05 ` Wei Liu
31 siblings, 0 replies; 70+ messages in thread
From: Wei Liu @ 2016-02-25 16:05 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
I managed to go through up to patch 21 today. I will see if I can get
some more time to look at the rest.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state
2016-02-25 15:53 ` Wei Liu
@ 2016-02-26 1:55 ` Wen Congyang
0 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-26 1:55 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Anthony Perard,
Gui Jianfeng, Shriram Rajagopalan, Ian Jackson, Yang Hongyang
On 02/25/2016 11:53 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:05AM +0800, Wen Congyang wrote:
>> In normal migration, the qemu state is passed to qemu as a parameter.
>> With COLO, secondary vm is running. So we will do the following steps
>> at every checkpoint:
>> 1. suspend both primary vm and secondary vm
>> 2. sync the state
>> 3. resume both primary vm and secondary vm
>> Primary will send qemu's state in step2, and secondary's qemu should
>> read it and restore the state before it is resumed. We can not pass
>> the state to qemu as a parameter because secondary QEMU already started
>
> is already started.
>
>> at this point, so we introduce libxl__domain_restore_device_model() to
>> do it. This API MUST be called before resuming secondary vm.
>>
>
> The last sentence is of no relevancy of this function itself, so I would
> just remove it if this patch will be resent in its current form.
>
> And some comments below.
>
>> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Cc: Anthony Perard <anthony.perard@citrix.com>
>> Reviewed-by: Konrad Rzeszutek Wilk <konrad.wilk@oracle.com>
>> ---
>> tools/libxl/libxl_dom_save.c | 20 ++++++++++++++++++++
>> tools/libxl/libxl_internal.h | 4 ++++
>> tools/libxl/libxl_qmp.c | 10 ++++++++++
>> 3 files changed, 34 insertions(+)
>>
>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>> index 4eb7960..7d345f9 100644
>> --- a/tools/libxl/libxl_dom_save.c
>> +++ b/tools/libxl/libxl_dom_save.c
>> @@ -512,6 +512,26 @@ int libxl__restore_emulator_xenstore_data(libxl__domain_create_state *dcs,
>> return rc;
>> }
>>
>> +int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
>> + const char *restore_file)
>> +{
>> + int rc;
>> +
>> + switch (libxl__device_model_version_running(gc, domid)) {
>> + case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
>> + /* Will never be supported. */
>> + rc = ERROR_INVAL;
>> + break;
>> + case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
>> + rc = libxl__qmp_restore(gc, domid, restore_file);
>> + break;
>> + default:
>> + rc = ERROR_INVAL;
>> + }
>> +
>> + return rc;
>> +}
>> +
>
> The function name suggests that it should be universally applied to all
> device model cases and all places that involves restoring device model.
> This is not the case in this series. I search for this function in the
> rest of this series, it seems to be only used by COLO (in patch #10).
>
> It's also unclear where the other half libxl__domain_save_device_model
> is. I don't think this is your problem, though.
>
> Based on the two reasons above, I would say let's not have this function
> at all. You can call libxl__qmp_restore in COLO code directly.
>
> Does this sound plausible?
>
> If you agree, this patch can be turned into introduction of
> libxl__qmp_restore.
Yes, it is only used by COLO. I will update it in the next version.
Thanks
Wen Congyang
>
>> /*
>> * Local variables:
>> * mode: C
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index a1aae97..5320e5e 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -1119,6 +1119,8 @@ _hidden int libxl__domain_rename(libxl__gc *gc, uint32_t domid,
>> const char *old_name, const char *new_name,
>> xs_transaction_t trans);
>>
>> +_hidden int libxl__domain_restore_device_model(libxl__gc *gc, uint32_t domid,
>> + const char *restore_file);
>> _hidden int libxl__domain_resume_device_model(libxl__gc *gc, uint32_t domid);
>>
>> _hidden const char *libxl__userdata_path(libxl__gc *gc, uint32_t domid,
>> @@ -1762,6 +1764,8 @@ _hidden int libxl__qmp_stop(libxl__gc *gc, int domid);
>> _hidden int libxl__qmp_resume(libxl__gc *gc, int domid);
>> /* Save current QEMU state into fd. */
>> _hidden int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename);
>> +/* Load current QEMU state from file. */
>> +_hidden int libxl__qmp_restore(libxl__gc *gc, int domid, const char *filename);
>> /* Set dirty bitmap logging status */
>> _hidden int libxl__qmp_set_global_dirty_log(libxl__gc *gc, int domid, bool enable);
>> _hidden int libxl__qmp_insert_cdrom(libxl__gc *gc, int domid, const libxl_device_disk *disk);
>> diff --git a/tools/libxl/libxl_qmp.c b/tools/libxl/libxl_qmp.c
>> index 714038b..eec8a44 100644
>> --- a/tools/libxl/libxl_qmp.c
>> +++ b/tools/libxl/libxl_qmp.c
>> @@ -905,6 +905,16 @@ int libxl__qmp_save(libxl__gc *gc, int domid, const char *filename)
>> NULL, NULL);
>> }
>>
>> +int libxl__qmp_restore(libxl__gc *gc, int domid, const char *state_file)
>> +{
>> + libxl__json_object *args = NULL;
>> +
>> + qmp_parameters_add_string(gc, &args, "filename", state_file);
>> +
>> + return qmp_run_command(gc, domid, "xen-load-devices-state", args,
>> + NULL, NULL);
>> +}
>> +
>
> This looks OK.
>
>> static int qmp_change(libxl__gc *gc, libxl__qmp_handler *qmp,
>> char *device, char *target, char *arg)
>> {
>> --
>> 2.5.0
>>
>>
>>
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 07/31] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams
2016-02-25 15:54 ` Wei Liu
@ 2016-02-26 1:59 ` Wen Congyang
0 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-26 1:59 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 02/25/2016 11:54 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:11AM +0800, Wen Congyang wrote:
>> It is the negotiation record for COLO.
>> Primary->Secondary:
>> control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
>> Secondary->Primary:
>> 0x00000001: Secondary VM is suspended
>> 0x00000002: Secondary VM is ready
>> 0x00000003: Secondary VM is resumed
>>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
>> ---
>> docs/specs/libxl-migration-stream.pandoc | 25 +++++++++++++++++++++++--
>> tools/libxl/libxl_sr_stream_format.h | 11 +++++++++++
>> tools/python/xen/migration/libxl.py | 9 +++++++++
>> 3 files changed, 43 insertions(+), 2 deletions(-)
>>
>> diff --git a/docs/specs/libxl-migration-stream.pandoc b/docs/specs/libxl-migration-stream.pandoc
>> index 2c97d86..5166d66 100644
>> --- a/docs/specs/libxl-migration-stream.pandoc
>> +++ b/docs/specs/libxl-migration-stream.pandoc
>> @@ -1,6 +1,6 @@
>> % LibXenLight Domain Image Format
>> % Andrew Cooper <<andrew.cooper3@citrix.com>>
>
> You can add yours and Hongyang's name and email here as well.
OK
>
>> -% Revision 1
>> +% Revision 2
>>
>> Introduction
>> ============
>> @@ -119,7 +119,9 @@ type 0x00000000: END
>>
>> 0x00000004: CHECKPOINT_END
>>
>> - 0x00000005 - 0x7FFFFFFF: Reserved for future _mandatory_
>> + 0x00000005: CHECKPOINT_STATE
>> +
>> + 0x00000006 - 0x7FFFFFFF: Reserved for future _mandatory_
>> records.
>>
>> 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
>> @@ -249,6 +251,25 @@ A checkpoint end record marks the end of a checkpoint in the image.
>> The end record contains no fields; its body_length is 0.
>>
>>
>> +CHECKPOINT\_STATE
>> +--------------
>> +
>> +A checkpoint state record contains the control information for checkpoint.
>> +
>> + 0 1 2 3 4 5 6 7 octet
>> + +------------------------+------------------------+
>> + | control_id | padding |
>> + +------------------------+------------------------+
>> +
>> +--------------------------------------------------------------------
>> +Field Description
>> +------------ ---------------------------------------------------
>> +control_id 0x00000000: Secondary VM is out of sync, start a new checkpoint
>> + 0x00000001: Secondary VM is suspended
>> + 0x00000002: Secondary VM is ready
>> + 0x00000003: Secondary VM is resumed
>
> You commit message seems to suggest 0 can only go from primary to
> secondary, while the other three only go from secondary to primary. You
> probably need to spell out the direction here (and later enforce this
> behaviour in code).
OK, will fix it in the next version.
Thanks
Wen Congyang
>
> Wei.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 10/31] tools/libxl: add back channel support to write stream
2016-02-25 15:54 ` Wei Liu
@ 2016-02-26 2:11 ` Wen Congyang
2016-03-02 15:02 ` Wei Liu
0 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-26 2:11 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 02/25/2016 11:54 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:14AM +0800, Wen Congyang wrote:
>> Add back channel support to write stream. If the write stream is
>> a back channel stream, this means the write stream is used by
>> Secondary to send some records back.
>>
>> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> ---
>> tools/libxl/libxl_dom_save.c | 1 +
>> tools/libxl/libxl_internal.h | 1 +
>> tools/libxl/libxl_stream_write.c | 26 ++++++++++++++++++++------
>> 3 files changed, 22 insertions(+), 6 deletions(-)
>>
>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>> index 72b61c7..18946c5 100644
>> --- a/tools/libxl/libxl_dom_save.c
>> +++ b/tools/libxl/libxl_dom_save.c
>> @@ -404,6 +404,7 @@ void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
>> dss->sws.ao = dss->ao;
>> dss->sws.dss = dss;
>> dss->sws.fd = dss->fd;
>> + dss->sws.back_channel = false;
>> dss->sws.completion_callback = stream_done;
>>
>> libxl__stream_write_start(egc, &dss->sws);
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 3d3e8e8..e02e554 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -3044,6 +3044,7 @@ struct libxl__stream_write_state {
>> libxl__ao *ao;
>> libxl__domain_save_state *dss;
>> int fd;
>> + bool back_channel;
>> void (*completion_callback)(libxl__egc *egc,
>> libxl__stream_write_state *sws,
>> int rc);
>> diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
>> index f6ea55d..5379126 100644
>> --- a/tools/libxl/libxl_stream_write.c
>> +++ b/tools/libxl/libxl_stream_write.c
>> @@ -49,6 +49,13 @@
>> * - if (hvm)
>> * - Emulator context record
>> * - Checkpoint end record
>> + *
>> + * For back channel stream:
>> + * - libxl__stream_write_start()
>> + * - Set up the stream to running state
>> + *
>> + * - Add a new API to write the record. When the record is written
>> + * out, call stream->checkpoint_callback() to return.
>
> What does this mean? Which new API?
The next patch introduces this API. The commits is very old.
I think I can merge these two patches into one patch.
>
>> */
>>
>> /* Success/error/cleanup handling. */
>> @@ -225,6 +232,15 @@ void libxl__stream_write_start(libxl__egc *egc,
>>
>> stream->running = true;
>>
>> + dc->ao = ao;
>> + dc->readfd = -1;
>> + dc->copywhat = "save v2 stream";
>> + dc->writefd = stream->fd;
>> + dc->maxsz = -1;
>> +
>> + if (stream->back_channel)
>> + return;
>> +
>
> There seems to be very subtle change of behaviour.
>
> Before this patch, the dc->* are not set until the emulator check is
> done. With this path, it is possible in the normal case some of the
> fields are initialised in the error path.
>
> I think this is OK given the callbacks in the upper layer and in
> the writer don't rely on those fields to clean up. Just one thing to
> note.
>
> That said, I suggest you move all initialisation of dc->* in one place.
OK, I will do it.
>
>> if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
>> stream->device_model_version =
>> libxl__device_model_version_running(gc, dss->domid);
>> @@ -249,12 +265,7 @@ void libxl__stream_write_start(libxl__egc *egc,
>> stream->emu_sub_hdr.index = 0;
>> }
>>
>> - dc->ao = ao;
>> - dc->readfd = -1;
>> dc->writewhat = "stream header";
>> - dc->copywhat = "save v2 stream";
>> - dc->writefd = stream->fd;
>> - dc->maxsz = -1;
>> dc->callback = stream_header_done;
>>
>> rc = libxl__datacopier_start(dc);
>> @@ -279,6 +290,7 @@ void libxl__stream_write_start_checkpoint(libxl__egc *egc,
>> {
>> assert(stream->running);
>> assert(!stream->in_checkpoint);
>> + assert(!stream->back_channel);
>> stream->in_checkpoint = true;
>>
>> write_emulator_xenstore_record(egc, stream);
>> @@ -590,7 +602,9 @@ static void stream_done(libxl__egc *egc,
>> libxl__carefd_close(stream->emu_carefd);
>> free(stream->emu_body);
>>
>> - check_all_finished(egc, stream, rc);
>> + if (!stream->back_channel)
>> + /* back channel stream doesn't have save helper */
>> + check_all_finished(egc, stream, rc);
>
> Though it doesn't have helper, do you not need to check if the back
> channel stream itself is OK? The comment itself doesn't seem to justify
> this change.
What do you want to check? assert(!stream->shs) or assert(!libxl__stream_write_inuse(stream))?
Thanks
Wen Congyang
>
> Wei.
>
>> }
>>
>> static void checkpoint_done(libxl__egc *egc,
>> --
>> 2.5.0
>>
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 12/31] tools/libxl: add back channel support to read stream
2016-02-25 15:54 ` Wei Liu
@ 2016-02-26 2:16 ` Wen Congyang
2016-03-02 15:03 ` Wei Liu
0 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-02-26 2:16 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 02/25/2016 11:54 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:16AM +0800, Wen Congyang wrote:
>> This is used by primay to read records sent by secondary.
>>
>> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> ---
>> tools/libxl/libxl_create.c | 1 +
>> tools/libxl/libxl_internal.h | 1 +
>> tools/libxl/libxl_stream_read.c | 27 +++++++++++++++++++++++----
>> 3 files changed, 25 insertions(+), 4 deletions(-)
>>
>> diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
>> index 525bf85..fc746fb 100644
>> --- a/tools/libxl/libxl_create.c
>> +++ b/tools/libxl/libxl_create.c
>> @@ -1030,6 +1030,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
>> dcs->srs.dcs = dcs;
>> dcs->srs.fd = restore_fd;
>> dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
>> + dcs->srs.back_channel = false;
>> dcs->srs.completion_callback = domcreate_stream_done;
>>
>> if (restore_fd >= 0) {
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index cdf6751..32957ca 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -3420,6 +3420,7 @@ struct libxl__stream_read_state {
>> libxl__domain_create_state *dcs;
>> int fd;
>> bool legacy;
>> + bool back_channel;
>> void (*completion_callback)(libxl__egc *egc,
>> libxl__stream_read_state *srs,
>> int rc);
>> diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
>> index f4781eb..02a2c46 100644
>> --- a/tools/libxl/libxl_stream_read.c
>> +++ b/tools/libxl/libxl_stream_read.c
>> @@ -118,6 +118,15 @@
>> * record, and therefore the buffered state is inconsistent. In
>> * libxl__xc_domain_restore_done(), we just complete the stream and
>> * stream->completion_callback() will be called to resume the guest
>> + *
>> + * For back channel stream:
>> + * - libxl__stream_read_start()
>> + * - Set up the stream to running state
>> + *
>> + * - libxl__stream_read_continue()
>> + * - Set up reading the next record from a started stream.
>> + * Add some codes to process_record() to handle the record.
>> + * Then call stream->checkpoint_callback() to return.
>> */
>>
>> /* Success/error/cleanup handling. */
>> @@ -221,6 +230,17 @@ void libxl__stream_read_start(libxl__egc *egc,
>> stream->running = true;
>> stream->phase = SRS_PHASE_NORMAL;
>>
>> + dc->ao = stream->ao;
>> + dc->copywhat = "restore v2 stream";
>> + dc->writefd = -1;
>> +
>> + if (stream->back_channel) {
>> + assert(!stream->legacy);
>> +
>> + dc->readfd = stream->fd;
>
> Why is this needed? dc->readfd is set to stream->fd no matter it is back
> channel or not. This can be moved outside this if {}.
Yes, will fix it in the next version.
>
>> + return;
>> + }
>> +
>> if (stream->legacy) {
>> /* Convert the legacy stream. */
>> libxl__conversion_helper_state *chs = &stream->chs;
>> @@ -243,10 +263,7 @@ void libxl__stream_read_start(libxl__egc *egc,
>> }
>> /* stream->fd is now a v2 stream. */
>>
>> - dc->ao = stream->ao;
>> - dc->copywhat = "restore v2 stream";
>> dc->readfd = stream->fd;
>> - dc->writefd = -1;
>>
>> /* Start reading the stream header. */
>> rc = setup_read(stream, "stream header",
>> @@ -762,7 +779,9 @@ static void stream_done(libxl__egc *egc,
>> LIBXL_STAILQ_FOREACH_SAFE(rec, &stream->record_queue, entry, trec)
>> free_record(rec);
>>
>> - check_all_finished(egc, stream, rc);
>> + if (!stream->back_channel)
>> + /* back channel stream doesn't have restore helper */
>> + check_all_finished(egc, stream, rc);
>
> Even if it doesn't have restore helper, check_all_finished also checks
> if the stream and the conversion helper are till in use. The
> explanation in the comment doesn't seem to justify this change.
In stream_done(), stream->running is set to false, so libxl__stream_read_in_use()
returns false.
Back channel stream doesn't support legacy stream, so there is no conversion helper.
I will update the comments in the next version.
Thanks
Wen Congyang
>
> Wei.
>
>> }
>>
>> void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
>> --
>> 2.5.0
>>
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code
2016-02-25 15:56 ` Wei Liu
@ 2016-02-26 2:30 ` Wen Congyang
2016-03-01 10:06 ` Wen Congyang
1 sibling, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-26 2:30 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 02/25/2016 11:56 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:20AM +0800, Wen Congyang wrote:
>> Secondary vm is running in colo mode. So we will do
>> the following things again and again:
>> 1. Resume secondary vm
>> a. Send CHECKPOINT_SVM_READY to master.
>> b. If it is not the first resume, call libxl__checkpoint_devices_preresume().
>> c. If it is the first resume(resume right after live migration),
>> - call libxl__xc_domain_restore_done() to build the secondary vm.
>> - enable secondary vm's logdirty.
>> - call libxl__domain_resume() to resume secondary vm.
>> - call libxl__checkpoint_devices_setup() to setup checkpoint devices.
>> d. Send CHECKPOINT_SVM_RESUMED to master.
>> 2. Wait a new checkpoint
>> a. Call libxl__checkpoint_devices_commit().
>> b. Read CHECKPOINT_NEW from master.
>> 3. Suspend secondary vm
>> a. Suspend secondary vm.
>> b. Call libxl__checkpoint_devices_postsuspend().
>> c. Send CHECKPOINT_SVM_SUSPENDED to master.
>>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
>> ---
>> tools/libxc/xc_sr_common.h | 2 +
>> tools/libxc/xc_sr_save.c | 3 +-
>> tools/libxl/Makefile | 1 +
>> tools/libxl/libxl_colo.h | 24 +
>> tools/libxl/libxl_colo_restore.c | 1038 ++++++++++++++++++++++++++++++++++++++
>> tools/libxl/libxl_create.c | 37 ++
>> tools/libxl/libxl_internal.h | 19 +
>> tools/libxl/libxl_save_callout.c | 7 +-
>> tools/libxl/libxl_stream_read.c | 12 +
>> tools/libxl/libxl_types.idl | 1 +
>
> There is a bunch of TODOs in libxl_colo.c but I don't think you're in a
> better position to judge whether they should be blocker or not.
>
>> 10 files changed, 1142 insertions(+), 2 deletions(-)
>> create mode 100644 tools/libxl/libxl_colo.h
>> create mode 100644 tools/libxl/libxl_colo_restore.c
>>
>> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
>> index 5d9f497..2bfed64 100644
>> --- a/tools/libxc/xc_sr_common.h
>> +++ b/tools/libxc/xc_sr_common.h
>> @@ -184,10 +184,12 @@ struct xc_sr_context
>> * migration stream
>> * 0: Plain VM
>> * 1: Remus
>> + * 2: COLO
>> */
>> enum {
>> MIG_STREAM_NONE, /* plain stream */
>> MIG_STREAM_REMUS,
>> + MIG_STREAM_COLO,
>> } migration_stream;
>>
>> union /* Common save or restore data. */
>> diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
>> index fe210cc..7393355 100644
>> --- a/tools/libxc/xc_sr_save.c
>> +++ b/tools/libxc/xc_sr_save.c
>> @@ -846,7 +846,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
>>
>> /* If altering migration_stream update this assert too. */
>> assert(checkpointed_stream == MIG_STREAM_NONE ||
>> - checkpointed_stream == MIG_STREAM_REMUS);
>> + checkpointed_stream == MIG_STREAM_REMUS ||
>> + checkpointed_stream == MIG_STREAM_COLO);
>>
>> /*
>> * TODO: Find some time to better tweak the live migration algorithm.
>
> [...]
>
>> +
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +#include "libxl_colo.h"
>> +#include "libxl_sr_stream_format.h"
>> +
>> +enum {
>> + LIBXL_COLO_SETUPED,
>> + LIBXL_COLO_SUSPENDED,
>> + LIBXL_COLO_RESUMED,
>> +};
>> +
>> +typedef struct libxl__colo_restore_checkpoint_state libxl__colo_restore_checkpoint_state;
>> +struct libxl__colo_restore_checkpoint_state {
>> + libxl__domain_suspend_state dsps;
>> + libxl__logdirty_switch lds;
>> + libxl__colo_restore_state *crs;
>> + libxl__stream_write_state sws;
>> + int status;
>> + bool preresume;
>> + /* used for teardown */
>> + int teardown_devices;
>> + int saved_rc;
>> + char *state_file;
>> +
>> + void (*callback)(libxl__egc *,
>> + libxl__colo_restore_checkpoint_state *,
>> + int);
>> +};
>> +
>
> Shouldn't the enum and struct belong to libxl_colo.h ?
It only be used by restore side. I think it is OK to move them to libxl_colo.h.
>
>> +
>> +static void libxl__colo_restore_domain_resume_callback(void *data);
>> +static void libxl__colo_restore_domain_checkpoint_callback(void *data);
>> +static void libxl__colo_restore_domain_wait_checkpoint_callback(void *data);
>> +static void libxl__colo_restore_domain_suspend_callback(void *data);
>> +
>> +static const libxl__checkpoint_device_instance_ops *colo_restore_ops[] = {
>> + NULL,
>> +};
>> +
>
> It would be helpful to list the callbacks at the beginning of the time
> in the order they are supposed to occur.
>
> See libxl_create.c for example. Search for "Event callbacks, in this
> order".
OK, will fix it in the next version.
>
> I've tried to map the algorithm you described in commit message to all
> the callbacks, but without some references it is just too time consuming
> from my end.
>
> I think what I'm going to do is to make sure the normal path that
> doesn't use COLO is not broken and leave the internal to you and Ian (if
> he fancies to dig into details).
>
> [...]
>> +
>> +void libxl__colo_restore_setup(libxl__egc *egc,
>> + libxl__colo_restore_state *crs)
>> +{
>> + libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
>> + libxl__colo_restore_checkpoint_state *crcs;
>> + int rc = ERROR_FAIL;
>> +
>> + /* Convenience aliases */
>> + libxl__srm_restore_autogen_callbacks *const callbacks =
>> + &dcs->srs.shs.callbacks.restore.a;
>> + const int domid = crs->domid;
>> +
>> + STATE_AO_GC(crs->ao);
>> +
>> + GCNEW(crcs);
>> + crs->crcs = crcs;
>> + crcs->crs = crs;
>> +
>> + /* setup dsps */
>> + crcs->dsps.ao = ao;
>> + crcs->dsps.domid = domid;
>> + if (init_dsps(&crcs->dsps))
>> + goto err;
>> +
>> + callbacks->suspend = libxl__colo_restore_domain_suspend_callback;
>> + callbacks->postcopy = libxl__colo_restore_domain_resume_callback;
>> + callbacks->checkpoint = libxl__colo_restore_domain_checkpoint_callback;
>> + callbacks->wait_checkpoint = libxl__colo_restore_domain_wait_checkpoint_callback;
>> +
>> + /*
>> + * Secondary vm is running in colo mode, so we need to call
>> + * libxl__xc_domain_restore_done() to create secondary vm.
>> + * But we will exit in domain_create_cb(). So replace the
>> + * callback here.
>> + */
>> + crs->saved_cb = dcs->callback;
>> + dcs->callback = libxl__colo_domain_create_cb;
>> + crcs->state_file = GCSPRINTF(LIBXL_DEVICE_MODEL_RESTORE_FILE".%d", domid);
>
> Can you use a different name space from the normal one?
We write the qemu state into the restore file in write_emulator_blob().
>
> For example, you can put
>
> #define LIBXL_COLO_DEVICE_MODEL_RESTORE_FILE XXXX
So if we use a different name space, we should do
#define LIBXL_COLO_DEVICE_MODEL_RESTORE_FILE LIBXL_DEVICE_MODEL_RESTORE_FILE
In colo codes, IIRC there is no other code use it.
>
> in libxl_colo.h and use it in all COLO code.
>
>
>> + crcs->status = LIBXL_COLO_SETUPED;
>> +
>> + libxl__logdirty_init(&crcs->lds);
>> + crcs->lds.ao = ao;
>> +
>> + crcs->sws.fd = crs->send_back_fd;
>> + crcs->sws.ao = ao;
>> + crcs->sws.back_channel = true;
>> +
>> + dcs->cds.concrete_data = crs;
>> +
>> + libxl__stream_write_start(egc, &crcs->sws);
>> +
>> + rc = 0;
>> +
>> +out:
>> + crs->callback(egc, crs, rc);
>> + return;
>> +
>> +err:
>> + goto out;
>> +}
>> +
>> +static void libxl__colo_domain_create_cb(libxl__egc *egc,
>> + libxl__domain_create_state *dcs,
>> + int rc, uint32_t domid)
>> +{
>> + libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
>> +
>> + crcs->callback(egc, crcs, rc);
>> +}
>> +
>> +
> [...]
>> +
>> +static void colo_disable_logdirty_done(libxl__egc *egc,
>> + libxl__logdirty_switch *lds,
>> + int rc)
>> +{
>> + libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
>> +
>> + EGC_GC;
>> +
>> + if (rc)
>> + LOG(WARN, "cannot disable logdirty");
>> +
>> + if (crcs->status == LIBXL_COLO_SUSPENDED) {
>> + /*
>> + * failover when reading state from master, so no need to
>> + * call libxl__domain_restore().
>
> You need to update this comment to the right function name.
OK, will fix it in the next version.
>
>> + */
>> + colo_resume_vm(egc, crcs, 0);
>> + return;
>> + }
>> +
>> + /* If we cannot disable logdirty, we still can do failover */
>> + crcs->callback(egc, crcs, 0);
>> +}
>> +
> [...]
>>
>> +/* colo related structure */
>> +typedef struct libxl__colo_restore_state libxl__colo_restore_state;
>> +typedef void libxl__colo_callback(libxl__egc *,
>> + libxl__colo_restore_state *, int rc);
>> +struct libxl__colo_restore_state {
>> + /* must set by caller of libxl__colo_(setup|teardown) */
>> + libxl__ao *ao;
>> + uint32_t domid;
>> + int send_back_fd;
>> + int recv_fd;
>> + int hvm;
>> + libxl__colo_callback *callback;
>> +
>> + /* private, colo restore checkpoint state */
>> + libxl__domain_create_cb *saved_cb;
>> + void *crcs;
>> +};
>>
>
> And this should go to libxl_colo.h, too? And libxl_internal.h includes
> libxl_colo.h?
>
> I just don't want to colo structures and functions scatter in
> different places.
OK, will fix it in the next version.
>
>> struct libxl__domain_create_state {
>> /* filled in by user */
>> @@ -3486,6 +3503,8 @@ struct libxl__domain_create_state {
>> /* private to domain_create */
>> int guest_domid;
>> libxl__domain_build_state build_state;
>> + libxl__colo_restore_state crs;
>> + libxl__checkpoint_devices_state cds;
>> libxl__bootloader_state bl;
>> libxl__stub_dm_spawn_state dmss;
>> /* If we're not doing stubdom, we use only dmss.dm,
>> diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
>> index 0d6949a..b1810b2 100644
>> --- a/tools/libxl/libxl_save_callout.c
>> +++ b/tools/libxl/libxl_save_callout.c
>> @@ -15,6 +15,7 @@
>> #include "libxl_osdeps.h"
>>
>> #include "libxl_internal.h"
>> +#include "libxl_colo.h"
>>
>> /* stream_fd is as from the caller (eventually, the application).
>> * It may be 0, 1 or 2, in which case we need to dup it elsewhere.
>> @@ -68,7 +69,11 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
>> shs->ao = ao;
>> shs->domid = domid;
>> shs->recv_callback = libxl__srm_callout_received_restore;
>> - shs->completion_callback = libxl__xc_domain_restore_done;
>> + if (dcs->restore_params.checkpointed_stream ==
>> + LIBXL_CHECKPOINTED_STREAM_COLO)
>
> This is very strange line wrap.
Yes, will fix it in the next version.
Thanks
Wen Congyang
>
>> + shs->completion_callback = libxl__colo_restore_teardown;
>> + else
>> + shs->completion_callback = libxl__xc_domain_restore_done;
>> shs->caller_state = dcs;
>> shs->need_results = 1;
>>
>> diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
>> index 5d980d9..d6bd2fe 100644
>> --- a/tools/libxl/libxl_stream_read.c
>> +++ b/tools/libxl/libxl_stream_read.c
>> @@ -846,6 +846,18 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
>> */
>> if (libxl__stream_read_inuse(stream)) {
>> switch (checkpointed_stream) {
>> + case LIBXL_CHECKPOINTED_STREAM_COLO:
>> + if (stream->completion_callback) {
>> + /*
>> + * restore, just build the secondary vm, don't close
>> + * the stream
>> + */
>> + stream->completion_callback(egc, stream, 0);
>> + } else {
>> + /* failover, just close the stream */
>> + stream_complete(egc, stream, 0);
>> + }
>> + break;
>> case LIBXL_CHECKPOINTED_STREAM_REMUS:
>> /*
>> * Failover from primary. Domain state is currently at a
>> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
>> index 632c009..33f4a90 100644
>> --- a/tools/libxl/libxl_types.idl
>> +++ b/tools/libxl/libxl_types.idl
>> @@ -232,6 +232,7 @@ libxl_hdtype = Enumeration("hdtype", [
>> libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
>> (0, "NONE"),
>> (1, "REMUS"),
>> + (2, "COLO"),
>> ])
>>
>> #
>> --
>> 2.5.0
>>
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 17/31] primary vm suspend/resume/checkpoint code
2016-02-25 15:57 ` Wei Liu
@ 2016-02-26 2:32 ` Wen Congyang
0 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-26 2:32 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 02/25/2016 11:57 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:21AM +0800, Wen Congyang wrote:
> [...]
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +#include "libxl_colo.h"
>> +
>> +static const libxl__checkpoint_device_instance_ops *colo_ops[] = {
>> + NULL,
>> +};
>> +
>> +/* ================= helper functions ================= */
>> +static int init_device_subkind(libxl__checkpoint_devices_state *cds)
>> +{
>> + /* init device subkind-specific state in the libxl ctx */
>> + int rc;
>> + STATE_AO_GC(cds->ao);
>> +
>> + rc = 0;
>> + return rc;
>> +}
>> +
>> +static void cleanup_device_subkind(libxl__checkpoint_devices_state *cds)
>> +{
>> + /* cleanup device subkind-specific state in the libxl ctx */
>> + STATE_AO_GC(cds->ao);
>> +}
>> +
>> +/* ================= colo: setup save environment ================= */
>> +static void colo_save_setup_done(libxl__egc *egc,
>> + libxl__checkpoint_devices_state *cds,
>> + int rc);
>> +static void colo_save_setup_failed(libxl__egc *egc,
>> + libxl__checkpoint_devices_state *cds,
>> + int rc);
>> +static void libxl__colo_save_domain_suspend_callback(void *data);
>> +static void libxl__colo_save_domain_checkpoint_callback(void *data);
>> +static void libxl__colo_save_domain_resume_callback(void *data);
>> +static void libxl__colo_save_domain_wait_checkpoint_callback(void *data);
>> +
>
>
> An ordered list of callbacks is appreciated. This doesn't seem to cover
> all callbacks and I'm not sure if this is the order they are supposed to
> fire either.
OK, I will check it and fix it in the next version.
Thanks
Wen Congyang
>
> Wei.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 18/31] libxc/restore: support COLO restore
2016-02-25 15:57 ` Wei Liu
@ 2016-02-26 2:33 ` Wen Congyang
0 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-26 2:33 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 02/25/2016 11:57 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:22AM +0800, Wen Congyang wrote:
> [...]
>> - * With Remus, we buffer the records sent by the primary at checkpoint,
>> + * With Remus/COLO, we buffer the records sent by the primary at checkpoint,
>> * in case the primary will fail, we can recover from the last
>> * checkpoint state.
>> * This should be enough for most of the cases because primary only send
>> diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
>> index aef9bca..2ae8154 100644
>> --- a/tools/libxc/xc_sr_restore.c
>> +++ b/tools/libxc/xc_sr_restore.c
>> @@ -460,6 +460,49 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
>> else
>> ctx->restore.buffer_all_records = true;
>>
>> + if ( ctx->restore.checkpointed == MIG_STREAM_COLO )
>> + {
>> +#define HANDLE_CALLBACK_RETURN_VALUE(ret) \
>> + do { \
>> + if ( ret == 1 ) \
>> + rc = 0; /* Success */ \
>> + else \
>> + { \
>> + if ( ret == 2 ) \
>> + rc = BROKEN_CHANNEL; \
>> + else \
>> + rc = -1; /* Some unspecified error */ \
>> + goto err; \
>> + } \
>> + } while (0)
>> +
>> + /* COLO */
>> +
>> + /* We need to resume guest */
>> + rc = ctx->restore.ops.stream_complete(ctx);
>> + if ( rc )
>> + goto err;
>> +
>> + /* TODO: call restore_results */
>> +
>> + /* Resume secondary vm */
>> + ret = ctx->restore.callbacks->postcopy(ctx->restore.callbacks->data);
>> + HANDLE_CALLBACK_RETURN_VALUE(ret);
>> +
>> + /* Wait for a new checkpoint */
>> + ret = ctx->restore.callbacks->wait_checkpoint(
>> + ctx->restore.callbacks->data);
>> + HANDLE_CALLBACK_RETURN_VALUE(ret);
>> +
>> + /* suspend secondary vm */
>> + ret = ctx->restore.callbacks->suspend(ctx->restore.callbacks->data);
>> + HANDLE_CALLBACK_RETURN_VALUE(ret);
>> +
>> +#undef HANDLE_CALLBACK_RETURN_VALUE
>> +
>> + /* TODO: send dirty pfn list to primary */
>
> You replace the TODOs with actual code in the next two patches.
>
> You can rearrange them a bit so that you don't need to add TODOs at all.
Yes, will fix it in the next version.
Thanks
Wen Congyang
>
>
> Wei.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 21/31] libxc/save: support COLO save
2016-02-25 15:58 ` Wei Liu
@ 2016-02-26 2:35 ` Wen Congyang
0 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-02-26 2:35 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 02/25/2016 11:58 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:25AM +0800, Wen Congyang wrote:
> [...]
>> /*
>> * Suspend the domain and send dirty memory.
>> * This is the last iteration of the live migration and the
>> @@ -558,6 +610,16 @@ static int suspend_and_send_dirty(struct xc_sr_context *ctx)
>>
>> bitmap_or(dirty_bitmap, ctx->save.deferred_pages, ctx->save.p2m_size);
>>
>> + if ( !ctx->save.live && ctx->save.checkpointed == MIG_STREAM_COLO )
>> + {
>> + rc = merge_secondary_dirty_bitmap(ctx);
>> + if ( rc )
>> + {
>> + PERROR("Failed to get secondary vm's dirty pages");
>> + goto out;
>> + }
>> + }
>> +
>> rc = send_dirty_pages(ctx, stats.dirty_count + ctx->save.nr_deferred_pages);
>> if ( rc )
>> goto out;
>> @@ -791,13 +853,42 @@ static int save(struct xc_sr_context *ctx, uint16_t guest_type)
>> if ( rc )
>> goto err;
>>
>> - rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
>> - if ( rc <= 0 )
>> - goto err;
>> + if ( ctx->save.checkpointed == MIG_STREAM_COLO )
>> + {
>> + rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
>> + if ( !rc )
>> + {
>> + rc = -1;
>> + goto err;
>> + }
>> + }
>>
>> - rc = ctx->save.callbacks->checkpoint(ctx->save.callbacks->data);
>> - if ( rc <= 0 )
>> + rc = ctx->save.callbacks->postcopy(ctx->save.callbacks->data);
>> + if ( !rc )
>
> This original code for checking postcopy return value is if ( rc <= 0 ).
OK, I will check it.
Thanks
Wen Congyang
>
> Wei.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code
2016-02-25 15:56 ` Wei Liu
2016-02-26 2:30 ` Wen Congyang
@ 2016-03-01 10:06 ` Wen Congyang
1 sibling, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-03-01 10:06 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 02/25/2016 11:56 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:20AM +0800, Wen Congyang wrote:
>> Secondary vm is running in colo mode. So we will do
>> the following things again and again:
>> 1. Resume secondary vm
>> a. Send CHECKPOINT_SVM_READY to master.
>> b. If it is not the first resume, call libxl__checkpoint_devices_preresume().
>> c. If it is the first resume(resume right after live migration),
>> - call libxl__xc_domain_restore_done() to build the secondary vm.
>> - enable secondary vm's logdirty.
>> - call libxl__domain_resume() to resume secondary vm.
>> - call libxl__checkpoint_devices_setup() to setup checkpoint devices.
>> d. Send CHECKPOINT_SVM_RESUMED to master.
>> 2. Wait a new checkpoint
>> a. Call libxl__checkpoint_devices_commit().
>> b. Read CHECKPOINT_NEW from master.
>> 3. Suspend secondary vm
>> a. Suspend secondary vm.
>> b. Call libxl__checkpoint_devices_postsuspend().
>> c. Send CHECKPOINT_SVM_SUSPENDED to master.
>>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
>> ---
>> tools/libxc/xc_sr_common.h | 2 +
>> tools/libxc/xc_sr_save.c | 3 +-
>> tools/libxl/Makefile | 1 +
>> tools/libxl/libxl_colo.h | 24 +
>> tools/libxl/libxl_colo_restore.c | 1038 ++++++++++++++++++++++++++++++++++++++
>> tools/libxl/libxl_create.c | 37 ++
>> tools/libxl/libxl_internal.h | 19 +
>> tools/libxl/libxl_save_callout.c | 7 +-
>> tools/libxl/libxl_stream_read.c | 12 +
>> tools/libxl/libxl_types.idl | 1 +
>
> There is a bunch of TODOs in libxl_colo.c but I don't think you're in a
> better position to judge whether they should be blocker or not.
>
>> 10 files changed, 1142 insertions(+), 2 deletions(-)
>> create mode 100644 tools/libxl/libxl_colo.h
>> create mode 100644 tools/libxl/libxl_colo_restore.c
>>
>> diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
>> index 5d9f497..2bfed64 100644
>> --- a/tools/libxc/xc_sr_common.h
>> +++ b/tools/libxc/xc_sr_common.h
>> @@ -184,10 +184,12 @@ struct xc_sr_context
>> * migration stream
>> * 0: Plain VM
>> * 1: Remus
>> + * 2: COLO
>> */
>> enum {
>> MIG_STREAM_NONE, /* plain stream */
>> MIG_STREAM_REMUS,
>> + MIG_STREAM_COLO,
>> } migration_stream;
>>
>> union /* Common save or restore data. */
>> diff --git a/tools/libxc/xc_sr_save.c b/tools/libxc/xc_sr_save.c
>> index fe210cc..7393355 100644
>> --- a/tools/libxc/xc_sr_save.c
>> +++ b/tools/libxc/xc_sr_save.c
>> @@ -846,7 +846,8 @@ int xc_domain_save(xc_interface *xch, int io_fd, uint32_t dom,
>>
>> /* If altering migration_stream update this assert too. */
>> assert(checkpointed_stream == MIG_STREAM_NONE ||
>> - checkpointed_stream == MIG_STREAM_REMUS);
>> + checkpointed_stream == MIG_STREAM_REMUS ||
>> + checkpointed_stream == MIG_STREAM_COLO);
>>
>> /*
>> * TODO: Find some time to better tweak the live migration algorithm.
>
> [...]
>
>> +
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +#include "libxl_colo.h"
>> +#include "libxl_sr_stream_format.h"
>> +
>> +enum {
>> + LIBXL_COLO_SETUPED,
>> + LIBXL_COLO_SUSPENDED,
>> + LIBXL_COLO_RESUMED,
>> +};
>> +
>> +typedef struct libxl__colo_restore_checkpoint_state libxl__colo_restore_checkpoint_state;
>> +struct libxl__colo_restore_checkpoint_state {
>> + libxl__domain_suspend_state dsps;
>> + libxl__logdirty_switch lds;
>> + libxl__colo_restore_state *crs;
>> + libxl__stream_write_state sws;
>> + int status;
>> + bool preresume;
>> + /* used for teardown */
>> + int teardown_devices;
>> + int saved_rc;
>> + char *state_file;
>> +
>> + void (*callback)(libxl__egc *,
>> + libxl__colo_restore_checkpoint_state *,
>> + int);
>> +};
>> +
>
> Shouldn't the enum and struct belong to libxl_colo.h ?
If we inlucde libxl_colo.h in libxl_internal.h, we cannot move this into colo.h, because
this structure needs libxl__domain_suspend_state, libxl__logdirty_switch, ...
We cannot just declare it, because this structure needs know there size.
>
>> +
>> +static void libxl__colo_restore_domain_resume_callback(void *data);
>> +static void libxl__colo_restore_domain_checkpoint_callback(void *data);
>> +static void libxl__colo_restore_domain_wait_checkpoint_callback(void *data);
>> +static void libxl__colo_restore_domain_suspend_callback(void *data);
>> +
>> +static const libxl__checkpoint_device_instance_ops *colo_restore_ops[] = {
>> + NULL,
>> +};
>> +
>
> It would be helpful to list the callbacks at the beginning of the time
> in the order they are supposed to occur.
>
> See libxl_create.c for example. Search for "Event callbacks, in this
> order".
>
> I've tried to map the algorithm you described in commit message to all
> the callbacks, but without some references it is just too time consuming
> from my end.
>
> I think what I'm going to do is to make sure the normal path that
> doesn't use COLO is not broken and leave the internal to you and Ian (if
> he fancies to dig into details).
>
> [...]
>> +
>> +void libxl__colo_restore_setup(libxl__egc *egc,
>> + libxl__colo_restore_state *crs)
>> +{
>> + libxl__domain_create_state *dcs = CONTAINER_OF(crs, *dcs, crs);
>> + libxl__colo_restore_checkpoint_state *crcs;
>> + int rc = ERROR_FAIL;
>> +
>> + /* Convenience aliases */
>> + libxl__srm_restore_autogen_callbacks *const callbacks =
>> + &dcs->srs.shs.callbacks.restore.a;
>> + const int domid = crs->domid;
>> +
>> + STATE_AO_GC(crs->ao);
>> +
>> + GCNEW(crcs);
>> + crs->crcs = crcs;
>> + crcs->crs = crs;
>> +
>> + /* setup dsps */
>> + crcs->dsps.ao = ao;
>> + crcs->dsps.domid = domid;
>> + if (init_dsps(&crcs->dsps))
>> + goto err;
>> +
>> + callbacks->suspend = libxl__colo_restore_domain_suspend_callback;
>> + callbacks->postcopy = libxl__colo_restore_domain_resume_callback;
>> + callbacks->checkpoint = libxl__colo_restore_domain_checkpoint_callback;
>> + callbacks->wait_checkpoint = libxl__colo_restore_domain_wait_checkpoint_callback;
>> +
>> + /*
>> + * Secondary vm is running in colo mode, so we need to call
>> + * libxl__xc_domain_restore_done() to create secondary vm.
>> + * But we will exit in domain_create_cb(). So replace the
>> + * callback here.
>> + */
>> + crs->saved_cb = dcs->callback;
>> + dcs->callback = libxl__colo_domain_create_cb;
>> + crcs->state_file = GCSPRINTF(LIBXL_DEVICE_MODEL_RESTORE_FILE".%d", domid);
>
> Can you use a different name space from the normal one?
>
> For example, you can put
>
> #define LIBXL_COLO_DEVICE_MODEL_RESTORE_FILE XXXX
>
> in libxl_colo.h and use it in all COLO code.
>
>
>> + crcs->status = LIBXL_COLO_SETUPED;
>> +
>> + libxl__logdirty_init(&crcs->lds);
>> + crcs->lds.ao = ao;
>> +
>> + crcs->sws.fd = crs->send_back_fd;
>> + crcs->sws.ao = ao;
>> + crcs->sws.back_channel = true;
>> +
>> + dcs->cds.concrete_data = crs;
>> +
>> + libxl__stream_write_start(egc, &crcs->sws);
>> +
>> + rc = 0;
>> +
>> +out:
>> + crs->callback(egc, crs, rc);
>> + return;
>> +
>> +err:
>> + goto out;
>> +}
>> +
>> +static void libxl__colo_domain_create_cb(libxl__egc *egc,
>> + libxl__domain_create_state *dcs,
>> + int rc, uint32_t domid)
>> +{
>> + libxl__colo_restore_checkpoint_state *crcs = dcs->crs.crcs;
>> +
>> + crcs->callback(egc, crcs, rc);
>> +}
>> +
>> +
> [...]
>> +
>> +static void colo_disable_logdirty_done(libxl__egc *egc,
>> + libxl__logdirty_switch *lds,
>> + int rc)
>> +{
>> + libxl__colo_restore_checkpoint_state *crcs = CONTAINER_OF(lds, *crcs, lds);
>> +
>> + EGC_GC;
>> +
>> + if (rc)
>> + LOG(WARN, "cannot disable logdirty");
>> +
>> + if (crcs->status == LIBXL_COLO_SUSPENDED) {
>> + /*
>> + * failover when reading state from master, so no need to
>> + * call libxl__domain_restore().
>
> You need to update this comment to the right function name.
>
>> + */
>> + colo_resume_vm(egc, crcs, 0);
>> + return;
>> + }
>> +
>> + /* If we cannot disable logdirty, we still can do failover */
>> + crcs->callback(egc, crcs, 0);
>> +}
>> +
> [...]
>>
>> +/* colo related structure */
>> +typedef struct libxl__colo_restore_state libxl__colo_restore_state;
>> +typedef void libxl__colo_callback(libxl__egc *,
>> + libxl__colo_restore_state *, int rc);
>> +struct libxl__colo_restore_state {
>> + /* must set by caller of libxl__colo_(setup|teardown) */
>> + libxl__ao *ao;
>> + uint32_t domid;
>> + int send_back_fd;
>> + int recv_fd;
>> + int hvm;
>> + libxl__colo_callback *callback;
>> +
>> + /* private, colo restore checkpoint state */
>> + libxl__domain_create_cb *saved_cb;
>> + void *crcs;
>> +};
>>
>
> And this should go to libxl_colo.h, too? And libxl_internal.h includes
> libxl_colo.h?
If we do so, we should declare libxl__ao, libxl__domain_create_cb in libxl_colo.h
Thanks
Wen Congyang
>
> I just don't want to colo structures and functions scatter in
> different places.
>
>> struct libxl__domain_create_state {
>> /* filled in by user */
>> @@ -3486,6 +3503,8 @@ struct libxl__domain_create_state {
>> /* private to domain_create */
>> int guest_domid;
>> libxl__domain_build_state build_state;
>> + libxl__colo_restore_state crs;
>> + libxl__checkpoint_devices_state cds;
>> libxl__bootloader_state bl;
>> libxl__stub_dm_spawn_state dmss;
>> /* If we're not doing stubdom, we use only dmss.dm,
>> diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
>> index 0d6949a..b1810b2 100644
>> --- a/tools/libxl/libxl_save_callout.c
>> +++ b/tools/libxl/libxl_save_callout.c
>> @@ -15,6 +15,7 @@
>> #include "libxl_osdeps.h"
>>
>> #include "libxl_internal.h"
>> +#include "libxl_colo.h"
>>
>> /* stream_fd is as from the caller (eventually, the application).
>> * It may be 0, 1 or 2, in which case we need to dup it elsewhere.
>> @@ -68,7 +69,11 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
>> shs->ao = ao;
>> shs->domid = domid;
>> shs->recv_callback = libxl__srm_callout_received_restore;
>> - shs->completion_callback = libxl__xc_domain_restore_done;
>> + if (dcs->restore_params.checkpointed_stream ==
>> + LIBXL_CHECKPOINTED_STREAM_COLO)
>
> This is very strange line wrap.
>
>> + shs->completion_callback = libxl__colo_restore_teardown;
>> + else
>> + shs->completion_callback = libxl__xc_domain_restore_done;
>> shs->caller_state = dcs;
>> shs->need_results = 1;
>>
>> diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
>> index 5d980d9..d6bd2fe 100644
>> --- a/tools/libxl/libxl_stream_read.c
>> +++ b/tools/libxl/libxl_stream_read.c
>> @@ -846,6 +846,18 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
>> */
>> if (libxl__stream_read_inuse(stream)) {
>> switch (checkpointed_stream) {
>> + case LIBXL_CHECKPOINTED_STREAM_COLO:
>> + if (stream->completion_callback) {
>> + /*
>> + * restore, just build the secondary vm, don't close
>> + * the stream
>> + */
>> + stream->completion_callback(egc, stream, 0);
>> + } else {
>> + /* failover, just close the stream */
>> + stream_complete(egc, stream, 0);
>> + }
>> + break;
>> case LIBXL_CHECKPOINTED_STREAM_REMUS:
>> /*
>> * Failover from primary. Domain state is currently at a
>> diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
>> index 632c009..33f4a90 100644
>> --- a/tools/libxl/libxl_types.idl
>> +++ b/tools/libxl/libxl_types.idl
>> @@ -232,6 +232,7 @@ libxl_hdtype = Enumeration("hdtype", [
>> libxl_checkpointed_stream = Enumeration("checkpointed_stream", [
>> (0, "NONE"),
>> (1, "REMUS"),
>> + (2, "COLO"),
>> ])
>>
>> #
>> --
>> 2.5.0
>>
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 10/31] tools/libxl: add back channel support to write stream
2016-02-26 2:11 ` Wen Congyang
@ 2016-03-02 15:02 ` Wei Liu
2016-03-03 1:25 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-03-02 15:02 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Fri, Feb 26, 2016 at 10:11:27AM +0800, Wen Congyang wrote:
> On 02/25/2016 11:54 PM, Wei Liu wrote:
> > On Mon, Feb 22, 2016 at 10:52:14AM +0800, Wen Congyang wrote:
> >> Add back channel support to write stream. If the write stream is
> >> a back channel stream, this means the write stream is used by
> >> Secondary to send some records back.
> >>
> >> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> >> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> >> ---
> >> tools/libxl/libxl_dom_save.c | 1 +
> >> tools/libxl/libxl_internal.h | 1 +
> >> tools/libxl/libxl_stream_write.c | 26 ++++++++++++++++++++------
> >> 3 files changed, 22 insertions(+), 6 deletions(-)
> >>
> >> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
> >> index 72b61c7..18946c5 100644
> >> --- a/tools/libxl/libxl_dom_save.c
> >> +++ b/tools/libxl/libxl_dom_save.c
> >> @@ -404,6 +404,7 @@ void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
> >> dss->sws.ao = dss->ao;
> >> dss->sws.dss = dss;
> >> dss->sws.fd = dss->fd;
> >> + dss->sws.back_channel = false;
> >> dss->sws.completion_callback = stream_done;
> >>
> >> libxl__stream_write_start(egc, &dss->sws);
> >> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> >> index 3d3e8e8..e02e554 100644
> >> --- a/tools/libxl/libxl_internal.h
> >> +++ b/tools/libxl/libxl_internal.h
> >> @@ -3044,6 +3044,7 @@ struct libxl__stream_write_state {
> >> libxl__ao *ao;
> >> libxl__domain_save_state *dss;
> >> int fd;
> >> + bool back_channel;
> >> void (*completion_callback)(libxl__egc *egc,
> >> libxl__stream_write_state *sws,
> >> int rc);
> >> diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
> >> index f6ea55d..5379126 100644
> >> --- a/tools/libxl/libxl_stream_write.c
> >> +++ b/tools/libxl/libxl_stream_write.c
> >> @@ -49,6 +49,13 @@
> >> * - if (hvm)
> >> * - Emulator context record
> >> * - Checkpoint end record
> >> + *
> >> + * For back channel stream:
> >> + * - libxl__stream_write_start()
> >> + * - Set up the stream to running state
> >> + *
> >> + * - Add a new API to write the record. When the record is written
> >> + * out, call stream->checkpoint_callback() to return.
> >
> > What does this mean? Which new API?
>
> The next patch introduces this API. The commits is very old.
>
> I think I can merge these two patches into one patch.
>
Please reference the actual function / API.
> >
> >> */
> >>
> >> /* Success/error/cleanup handling. */
> >> @@ -225,6 +232,15 @@ void libxl__stream_write_start(libxl__egc *egc,
> >>
> >> stream->running = true;
> >>
> >> + dc->ao = ao;
> >> + dc->readfd = -1;
> >> + dc->copywhat = "save v2 stream";
> >> + dc->writefd = stream->fd;
> >> + dc->maxsz = -1;
> >> +
> >> + if (stream->back_channel)
> >> + return;
> >> +
> >
> > There seems to be very subtle change of behaviour.
> >
> > Before this patch, the dc->* are not set until the emulator check is
> > done. With this path, it is possible in the normal case some of the
> > fields are initialised in the error path.
> >
> > I think this is OK given the callbacks in the upper layer and in
> > the writer don't rely on those fields to clean up. Just one thing to
> > note.
> >
> > That said, I suggest you move all initialisation of dc->* in one place.
>
> OK, I will do it.
>
> >
> >> if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
> >> stream->device_model_version =
> >> libxl__device_model_version_running(gc, dss->domid);
> >> @@ -249,12 +265,7 @@ void libxl__stream_write_start(libxl__egc *egc,
> >> stream->emu_sub_hdr.index = 0;
> >> }
> >>
> >> - dc->ao = ao;
> >> - dc->readfd = -1;
> >> dc->writewhat = "stream header";
> >> - dc->copywhat = "save v2 stream";
> >> - dc->writefd = stream->fd;
> >> - dc->maxsz = -1;
> >> dc->callback = stream_header_done;
> >>
> >> rc = libxl__datacopier_start(dc);
> >> @@ -279,6 +290,7 @@ void libxl__stream_write_start_checkpoint(libxl__egc *egc,
> >> {
> >> assert(stream->running);
> >> assert(!stream->in_checkpoint);
> >> + assert(!stream->back_channel);
> >> stream->in_checkpoint = true;
> >>
> >> write_emulator_xenstore_record(egc, stream);
> >> @@ -590,7 +602,9 @@ static void stream_done(libxl__egc *egc,
> >> libxl__carefd_close(stream->emu_carefd);
> >> free(stream->emu_body);
> >>
> >> - check_all_finished(egc, stream, rc);
> >> + if (!stream->back_channel)
> >> + /* back channel stream doesn't have save helper */
> >> + check_all_finished(egc, stream, rc);
> >
> > Though it doesn't have helper, do you not need to check if the back
> > channel stream itself is OK? The comment itself doesn't seem to justify
> > this change.
>
> What do you want to check? assert(!stream->shs) or
> assert(!libxl__stream_write_inuse(stream))?
The stream itself is not in use.
Also check_all_finished calls completion_callback. Don't you need that,
too?
Wei.
>
> Thanks
> Wen Congyang
>
> >
> > Wei.
> >
> >> }
> >>
> >> static void checkpoint_done(libxl__egc *egc,
> >> --
> >> 2.5.0
> >>
> >>
> >>
> >>
> >> _______________________________________________
> >> Xen-devel mailing list
> >> Xen-devel@lists.xen.org
> >> http://lists.xen.org/xen-devel
> >
> >
> > .
> >
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 12/31] tools/libxl: add back channel support to read stream
2016-02-26 2:16 ` Wen Congyang
@ 2016-03-02 15:03 ` Wei Liu
0 siblings, 0 replies; 70+ messages in thread
From: Wei Liu @ 2016-03-02 15:03 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Fri, Feb 26, 2016 at 10:16:43AM +0800, Wen Congyang wrote:
[...]
> >
> > Even if it doesn't have restore helper, check_all_finished also checks
> > if the stream and the conversion helper are till in use. The
> > explanation in the comment doesn't seem to justify this change.
>
> In stream_done(), stream->running is set to false, so libxl__stream_read_in_use()
> returns false.
>
> Back channel stream doesn't support legacy stream, so there is no conversion helper.
>
> I will update the comments in the next version.
>
Yes, please do. Say clearly why it is not needed.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 22/31] implement the cmdline for COLO
2016-02-22 2:52 ` [PATCH v10 22/31] implement the cmdline for COLO Wen Congyang
@ 2016-03-02 15:03 ` Wei Liu
2016-03-03 1:30 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-03-02 15:03 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:26AM +0800, Wen Congyang wrote:
[...]
> + if (libxl_defbool_val(info->colo)) {
> + if (libxl_defbool_val(info->compression)) {
This can be simplified as
if (libxl_defbool_val(xxx) && libxl_defbool_val(yyy))
> + LOG(ERROR, "cannot use memory checkpoint compression in COLO mode");
> + rc = ERROR_FAIL;
> + goto out;
> + }
> + }
> +
> if (!libxl_defbool_val(info->allow_unsafe) &&
> (libxl_defbool_val(info->blackhole) ||
> !libxl_defbool_val(info->netbuf) ||
> @@ -876,7 +892,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
> dss->live = 1;
> dss->debug = 0;
> dss->remus = info;
> - dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
> + if (libxl_defbool_val(info->colo))
> + dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
> + else
> + dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>
> assert(info);
>
> diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
> index df7268b..0dc7220 100644
> --- a/tools/libxl/xl_cmdimpl.c
> +++ b/tools/libxl/xl_cmdimpl.c
> @@ -4440,6 +4440,8 @@ static void migrate_receive(int debug, int daemonize, int monitor,
> char rc_buf;
> char *migration_domname;
> struct domain_create dom_info;
> + const char *ha = checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO ?
> + "COLO" : "Remus";
>
> signal(SIGPIPE, SIG_IGN);
> /* if we get SIGPIPE we'd rather just have it as an error */
> @@ -4460,6 +4462,9 @@ static void migrate_receive(int debug, int daemonize, int monitor,
> dom_info.send_back_fd = send_fd;
> dom_info.migration_domname_r = &migration_domname;
> dom_info.checkpointed_stream = checkpointed;
> + if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
> + /* COLO uses stdout to send control message to master */
> + dom_info.quiet = 1;
>
It seems that dom_info->quiet affects stderr, not stdout. See the only
place that checks this in xl_cmdimpl.c.
> rc = create_domain(&dom_info);
> if (rc < 0) {
> @@ -4472,11 +4477,12 @@ static void migrate_receive(int debug, int daemonize, int monitor,
>
> switch (checkpointed) {
> case LIBXL_CHECKPOINTED_STREAM_REMUS:
> + case LIBXL_CHECKPOINTED_STREAM_COLO:
> /* If we are here, it means that the sender (primary) has crashed.
> * TODO: Split-Brain Check.
> */
> - fprintf(stderr, "migration target: Remus Failover for domain %u\n",
> - domid);
> + fprintf(stderr, "migration target: %s Failover for domain %u\n",
> + ha, domid);
>
> /*
> * If domain renaming fails, lets just continue (as we need the domain
> @@ -4492,16 +4498,20 @@ static void migrate_receive(int debug, int daemonize, int monitor,
> rc = libxl_domain_rename(ctx, domid, migration_domname,
> common_domname);
> if (rc)
> - fprintf(stderr, "migration target (Remus): "
> + fprintf(stderr, "migration target (%s): "
> "Failed to rename domain from %s to %s:%d\n",
> - migration_domname, common_domname, rc);
> + ha, migration_domname, common_domname, rc);
> }
>
> + if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
> + /* The guest is running after failover in COLO mode */
> + exit(rc ? -ERROR_FAIL: 0);
> +
> rc = libxl_domain_unpause(ctx, domid);
> if (rc)
> - fprintf(stderr, "migration target (Remus): "
> + fprintf(stderr, "migration target (%s): "
> "Failed to unpause domain %s (id: %u):%d\n",
> - common_domname, domid, rc);
> + ha, common_domname, domid, rc);
>
> exit(rc ? -ERROR_FAIL: 0);
> default:
> @@ -4649,7 +4659,7 @@ int main_migrate_receive(int argc, char **argv)
> libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
> int opt;
>
> - SWITCH_FOREACH_OPT(opt, "Fedr", NULL, "migrate-receive", 0) {
> + SWITCH_FOREACH_OPT(opt, "Fedrc", NULL, "migrate-receive", 0) {
> case 'F':
> daemonize = 0;
> break;
> @@ -4663,6 +4673,9 @@ int main_migrate_receive(int argc, char **argv)
> case 'r':
> checkpointed = LIBXL_CHECKPOINTED_STREAM_REMUS;
> break;
> + case 'c':
> + checkpointed = LIBXL_CHECKPOINTED_STREAM_COLO;
> + break;
> }
>
> if (argc-optind != 0) {
> @@ -8032,11 +8045,8 @@ int main_remus(int argc, char **argv)
> int config_len;
>
> memset(&r_info, 0, sizeof(libxl_domain_remus_info));
> - /* Defaults */
> - r_info.interval = 200;
> - libxl_defbool_setdefault(&r_info.blackhole, false);
>
> - SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:e", NULL, "remus", 2) {
> + SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:ec", NULL, "remus", 2) {
> case 'i':
> r_info.interval = atoi(optarg);
> break;
> @@ -8064,11 +8074,32 @@ int main_remus(int argc, char **argv)
> case 'e':
> daemonize = 0;
> break;
> + case 'c':
> + libxl_defbool_set(&r_info.colo, true);
> }
>
> domid = find_domain(argv[optind]);
> host = argv[optind + 1];
>
> + /* Defaults */
> + libxl_defbool_setdefault(&r_info.blackhole, false);
> + libxl_defbool_setdefault(&r_info.colo, false);
> + if (!libxl_defbool_val(r_info.colo) && !r_info.interval)
> + r_info.interval = 200;
> +
> + if (libxl_defbool_val(r_info.colo)) {
> + if (r_info.interval || libxl_defbool_val(r_info.blackhole)) {
> + perror("Option -c conflicts with -i or -b");
> + exit(-1);
> + }
> +
> + if (libxl_defbool_is_default(r_info.compression)) {
> + perror("COLO can't be used with memory compression. "
> + "Disable memory checkpoint compression now...");
> + libxl_defbool_set(&r_info.compression, false);
> + }
> + }
> +
I don't think I'm entirely happy with how these things are arranged.
Remus and COLO don't seem to have a set of consistent APIs that
arbitrary users can call.
But for the sake of not growing this series any longer let's leave it
like this for the moment. I think COLO at best is going to be (as you
stated in manpage) experimental at this stage.
> if (!r_info.netbufscript)
> r_info.netbufscript = default_remus_netbufscript;
>
> @@ -8083,8 +8114,9 @@ int main_remus(int argc, char **argv)
> if (!ssh_command[0]) {
> rune = host;
> } else {
> - xasprintf(&rune, "exec %s %s xl migrate-receive -r %s",
> + xasprintf(&rune, "exec %s %s xl migrate-receive %s %s",
> ssh_command, host,
> + libxl_defbool_val(r_info.colo) ? "-c" : "-r",
> daemonize ? "" : " -e");
> }
>
> @@ -8112,7 +8144,8 @@ int main_remus(int argc, char **argv)
> * domain to force failover
> */
> if (libxl_domain_info(ctx, 0, domid)) {
> - fprintf(stderr, "Remus: Primary domain has been destroyed.\n");
> + fprintf(stderr, "%s: Primary domain has been destroyed.\n",
> + libxl_defbool_val(r_info.colo) ? "COLO" : "Remus");
> close(send_fd);
> return 0;
> }
> @@ -8124,7 +8157,8 @@ int main_remus(int argc, char **argv)
> if (rc == ERROR_GUEST_TIMEDOUT)
> fprintf(stderr, "Failed to suspend domain at primary.\n");
> else {
> - fprintf(stderr, "Remus: Backup failed? resuming domain at primary.\n");
> + fprintf(stderr, "%s: Backup failed? resuming domain at primary.\n",
> + libxl_defbool_val(r_info.colo) ? "COLO" : "Remus");
> libxl_domain_resume(ctx, domid, 1, 0);
> }
>
> diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
> index fdc1ac6..b6b630c 100644
> --- a/tools/libxl/xl_cmdtable.c
> +++ b/tools/libxl/xl_cmdtable.c
> @@ -499,7 +499,9 @@ struct cmd_spec cmd_table[] = {
> "-b Replicate memory checkpoints to /dev/null (blackhole).\n"
> " Works only in unsafe mode.\n"
> "-n Disable network output buffering. Works only in unsafe mode.\n"
> - "-d Disable disk replication. Works only in unsafe mode."
> + "-d Disable disk replication. Works only in unsafe mode.\n"
> + "-c Enable COLO HA. It is conflict with -i and -b, and memory\n"
> + " checkpoint must be disabled"
> },
> #endif
> { "devd",
> --
> 2.5.0
>
>
>
>
> _______________________________________________
> Xen-devel mailing list
> Xen-devel@lists.xen.org
> http://lists.xen.org/xen-devel
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 23/31] COLO: introduce new API to prepare/start/do/get_error/stop replication
2016-02-22 2:52 ` [PATCH v10 23/31] COLO: introduce new API to prepare/start/do/get_error/stop replication Wen Congyang
@ 2016-03-02 15:03 ` Wei Liu
0 siblings, 0 replies; 70+ messages in thread
From: Wei Liu @ 2016-03-02 15:03 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:27AM +0800, Wen Congyang wrote:
> We will use qemu block replication, and qemu provides some qmp commands
> to prepare replication, start replication, get replication error, and
> stop replication. Introduce new API to execute these qmp commands.
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 25/31] COLO: use qemu block replication
2016-02-22 2:52 ` [PATCH v10 25/31] COLO: use qemu block replication Wen Congyang
@ 2016-03-02 15:03 ` Wei Liu
0 siblings, 0 replies; 70+ messages in thread
From: Wei Liu @ 2016-03-02 15:03 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:29AM +0800, Wen Congyang wrote:
> Use qemu block replication as our block replication solution.
> Note that guest must be paused before starting COLO, otherwise,
> the disk won't be consistent between primary and secondary.
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> ---
> tools/libxl/Makefile | 1 +
> tools/libxl/libxl_colo_qdisk.c | 226 +++++++++++++++++++++++++++++++++++++++
> tools/libxl/libxl_colo_restore.c | 42 +++++++-
> tools/libxl/libxl_colo_save.c | 54 +++++++++-
> tools/libxl/libxl_internal.h | 13 +++
All the changes look internal to COLO and are trying to manipulate QEMU
or whatnot. I don't think I can provide very detailed feedback. I am
tempted to just ack this patch.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module
2016-02-22 2:52 ` [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module Wen Congyang
@ 2016-03-02 15:04 ` Wei Liu
2016-03-11 22:25 ` Konrad Rzeszutek Wilk
1 sibling, 0 replies; 70+ messages in thread
From: Wei Liu @ 2016-03-02 15:04 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:30AM +0800, Wen Congyang wrote:
> setup/teardown of COLO proxy module.
> we use netlink to communicate with proxy module.
> About colo-proxy module:
> https://lkml.org/lkml/2015/6/18/32
> How to use:
> http://wiki.xen.org/wiki/COLO_-_Coarse_Grain_Lock_Stepping
>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
I'm tempted to just ack this patch as well.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 27/31] COLO proxy: preresume, postresume and checkpoint
2016-02-22 2:52 ` [PATCH v10 27/31] COLO proxy: preresume, postresume and checkpoint Wen Congyang
@ 2016-03-02 15:04 ` Wei Liu
0 siblings, 0 replies; 70+ messages in thread
From: Wei Liu @ 2016-03-02 15:04 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:31AM +0800, Wen Congyang wrote:
> preresume, postresume and checkpoint
>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Same as last patch...
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 24/31] Support colo mode for qemu disk
2016-02-22 2:52 ` [PATCH v10 24/31] Support colo mode for qemu disk Wen Congyang
@ 2016-03-02 15:04 ` Wei Liu
2016-03-03 1:40 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-03-02 15:04 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:28AM +0800, Wen Congyang wrote:
> Usage: disk = ['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
> For QEMU block replication details:
> http://wiki.qemu.org/Features/BlockReplication
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> ---
> docs/man/xl.pod.1 | 2 +-
> docs/misc/xl-disk-configuration.txt | 50 ++++++++++
> tools/libxl/libxl.c | 62 +++++++++++-
> tools/libxl/libxl_create.c | 25 ++++-
> tools/libxl/libxl_device.c | 54 +++++++++++
> tools/libxl/libxl_dm.c | 184 ++++++++++++++++++++++++++++++++++--
> tools/libxl/libxl_types.idl | 7 ++
> tools/libxl/libxlu_disk_l.l | 7 ++
> 8 files changed, 382 insertions(+), 9 deletions(-)
>
> diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
> index 1c6dd87..4f1901d 100644
> --- a/docs/man/xl.pod.1
> +++ b/docs/man/xl.pod.1
> @@ -454,7 +454,7 @@ N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
> Disk replication support is limited to DRBD disks.
>
> COLO support in xl is still in experimental (proof-of-concept) phase.
> - There is no support for network or disk at the moment.
> + There is no support for network at the moment.
You need some document here for the syntax, otherwise users have no clue
how to configure disk replicate support. I also won't be able to
meaningfully review this patch without a reference.
>
> B<OPTIONS>
>
> diff --git a/docs/misc/xl-disk-configuration.txt b/docs/misc/xl-disk-configuration.txt
> index 29f6ddb..6f23c2d 100644
> --- a/docs/misc/xl-disk-configuration.txt
> +++ b/docs/misc/xl-disk-configuration.txt
> @@ -234,6 +234,56 @@ were intentionally created non-sparse to avoid fragmentation of the
> file.
>
>
Some nitpicking about the format below.
> +===============
> +COLO PARAMETERS
> +===============
> +
> +
> +colo
> +----
> +
> +Enable COLO HA for disk. For better understanding block replication on
> +QEMU, please refer to:
> +http://wiki.qemu.org/Features/BlockReplication
> +
> +
> +colo-host
> +---------
Blank line here please.
> +Description: Secondary host's address
> +Mandatory: Yes when COLO enabled
> +
> +
> +colo-port
> +---------
Ditto.
> +Description: Secondary port
> + We will run a nbd server on secondary host,
> + and the nbd server will listen this port.
> +Mandatory: Yes when COLO enabled
> +
> +
> +colo-export
> +---------
Here as well. And some more "-"s to match "colo-export".
> +Description: We will run a nbd server on secondary host,
> + exportname is the nbd server's disk export name.
> +Mandatory: Yes when COLO enabled
> +
> +
> +active-disk
> +-----------
> +
> +Description: This is used by secondary. Secondary guest's write
> + will be buffered in this disk.
> +Mandatory: Yes when COLO enabled
> +
> +
> +hidden-disk
> +-----------
> +
> +Description: This is used by secondary. It buffers the original
> + content that is modified by the primary VM.
> +Mandatory: Yes when COLO enabled
> +
> +
The rest of the patch is mainly for manipulating QEMU parameters. I've
skipped it for now.
> ============================================
> DEPRECATED PARAMETERS, PREFIXES AND SYNTAXES
> ============================================
> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
> index 12df81a..f691628 100644
> --- a/tools/libxl/libxl.c
> +++ b/tools/libxl/libxl.c
> @@ -2309,6 +2309,8 @@ int libxl__device_disk_setdefault(libxl__gc *gc, libxl_device_disk *disk)
> int rc;
>
> libxl_defbool_setdefault(&disk->discard_enable, !!disk->readwrite);
> + libxl_defbool_setdefault(&disk->colo_enable, false);
> + libxl_defbool_setdefault(&disk->colo_restore_enable, false);
>
> rc = libxl__resolve_domid(gc, disk->backend_domname, &disk->backend_domid);
> if (rc < 0) return rc;
> @@ -2507,6 +2509,18 @@ static void device_disk_add(libxl__egc *egc, uint32_t domid,
> flexarray_append(back, "params");
> flexarray_append(back, GCSPRINTF("%s:%s",
> libxl__device_disk_string_of_format(disk->format), disk->pdev_path));
> + if (libxl_defbool_val(disk->colo_enable)) {
> + flexarray_append(back, "colo-host");
> + flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_host));
> + flexarray_append(back, "colo-port");
> + flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_port));
> + flexarray_append(back, "colo-export");
> + flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_export));
> + flexarray_append(back, "active-disk");
> + flexarray_append(back, libxl__sprintf(gc, "%s", disk->active_disk));
> + flexarray_append(back, "hidden-disk");
> + flexarray_append(back, libxl__sprintf(gc, "%s", disk->hidden_disk));
> + }
> assert(device->backend_kind == LIBXL__DEVICE_KIND_QDISK);
> break;
> default:
> @@ -2622,7 +2636,10 @@ static int libxl__device_disk_from_xs_be(libxl__gc *gc,
> goto cleanup;
> }
>
> - /* "params" may not be present; but everything else must be. */
> + /*
> + * "params" and "colo-host" may not be present; but everything
> + * else must be.
I don't think this is correct. You have certainly have colo disabled.
I think you can just avoid modifying this comment.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 28/31] COLO nic: implement COLO nic subkind
2016-02-22 2:52 ` [PATCH v10 28/31] COLO nic: implement COLO nic subkind Wen Congyang
@ 2016-03-02 15:04 ` Wei Liu
0 siblings, 0 replies; 70+ messages in thread
From: Wei Liu @ 2016-03-02 15:04 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:32AM +0800, Wen Congyang wrote:
> implement COLO nic subkind.
>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> ---
> tools/hotplug/Linux/Makefile | 1 +
> tools/hotplug/Linux/colo-proxy-setup | 135 +++++++++++++++
> tools/libxl/Makefile | 1 +
> tools/libxl/libxl_colo_nic.c | 321 +++++++++++++++++++++++++++++++++++
> tools/libxl/libxl_internal.h | 5 +
> tools/libxl/libxl_types.idl | 1 +
I skipped this patch because it looks mostly internal to COLO.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 31/31] cmdline switches and config vars to control colo-proxy
2016-02-22 2:52 ` [PATCH v10 31/31] cmdline switches and config vars to control colo-proxy Wen Congyang
@ 2016-03-02 15:05 ` Wei Liu
2016-03-03 1:41 ` Wen Congyang
0 siblings, 1 reply; 70+ messages in thread
From: Wei Liu @ 2016-03-02 15:05 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On Mon, Feb 22, 2016 at 10:52:35AM +0800, Wen Congyang wrote:
> Add cmdline switches to 'xl migrate-receive' command to specify
> a domain-specific hotplug script to setup COLO proxy.
>
> Add a new config var 'colo.default.agentscript' to xl.conf, that
> allows the user to override the default global script used to
> setup COLO proxy.
>
> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> ---
> docs/man/xl.conf.pod.5 | 6 ++++++
> docs/man/xl.pod.1 | 1 -
> tools/libxl/libxl.c | 6 ++++++
> tools/libxl/libxl_create.c | 14 ++++++++++++--
> tools/libxl/libxl_types.idl | 1 +
> tools/libxl/xl.c | 3 +++
> tools/libxl/xl.h | 1 +
> tools/libxl/xl_cmdimpl.c | 47 ++++++++++++++++++++++++++++++++++-----------
> 8 files changed, 65 insertions(+), 14 deletions(-)
>
> diff --git a/docs/man/xl.conf.pod.5 b/docs/man/xl.conf.pod.5
> index 8ae19bb..8f7fd28 100644
> --- a/docs/man/xl.conf.pod.5
> +++ b/docs/man/xl.conf.pod.5
> @@ -111,6 +111,12 @@ Configures the default script used by Remus to setup network buffering.
>
> Default: C</etc/xen/scripts/remus-netbuf-setup>
>
> +=item B<colo.default.proxyscript="PATH">
> +
> +Configures the default script used by COLO to setup colo-proxy.
> +
> +Default: C</etc/xen/scripts/colo-proxy-setup>
> +
> =item B<output_format="json|sxp">
>
> Configures the default output format used by xl when printing "machine
> diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
> index 4f1901d..edeafcf 100644
> --- a/docs/man/xl.pod.1
> +++ b/docs/man/xl.pod.1
> @@ -454,7 +454,6 @@ N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
> Disk replication support is limited to DRBD disks.
>
> COLO support in xl is still in experimental (proof-of-concept) phase.
> - There is no support for network at the moment.
Same here, missing documentation on how to use the new parameters (if
any). Please provide adequate documentation otherwise we can't
meaningfully review the rest of this patch.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 10/31] tools/libxl: add back channel support to write stream
2016-03-02 15:02 ` Wei Liu
@ 2016-03-03 1:25 ` Wen Congyang
0 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-03-03 1:25 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/02/2016 11:02 PM, Wei Liu wrote:
> On Fri, Feb 26, 2016 at 10:11:27AM +0800, Wen Congyang wrote:
>> On 02/25/2016 11:54 PM, Wei Liu wrote:
>>> On Mon, Feb 22, 2016 at 10:52:14AM +0800, Wen Congyang wrote:
>>>> Add back channel support to write stream. If the write stream is
>>>> a back channel stream, this means the write stream is used by
>>>> Secondary to send some records back.
>>>>
>>>> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
>>>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>>>> ---
>>>> tools/libxl/libxl_dom_save.c | 1 +
>>>> tools/libxl/libxl_internal.h | 1 +
>>>> tools/libxl/libxl_stream_write.c | 26 ++++++++++++++++++++------
>>>> 3 files changed, 22 insertions(+), 6 deletions(-)
>>>>
>>>> diff --git a/tools/libxl/libxl_dom_save.c b/tools/libxl/libxl_dom_save.c
>>>> index 72b61c7..18946c5 100644
>>>> --- a/tools/libxl/libxl_dom_save.c
>>>> +++ b/tools/libxl/libxl_dom_save.c
>>>> @@ -404,6 +404,7 @@ void libxl__domain_save(libxl__egc *egc, libxl__domain_save_state *dss)
>>>> dss->sws.ao = dss->ao;
>>>> dss->sws.dss = dss;
>>>> dss->sws.fd = dss->fd;
>>>> + dss->sws.back_channel = false;
>>>> dss->sws.completion_callback = stream_done;
>>>>
>>>> libxl__stream_write_start(egc, &dss->sws);
>>>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>>>> index 3d3e8e8..e02e554 100644
>>>> --- a/tools/libxl/libxl_internal.h
>>>> +++ b/tools/libxl/libxl_internal.h
>>>> @@ -3044,6 +3044,7 @@ struct libxl__stream_write_state {
>>>> libxl__ao *ao;
>>>> libxl__domain_save_state *dss;
>>>> int fd;
>>>> + bool back_channel;
>>>> void (*completion_callback)(libxl__egc *egc,
>>>> libxl__stream_write_state *sws,
>>>> int rc);
>>>> diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
>>>> index f6ea55d..5379126 100644
>>>> --- a/tools/libxl/libxl_stream_write.c
>>>> +++ b/tools/libxl/libxl_stream_write.c
>>>> @@ -49,6 +49,13 @@
>>>> * - if (hvm)
>>>> * - Emulator context record
>>>> * - Checkpoint end record
>>>> + *
>>>> + * For back channel stream:
>>>> + * - libxl__stream_write_start()
>>>> + * - Set up the stream to running state
>>>> + *
>>>> + * - Add a new API to write the record. When the record is written
>>>> + * out, call stream->checkpoint_callback() to return.
>>>
>>> What does this mean? Which new API?
>>
>> The next patch introduces this API. The commits is very old.
>>
>> I think I can merge these two patches into one patch.
>>
>
> Please reference the actual function / API.
>
>>>
>>>> */
>>>>
>>>> /* Success/error/cleanup handling. */
>>>> @@ -225,6 +232,15 @@ void libxl__stream_write_start(libxl__egc *egc,
>>>>
>>>> stream->running = true;
>>>>
>>>> + dc->ao = ao;
>>>> + dc->readfd = -1;
>>>> + dc->copywhat = "save v2 stream";
>>>> + dc->writefd = stream->fd;
>>>> + dc->maxsz = -1;
>>>> +
>>>> + if (stream->back_channel)
>>>> + return;
>>>> +
>>>
>>> There seems to be very subtle change of behaviour.
>>>
>>> Before this patch, the dc->* are not set until the emulator check is
>>> done. With this path, it is possible in the normal case some of the
>>> fields are initialised in the error path.
>>>
>>> I think this is OK given the callbacks in the upper layer and in
>>> the writer don't rely on those fields to clean up. Just one thing to
>>> note.
>>>
>>> That said, I suggest you move all initialisation of dc->* in one place.
>>
>> OK, I will do it.
>>
>>>
>>>> if (dss->type == LIBXL_DOMAIN_TYPE_HVM) {
>>>> stream->device_model_version =
>>>> libxl__device_model_version_running(gc, dss->domid);
>>>> @@ -249,12 +265,7 @@ void libxl__stream_write_start(libxl__egc *egc,
>>>> stream->emu_sub_hdr.index = 0;
>>>> }
>>>>
>>>> - dc->ao = ao;
>>>> - dc->readfd = -1;
>>>> dc->writewhat = "stream header";
>>>> - dc->copywhat = "save v2 stream";
>>>> - dc->writefd = stream->fd;
>>>> - dc->maxsz = -1;
>>>> dc->callback = stream_header_done;
>>>>
>>>> rc = libxl__datacopier_start(dc);
>>>> @@ -279,6 +290,7 @@ void libxl__stream_write_start_checkpoint(libxl__egc *egc,
>>>> {
>>>> assert(stream->running);
>>>> assert(!stream->in_checkpoint);
>>>> + assert(!stream->back_channel);
>>>> stream->in_checkpoint = true;
>>>>
>>>> write_emulator_xenstore_record(egc, stream);
>>>> @@ -590,7 +602,9 @@ static void stream_done(libxl__egc *egc,
>>>> libxl__carefd_close(stream->emu_carefd);
>>>> free(stream->emu_body);
>>>>
>>>> - check_all_finished(egc, stream, rc);
>>>> + if (!stream->back_channel)
>>>> + /* back channel stream doesn't have save helper */
>>>> + check_all_finished(egc, stream, rc);
>>>
>>> Though it doesn't have helper, do you not need to check if the back
>>> channel stream itself is OK? The comment itself doesn't seem to justify
>>> this change.
>>
>> What do you want to check? assert(!stream->shs) or
>> assert(!libxl__stream_write_inuse(stream))?
>
>
> The stream itself is not in use.
We set stream->running to false in stream_done, so libxl__stream_write_inuse()
will return false. So I think there is no need to check it. I will update
the comments to document it.
>
> Also check_all_finished calls completion_callback. Don't you need that,
> too?
back channel stream doesn't have completion_callback. See the comments in
check_all_finished().
Thanks
Wen Congyang
>
> Wei.
>
>>
>> Thanks
>> Wen Congyang
>>
>>>
>>> Wei.
>>>
>>>> }
>>>>
>>>> static void checkpoint_done(libxl__egc *egc,
>>>> --
>>>> 2.5.0
>>>>
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> Xen-devel mailing list
>>>> Xen-devel@lists.xen.org
>>>> http://lists.xen.org/xen-devel
>>>
>>>
>>> .
>>>
>>
>>
>>
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 22/31] implement the cmdline for COLO
2016-03-02 15:03 ` Wei Liu
@ 2016-03-03 1:30 ` Wen Congyang
0 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-03-03 1:30 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/02/2016 11:03 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:26AM +0800, Wen Congyang wrote:
> [...]
>> + if (libxl_defbool_val(info->colo)) {
>> + if (libxl_defbool_val(info->compression)) {
>
> This can be simplified as
>
> if (libxl_defbool_val(xxx) && libxl_defbool_val(yyy))
OK. will fix it in the next version.
>
>> + LOG(ERROR, "cannot use memory checkpoint compression in COLO mode");
>> + rc = ERROR_FAIL;
>> + goto out;
>> + }
>> + }
>> +
>> if (!libxl_defbool_val(info->allow_unsafe) &&
>> (libxl_defbool_val(info->blackhole) ||
>> !libxl_defbool_val(info->netbuf) ||
>> @@ -876,7 +892,10 @@ int libxl_domain_remus_start(libxl_ctx *ctx, libxl_domain_remus_info *info,
>> dss->live = 1;
>> dss->debug = 0;
>> dss->remus = info;
>> - dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>> + if (libxl_defbool_val(info->colo))
>> + dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_COLO;
>> + else
>> + dss->checkpointed_stream = LIBXL_CHECKPOINTED_STREAM_REMUS;
>>
>> assert(info);
>>
>> diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
>> index df7268b..0dc7220 100644
>> --- a/tools/libxl/xl_cmdimpl.c
>> +++ b/tools/libxl/xl_cmdimpl.c
>> @@ -4440,6 +4440,8 @@ static void migrate_receive(int debug, int daemonize, int monitor,
>> char rc_buf;
>> char *migration_domname;
>> struct domain_create dom_info;
>> + const char *ha = checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO ?
>> + "COLO" : "Remus";
>>
>> signal(SIGPIPE, SIG_IGN);
>> /* if we get SIGPIPE we'd rather just have it as an error */
>> @@ -4460,6 +4462,9 @@ static void migrate_receive(int debug, int daemonize, int monitor,
>> dom_info.send_back_fd = send_fd;
>> dom_info.migration_domname_r = &migration_domname;
>> dom_info.checkpointed_stream = checkpointed;
>> + if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
>> + /* COLO uses stdout to send control message to master */
>> + dom_info.quiet = 1;
>>
>
> It seems that dom_info->quiet affects stderr, not stdout. See the only
> place that checks this in xl_cmdimpl.c.
>
>> rc = create_domain(&dom_info);
>> if (rc < 0) {
>> @@ -4472,11 +4477,12 @@ static void migrate_receive(int debug, int daemonize, int monitor,
>>
>> switch (checkpointed) {
>> case LIBXL_CHECKPOINTED_STREAM_REMUS:
>> + case LIBXL_CHECKPOINTED_STREAM_COLO:
>> /* If we are here, it means that the sender (primary) has crashed.
>> * TODO: Split-Brain Check.
>> */
>> - fprintf(stderr, "migration target: Remus Failover for domain %u\n",
>> - domid);
>> + fprintf(stderr, "migration target: %s Failover for domain %u\n",
>> + ha, domid);
>>
>> /*
>> * If domain renaming fails, lets just continue (as we need the domain
>> @@ -4492,16 +4498,20 @@ static void migrate_receive(int debug, int daemonize, int monitor,
>> rc = libxl_domain_rename(ctx, domid, migration_domname,
>> common_domname);
>> if (rc)
>> - fprintf(stderr, "migration target (Remus): "
>> + fprintf(stderr, "migration target (%s): "
>> "Failed to rename domain from %s to %s:%d\n",
>> - migration_domname, common_domname, rc);
>> + ha, migration_domname, common_domname, rc);
>> }
>>
>> + if (checkpointed == LIBXL_CHECKPOINTED_STREAM_COLO)
>> + /* The guest is running after failover in COLO mode */
>> + exit(rc ? -ERROR_FAIL: 0);
>> +
>> rc = libxl_domain_unpause(ctx, domid);
>> if (rc)
>> - fprintf(stderr, "migration target (Remus): "
>> + fprintf(stderr, "migration target (%s): "
>> "Failed to unpause domain %s (id: %u):%d\n",
>> - common_domname, domid, rc);
>> + ha, common_domname, domid, rc);
>>
>> exit(rc ? -ERROR_FAIL: 0);
>> default:
>> @@ -4649,7 +4659,7 @@ int main_migrate_receive(int argc, char **argv)
>> libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
>> int opt;
>>
>> - SWITCH_FOREACH_OPT(opt, "Fedr", NULL, "migrate-receive", 0) {
>> + SWITCH_FOREACH_OPT(opt, "Fedrc", NULL, "migrate-receive", 0) {
>> case 'F':
>> daemonize = 0;
>> break;
>> @@ -4663,6 +4673,9 @@ int main_migrate_receive(int argc, char **argv)
>> case 'r':
>> checkpointed = LIBXL_CHECKPOINTED_STREAM_REMUS;
>> break;
>> + case 'c':
>> + checkpointed = LIBXL_CHECKPOINTED_STREAM_COLO;
>> + break;
>> }
>>
>> if (argc-optind != 0) {
>> @@ -8032,11 +8045,8 @@ int main_remus(int argc, char **argv)
>> int config_len;
>>
>> memset(&r_info, 0, sizeof(libxl_domain_remus_info));
>> - /* Defaults */
>> - r_info.interval = 200;
>> - libxl_defbool_setdefault(&r_info.blackhole, false);
>>
>> - SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:e", NULL, "remus", 2) {
>> + SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:ec", NULL, "remus", 2) {
>> case 'i':
>> r_info.interval = atoi(optarg);
>> break;
>> @@ -8064,11 +8074,32 @@ int main_remus(int argc, char **argv)
>> case 'e':
>> daemonize = 0;
>> break;
>> + case 'c':
>> + libxl_defbool_set(&r_info.colo, true);
>> }
>>
>> domid = find_domain(argv[optind]);
>> host = argv[optind + 1];
>>
>> + /* Defaults */
>> + libxl_defbool_setdefault(&r_info.blackhole, false);
>> + libxl_defbool_setdefault(&r_info.colo, false);
>> + if (!libxl_defbool_val(r_info.colo) && !r_info.interval)
>> + r_info.interval = 200;
>> +
>> + if (libxl_defbool_val(r_info.colo)) {
>> + if (r_info.interval || libxl_defbool_val(r_info.blackhole)) {
>> + perror("Option -c conflicts with -i or -b");
>> + exit(-1);
>> + }
>> +
>> + if (libxl_defbool_is_default(r_info.compression)) {
>> + perror("COLO can't be used with memory compression. "
>> + "Disable memory checkpoint compression now...");
>> + libxl_defbool_set(&r_info.compression, false);
>> + }
>> + }
>> +
>
> I don't think I'm entirely happy with how these things are arranged.
> Remus and COLO don't seem to have a set of consistent APIs that
> arbitrary users can call.
>
> But for the sake of not growing this series any longer let's leave it
> like this for the moment. I think COLO at best is going to be (as you
> stated in manpage) experimental at this stage.
Yes, it is experimental now.
Thanks
Wen Congyang
>
>
>> if (!r_info.netbufscript)
>> r_info.netbufscript = default_remus_netbufscript;
>>
>> @@ -8083,8 +8114,9 @@ int main_remus(int argc, char **argv)
>> if (!ssh_command[0]) {
>> rune = host;
>> } else {
>> - xasprintf(&rune, "exec %s %s xl migrate-receive -r %s",
>> + xasprintf(&rune, "exec %s %s xl migrate-receive %s %s",
>> ssh_command, host,
>> + libxl_defbool_val(r_info.colo) ? "-c" : "-r",
>> daemonize ? "" : " -e");
>> }
>>
>> @@ -8112,7 +8144,8 @@ int main_remus(int argc, char **argv)
>> * domain to force failover
>> */
>> if (libxl_domain_info(ctx, 0, domid)) {
>> - fprintf(stderr, "Remus: Primary domain has been destroyed.\n");
>> + fprintf(stderr, "%s: Primary domain has been destroyed.\n",
>> + libxl_defbool_val(r_info.colo) ? "COLO" : "Remus");
>> close(send_fd);
>> return 0;
>> }
>> @@ -8124,7 +8157,8 @@ int main_remus(int argc, char **argv)
>> if (rc == ERROR_GUEST_TIMEDOUT)
>> fprintf(stderr, "Failed to suspend domain at primary.\n");
>> else {
>> - fprintf(stderr, "Remus: Backup failed? resuming domain at primary.\n");
>> + fprintf(stderr, "%s: Backup failed? resuming domain at primary.\n",
>> + libxl_defbool_val(r_info.colo) ? "COLO" : "Remus");
>> libxl_domain_resume(ctx, domid, 1, 0);
>> }
>>
>> diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
>> index fdc1ac6..b6b630c 100644
>> --- a/tools/libxl/xl_cmdtable.c
>> +++ b/tools/libxl/xl_cmdtable.c
>> @@ -499,7 +499,9 @@ struct cmd_spec cmd_table[] = {
>> "-b Replicate memory checkpoints to /dev/null (blackhole).\n"
>> " Works only in unsafe mode.\n"
>> "-n Disable network output buffering. Works only in unsafe mode.\n"
>> - "-d Disable disk replication. Works only in unsafe mode."
>> + "-d Disable disk replication. Works only in unsafe mode.\n"
>> + "-c Enable COLO HA. It is conflict with -i and -b, and memory\n"
>> + " checkpoint must be disabled"
>> },
>> #endif
>> { "devd",
>> --
>> 2.5.0
>>
>>
>>
>>
>> _______________________________________________
>> Xen-devel mailing list
>> Xen-devel@lists.xen.org
>> http://lists.xen.org/xen-devel
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 24/31] Support colo mode for qemu disk
2016-03-02 15:04 ` Wei Liu
@ 2016-03-03 1:40 ` Wen Congyang
0 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-03-03 1:40 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/02/2016 11:04 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:28AM +0800, Wen Congyang wrote:
>> Usage: disk = ['...,colo,colo-host=xxx,colo-port=xxx,colo-export=xxx,active-disk=xxx,hidden-disk=xxx...']
>> For QEMU block replication details:
>> http://wiki.qemu.org/Features/BlockReplication
>>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
>> ---
>> docs/man/xl.pod.1 | 2 +-
>> docs/misc/xl-disk-configuration.txt | 50 ++++++++++
>> tools/libxl/libxl.c | 62 +++++++++++-
>> tools/libxl/libxl_create.c | 25 ++++-
>> tools/libxl/libxl_device.c | 54 +++++++++++
>> tools/libxl/libxl_dm.c | 184 ++++++++++++++++++++++++++++++++++--
>> tools/libxl/libxl_types.idl | 7 ++
>> tools/libxl/libxlu_disk_l.l | 7 ++
>> 8 files changed, 382 insertions(+), 9 deletions(-)
>>
>> diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
>> index 1c6dd87..4f1901d 100644
>> --- a/docs/man/xl.pod.1
>> +++ b/docs/man/xl.pod.1
>> @@ -454,7 +454,7 @@ N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
>> Disk replication support is limited to DRBD disks.
>>
>> COLO support in xl is still in experimental (proof-of-concept) phase.
>> - There is no support for network or disk at the moment.
>> + There is no support for network at the moment.
>
> You need some document here for the syntax, otherwise users have no clue
> how to configure disk replicate support. I also won't be able to
> meaningfully review this patch without a reference.
OK. will fix it in the next version.
>
>>
>> B<OPTIONS>
>>
>> diff --git a/docs/misc/xl-disk-configuration.txt b/docs/misc/xl-disk-configuration.txt
>> index 29f6ddb..6f23c2d 100644
>> --- a/docs/misc/xl-disk-configuration.txt
>> +++ b/docs/misc/xl-disk-configuration.txt
>> @@ -234,6 +234,56 @@ were intentionally created non-sparse to avoid fragmentation of the
>> file.
>>
>>
>
> Some nitpicking about the format below.
>
>> +===============
>> +COLO PARAMETERS
>> +===============
>> +
>> +
>> +colo
>> +----
>> +
>> +Enable COLO HA for disk. For better understanding block replication on
>> +QEMU, please refer to:
>> +http://wiki.qemu.org/Features/BlockReplication
>> +
>> +
>> +colo-host
>> +---------
>
> Blank line here please.
>
>> +Description: Secondary host's address
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +colo-port
>> +---------
>
> Ditto.
>
>> +Description: Secondary port
>> + We will run a nbd server on secondary host,
>> + and the nbd server will listen this port.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +colo-export
>> +---------
>
> Here as well. And some more "-"s to match "colo-export".
>
>> +Description: We will run a nbd server on secondary host,
>> + exportname is the nbd server's disk export name.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +active-disk
>> +-----------
>> +
>> +Description: This is used by secondary. Secondary guest's write
>> + will be buffered in this disk.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>> +hidden-disk
>> +-----------
>> +
>> +Description: This is used by secondary. It buffers the original
>> + content that is modified by the primary VM.
>> +Mandatory: Yes when COLO enabled
>> +
>> +
>
> The rest of the patch is mainly for manipulating QEMU parameters. I've
> skipped it for now.
If you want to know about how qemu block repication works, you can see:
http://wiki.qemu.org/Features/BlockReplication
>
>> ============================================
>> DEPRECATED PARAMETERS, PREFIXES AND SYNTAXES
>> ============================================
>> diff --git a/tools/libxl/libxl.c b/tools/libxl/libxl.c
>> index 12df81a..f691628 100644
>> --- a/tools/libxl/libxl.c
>> +++ b/tools/libxl/libxl.c
>> @@ -2309,6 +2309,8 @@ int libxl__device_disk_setdefault(libxl__gc *gc, libxl_device_disk *disk)
>> int rc;
>>
>> libxl_defbool_setdefault(&disk->discard_enable, !!disk->readwrite);
>> + libxl_defbool_setdefault(&disk->colo_enable, false);
>> + libxl_defbool_setdefault(&disk->colo_restore_enable, false);
>>
>> rc = libxl__resolve_domid(gc, disk->backend_domname, &disk->backend_domid);
>> if (rc < 0) return rc;
>> @@ -2507,6 +2509,18 @@ static void device_disk_add(libxl__egc *egc, uint32_t domid,
>> flexarray_append(back, "params");
>> flexarray_append(back, GCSPRINTF("%s:%s",
>> libxl__device_disk_string_of_format(disk->format), disk->pdev_path));
>> + if (libxl_defbool_val(disk->colo_enable)) {
>> + flexarray_append(back, "colo-host");
>> + flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_host));
>> + flexarray_append(back, "colo-port");
>> + flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_port));
>> + flexarray_append(back, "colo-export");
>> + flexarray_append(back, libxl__sprintf(gc, "%s", disk->colo_export));
>> + flexarray_append(back, "active-disk");
>> + flexarray_append(back, libxl__sprintf(gc, "%s", disk->active_disk));
>> + flexarray_append(back, "hidden-disk");
>> + flexarray_append(back, libxl__sprintf(gc, "%s", disk->hidden_disk));
>> + }
>> assert(device->backend_kind == LIBXL__DEVICE_KIND_QDISK);
>> break;
>> default:
>> @@ -2622,7 +2636,10 @@ static int libxl__device_disk_from_xs_be(libxl__gc *gc,
>> goto cleanup;
>> }
>>
>> - /* "params" may not be present; but everything else must be. */
>> + /*
>> + * "params" and "colo-host" may not be present; but everything
>> + * else must be.
>
> I don't think this is correct. You have certainly have colo disabled.
> I think you can just avoid modifying this comment.
Hmm, add the following comments:
/* colo releated entries(colo-host, colo-port, colo-export,
* active-disk and hidden-disk) are present only if colo is
* disabled.
*/
Thanks
Wen Congyang
>
> Wei.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 31/31] cmdline switches and config vars to control colo-proxy
2016-03-02 15:05 ` Wei Liu
@ 2016-03-03 1:41 ` Wen Congyang
0 siblings, 0 replies; 70+ messages in thread
From: Wen Congyang @ 2016-03-03 1:41 UTC (permalink / raw)
To: Wei Liu
Cc: Lars Kurth, Changlong Xie, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Ian Jackson, xen devel, Dong Eddie, Gui Jianfeng,
Shriram Rajagopalan, Yang Hongyang
On 03/02/2016 11:05 PM, Wei Liu wrote:
> On Mon, Feb 22, 2016 at 10:52:35AM +0800, Wen Congyang wrote:
>> Add cmdline switches to 'xl migrate-receive' command to specify
>> a domain-specific hotplug script to setup COLO proxy.
>>
>> Add a new config var 'colo.default.agentscript' to xl.conf, that
>> allows the user to override the default global script used to
>> setup COLO proxy.
>>
>> Signed-off-by: Yang Hongyang <hongyang.yang@easystack.cn>
>> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
>> ---
>> docs/man/xl.conf.pod.5 | 6 ++++++
>> docs/man/xl.pod.1 | 1 -
>> tools/libxl/libxl.c | 6 ++++++
>> tools/libxl/libxl_create.c | 14 ++++++++++++--
>> tools/libxl/libxl_types.idl | 1 +
>> tools/libxl/xl.c | 3 +++
>> tools/libxl/xl.h | 1 +
>> tools/libxl/xl_cmdimpl.c | 47 ++++++++++++++++++++++++++++++++++-----------
>> 8 files changed, 65 insertions(+), 14 deletions(-)
>>
>> diff --git a/docs/man/xl.conf.pod.5 b/docs/man/xl.conf.pod.5
>> index 8ae19bb..8f7fd28 100644
>> --- a/docs/man/xl.conf.pod.5
>> +++ b/docs/man/xl.conf.pod.5
>> @@ -111,6 +111,12 @@ Configures the default script used by Remus to setup network buffering.
>>
>> Default: C</etc/xen/scripts/remus-netbuf-setup>
>>
>> +=item B<colo.default.proxyscript="PATH">
>> +
>> +Configures the default script used by COLO to setup colo-proxy.
>> +
>> +Default: C</etc/xen/scripts/colo-proxy-setup>
>> +
>> =item B<output_format="json|sxp">
>>
>> Configures the default output format used by xl when printing "machine
>> diff --git a/docs/man/xl.pod.1 b/docs/man/xl.pod.1
>> index 4f1901d..edeafcf 100644
>> --- a/docs/man/xl.pod.1
>> +++ b/docs/man/xl.pod.1
>> @@ -454,7 +454,6 @@ N.B: Remus support in xl is still in experimental (proof-of-concept) phase.
>> Disk replication support is limited to DRBD disks.
>>
>> COLO support in xl is still in experimental (proof-of-concept) phase.
>> - There is no support for network at the moment.
>
>
> Same here, missing documentation on how to use the new parameters (if
> any). Please provide adequate documentation otherwise we can't
> meaningfully review the rest of this patch.
OK, will fix it in the next version.
Thanks
Wen Congyang
>
> Wei.
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module
2016-02-22 2:52 ` [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module Wen Congyang
2016-03-02 15:04 ` Wei Liu
@ 2016-03-11 22:25 ` Konrad Rzeszutek Wilk
2016-03-14 9:13 ` Wen Congyang
1 sibling, 1 reply; 70+ messages in thread
From: Konrad Rzeszutek Wilk @ 2016-03-11 22:25 UTC (permalink / raw)
To: Wen Congyang
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Gui Jianfeng,
Shriram Rajagopalan, Ian Jackson, Yang Hongyang
> +extern int colo_proxy_setup(libxl__colo_proxy_state *cps);
> +extern void colo_proxy_teardown(libxl__colo_proxy_state *cps);
> #endif
> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
> new file mode 100644
> index 0000000..e07e640
> --- /dev/null
> +++ b/tools/libxl/libxl_colo_proxy.c
> @@ -0,0 +1,230 @@
> +/*
> + * Copyright (C) 2015 FUJITSU LIMITED
2016?
> + * Author: Yang Hongyang <hongyang.yang@easystack.cn>
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU Lesser General Public License as published
> + * by the Free Software Foundation; version 2.1 only. with the special
> + * exception on linking described in file LICENSE.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> + * GNU Lesser General Public License for more details.
> + */
> +
> +#include "libxl_osdeps.h" /* must come before any other headers */
> +
> +#include "libxl_internal.h"
> +#include "libxl_colo.h"
> +#include <linux/netlink.h>
> +
> +#define NETLINK_COLO 28
Can you include a comment why 28? Why not 31415?
> +
> +enum colo_netlink_op {
> + COLO_QUERY_CHECKPOINT = (NLMSG_MIN_TYPE + 1),
> + COLO_CHECKPOINT,
> + COLO_FAILOVER,
> + COLO_PROXY_INIT,
> + COLO_PROXY_RESET, /* UNUSED, will be used for continuous FT */
> +};
> +
> +/* ========= colo-proxy: helper functions ========== */
> +
> +static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
> + uint64_t size, int type)
> +{
> + struct sockaddr_nl sa;
> + struct nlmsghdr msg;
> + struct iovec iov;
> + struct msghdr mh;
> + int ret;
> +
> + STATE_AO_GC(cps->ao);
> +
> + memset(&sa, 0, sizeof(sa));
> + sa.nl_family = AF_NETLINK;
> + sa.nl_pid = 0;
> + sa.nl_groups = 0;
> +
> + msg.nlmsg_len = NLMSG_SPACE(0);
> + msg.nlmsg_flags = NLM_F_REQUEST;
> + if (type == COLO_PROXY_INIT) {
> + msg.nlmsg_flags |= NLM_F_ACK;
> + }
I don't think you need the { }?
Ah, yup:
5. Block structure
Every indented statement is braced apart from blocks that contain just
one statement.
> + msg.nlmsg_seq = 0;
> + /* This is untrusty */
Umm, can you be more specific pls?
> + msg.nlmsg_pid = cps->index;
> + msg.nlmsg_type = type;
> +
> + iov.iov_base = &msg;
> + iov.iov_len = msg.nlmsg_len;
> +
> + mh.msg_name = &sa;
> + mh.msg_namelen = sizeof(sa);
> + mh.msg_iov = &iov;
> + mh.msg_iovlen = 1;
> + mh.msg_control = NULL;
> + mh.msg_controllen = 0;
> + mh.msg_flags = 0;
> +
> + ret = sendmsg(cps->sock_fd, &mh, 0);
> + if (ret <= 0) {
> + LOG(ERROR, "can't send msg to kernel by netlink: %s",
> + strerror(errno));
> + }
> +
> + return ret;
> +}
> +
> +/* error: return -1, otherwise return 0 */
> +static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
> + unsigned int timeout_us)
> +{
> + struct sockaddr_nl sa;
> + struct iovec iov;
> + struct msghdr mh = {
> + .msg_name = &sa,
> + .msg_namelen = sizeof(sa),
> + .msg_iov = &iov,
> + .msg_iovlen = 1,
> + };
> + struct timeval tv;
> + uint32_t size = 16384;
> + int64_t len = 0;
> + int ret;
> +
> + STATE_AO_GC(cps->ao);
> + uint8_t *tmp = libxl__malloc(NOGC, size);
> +
> + if (timeout_us) {
> + tv.tv_sec = timeout_us / 1000000;
> + tv.tv_usec = timeout_us % 1000000;
> + setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
> + }
> +
> + iov.iov_base = tmp;
> + iov.iov_len = size;
> +next:
> + ret = recvmsg(cps->sock_fd, &mh, 0);
> + if (ret <= 0) {
> + if (errno != EAGAIN && errno != EWOULDBLOCK)
-EINTR ?
> + LOGE(ERROR, "can't recv msg from kernel by netlink");
> + goto err;
> + }
> +
> + len += ret;
> + if (mh.msg_flags & MSG_TRUNC) {
> + size += 16384;
> + tmp = libxl__realloc(NOGC, tmp, size);
You really should check 'tmp'.
If this loop continues on for some time the 'size' may be
in milions and this realloc will fail.
> + iov.iov_base = tmp + len;
> + iov.iov_len = size - len;
> + goto next;
> + }
> +
> + *buff = tmp;
> + ret = len;
> + goto out;
> +
> +err:
> + free(tmp);
> + *buff = NULL;
> +
> +out:
> + if (timeout_us) {
> + tv.tv_sec = 0;
> + tv.tv_usec = 0;
> + setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
> + }
> + return ret;
> +}
> +
> +/* ========= colo-proxy: setup and teardown ========== */
> +
> +int colo_proxy_setup(libxl__colo_proxy_state *cps)
> +{
> + int skfd = 0;
> + struct sockaddr_nl sa;
> + struct nlmsghdr *h;
> + int i = 1;
> + int ret = ERROR_FAIL;
> + uint8_t *buff = NULL;
> + int64_t size;
> +
> + STATE_AO_GC(cps->ao);
> +
> + skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
> + if (skfd < 0) {
> + LOG(ERROR, "can not create a netlink socket: %s", strerror(errno));
> + goto out;
> + }
> + cps->sock_fd = skfd;
> + memset(&sa, 0, sizeof(sa));
> + sa.nl_family = AF_NETLINK;
> + sa.nl_groups = 0;
> +retry:
> + sa.nl_pid = i++;
> +
> + if (i > 10) {
> + LOG(ERROR, "netlink bind error");
> + goto out;
> + }
> +
> + ret = bind(skfd, (struct sockaddr *)&sa, sizeof(sa));
> + if (ret < 0 && errno == EADDRINUSE) {
> + LOG(ERROR, "colo index %d has already in used", sa.nl_pid);
> + goto retry;
> + } else if (ret < 0) {
> + LOG(ERROR, "netlink bind error");
> + goto out;
> + }
> +
> + cps->index = sa.nl_pid;
> + ret = colo_proxy_send(cps, NULL, 0, COLO_PROXY_INIT);
> + if (ret < 0) {
> + goto out;
> + }
Ditto. You can remove it.
> + /* receive ack */
> + size = colo_proxy_recv(cps, &buff, 500000);
> + if (size < 0) {
> + LOG(ERROR, "Can't recv msg from kernel by netlink: %s",
> + strerror(errno));
> + goto out;
> + }
> +
> + if (size) {
> + h = (struct nlmsghdr *)buff;
> + if (h->nlmsg_type == NLMSG_ERROR) {
> + /* ack's type is NLMSG_ERROR */
> + struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(h);
> +
> + if (size - sizeof(*h) < sizeof(*err)) {
> + LOG(ERROR, "NLMSG_LENGTH is too short");
> + goto out;
> + }
> +
> + if (err->error) {
> + LOG(ERROR, "NLMSG_ERROR contains error %d", err->error);
> + goto out;
> + }
> + }
> + }
> +
> + ret = 0;
> +
> +out:
> + free(buff);
> + if (ret) {
> + close(cps->sock_fd);
> + cps->sock_fd = -1;
> + }
> + return ret;
> +}
> +
> +void colo_proxy_teardown(libxl__colo_proxy_state *cps)
> +{
> + if (cps->sock_fd >= 0) {
> + close(cps->sock_fd);
> + cps->sock_fd = -1;
> + }
> +}
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 3af5fdd..3b44b09 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -3112,6 +3112,15 @@ libxl__stream_read_inuse(const libxl__stream_read_state *stream)
> }
>
> /*----- colo related state structure -----*/
> +typedef struct libxl__colo_proxy_state libxl__colo_proxy_state;
> +struct libxl__colo_proxy_state {
> + /* set by caller of colo_proxy_setup */
> + libxl__ao *ao;
> +
> + int sock_fd;
> + int index;
> +};
> +
> typedef struct libxl__colo_save_state libxl__colo_save_state;
> struct libxl__colo_save_state {
> int send_fd;
> @@ -3126,6 +3135,9 @@ struct libxl__colo_save_state {
> /* private, used by qdisk block replication */
> bool qdisk_used;
> bool qdisk_setuped;
> +
> + /* private, used by colo-proxy */
> + libxl__colo_proxy_state cps;
> };
>
> /*----- Domain suspend (save) state structure -----*/
> @@ -3535,6 +3547,9 @@ struct libxl__colo_restore_state {
> bool qdisk_setuped;
> const char *host;
> const char *port;
> +
> + /* private, used by colo-proxy */
> + libxl__colo_proxy_state cps;
> };
>
> struct libxl__domain_create_state {
> --
> 2.5.0
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module
2016-03-11 22:25 ` Konrad Rzeszutek Wilk
@ 2016-03-14 9:13 ` Wen Congyang
2016-03-22 3:40 ` Changlong Xie
0 siblings, 1 reply; 70+ messages in thread
From: Wen Congyang @ 2016-03-14 9:13 UTC (permalink / raw)
To: Konrad Rzeszutek Wilk
Cc: Lars Kurth, Changlong Xie, Wei Liu, Ian Campbell, Andrew Cooper,
Jiang Yunhong, Dong Eddie, xen devel, Gui Jianfeng,
Shriram Rajagopalan, Ian Jackson, Yang Hongyang
On 03/12/2016 06:25 AM, Konrad Rzeszutek Wilk wrote:
>> +extern int colo_proxy_setup(libxl__colo_proxy_state *cps);
>> +extern void colo_proxy_teardown(libxl__colo_proxy_state *cps);
>> #endif
>> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
>> new file mode 100644
>> index 0000000..e07e640
>> --- /dev/null
>> +++ b/tools/libxl/libxl_colo_proxy.c
>> @@ -0,0 +1,230 @@
>> +/*
>> + * Copyright (C) 2015 FUJITSU LIMITED
>
> 2016?
Yes, I will check all new files.
>> + * Author: Yang Hongyang <hongyang.yang@easystack.cn>
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU Lesser General Public License as published
>> + * by the Free Software Foundation; version 2.1 only. with the special
>> + * exception on linking described in file LICENSE.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> + * GNU Lesser General Public License for more details.
>> + */
>> +
>> +#include "libxl_osdeps.h" /* must come before any other headers */
>> +
>> +#include "libxl_internal.h"
>> +#include "libxl_colo.h"
>> +#include <linux/netlink.h>
>> +
>> +#define NETLINK_COLO 28
>
> Can you include a comment why 28? Why not 31415?
OK, will add a comment to describe it.
>
>> +
>> +enum colo_netlink_op {
>> + COLO_QUERY_CHECKPOINT = (NLMSG_MIN_TYPE + 1),
>> + COLO_CHECKPOINT,
>> + COLO_FAILOVER,
>> + COLO_PROXY_INIT,
>> + COLO_PROXY_RESET, /* UNUSED, will be used for continuous FT */
>> +};
>> +
>> +/* ========= colo-proxy: helper functions ========== */
>> +
>> +static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
>> + uint64_t size, int type)
>> +{
>> + struct sockaddr_nl sa;
>> + struct nlmsghdr msg;
>> + struct iovec iov;
>> + struct msghdr mh;
>> + int ret;
>> +
>> + STATE_AO_GC(cps->ao);
>> +
>> + memset(&sa, 0, sizeof(sa));
>> + sa.nl_family = AF_NETLINK;
>> + sa.nl_pid = 0;
>> + sa.nl_groups = 0;
>> +
>> + msg.nlmsg_len = NLMSG_SPACE(0);
>> + msg.nlmsg_flags = NLM_F_REQUEST;
>> + if (type == COLO_PROXY_INIT) {
>> + msg.nlmsg_flags |= NLM_F_ACK;
>> + }
>
> I don't think you need the { }?
Yes, will fix it in the next version.
>
> Ah, yup:
>
> 5. Block structure
>
> Every indented statement is braced apart from blocks that contain just
> one statement.
>
>> + msg.nlmsg_seq = 0;
>> + /* This is untrusty */
>
> Umm, can you be more specific pls?
>
>> + msg.nlmsg_pid = cps->index;
>> + msg.nlmsg_type = type;
>> +
>> + iov.iov_base = &msg;
>> + iov.iov_len = msg.nlmsg_len;
>> +
>> + mh.msg_name = &sa;
>> + mh.msg_namelen = sizeof(sa);
>> + mh.msg_iov = &iov;
>> + mh.msg_iovlen = 1;
>> + mh.msg_control = NULL;
>> + mh.msg_controllen = 0;
>> + mh.msg_flags = 0;
>> +
>> + ret = sendmsg(cps->sock_fd, &mh, 0);
>> + if (ret <= 0) {
>> + LOG(ERROR, "can't send msg to kernel by netlink: %s",
>> + strerror(errno));
>> + }
>> +
>> + return ret;
>> +}
>> +
>> +/* error: return -1, otherwise return 0 */
>> +static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
>> + unsigned int timeout_us)
>> +{
>> + struct sockaddr_nl sa;
>> + struct iovec iov;
>> + struct msghdr mh = {
>> + .msg_name = &sa,
>> + .msg_namelen = sizeof(sa),
>> + .msg_iov = &iov,
>> + .msg_iovlen = 1,
>> + };
>> + struct timeval tv;
>> + uint32_t size = 16384;
>> + int64_t len = 0;
>> + int ret;
>> +
>> + STATE_AO_GC(cps->ao);
>> + uint8_t *tmp = libxl__malloc(NOGC, size);
>> +
>> + if (timeout_us) {
>> + tv.tv_sec = timeout_us / 1000000;
>> + tv.tv_usec = timeout_us % 1000000;
>> + setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
>> + }
>> +
>> + iov.iov_base = tmp;
>> + iov.iov_len = size;
>> +next:
>> + ret = recvmsg(cps->sock_fd, &mh, 0);
>> + if (ret <= 0) {
>> + if (errno != EAGAIN && errno != EWOULDBLOCK)
>
> -EINTR ?
IIRC, WAGAIN and EWOULDBLOCK may have different value in some system.
EINTR is not handled here.
>
>> + LOGE(ERROR, "can't recv msg from kernel by netlink");
>> + goto err;
>> + }
>> +
>> + len += ret;
>> + if (mh.msg_flags & MSG_TRUNC) {
>> + size += 16384;
>> + tmp = libxl__realloc(NOGC, tmp, size);
>
> You really should check 'tmp'.
>
> If this loop continues on for some time the 'size' may be
> in milions and this realloc will fail.
OK, will fix it in the next version.
>
>> + iov.iov_base = tmp + len;
>> + iov.iov_len = size - len;
>> + goto next;
>
>> + }
>> +
>> + *buff = tmp;
>> + ret = len;
>> + goto out;
>> +
>> +err:
>> + free(tmp);
>> + *buff = NULL;
>> +
>> +out:
>> + if (timeout_us) {
>> + tv.tv_sec = 0;
>> + tv.tv_usec = 0;
>> + setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
>> + }
>> + return ret;
>> +}
>> +
>> +/* ========= colo-proxy: setup and teardown ========== */
>> +
>> +int colo_proxy_setup(libxl__colo_proxy_state *cps)
>> +{
>> + int skfd = 0;
>> + struct sockaddr_nl sa;
>> + struct nlmsghdr *h;
>> + int i = 1;
>> + int ret = ERROR_FAIL;
>> + uint8_t *buff = NULL;
>> + int64_t size;
>> +
>> + STATE_AO_GC(cps->ao);
>> +
>> + skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
>> + if (skfd < 0) {
>> + LOG(ERROR, "can not create a netlink socket: %s", strerror(errno));
>> + goto out;
>> + }
>> + cps->sock_fd = skfd;
>> + memset(&sa, 0, sizeof(sa));
>> + sa.nl_family = AF_NETLINK;
>> + sa.nl_groups = 0;
>> +retry:
>> + sa.nl_pid = i++;
>> +
>> + if (i > 10) {
>> + LOG(ERROR, "netlink bind error");
>> + goto out;
>> + }
>> +
>> + ret = bind(skfd, (struct sockaddr *)&sa, sizeof(sa));
>> + if (ret < 0 && errno == EADDRINUSE) {
>> + LOG(ERROR, "colo index %d has already in used", sa.nl_pid);
>> + goto retry;
>> + } else if (ret < 0) {
>> + LOG(ERROR, "netlink bind error");
>> + goto out;
>> + }
>> +
>> + cps->index = sa.nl_pid;
>> + ret = colo_proxy_send(cps, NULL, 0, COLO_PROXY_INIT);
>> + if (ret < 0) {
>> + goto out;
>> + }
>
> Ditto. You can remove it.
OK, will check all codes.
Thanks
Wen Congyang
>
>> + /* receive ack */
>> + size = colo_proxy_recv(cps, &buff, 500000);
>> + if (size < 0) {
>> + LOG(ERROR, "Can't recv msg from kernel by netlink: %s",
>> + strerror(errno));
>> + goto out;
>> + }
>> +
>> + if (size) {
>> + h = (struct nlmsghdr *)buff;
>> + if (h->nlmsg_type == NLMSG_ERROR) {
>> + /* ack's type is NLMSG_ERROR */
>> + struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(h);
>> +
>> + if (size - sizeof(*h) < sizeof(*err)) {
>> + LOG(ERROR, "NLMSG_LENGTH is too short");
>> + goto out;
>> + }
>> +
>> + if (err->error) {
>> + LOG(ERROR, "NLMSG_ERROR contains error %d", err->error);
>> + goto out;
>> + }
>> + }
>> + }
>> +
>> + ret = 0;
>> +
>> +out:
>> + free(buff);
>> + if (ret) {
>> + close(cps->sock_fd);
>> + cps->sock_fd = -1;
>> + }
>> + return ret;
>> +}
>> +
>> +void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>> +{
>> + if (cps->sock_fd >= 0) {
>> + close(cps->sock_fd);
>> + cps->sock_fd = -1;
>> + }
>> +}
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 3af5fdd..3b44b09 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -3112,6 +3112,15 @@ libxl__stream_read_inuse(const libxl__stream_read_state *stream)
>> }
>>
>> /*----- colo related state structure -----*/
>> +typedef struct libxl__colo_proxy_state libxl__colo_proxy_state;
>> +struct libxl__colo_proxy_state {
>> + /* set by caller of colo_proxy_setup */
>> + libxl__ao *ao;
>> +
>> + int sock_fd;
>> + int index;
>> +};
>> +
>> typedef struct libxl__colo_save_state libxl__colo_save_state;
>> struct libxl__colo_save_state {
>> int send_fd;
>> @@ -3126,6 +3135,9 @@ struct libxl__colo_save_state {
>> /* private, used by qdisk block replication */
>> bool qdisk_used;
>> bool qdisk_setuped;
>> +
>> + /* private, used by colo-proxy */
>> + libxl__colo_proxy_state cps;
>> };
>>
>> /*----- Domain suspend (save) state structure -----*/
>> @@ -3535,6 +3547,9 @@ struct libxl__colo_restore_state {
>> bool qdisk_setuped;
>> const char *host;
>> const char *port;
>> +
>> + /* private, used by colo-proxy */
>> + libxl__colo_proxy_state cps;
>> };
>>
>> struct libxl__domain_create_state {
>> --
>> 2.5.0
>>
>>
>>
>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
* Re: [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module
2016-03-14 9:13 ` Wen Congyang
@ 2016-03-22 3:40 ` Changlong Xie
0 siblings, 0 replies; 70+ messages in thread
From: Changlong Xie @ 2016-03-22 3:40 UTC (permalink / raw)
To: Wen Congyang, Konrad Rzeszutek Wilk
Cc: Lars Kurth, Wei Liu, Ian Campbell, Andrew Cooper, Jiang Yunhong,
Dong Eddie, xen devel, Gui Jianfeng, Shriram Rajagopalan,
Ian Jackson, Yang Hongyang
On 03/14/2016 05:13 PM, Wen Congyang wrote:
> On 03/12/2016 06:25 AM, Konrad Rzeszutek Wilk wrote:
>>> +extern int colo_proxy_setup(libxl__colo_proxy_state *cps);
>>> +extern void colo_proxy_teardown(libxl__colo_proxy_state *cps);
>>> #endif
>>> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
>>> new file mode 100644
>>> index 0000000..e07e640
>>> --- /dev/null
>>> +++ b/tools/libxl/libxl_colo_proxy.c
>>> @@ -0,0 +1,230 @@
>>> +/*
>>> + * Copyright (C) 2015 FUJITSU LIMITED
>>
>> 2016?
>
> Yes, I will check all new files.
>
>>> + * Author: Yang Hongyang <hongyang.yang@easystack.cn>
>>> + *
>>> + * This program is free software; you can redistribute it and/or modify
>>> + * it under the terms of the GNU Lesser General Public License as published
>>> + * by the Free Software Foundation; version 2.1 only. with the special
>>> + * exception on linking described in file LICENSE.
>>> + *
>>> + * This program is distributed in the hope that it will be useful,
>>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>>> + * GNU Lesser General Public License for more details.
>>> + */
>>> +
>>> +#include "libxl_osdeps.h" /* must come before any other headers */
>>> +
>>> +#include "libxl_internal.h"
>>> +#include "libxl_colo.h"
>>> +#include <linux/netlink.h>
>>> +
>>> +#define NETLINK_COLO 28
>>
>> Can you include a comment why 28? Why not 31415?
>
> OK, will add a comment to describe it.
>
>>
>>> +
>>> +enum colo_netlink_op {
>>> + COLO_QUERY_CHECKPOINT = (NLMSG_MIN_TYPE + 1),
>>> + COLO_CHECKPOINT,
>>> + COLO_FAILOVER,
>>> + COLO_PROXY_INIT,
>>> + COLO_PROXY_RESET, /* UNUSED, will be used for continuous FT */
>>> +};
>>> +
>>> +/* ========= colo-proxy: helper functions ========== */
>>> +
>>> +static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
>>> + uint64_t size, int type)
>>> +{
>>> + struct sockaddr_nl sa;
>>> + struct nlmsghdr msg;
>>> + struct iovec iov;
>>> + struct msghdr mh;
>>> + int ret;
>>> +
>>> + STATE_AO_GC(cps->ao);
>>> +
>>> + memset(&sa, 0, sizeof(sa));
>>> + sa.nl_family = AF_NETLINK;
>>> + sa.nl_pid = 0;
>>> + sa.nl_groups = 0;
>>> +
>>> + msg.nlmsg_len = NLMSG_SPACE(0);
>>> + msg.nlmsg_flags = NLM_F_REQUEST;
>>> + if (type == COLO_PROXY_INIT) {
>>> + msg.nlmsg_flags |= NLM_F_ACK;
>>> + }
>>
>> I don't think you need the { }?
>
> Yes, will fix it in the next version.
>
>>
>> Ah, yup:
>>
>> 5. Block structure
>>
>> Every indented statement is braced apart from blocks that contain just
>> one statement.
>>
>>> + msg.nlmsg_seq = 0;
>>> + /* This is untrusty */
>>
>> Umm, can you be more specific pls?
>>
>>> + msg.nlmsg_pid = cps->index;
>>> + msg.nlmsg_type = type;
>>> +
>>> + iov.iov_base = &msg;
>>> + iov.iov_len = msg.nlmsg_len;
>>> +
>>> + mh.msg_name = &sa;
>>> + mh.msg_namelen = sizeof(sa);
>>> + mh.msg_iov = &iov;
>>> + mh.msg_iovlen = 1;
>>> + mh.msg_control = NULL;
>>> + mh.msg_controllen = 0;
>>> + mh.msg_flags = 0;
>>> +
>>> + ret = sendmsg(cps->sock_fd, &mh, 0);
>>> + if (ret <= 0) {
>>> + LOG(ERROR, "can't send msg to kernel by netlink: %s",
>>> + strerror(errno));
>>> + }
>>> +
>>> + return ret;
>>> +}
>>> +
>>> +/* error: return -1, otherwise return 0 */
>>> +static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
>>> + unsigned int timeout_us)
>>> +{
>>> + struct sockaddr_nl sa;
>>> + struct iovec iov;
>>> + struct msghdr mh = {
>>> + .msg_name = &sa,
>>> + .msg_namelen = sizeof(sa),
>>> + .msg_iov = &iov,
>>> + .msg_iovlen = 1,
>>> + };
>>> + struct timeval tv;
>>> + uint32_t size = 16384;
>>> + int64_t len = 0;
>>> + int ret;
>>> +
>>> + STATE_AO_GC(cps->ao);
>>> + uint8_t *tmp = libxl__malloc(NOGC, size);
>>> +
>>> + if (timeout_us) {
>>> + tv.tv_sec = timeout_us / 1000000;
>>> + tv.tv_usec = timeout_us % 1000000;
>>> + setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
>>> + }
>>> +
>>> + iov.iov_base = tmp;
>>> + iov.iov_len = size;
>>> +next:
>>> + ret = recvmsg(cps->sock_fd, &mh, 0);
>>> + if (ret <= 0) {
>>> + if (errno != EAGAIN && errno != EWOULDBLOCK)
>>
>> -EINTR ?
>
> IIRC, WAGAIN and EWOULDBLOCK may have different value in some system.
> EINTR is not handled here.
>
>>
>>> + LOGE(ERROR, "can't recv msg from kernel by netlink");
>>> + goto err;
>>> + }
>>> +
>>> + len += ret;
>>> + if (mh.msg_flags & MSG_TRUNC) {
>>> + size += 16384;
>>> + tmp = libxl__realloc(NOGC, tmp, size);
>>
>> You really should check 'tmp'.
>>
>> If this loop continues on for some time the 'size' may be
>> in milions and this realloc will fail.
As i see from the code, libxl_realloc will invoke _exit(-1) if malloc
failed, so we'll keep it.
Thanks
-Xie
>
> OK, will fix it in the next version.
>
>>
>>> + iov.iov_base = tmp + len;
>>> + iov.iov_len = size - len;
>>> + goto next;
>>
>>> + }
>>> +
>>> + *buff = tmp;
>>> + ret = len;
>>> + goto out;
>>> +
>>> +err:
>>> + free(tmp);
>>> + *buff = NULL;
>>> +
>>> +out:
>>> + if (timeout_us) {
>>> + tv.tv_sec = 0;
>>> + tv.tv_usec = 0;
>>> + setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
>>> + }
>>> + return ret;
>>> +}
>>> +
>>> +/* ========= colo-proxy: setup and teardown ========== */
>>> +
>>> +int colo_proxy_setup(libxl__colo_proxy_state *cps)
>>> +{
>>> + int skfd = 0;
>>> + struct sockaddr_nl sa;
>>> + struct nlmsghdr *h;
>>> + int i = 1;
>>> + int ret = ERROR_FAIL;
>>> + uint8_t *buff = NULL;
>>> + int64_t size;
>>> +
>>> + STATE_AO_GC(cps->ao);
>>> +
>>> + skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
>>> + if (skfd < 0) {
>>> + LOG(ERROR, "can not create a netlink socket: %s", strerror(errno));
>>> + goto out;
>>> + }
>>> + cps->sock_fd = skfd;
>>> + memset(&sa, 0, sizeof(sa));
>>> + sa.nl_family = AF_NETLINK;
>>> + sa.nl_groups = 0;
>>> +retry:
>>> + sa.nl_pid = i++;
>>> +
>>> + if (i > 10) {
>>> + LOG(ERROR, "netlink bind error");
>>> + goto out;
>>> + }
>>> +
>>> + ret = bind(skfd, (struct sockaddr *)&sa, sizeof(sa));
>>> + if (ret < 0 && errno == EADDRINUSE) {
>>> + LOG(ERROR, "colo index %d has already in used", sa.nl_pid);
>>> + goto retry;
>>> + } else if (ret < 0) {
>>> + LOG(ERROR, "netlink bind error");
>>> + goto out;
>>> + }
>>> +
>>> + cps->index = sa.nl_pid;
>>> + ret = colo_proxy_send(cps, NULL, 0, COLO_PROXY_INIT);
>>> + if (ret < 0) {
>>> + goto out;
>>> + }
>>
>> Ditto. You can remove it.
>
> OK, will check all codes.
>
> Thanks
> Wen Congyang
>
>>
>>> + /* receive ack */
>>> + size = colo_proxy_recv(cps, &buff, 500000);
>>> + if (size < 0) {
>>> + LOG(ERROR, "Can't recv msg from kernel by netlink: %s",
>>> + strerror(errno));
>>> + goto out;
>>> + }
>>> +
>>> + if (size) {
>>> + h = (struct nlmsghdr *)buff;
>>> + if (h->nlmsg_type == NLMSG_ERROR) {
>>> + /* ack's type is NLMSG_ERROR */
>>> + struct nlmsgerr *err = (struct nlmsgerr *)NLMSG_DATA(h);
>>> +
>>> + if (size - sizeof(*h) < sizeof(*err)) {
>>> + LOG(ERROR, "NLMSG_LENGTH is too short");
>>> + goto out;
>>> + }
>>> +
>>> + if (err->error) {
>>> + LOG(ERROR, "NLMSG_ERROR contains error %d", err->error);
>>> + goto out;
>>> + }
>>> + }
>>> + }
>>> +
>>> + ret = 0;
>>> +
>>> +out:
>>> + free(buff);
>>> + if (ret) {
>>> + close(cps->sock_fd);
>>> + cps->sock_fd = -1;
>>> + }
>>> + return ret;
>>> +}
>>> +
>>> +void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>>> +{
>>> + if (cps->sock_fd >= 0) {
>>> + close(cps->sock_fd);
>>> + cps->sock_fd = -1;
>>> + }
>>> +}
>>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>>> index 3af5fdd..3b44b09 100644
>>> --- a/tools/libxl/libxl_internal.h
>>> +++ b/tools/libxl/libxl_internal.h
>>> @@ -3112,6 +3112,15 @@ libxl__stream_read_inuse(const libxl__stream_read_state *stream)
>>> }
>>>
>>> /*----- colo related state structure -----*/
>>> +typedef struct libxl__colo_proxy_state libxl__colo_proxy_state;
>>> +struct libxl__colo_proxy_state {
>>> + /* set by caller of colo_proxy_setup */
>>> + libxl__ao *ao;
>>> +
>>> + int sock_fd;
>>> + int index;
>>> +};
>>> +
>>> typedef struct libxl__colo_save_state libxl__colo_save_state;
>>> struct libxl__colo_save_state {
>>> int send_fd;
>>> @@ -3126,6 +3135,9 @@ struct libxl__colo_save_state {
>>> /* private, used by qdisk block replication */
>>> bool qdisk_used;
>>> bool qdisk_setuped;
>>> +
>>> + /* private, used by colo-proxy */
>>> + libxl__colo_proxy_state cps;
>>> };
>>>
>>> /*----- Domain suspend (save) state structure -----*/
>>> @@ -3535,6 +3547,9 @@ struct libxl__colo_restore_state {
>>> bool qdisk_setuped;
>>> const char *host;
>>> const char *port;
>>> +
>>> + /* private, used by colo-proxy */
>>> + libxl__colo_proxy_state cps;
>>> };
>>>
>>> struct libxl__domain_create_state {
>>> --
>>> 2.5.0
>>>
>>>
>>>
>>
>>
>> .
>>
>
> .
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
http://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 70+ messages in thread
end of thread, other threads:[~2016-03-22 3:40 UTC | newest]
Thread overview: 70+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2016-02-22 2:52 [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wen Congyang
2016-02-22 2:52 ` [PATCH v10 01/31] tools/libxl: introduce libxl__domain_restore_device_model to load qemu state Wen Congyang
2016-02-25 15:53 ` Wei Liu
2016-02-26 1:55 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 02/31] tools/libxl: introduce libxl__domain_common_switch_qemu_logdirty() Wen Congyang
2016-02-22 2:52 ` [PATCH v10 03/31] tools/libxl: Add back channel to allow migration target send data back Wen Congyang
2016-02-22 2:52 ` [PATCH v10 04/31] tools/libxl: Introduce new helper function dup_fd_helper() Wen Congyang
2016-02-25 15:53 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 05/31] tools/libx{l, c}: add back channel to libxc Wen Congyang
2016-02-25 15:54 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 06/31] docs: add colo readme Wen Congyang
2016-02-25 15:54 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 07/31] docs/libxl: Introduce CHECKPOINT_CONTEXT to support migration v2 colo streams Wen Congyang
2016-02-25 15:54 ` Wei Liu
2016-02-26 1:59 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 08/31] libxc/migration: Specification update for DIRTY_PFN_LIST records Wen Congyang
2016-02-22 2:52 ` [PATCH v10 09/31] libxc/migration: export read_record for common use Wen Congyang
2016-02-22 2:52 ` [PATCH v10 10/31] tools/libxl: add back channel support to write stream Wen Congyang
2016-02-25 15:54 ` Wei Liu
2016-02-26 2:11 ` Wen Congyang
2016-03-02 15:02 ` Wei Liu
2016-03-03 1:25 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 11/31] tools/libxl: write checkpoint_state records into the stream Wen Congyang
2016-02-22 2:52 ` [PATCH v10 12/31] tools/libxl: add back channel support to read stream Wen Congyang
2016-02-25 15:54 ` Wei Liu
2016-02-26 2:16 ` Wen Congyang
2016-03-02 15:03 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 13/31] tools/libxl: handle checkpoint_state records in a libxl migration v2 " Wen Congyang
2016-02-22 2:52 ` [PATCH v10 14/31] tools/libx{l, c}: introduce wait_checkpoint callback Wen Congyang
2016-02-22 2:52 ` [PATCH v10 15/31] tools/libx{l, c}: add postcopy/suspend callback to restore side Wen Congyang
2016-02-22 2:52 ` [PATCH v10 16/31] secondary vm suspend/resume/checkpoint code Wen Congyang
2016-02-25 15:56 ` Wei Liu
2016-02-26 2:30 ` Wen Congyang
2016-03-01 10:06 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 17/31] primary " Wen Congyang
2016-02-25 15:57 ` Wei Liu
2016-02-26 2:32 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 18/31] libxc/restore: support COLO restore Wen Congyang
2016-02-25 15:57 ` Wei Liu
2016-02-26 2:33 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 19/31] libxc/restore: send dirty pfn list to primary when checkpoint under colo Wen Congyang
2016-02-22 2:52 ` [PATCH v10 20/31] send store gfn and console gfn to xl before resuming secondary vm Wen Congyang
2016-02-22 2:52 ` [PATCH v10 21/31] libxc/save: support COLO save Wen Congyang
2016-02-25 15:58 ` Wei Liu
2016-02-26 2:35 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 22/31] implement the cmdline for COLO Wen Congyang
2016-03-02 15:03 ` Wei Liu
2016-03-03 1:30 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 23/31] COLO: introduce new API to prepare/start/do/get_error/stop replication Wen Congyang
2016-03-02 15:03 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 24/31] Support colo mode for qemu disk Wen Congyang
2016-03-02 15:04 ` Wei Liu
2016-03-03 1:40 ` Wen Congyang
2016-02-22 2:52 ` [PATCH v10 25/31] COLO: use qemu block replication Wen Congyang
2016-03-02 15:03 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 26/31] COLO proxy: implement setup/teardown of COLO proxy module Wen Congyang
2016-03-02 15:04 ` Wei Liu
2016-03-11 22:25 ` Konrad Rzeszutek Wilk
2016-03-14 9:13 ` Wen Congyang
2016-03-22 3:40 ` Changlong Xie
2016-02-22 2:52 ` [PATCH v10 27/31] COLO proxy: preresume, postresume and checkpoint Wen Congyang
2016-03-02 15:04 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 28/31] COLO nic: implement COLO nic subkind Wen Congyang
2016-03-02 15:04 ` Wei Liu
2016-02-22 2:52 ` [PATCH v10 29/31] setup and control colo proxy on primary side Wen Congyang
2016-02-22 2:52 ` [PATCH v10 30/31] setup and control colo proxy on secondary side Wen Congyang
2016-02-22 2:52 ` [PATCH v10 31/31] cmdline switches and config vars to control colo-proxy Wen Congyang
2016-03-02 15:05 ` Wei Liu
2016-03-03 1:41 ` Wen Congyang
2016-02-25 16:05 ` [PATCH v10 00/31] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Wei Liu
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).