* [PATCH V3 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy
@ 2017-02-17 2:18 Zhang Chen
2017-02-17 2:18 ` [PATCH V3 1/7] COLO-Proxy: Add remus command to open userspace proxy Zhang Chen
` (6 more replies)
0 siblings, 7 replies; 23+ messages in thread
From: Zhang Chen @ 2017-02-17 2:18 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Zhang Chen, Li Zhijian, eddie, Yang Hongyang, Bian Naimeng
Because of some reason, We no longer support COLO kernel proxy.
So we send this patch set to make Xen use userspace colo-proxy in qemu.
Below is a COLO userspace proxy ascii figure:
Primary qemu Secondary qemu
+--------------------------------------------------------------+ +----------------------------------------------------------------+
| +----------------------------------------------------------+ | | +-----------------------------------------------------------+ |
| | | | | | | |
| | guest | | | | guest | |
| | | | | | | |
| +-------^--------------------------+-----------------------+ | | +---------------------+--------+----------------------------+ |
| | | | | ^ | |
| | | | | | | |
| | +------------------------------------------------------+ | | | |
|netfilter| | | | | | netfilter | | |
| +----------+ +----------------------------+ | | | +-----------------------------------------------------------+ |
| | | | | | out | | | | | | filter excute order | |
| | | | +-----------------------------+ | | | | | | +-------------------> | |
| | | | | | | | | | | | | | TCP | |
| | +-----+--+-+ +-----v----+ +-----v----+ |pri +----+----+sec| | | | +------------+ +---+----+---v+rewriter++ +------------+ | |
| | | | | | | | |in | |in | | | | | | | | | | | | |
| | | filter | | filter | | filter +------> colo <------+ +--------> filter +--> adjust | adjust +--> filter | | |
| | | mirror | |redirector| |redirector| | | compare | | | | | | redirector | | ack | seq | | redirector | | |
| | | | | | | | | | | | | | | | | | | | | | | |
| | +----^-----+ +----+-----+ +----------+ | +---------+ | | | | +------------+ +--------+--------------+ +---+--------+ | |
| | | tx | rx rx | | | | | tx all | rx | |
| | | | | | | | +-----------------------------------------------------------+ |
| | | +--------------+ | | | | | |
| | | filter excute order | | | | | | |
| | | +----------------> | | | +--------------------------------------------------------+ |
| +-----------------------------------------+ | | |
| | | | | |
+--------------------------------------------------------------+ +----------------------------------------------------------------+
|guest receive | guest send
| |
+--------+----------------------------v------------------------+
| | NOTE: filter direction is rx/tx/all
| tap | rx:receive packets sent to the netdev
| | tx:receive packets sent by the netdev
+--------------------------------------------------------------+
You can know the detail from here:
http://wiki.qemu.org/Features/COLO
https://github.com/qemu/qemu/blob/master/docs/colo-proxy.txt
V3:
- remove the 'RFC' tag.
- fix some bug in patch 7/7.
- fix codestyle.
V2:
- Address wei's comments, use macro to reuse codes.
- Do some work on the last patch.
- Fix some typo.
- Add LIBXL_HAVE_COLO_USERSPACE_PROXY macro in libxl.h.
- Some bug fix.
V1:
- Initial patch
Zhang Chen (7):
COLO-Proxy: Add remus command to open userspace proxy
COLO-Proxy: Setup userspace colo-proxy on primary side
tools/libxl: refactor do_domain_create()
COLO-Proxy: Setup userspace colo-proxy on secondary side
COLO-Proxy: Add primary userspace colo proxy start args
COLO-Proxy: Add secondary userspace colo-proxy start args
COLO-Proxy: Use socket to get checkpoint event.
docs/man/xl.pod.1.in | 5 ++
tools/libxl/libxl.h | 6 ++
tools/libxl/libxl_colo.h | 7 +++
tools/libxl/libxl_colo_proxy.c | 100 +++++++++++++++++++++++++++++
tools/libxl/libxl_colo_restore.c | 19 ++++--
tools/libxl/libxl_colo_save.c | 19 +++++-
tools/libxl/libxl_create.c | 21 ++++---
tools/libxl/libxl_dm.c | 132 +++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_nic.c | 109 ++++++++++++++++++++++++++++++++
tools/libxl/libxl_types.idl | 64 ++++++++++++++++---
tools/libxl/xl_cmdimpl.c | 120 +++++++++++++++++++++++++++++++++--
tools/libxl/xl_cmdtable.c | 3 +-
12 files changed, 576 insertions(+), 29 deletions(-)
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* [PATCH V3 1/7] COLO-Proxy: Add remus command to open userspace proxy
2017-02-17 2:18 [PATCH V3 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
@ 2017-02-17 2:18 ` Zhang Chen
2017-02-20 15:50 ` Wei Liu
2017-02-17 2:18 ` [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side Zhang Chen
` (5 subsequent siblings)
6 siblings, 1 reply; 23+ messages in thread
From: Zhang Chen @ 2017-02-17 2:18 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Zhang Chen, Li Zhijian, eddie, Yang Hongyang, Bian Naimeng
Add remus '-p' to enable userspace colo proxy(in qemu).
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
docs/man/xl.pod.1.in | 5 +++++
tools/libxl/libxl.h | 6 ++++++
tools/libxl/libxl_colo.h | 5 +++++
tools/libxl/libxl_colo_save.c | 2 ++
tools/libxl/libxl_types.idl | 17 +++++++++--------
tools/libxl/xl_cmdimpl.c | 13 ++++++++++++-
tools/libxl/xl_cmdtable.c | 3 ++-
7 files changed, 41 insertions(+), 10 deletions(-)
diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
index 09c1faa..4260777 100644
--- a/docs/man/xl.pod.1.in
+++ b/docs/man/xl.pod.1.in
@@ -553,6 +553,11 @@ Disable disk replication. Requires enabling unsafe mode.
Enable COLO HA. This conflicts with B<-i> and B<-b>, and memory
checkpoint compression must be disabled.
+=item B<-p>
+
+Use userspace COLO Proxy. This option must be used in conjunction
+with B<-c>.
+
=back
=item B<pause> I<domain-id>
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 3924464..fce7fab 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -870,6 +870,12 @@ typedef struct libxl__ctx libxl_ctx;
*/
#define LIBXL_HAVE_REMUS 1
+/*
+ * LIBXL_HAVE_COLO_USERSPACE_PROXY
+ * If this is defined, then libxl supports COLO userspace proxy.
+ */
+#define LIBXL_HAVE_COLO_USERSPACE_PROXY 1
+
typedef uint8_t libxl_mac[6];
#define LIBXL_MAC_FMT "%02hhx:%02hhx:%02hhx:%02hhx:%02hhx:%02hhx"
#define LIBXL_MAC_FMTLEN ((2*6)+5) /* 6 hex bytes plus 5 colons */
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 682275c..4746d8c 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -64,6 +64,11 @@ struct libxl__colo_proxy_state {
int sock_fd;
int index;
+ /*
+ * Private, True means use userspace colo proxy
+ * False means use kernel colo proxy.
+ */
+ bool is_userspace_proxy;
};
struct libxl__colo_save_state {
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index 6277770..eb8336c 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -101,6 +101,8 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->qdisk_setuped = false;
css->qdisk_used = false;
libxl__ev_child_init(&css->child);
+ css->cps.is_userspace_proxy =
+ libxl_defbool_val(dss->remus->userspace_colo_proxy);
if (dss->remus->netbufscript)
css->colo_proxy_script = libxl__strdup(gc, dss->remus->netbufscript);
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index a612d1f..1bd2057 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -844,14 +844,15 @@ libxl_sched_credit2_params = Struct("sched_credit2_params", [
], dispose_fn=None)
libxl_domain_remus_info = Struct("domain_remus_info",[
- ("interval", integer),
- ("allow_unsafe", libxl_defbool),
- ("blackhole", libxl_defbool),
- ("compression", libxl_defbool),
- ("netbuf", libxl_defbool),
- ("netbufscript", string),
- ("diskbuf", libxl_defbool),
- ("colo", libxl_defbool)
+ ("interval", integer),
+ ("allow_unsafe", libxl_defbool),
+ ("blackhole", libxl_defbool),
+ ("compression", libxl_defbool),
+ ("netbuf", libxl_defbool),
+ ("netbufscript", string),
+ ("diskbuf", libxl_defbool),
+ ("colo", libxl_defbool),
+ ("userspace_colo_proxy", libxl_defbool)
])
libxl_event_type = Enumeration("event_type", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 7e8a8ae..99baeef 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -8893,7 +8893,7 @@ int main_remus(int argc, char **argv)
memset(&r_info, 0, sizeof(libxl_domain_remus_info));
- SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:ec", NULL, "remus", 2) {
+ SWITCH_FOREACH_OPT(opt, "Fbundi:s:N:ecp", NULL, "remus", 2) {
case 'i':
r_info.interval = atoi(optarg);
break;
@@ -8923,6 +8923,9 @@ int main_remus(int argc, char **argv)
break;
case 'c':
libxl_defbool_set(&r_info.colo, true);
+ break;
+ case 'p':
+ libxl_defbool_set(&r_info.userspace_colo_proxy, true);
}
domid = find_domain(argv[optind]);
@@ -8931,9 +8934,17 @@ int main_remus(int argc, char **argv)
/* Defaults */
libxl_defbool_setdefault(&r_info.blackhole, false);
libxl_defbool_setdefault(&r_info.colo, false);
+ libxl_defbool_setdefault(&r_info.userspace_colo_proxy, false);
+
if (!libxl_defbool_val(r_info.colo) && !r_info.interval)
r_info.interval = 200;
+ if (libxl_defbool_val(r_info.userspace_colo_proxy) &&
+ !libxl_defbool_val(r_info.colo)) {
+ fprintf(stderr, "Option -p must be used in conjunction with -c");
+ exit(-1);
+ }
+
if (libxl_defbool_val(r_info.colo)) {
if (r_info.interval || libxl_defbool_val(r_info.blackhole) ||
!libxl_defbool_is_default(r_info.netbuf) ||
diff --git a/tools/libxl/xl_cmdtable.c b/tools/libxl/xl_cmdtable.c
index 588d5d9..409eceb 100644
--- a/tools/libxl/xl_cmdtable.c
+++ b/tools/libxl/xl_cmdtable.c
@@ -506,7 +506,8 @@ struct cmd_spec cmd_table[] = {
"-n Disable network output buffering. Works only in unsafe mode.\n"
"-d Disable disk replication. Works only in unsafe mode.\n"
"-c Enable COLO HA. It is conflict with -i and -b, and memory\n"
- " checkpoint must be disabled"
+ " checkpoint must be disabled.\n"
+ "-p Use COLO userspace proxy."
},
#endif
{ "devd",
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side
2017-02-17 2:18 [PATCH V3 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
2017-02-17 2:18 ` [PATCH V3 1/7] COLO-Proxy: Add remus command to open userspace proxy Zhang Chen
@ 2017-02-17 2:18 ` Zhang Chen
2017-02-20 15:55 ` Wei Liu
2017-02-17 2:18 ` [PATCH V3 3/7] tools/libxl: refactor do_domain_create() Zhang Chen
` (4 subsequent siblings)
6 siblings, 1 reply; 23+ messages in thread
From: Zhang Chen @ 2017-02-17 2:18 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Zhang Chen, Li Zhijian, eddie, Yang Hongyang, Bian Naimeng
In this patch we close kernel COLO-Proxy on primary side.
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_colo_proxy.c | 27 +++++++++++++++++++++++++++
tools/libxl/libxl_colo_save.c | 9 +++++++--
2 files changed, 34 insertions(+), 2 deletions(-)
diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
index 0983f42..dd902fc 100644
--- a/tools/libxl/libxl_colo_proxy.c
+++ b/tools/libxl/libxl_colo_proxy.c
@@ -152,6 +152,10 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
STATE_AO_GC(cps->ao);
+ /* If enable userspace proxy mode, we don't need setup kernel proxy */
+ if (cps->is_userspace_proxy)
+ return 0;
+
skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
if (skfd < 0) {
LOGD(ERROR, ao->domid, "can not create a netlink socket: %s", strerror(errno));
@@ -222,6 +226,13 @@ out:
void colo_proxy_teardown(libxl__colo_proxy_state *cps)
{
+ /*
+ * If enable userspace proxy mode,
+ * we don't need teardown kernel proxy
+ */
+ if (cps->is_userspace_proxy)
+ return;
+
if (cps->sock_fd >= 0) {
close(cps->sock_fd);
cps->sock_fd = -1;
@@ -232,6 +243,13 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
void colo_proxy_preresume(libxl__colo_proxy_state *cps)
{
+ /*
+ * If enable userspace proxy mode,
+ * we don't need preresume kernel proxy
+ */
+ if (cps->is_userspace_proxy)
+ return;
+
colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
/* TODO: need to handle if the call fails... */
}
@@ -262,6 +280,15 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
STATE_AO_GC(cps->ao);
+ /*
+ * enable userspace proxy mode, tmp sleep.
+ * then we will add qemu API support this func.
+ */
+ if (cps->is_userspace_proxy) {
+ sleep(timeout_us / 1000000);
+ return 0;
+ }
+
size = colo_proxy_recv(cps, &buff, timeout_us);
/* timeout, return no checkpoint message. */
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index eb8336c..91e3fce 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -110,8 +110,13 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
libxl__xen_script_dir_path());
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
- (1 << LIBXL__DEVICE_KIND_VBD);
+ /* If enable userspace proxy mode, we don't need VIF */
+ if (css->cps.is_userspace_proxy)
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+ else
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
+
cds->ops = colo_ops;
cds->callback = colo_save_setup_done;
cds->ao = ao;
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH V3 3/7] tools/libxl: refactor do_domain_create()
2017-02-17 2:18 [PATCH V3 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
2017-02-17 2:18 ` [PATCH V3 1/7] COLO-Proxy: Add remus command to open userspace proxy Zhang Chen
2017-02-17 2:18 ` [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side Zhang Chen
@ 2017-02-17 2:18 ` Zhang Chen
2017-02-20 15:50 ` Wei Liu
2017-02-17 2:18 ` [PATCH V3 4/7] COLO-Proxy: Setup userspace colo-proxy on secondary side Zhang Chen
` (3 subsequent siblings)
6 siblings, 1 reply; 23+ messages in thread
From: Zhang Chen @ 2017-02-17 2:18 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Zhang Chen, Li Zhijian, eddie, Yang Hongyang, Bian Naimeng
We use params->colo_proxy_script to make do_domain_create()
doesn't take "colo_proxy_script" anymore.
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_create.c | 17 ++++++++++-------
1 file changed, 10 insertions(+), 7 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index e3bc257..e741b9a 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1608,7 +1608,6 @@ static void domain_create_cb(libxl__egc *egc,
static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
uint32_t *domid, int restore_fd, int send_back_fd,
const libxl_domain_restore_params *params,
- const char *colo_proxy_script,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
@@ -1632,7 +1631,14 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
}
cdcs->dcs.callback = domain_create_cb;
cdcs->dcs.domid_soft_reset = INVALID_DOMID;
- cdcs->dcs.colo_proxy_script = colo_proxy_script;
+
+ if (cdcs->dcs.restore_params.checkpointed_stream ==
+ LIBXL_CHECKPOINTED_STREAM_COLO)
+ cdcs->dcs.colo_proxy_script =
+ cdcs->dcs.restore_params.colo_proxy_script;
+ else
+ cdcs->dcs.colo_proxy_script = NULL;
+
libxl__ao_progress_gethow(&cdcs->dcs.aop_console_how, aop_console_how);
cdcs->domid_out = domid;
@@ -1820,7 +1826,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncprogress_how *aop_console_how)
{
unset_disk_colo_restore(d_config);
- return do_domain_create(ctx, d_config, domid, -1, -1, NULL, NULL,
+ return do_domain_create(ctx, d_config, domid, -1, -1, NULL,
ao_how, aop_console_how);
}
@@ -1831,17 +1837,14 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
- char *colo_proxy_script = NULL;
-
if (params->checkpointed_stream == LIBXL_CHECKPOINTED_STREAM_COLO) {
- colo_proxy_script = params->colo_proxy_script;
set_disk_colo_restore(d_config);
} else {
unset_disk_colo_restore(d_config);
}
return do_domain_create(ctx, d_config, domid, restore_fd, send_back_fd,
- params, colo_proxy_script, ao_how, aop_console_how);
+ params, ao_how, aop_console_how);
}
int libxl_domain_soft_reset(libxl_ctx *ctx,
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH V3 4/7] COLO-Proxy: Setup userspace colo-proxy on secondary side
2017-02-17 2:18 [PATCH V3 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
` (2 preceding siblings ...)
2017-02-17 2:18 ` [PATCH V3 3/7] tools/libxl: refactor do_domain_create() Zhang Chen
@ 2017-02-17 2:18 ` Zhang Chen
2017-02-17 2:18 ` [PATCH V3 5/7] COLO-Proxy: Add primary userspace colo proxy start args Zhang Chen
` (2 subsequent siblings)
6 siblings, 0 replies; 23+ messages in thread
From: Zhang Chen @ 2017-02-17 2:18 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Zhang Chen, Li Zhijian, eddie, Yang Hongyang, Bian Naimeng
In this patch we add a function to close COLO kernel Proxy on secondary side.
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_colo_restore.c | 8 ++++++--
tools/libxl/libxl_create.c | 8 ++++++--
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl_cmdimpl.c | 18 +++++++++++++++---
4 files changed, 28 insertions(+), 7 deletions(-)
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index 6a96328..c6d239a 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -774,8 +774,12 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
STATE_AO_GC(crs->ao);
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
- (1 << LIBXL__DEVICE_KIND_VBD);
+ if (crs->cps.is_userspace_proxy)
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+ else
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
+
cds->callback = colo_restore_setup_cds_done;
cds->ao = ao;
cds->domid = crs->domid;
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index e741b9a..409945a 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1633,11 +1633,15 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
cdcs->dcs.domid_soft_reset = INVALID_DOMID;
if (cdcs->dcs.restore_params.checkpointed_stream ==
- LIBXL_CHECKPOINTED_STREAM_COLO)
+ LIBXL_CHECKPOINTED_STREAM_COLO) {
cdcs->dcs.colo_proxy_script =
cdcs->dcs.restore_params.colo_proxy_script;
- else
+ cdcs->dcs.crs.cps.is_userspace_proxy =
+ libxl_defbool_val(cdcs->dcs.restore_params.userspace_colo_proxy);
+ } else {
cdcs->dcs.colo_proxy_script = NULL;
+ cdcs->dcs.crs.cps.is_userspace_proxy = false;
+ }
libxl__ao_progress_gethow(&cdcs->dcs.aop_console_how, aop_console_how);
cdcs->domid_out = domid;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 1bd2057..89c2c9d 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -390,6 +390,7 @@ libxl_domain_restore_params = Struct("domain_restore_params", [
("checkpointed_stream", integer),
("stream_version", uint32, {'init_val': '1'}),
("colo_proxy_script", string),
+ ("userspace_colo_proxy", libxl_defbool),
])
libxl_sched_params = Struct("sched_params",[
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 99baeef..b286d47 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -162,6 +162,7 @@ struct domain_create {
char *extra_config; /* extra config string */
const char *restore_file;
char *colo_proxy_script;
+ bool userspace_colo_proxy;
int migrate_fd; /* -1 means none */
int send_back_fd; /* -1 means none */
char **migration_domname_r; /* from malloc */
@@ -3024,6 +3025,8 @@ start:
params.stream_version =
(hdr.mandatory_flags & XL_MANDATORY_FLAG_STREAMv2) ? 2 : 1;
params.colo_proxy_script = dom_info->colo_proxy_script;
+ libxl_defbool_set(¶ms.userspace_colo_proxy,
+ dom_info->userspace_colo_proxy);
ret = libxl_domain_create_restore(ctx, &d_config,
&domid, restore_fd,
@@ -4824,7 +4827,8 @@ static void migrate_receive(int debug, int daemonize, int monitor,
int pause_after_migration,
int send_fd, int recv_fd,
libxl_checkpointed_stream checkpointed,
- char *colo_proxy_script)
+ char *colo_proxy_script,
+ bool userspace_colo_proxy)
{
uint32_t domid;
int rc, rc2;
@@ -4852,6 +4856,7 @@ static void migrate_receive(int debug, int daemonize, int monitor,
dom_info.migration_domname_r = &migration_domname;
dom_info.checkpointed_stream = checkpointed;
dom_info.colo_proxy_script = colo_proxy_script;
+ dom_info.userspace_colo_proxy = userspace_colo_proxy;
rc = create_domain(&dom_info);
if (rc < 0) {
@@ -5051,11 +5056,13 @@ int main_migrate_receive(int argc, char **argv)
int debug = 0, daemonize = 1, monitor = 1, pause_after_migration = 0;
libxl_checkpointed_stream checkpointed = LIBXL_CHECKPOINTED_STREAM_NONE;
int opt;
+ bool userspace_colo_proxy = false;
char *script = NULL;
static struct option opts[] = {
{"colo", 0, 0, 0x100},
/* It is a shame that the management code for disk is not here. */
{"coloft-script", 1, 0, 0x200},
+ {"userspace-colo-proxy", 0, 0, 0x300},
COMMON_LONG_OPTS
};
@@ -5079,6 +5086,9 @@ int main_migrate_receive(int argc, char **argv)
case 0x200:
script = optarg;
break;
+ case 0x300:
+ userspace_colo_proxy = true;
+ break;
case 'p':
pause_after_migration = 1;
break;
@@ -5090,7 +5100,7 @@ int main_migrate_receive(int argc, char **argv)
}
migrate_receive(debug, daemonize, monitor, pause_after_migration,
STDOUT_FILENO, STDIN_FILENO,
- checkpointed, script);
+ checkpointed, script, userspace_colo_proxy);
return EXIT_SUCCESS;
}
@@ -8984,11 +8994,13 @@ int main_remus(int argc, char **argv)
"-r",
daemonize ? "" : " -e");
} else {
- xasprintf(&rune, "exec %s %s xl migrate-receive %s %s %s %s",
+ xasprintf(&rune, "exec %s %s xl migrate-receive %s %s %s %s %s",
ssh_command, host,
"--colo",
r_info.netbufscript ? "--coloft-script" : "",
r_info.netbufscript ? r_info.netbufscript : "",
+ libxl_defbool_val(r_info.userspace_colo_proxy) ?
+ "--userspace-colo-proxy" : "",
daemonize ? "" : " -e");
}
}
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH V3 5/7] COLO-Proxy: Add primary userspace colo proxy start args
2017-02-17 2:18 [PATCH V3 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
` (3 preceding siblings ...)
2017-02-17 2:18 ` [PATCH V3 4/7] COLO-Proxy: Setup userspace colo-proxy on secondary side Zhang Chen
@ 2017-02-17 2:18 ` Zhang Chen
2017-02-20 15:59 ` Wei Liu
2017-02-17 2:18 ` [PATCH V3 6/7] COLO-Proxy: Add secondary userspace colo-proxy " Zhang Chen
2017-02-17 2:18 ` [PATCH V3 7/7] COLO-Proxy: Use socket to get checkpoint event Zhang Chen
6 siblings, 1 reply; 23+ messages in thread
From: Zhang Chen @ 2017-02-17 2:18 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Zhang Chen, Li Zhijian, eddie, Yang Hongyang, Bian Naimeng
Qemu need this args to start userspace colo-proxy.
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_dm.c | 98 +++++++++++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_nic.c | 78 ++++++++++++++++++++++++++++++++++++
tools/libxl/libxl_types.idl | 31 +++++++++++++-
tools/libxl/xl_cmdimpl.c | 58 +++++++++++++++++++++++++++
4 files changed, 264 insertions(+), 1 deletion(-)
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index 281058d..abd4edd 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -1244,7 +1244,105 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
nics[i].devid, ifname,
libxl_tapif_script(gc),
libxl_tapif_script(gc)));
+
+ /* Userspace COLO Proxy need this */
+#define APPEND_COLO_SOCK_SERVER(sock_id, sock_ip, sock_port) ({ \
+ if (nics[i].colo_##sock_id && \
+ nics[i].colo_##sock_ip && \
+ nics[i].colo_##sock_port) { \
+ flexarray_append(dm_args, "-chardev"); \
+ flexarray_append(dm_args, \
+ GCSPRINTF("socket,id=%s,host=%s,port=%s,server,nowait", \
+ nics[i].colo_##sock_id, \
+ nics[i].colo_##sock_ip, \
+ nics[i].colo_##sock_port)); \
+ } \
+})
+
+#define APPEND_COLO_SOCK_CLIENT(sock_id, sock_ip, sock_port) ({ \
+ if (nics[i].colo_##sock_id && \
+ nics[i].colo_##sock_ip && \
+ nics[i].colo_##sock_port) { \
+ flexarray_append(dm_args, "-chardev"); \
+ flexarray_append(dm_args, \
+ GCSPRINTF("socket,id=%s,host=%s,port=%s", \
+ nics[i].colo_##sock_id, \
+ nics[i].colo_##sock_ip, \
+ nics[i].colo_##sock_port)); \
+ } \
+})
+
+ if (state->saved_state) {
+ /* secondary colo run */
+ } else {
+ /* primary colo run */
+
+ APPEND_COLO_SOCK_SERVER(sock_mirror_id,
+ sock_mirror_ip,
+ sock_mirror_port);
+
+ APPEND_COLO_SOCK_SERVER(sock_compare_pri_in_id,
+ sock_compare_pri_in_ip,
+ sock_compare_pri_in_port);
+
+ APPEND_COLO_SOCK_SERVER(sock_compare_sec_in_id,
+ sock_compare_sec_in_ip,
+ sock_compare_sec_in_port);
+
+ APPEND_COLO_SOCK_SERVER(sock_redirector0_id,
+ sock_redirector0_ip,
+ sock_redirector0_port);
+
+ APPEND_COLO_SOCK_CLIENT(sock_redirector1_id,
+ sock_redirector1_ip,
+ sock_redirector1_port);
+
+ APPEND_COLO_SOCK_CLIENT(sock_redirector2_id,
+ sock_redirector2_ip,
+ sock_redirector2_port);
+
+ if (nics[i].colo_filter_mirror_queue &&
+ nics[i].colo_filter_mirror_outdev) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-mirror,id=m1,netdev=net%d,queue=%s,outdev=%s",
+ nics[i].devid,
+ nics[i].colo_filter_mirror_queue,
+ nics[i].colo_filter_mirror_outdev));
+ }
+ if (nics[i].colo_filter_redirector0_queue &&
+ nics[i].colo_filter_redirector0_indev) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-redirector,id=r1,netdev=net%d,queue=%s,indev=%s",
+ nics[i].devid,
+ nics[i].colo_filter_redirector0_queue,
+ nics[i].colo_filter_redirector0_indev));
+ }
+ if (nics[i].colo_filter_redirector1_queue &&
+ nics[i].colo_filter_redirector1_outdev) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-redirector,id=r2,netdev=net%d,queue=%s,outdev=%s",
+ nics[i].devid,
+ nics[i].colo_filter_redirector1_queue,
+ nics[i].colo_filter_redirector1_outdev));
+ }
+ if (nics[i].colo_compare_pri_in &&
+ nics[i].colo_compare_sec_in &&
+ nics[i].colo_compare_out) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("colo-compare,id=c1,primary_in=%s,secondary_in=%s,outdev=%s",
+ nics[i].colo_compare_pri_in,
+ nics[i].colo_compare_sec_in,
+ nics[i].colo_compare_out));
+ }
+ }
ioemu_nics++;
+
+#undef APPEND_COLO_SOCK_SERVER
+#undef APPEND_COLO_SOCK_CLIENT
}
}
/* If we have no emulated nics, tell qemu not to create any */
diff --git a/tools/libxl/libxl_nic.c b/tools/libxl/libxl_nic.c
index 61b55ca..7c57bcf 100644
--- a/tools/libxl/libxl_nic.c
+++ b/tools/libxl/libxl_nic.c
@@ -196,6 +196,45 @@ static void libxl__device_nic_add(libxl__egc *egc, uint32_t domid,
flexarray_append(back, nic->coloft_forwarddev);
}
+#define MAYBE_ADD_COLO_ARGS(arg) ({ \
+ if (nic->colo_##arg) { \
+ flexarray_append(back, "colo_"#arg); \
+ flexarray_append(back, nic->colo_##arg); \
+ } \
+})
+
+ MAYBE_ADD_COLO_ARGS(sock_mirror_id);
+ MAYBE_ADD_COLO_ARGS(sock_mirror_ip);
+ MAYBE_ADD_COLO_ARGS(sock_mirror_port);
+ MAYBE_ADD_COLO_ARGS(sock_compare_pri_in_id);
+ MAYBE_ADD_COLO_ARGS(sock_compare_pri_in_ip);
+ MAYBE_ADD_COLO_ARGS(sock_compare_pri_in_port);
+ MAYBE_ADD_COLO_ARGS(sock_compare_sec_in_id);
+ MAYBE_ADD_COLO_ARGS(sock_compare_sec_in_ip);
+ MAYBE_ADD_COLO_ARGS(sock_compare_sec_in_port);
+ MAYBE_ADD_COLO_ARGS(sock_redirector0_id);
+ MAYBE_ADD_COLO_ARGS(sock_redirector0_ip);
+ MAYBE_ADD_COLO_ARGS(sock_redirector0_port);
+ MAYBE_ADD_COLO_ARGS(sock_redirector1_id);
+ MAYBE_ADD_COLO_ARGS(sock_redirector1_ip);
+ MAYBE_ADD_COLO_ARGS(sock_redirector1_port);
+ MAYBE_ADD_COLO_ARGS(sock_redirector2_id);
+ MAYBE_ADD_COLO_ARGS(sock_redirector2_ip);
+ MAYBE_ADD_COLO_ARGS(sock_redirector2_port);
+ MAYBE_ADD_COLO_ARGS(filter_mirror_queue);
+ MAYBE_ADD_COLO_ARGS(filter_mirror_outdev);
+ MAYBE_ADD_COLO_ARGS(filter_redirector0_queue);
+ MAYBE_ADD_COLO_ARGS(filter_redirector0_indev);
+ MAYBE_ADD_COLO_ARGS(filter_redirector0_outdev);
+ MAYBE_ADD_COLO_ARGS(filter_redirector1_queue);
+ MAYBE_ADD_COLO_ARGS(filter_redirector1_indev);
+ MAYBE_ADD_COLO_ARGS(filter_redirector1_outdev);
+ MAYBE_ADD_COLO_ARGS(compare_pri_in);
+ MAYBE_ADD_COLO_ARGS(compare_sec_in);
+ MAYBE_ADD_COLO_ARGS(compare_out);
+
+#undef MAYBE_ADD_COLO_ARGS
+
flexarray_append(back, "mac");
flexarray_append(back,GCSPRINTF(LIBXL_MAC_FMT, LIBXL_MAC_BYTES(nic->mac)));
if (nic->ip) {
@@ -349,6 +388,45 @@ static int libxl__device_nic_from_xenstore(libxl__gc *gc,
(const char **)(&nic->coloft_forwarddev));
if (rc) goto out;
+#define CHECK_COLO_ARGS(arg) ({ \
+ rc = libxl__xs_read_checked(NOGC, XBT_NULL, \
+ GCSPRINTF("%s/colo_"#arg, libxl_path), \
+ (const char **)(&nic->colo_##arg)); \
+ if (rc) goto out; \
+})
+
+ CHECK_COLO_ARGS(sock_mirror_id);
+ CHECK_COLO_ARGS(sock_mirror_ip);
+ CHECK_COLO_ARGS(sock_mirror_port);
+ CHECK_COLO_ARGS(sock_compare_pri_in_id);
+ CHECK_COLO_ARGS(sock_compare_pri_in_ip);
+ CHECK_COLO_ARGS(sock_compare_pri_in_port);
+ CHECK_COLO_ARGS(sock_compare_sec_in_id);
+ CHECK_COLO_ARGS(sock_compare_sec_in_ip);
+ CHECK_COLO_ARGS(sock_compare_sec_in_port);
+ CHECK_COLO_ARGS(sock_redirector0_id);
+ CHECK_COLO_ARGS(sock_redirector0_ip);
+ CHECK_COLO_ARGS(sock_redirector0_port);
+ CHECK_COLO_ARGS(sock_redirector1_id);
+ CHECK_COLO_ARGS(sock_redirector1_ip);
+ CHECK_COLO_ARGS(sock_redirector1_port);
+ CHECK_COLO_ARGS(sock_redirector2_id);
+ CHECK_COLO_ARGS(sock_redirector2_ip);
+ CHECK_COLO_ARGS(sock_redirector2_port);
+ CHECK_COLO_ARGS(filter_mirror_queue);
+ CHECK_COLO_ARGS(filter_mirror_outdev);
+ CHECK_COLO_ARGS(filter_redirector0_queue);
+ CHECK_COLO_ARGS(filter_redirector0_indev);
+ CHECK_COLO_ARGS(filter_redirector0_outdev);
+ CHECK_COLO_ARGS(filter_redirector1_queue);
+ CHECK_COLO_ARGS(filter_redirector1_indev);
+ CHECK_COLO_ARGS(filter_redirector1_outdev);
+ CHECK_COLO_ARGS(compare_pri_in);
+ CHECK_COLO_ARGS(compare_sec_in);
+ CHECK_COLO_ARGS(compare_out);
+
+#undef CHECK_COLO_ARGS
+
/* vif_ioemu nics use the same xenstore entries as vif interfaces */
rc = libxl__xs_read_checked(gc, XBT_NULL,
GCSPRINTF("%s/type", libxl_path), &tmp);
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 89c2c9d..07ce345 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -629,7 +629,36 @@ libxl_device_nic = Struct("device_nic", [
("gatewaydev", string),
# Note that the COLO configuration settings should be considered unstable.
# They may change incompatibly in future versions of Xen.
- ("coloft_forwarddev", string)
+ ("coloft_forwarddev", string),
+ ("colo_sock_mirror_id", string),
+ ("colo_sock_mirror_ip", string),
+ ("colo_sock_mirror_port", string),
+ ("colo_sock_compare_pri_in_id", string),
+ ("colo_sock_compare_pri_in_ip", string),
+ ("colo_sock_compare_pri_in_port", string),
+ ("colo_sock_compare_sec_in_id", string),
+ ("colo_sock_compare_sec_in_ip", string),
+ ("colo_sock_compare_sec_in_port", string),
+ ("colo_sock_redirector0_id", string),
+ ("colo_sock_redirector0_ip", string),
+ ("colo_sock_redirector0_port", string),
+ ("colo_sock_redirector1_id", string),
+ ("colo_sock_redirector1_ip", string),
+ ("colo_sock_redirector1_port", string),
+ ("colo_sock_redirector2_id", string),
+ ("colo_sock_redirector2_ip", string),
+ ("colo_sock_redirector2_port", string),
+ ("colo_filter_mirror_queue", string),
+ ("colo_filter_mirror_outdev", string),
+ ("colo_filter_redirector0_queue", string),
+ ("colo_filter_redirector0_indev", string),
+ ("colo_filter_redirector0_outdev", string),
+ ("colo_filter_redirector1_queue", string),
+ ("colo_filter_redirector1_indev", string),
+ ("colo_filter_redirector1_outdev", string),
+ ("colo_compare_pri_in", string),
+ ("colo_compare_sec_in", string),
+ ("colo_compare_out", string)
])
libxl_device_pci = Struct("device_pci", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index b286d47..e6cb015 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1060,6 +1060,64 @@ static int parse_nic_config(libxl_device_nic *nic, XLU_Config **config, char *to
parse_vif_rate(config, oparg, nic);
} else if (MATCH_OPTION("forwarddev", token, oparg)) {
replace_string(&nic->coloft_forwarddev, oparg);
+ } else if (MATCH_OPTION("colo_sock_mirror_id", token, oparg)) {
+ replace_string(&nic->colo_sock_mirror_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_mirror_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_mirror_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_mirror_port", token, oparg)) {
+ replace_string(&nic->colo_sock_mirror_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_sec_in_id", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_sec_in_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_sec_in_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_sec_in_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_sec_in_port", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_sec_in_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector0_id", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector0_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector0_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector0_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector0_port", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector0_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector1_id", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector1_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector1_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector1_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector1_port", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector1_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector2_id", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector2_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector2_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector2_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_redirector2_port", token, oparg)) {
+ replace_string(&nic->colo_sock_redirector2_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_pri_in_id", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_pri_in_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_pri_in_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_pri_in_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_compare_pri_in_port", token, oparg)) {
+ replace_string(&nic->colo_sock_compare_pri_in_port, oparg);
+ } else if (MATCH_OPTION("colo_filter_mirror_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_mirror_queue, oparg);
+ } else if (MATCH_OPTION("colo_filter_mirror_outdev", token, oparg)) {
+ replace_string(&nic->colo_filter_mirror_outdev, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector0_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector0_queue, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector0_indev", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector0_indev, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector0_outdev", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector0_outdev, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector1_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector1_queue, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector1_indev", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector1_indev, oparg);
+ } else if (MATCH_OPTION("colo_filter_redirector1_outdev", token, oparg)) {
+ replace_string(&nic->colo_filter_redirector1_outdev, oparg);
+ } else if (MATCH_OPTION("colo_compare_pri_in", token, oparg)) {
+ replace_string(&nic->colo_compare_pri_in, oparg);
+ } else if (MATCH_OPTION("colo_compare_sec_in", token, oparg)) {
+ replace_string(&nic->colo_compare_sec_in, oparg);
+ } else if (MATCH_OPTION("colo_compare_out", token, oparg)) {
+ replace_string(&nic->colo_compare_out, oparg);
} else if (MATCH_OPTION("accel", token, oparg)) {
fprintf(stderr, "the accel parameter for vifs is currently not supported\n");
} else {
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH V3 6/7] COLO-Proxy: Add secondary userspace colo-proxy start args
2017-02-17 2:18 [PATCH V3 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
` (4 preceding siblings ...)
2017-02-17 2:18 ` [PATCH V3 5/7] COLO-Proxy: Add primary userspace colo proxy start args Zhang Chen
@ 2017-02-17 2:18 ` Zhang Chen
2017-02-17 2:18 ` [PATCH V3 7/7] COLO-Proxy: Use socket to get checkpoint event Zhang Chen
6 siblings, 0 replies; 23+ messages in thread
From: Zhang Chen @ 2017-02-17 2:18 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Zhang Chen, Li Zhijian, eddie, Yang Hongyang, Bian Naimeng
Qemu need this args to start userspace colo-proxy.
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_dm.c | 34 ++++++++++++++++++++++++++++++++++
tools/libxl/libxl_nic.c | 27 +++++++++++++++++++++++++++
tools/libxl/libxl_types.idl | 15 ++++++++++++++-
tools/libxl/xl_cmdimpl.c | 27 +++++++++++++++++++++++++++
4 files changed, 102 insertions(+), 1 deletion(-)
diff --git a/tools/libxl/libxl_dm.c b/tools/libxl/libxl_dm.c
index abd4edd..0fabd64 100644
--- a/tools/libxl/libxl_dm.c
+++ b/tools/libxl/libxl_dm.c
@@ -1274,6 +1274,40 @@ static int libxl__build_device_model_args_new(libxl__gc *gc,
if (state->saved_state) {
/* secondary colo run */
+
+ APPEND_COLO_SOCK_CLIENT(sock_sec_redirector0_id,
+ sock_sec_redirector0_ip,
+ sock_sec_redirector0_port);
+
+ APPEND_COLO_SOCK_CLIENT(sock_sec_redirector1_id,
+ sock_sec_redirector1_ip,
+ sock_sec_redirector1_port);
+
+ if (nics[i].colo_filter_sec_redirector0_queue &&
+ nics[i].colo_filter_sec_redirector0_indev) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-redirector,id=rs1,netdev=net%d,queue=%s,indev=%s",
+ nics[i].devid,
+ nics[i].colo_filter_sec_redirector0_queue,
+ nics[i].colo_filter_sec_redirector0_indev));
+ }
+ if (nics[i].colo_filter_sec_redirector1_queue &&
+ nics[i].colo_filter_sec_redirector1_indev) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-redirector,id=rs2,netdev=net%d,queue=%s,outdev=%s",
+ nics[i].devid,
+ nics[i].colo_filter_sec_redirector1_queue,
+ nics[i].colo_filter_sec_redirector1_outdev));
+ }
+ if (nics[i].colo_filter_sec_rewriter0_queue) {
+ flexarray_append(dm_args, "-object");
+ flexarray_append(dm_args,
+ GCSPRINTF("filter-rewriter,id=rs3,netdev=net%d,queue=%s",
+ nics[i].devid,
+ nics[i].colo_filter_sec_rewriter0_queue));
+ }
} else {
/* primary colo run */
diff --git a/tools/libxl/libxl_nic.c b/tools/libxl/libxl_nic.c
index 7c57bcf..5e1fecd 100644
--- a/tools/libxl/libxl_nic.c
+++ b/tools/libxl/libxl_nic.c
@@ -233,6 +233,20 @@ static void libxl__device_nic_add(libxl__egc *egc, uint32_t domid,
MAYBE_ADD_COLO_ARGS(compare_sec_in);
MAYBE_ADD_COLO_ARGS(compare_out);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector0_id);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector0_ip);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector0_port);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector1_id);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector1_ip);
+ MAYBE_ADD_COLO_ARGS(sock_sec_redirector1_port);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector0_queue);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector0_indev);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector0_outdev);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_queue);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_indev);
+ MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_outdev);
+ MAYBE_ADD_COLO_ARGS(filter_sec_rewriter0_queue);
+
#undef MAYBE_ADD_COLO_ARGS
flexarray_append(back, "mac");
@@ -424,6 +438,19 @@ static int libxl__device_nic_from_xenstore(libxl__gc *gc,
CHECK_COLO_ARGS(compare_pri_in);
CHECK_COLO_ARGS(compare_sec_in);
CHECK_COLO_ARGS(compare_out);
+ CHECK_COLO_ARGS(sock_sec_redirector0_id);
+ CHECK_COLO_ARGS(sock_sec_redirector0_ip);
+ CHECK_COLO_ARGS(sock_sec_redirector0_port);
+ CHECK_COLO_ARGS(sock_sec_redirector1_id);
+ CHECK_COLO_ARGS(sock_sec_redirector1_ip);
+ CHECK_COLO_ARGS(sock_sec_redirector1_port);
+ CHECK_COLO_ARGS(filter_sec_redirector0_queue);
+ CHECK_COLO_ARGS(filter_sec_redirector0_indev);
+ CHECK_COLO_ARGS(filter_sec_redirector0_outdev);
+ CHECK_COLO_ARGS(filter_sec_redirector1_queue);
+ CHECK_COLO_ARGS(filter_sec_redirector1_indev);
+ CHECK_COLO_ARGS(filter_sec_redirector1_outdev);
+ CHECK_COLO_ARGS(filter_sec_rewriter0_queue);
#undef CHECK_COLO_ARGS
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 07ce345..47e96b1 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -658,7 +658,20 @@ libxl_device_nic = Struct("device_nic", [
("colo_filter_redirector1_outdev", string),
("colo_compare_pri_in", string),
("colo_compare_sec_in", string),
- ("colo_compare_out", string)
+ ("colo_compare_out", string),
+ ("colo_sock_sec_redirector0_id", string),
+ ("colo_sock_sec_redirector0_ip", string),
+ ("colo_sock_sec_redirector0_port", string),
+ ("colo_sock_sec_redirector1_id", string),
+ ("colo_sock_sec_redirector1_ip", string),
+ ("colo_sock_sec_redirector1_port", string),
+ ("colo_filter_sec_redirector0_queue", string),
+ ("colo_filter_sec_redirector0_indev", string),
+ ("colo_filter_sec_redirector0_outdev", string),
+ ("colo_filter_sec_redirector1_queue", string),
+ ("colo_filter_sec_redirector1_indev", string),
+ ("colo_filter_sec_redirector1_outdev", string),
+ ("colo_filter_sec_rewriter0_queue", string)
])
libxl_device_pci = Struct("device_pci", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index e6cb015..32a47f6 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1118,6 +1118,33 @@ static int parse_nic_config(libxl_device_nic *nic, XLU_Config **config, char *to
replace_string(&nic->colo_compare_sec_in, oparg);
} else if (MATCH_OPTION("colo_compare_out", token, oparg)) {
replace_string(&nic->colo_compare_out, oparg);
+
+ } else if (MATCH_OPTION("colo_sock_sec_redirector0_id", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector0_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_sec_redirector0_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector0_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_sec_redirector0_port", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector0_port, oparg);
+ } else if (MATCH_OPTION("colo_sock_sec_redirector1_id", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector1_id, oparg);
+ } else if (MATCH_OPTION("colo_sock_sec_redirector1_ip", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector1_ip, oparg);
+ } else if (MATCH_OPTION("colo_sock_sec_redirector1_port", token, oparg)) {
+ replace_string(&nic->colo_sock_sec_redirector1_port, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector0_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector0_queue, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector0_indev", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector0_indev, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector0_outdev", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector0_outdev, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector1_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector1_queue, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector1_indev", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector1_indev, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_redirector1_outdev", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_redirector1_outdev, oparg);
+ } else if (MATCH_OPTION("colo_filter_sec_rewriter0_queue", token, oparg)) {
+ replace_string(&nic->colo_filter_sec_rewriter0_queue, oparg);
} else if (MATCH_OPTION("accel", token, oparg)) {
fprintf(stderr, "the accel parameter for vifs is currently not supported\n");
} else {
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 23+ messages in thread
* [PATCH V3 7/7] COLO-Proxy: Use socket to get checkpoint event.
2017-02-17 2:18 [PATCH V3 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
` (5 preceding siblings ...)
2017-02-17 2:18 ` [PATCH V3 6/7] COLO-Proxy: Add secondary userspace colo-proxy " Zhang Chen
@ 2017-02-17 2:18 ` Zhang Chen
2017-02-20 16:07 ` Wei Liu
6 siblings, 1 reply; 23+ messages in thread
From: Zhang Chen @ 2017-02-17 2:18 UTC (permalink / raw)
To: Xen devel, Wei Liu, Ian Jackson
Cc: Zhang Chen, Li Zhijian, eddie, Yang Hongyang, Bian Naimeng
We use kernel colo proxy's way to get the checkpoint event
from qemu colo-compare.
Qemu colo-compare need add a API to support this(I will add this in qemu).
Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
---
tools/libxl/libxl_colo.h | 2 +
tools/libxl/libxl_colo_proxy.c | 85 +++++++++++++++++++++++++++++++++++++---
tools/libxl/libxl_colo_restore.c | 11 ++++--
tools/libxl/libxl_colo_save.c | 22 +++++++----
tools/libxl/libxl_nic.c | 4 ++
tools/libxl/libxl_types.idl | 4 +-
tools/libxl/xl_cmdimpl.c | 4 ++
7 files changed, 114 insertions(+), 18 deletions(-)
diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
index 4746d8c..6c01b55 100644
--- a/tools/libxl/libxl_colo.h
+++ b/tools/libxl/libxl_colo.h
@@ -69,6 +69,8 @@ struct libxl__colo_proxy_state {
* False means use kernel colo proxy.
*/
bool is_userspace_proxy;
+ const char *checkpoint_host;
+ const char *checkpoint_port;
};
struct libxl__colo_save_state {
diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
index dd902fc..a6436b0 100644
--- a/tools/libxl/libxl_colo_proxy.c
+++ b/tools/libxl/libxl_colo_proxy.c
@@ -18,9 +18,13 @@
#include "libxl_internal.h"
#include <netlink/netlink.h>
+#include <arpa/inet.h>
+#include <sys/socket.h>
+#include <netinet/in.h>
/* Consistent with the new COLO netlink channel in kernel side */
#define NETLINK_COLO 28
+#define COLO_DEFAULT_WAIT_TIME 500000
enum colo_netlink_op {
COLO_QUERY_CHECKPOINT = (NLMSG_MIN_TYPE + 1),
@@ -76,6 +80,26 @@ static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
return ret;
}
+static int colo_userspace_proxy_recv(libxl__colo_proxy_state *cps,
+ char *buff,
+ unsigned int timeout_us)
+{
+ struct timeval tv;
+ int ret;
+
+ STATE_AO_GC(cps->ao);
+
+ if (timeout_us) {
+ tv.tv_sec = timeout_us / 1000000;
+ tv.tv_usec = timeout_us % 1000000;
+ setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
+ }
+
+ ret = recv(cps->sock_fd, buff, sizeof(buff),0);
+
+ return ret;
+}
+
/* error: return -1, otherwise return 0 */
static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
unsigned int timeout_us)
@@ -153,8 +177,45 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
STATE_AO_GC(cps->ao);
/* If enable userspace proxy mode, we don't need setup kernel proxy */
- if (cps->is_userspace_proxy)
+ if (cps->is_userspace_proxy) {
+ struct sockaddr_in addr;
+ int port;
+ char recvbuff[1024];
+ char sendbuf[] = "COLO_USERSPACE_PROXY_INIT";
+
+ memset(&addr, 0, sizeof(addr));
+ port = atoi(cps->checkpoint_port);
+ addr.sin_family = AF_INET;
+ addr.sin_port = htons(port);
+ addr.sin_addr.s_addr = inet_addr(cps->checkpoint_host);
+
+ skfd = socket(AF_INET, SOCK_STREAM, 0);
+ if (skfd < 0) {
+ LOGD(ERROR, ao->domid, "can not create a TCP socket: %s",
+ strerror(errno));
+ goto out;
+ }
+
+ cps->sock_fd = skfd;
+
+ if (connect(skfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
+ LOGD(ERROR, ao->domid, "connect error");
+ goto out;
+ }
+
+ ret = send(skfd, sendbuf, strlen(sendbuf),0);
+ if (ret < 0)
+ goto out;
+
+ ret = colo_userspace_proxy_recv(cps, recvbuff, COLO_DEFAULT_WAIT_TIME);
+ if (ret < 0) {
+ LOGD(ERROR, ao->domid, "Can't recv msg from qemu colo-compare: %s",
+ strerror(errno));
+ goto out;
+ }
+
return 0;
+ }
skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
if (skfd < 0) {
@@ -247,8 +308,11 @@ void colo_proxy_preresume(libxl__colo_proxy_state *cps)
* If enable userspace proxy mode,
* we don't need preresume kernel proxy
*/
- if (cps->is_userspace_proxy)
+ if (cps->is_userspace_proxy) {
+ char sendbuf[] = "COLO_CHECKPOINT";
+ send(cps->sock_fd, sendbuf, strlen(sendbuf),0);
return;
+ }
colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
/* TODO: need to handle if the call fails... */
@@ -277,16 +341,25 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
struct nlmsghdr *h;
struct colo_msg *m;
int ret = -1;
+ char recvbuff[1024];
STATE_AO_GC(cps->ao);
/*
- * enable userspace proxy mode, tmp sleep.
- * then we will add qemu API support this func.
+ * enable userspace proxy mode.
+ * Then we will add qemu API support for this func.
*/
if (cps->is_userspace_proxy) {
- sleep(timeout_us / 1000000);
- return 0;
+ ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
+ if (ret <= 0)
+ return 0;
+
+ if (!strcmp(recvbuff, "DO_CHECKPOINT")) {
+ return 1;
+ } else {
+ LOGD(ERROR, ao->domid, "receive qemu colo-compare checkpoint error");
+ return -1;
+ }
}
size = colo_proxy_recv(cps, &buff, timeout_us);
diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
index c6d239a..065ea00 100644
--- a/tools/libxl/libxl_colo_restore.c
+++ b/tools/libxl/libxl_colo_restore.c
@@ -613,7 +613,8 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
}
}
- colo_proxy_preresume(&crs->cps);
+ if (!crs->cps.is_userspace_proxy)
+ colo_proxy_preresume(&crs->cps);
colo_restore_resume_vm(egc, crcs);
@@ -786,9 +787,11 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
cds->ops = colo_restore_ops;
crs->cps.ao = ao;
- if (colo_proxy_setup(&crs->cps)) {
- LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
- goto out;
+ if (!crs->cps.is_userspace_proxy) {
+ if (colo_proxy_setup(&crs->cps)) {
+ LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
+ goto out;
+ }
}
if (init_device_subkind(cds))
diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
index 91e3fce..a0cfc5a 100644
--- a/tools/libxl/libxl_colo_save.c
+++ b/tools/libxl/libxl_colo_save.c
@@ -86,6 +86,7 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
libxl__checkpoint_devices_state *const cds = &dss->cds;
libxl__srm_save_autogen_callbacks *const callbacks =
&dss->sws.shs.callbacks.save.a;
+ libxl_device_nic *nics;
STATE_AO_GC(dss->ao);
@@ -110,24 +111,31 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
libxl__xen_script_dir_path());
- /* If enable userspace proxy mode, we don't need VIF */
- if (css->cps.is_userspace_proxy)
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
- else
- cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
- (1 << LIBXL__DEVICE_KIND_VBD);
-
cds->ops = colo_ops;
cds->callback = colo_save_setup_done;
cds->ao = ao;
cds->domid = dss->domid;
cds->concrete_data = css;
+ /* If enable userspace proxy mode, we don't need VIF */
+ if (css->cps.is_userspace_proxy) {
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
+
+ /* Use this args we can connect to qemu colo-compare */
+ nics = libxl_device_nic_list(CTX, cds->domid, &cds->num_nics);
+ css->cps.checkpoint_host = nics->colo_checkpoint_host;
+ css->cps.checkpoint_port = nics->colo_checkpoint_port;
+ } else {
+ cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
+ (1 << LIBXL__DEVICE_KIND_VBD);
+ }
+
css->srs.ao = ao;
css->srs.fd = css->recv_fd;
css->srs.back_channel = true;
libxl__stream_read_start(egc, &css->srs);
css->cps.ao = ao;
+
if (colo_proxy_setup(&css->cps)) {
LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
goto out;
diff --git a/tools/libxl/libxl_nic.c b/tools/libxl/libxl_nic.c
index 5e1fecd..6bc6146 100644
--- a/tools/libxl/libxl_nic.c
+++ b/tools/libxl/libxl_nic.c
@@ -246,6 +246,8 @@ static void libxl__device_nic_add(libxl__egc *egc, uint32_t domid,
MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_indev);
MAYBE_ADD_COLO_ARGS(filter_sec_redirector1_outdev);
MAYBE_ADD_COLO_ARGS(filter_sec_rewriter0_queue);
+ MAYBE_ADD_COLO_ARGS(checkpoint_host);
+ MAYBE_ADD_COLO_ARGS(checkpoint_port);
#undef MAYBE_ADD_COLO_ARGS
@@ -451,6 +453,8 @@ static int libxl__device_nic_from_xenstore(libxl__gc *gc,
CHECK_COLO_ARGS(filter_sec_redirector1_indev);
CHECK_COLO_ARGS(filter_sec_redirector1_outdev);
CHECK_COLO_ARGS(filter_sec_rewriter0_queue);
+ CHECK_COLO_ARGS(checkpoint_host);
+ CHECK_COLO_ARGS(checkpoint_port);
#undef CHECK_COLO_ARGS
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 47e96b1..0b412ee 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -671,7 +671,9 @@ libxl_device_nic = Struct("device_nic", [
("colo_filter_sec_redirector1_queue", string),
("colo_filter_sec_redirector1_indev", string),
("colo_filter_sec_redirector1_outdev", string),
- ("colo_filter_sec_rewriter0_queue", string)
+ ("colo_filter_sec_rewriter0_queue", string),
+ ("colo_checkpoint_host", string),
+ ("colo_checkpoint_port", string)
])
libxl_device_pci = Struct("device_pci", [
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 32a47f6..ad5e193 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -1145,6 +1145,10 @@ static int parse_nic_config(libxl_device_nic *nic, XLU_Config **config, char *to
replace_string(&nic->colo_filter_sec_redirector1_outdev, oparg);
} else if (MATCH_OPTION("colo_filter_sec_rewriter0_queue", token, oparg)) {
replace_string(&nic->colo_filter_sec_rewriter0_queue, oparg);
+ } else if (MATCH_OPTION("colo_checkpoint_host", token, oparg)) {
+ replace_string(&nic->colo_checkpoint_host, oparg);
+ } else if (MATCH_OPTION("colo_checkpoint_port", token, oparg)) {
+ replace_string(&nic->colo_checkpoint_port, oparg);
} else if (MATCH_OPTION("accel", token, oparg)) {
fprintf(stderr, "the accel parameter for vifs is currently not supported\n");
} else {
--
2.7.4
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply related [flat|nested] 23+ messages in thread
* Re: [PATCH V3 3/7] tools/libxl: refactor do_domain_create()
2017-02-17 2:18 ` [PATCH V3 3/7] tools/libxl: refactor do_domain_create() Zhang Chen
@ 2017-02-20 15:50 ` Wei Liu
2017-02-21 2:48 ` Zhang Chen
0 siblings, 1 reply; 23+ messages in thread
From: Wei Liu @ 2017-02-20 15:50 UTC (permalink / raw)
To: Zhang Chen
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang,
Xen devel, Bian Naimeng
On Fri, Feb 17, 2017 at 10:18:25AM +0800, Zhang Chen wrote:
> We use params->colo_proxy_script to make do_domain_create()
> doesn't take "colo_proxy_script" anymore.
>
> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> ---
This patch is self-contained so:
Acked-by: Wei Liu <wei.liu2@citrix.com>
I will commit this soon. Please rebase your next version against
staging.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 1/7] COLO-Proxy: Add remus command to open userspace proxy
2017-02-17 2:18 ` [PATCH V3 1/7] COLO-Proxy: Add remus command to open userspace proxy Zhang Chen
@ 2017-02-20 15:50 ` Wei Liu
2017-02-21 2:49 ` Zhang Chen
0 siblings, 1 reply; 23+ messages in thread
From: Wei Liu @ 2017-02-20 15:50 UTC (permalink / raw)
To: Zhang Chen
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang,
Xen devel, Bian Naimeng
On Fri, Feb 17, 2017 at 10:18:23AM +0800, Zhang Chen wrote:
> Add remus '-p' to enable userspace colo proxy(in qemu).
>
> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> ---
> docs/man/xl.pod.1.in | 5 +++++
> tools/libxl/libxl.h | 6 ++++++
> tools/libxl/libxl_colo.h | 5 +++++
> tools/libxl/libxl_colo_save.c | 2 ++
> tools/libxl/libxl_types.idl | 17 +++++++++--------
> tools/libxl/xl_cmdimpl.c | 13 ++++++++++++-
> tools/libxl/xl_cmdtable.c | 3 ++-
> 7 files changed, 41 insertions(+), 10 deletions(-)
>
> diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
> index 09c1faa..4260777 100644
> --- a/docs/man/xl.pod.1.in
> +++ b/docs/man/xl.pod.1.in
> @@ -553,6 +553,11 @@ Disable disk replication. Requires enabling unsafe mode.
> Enable COLO HA. This conflicts with B<-i> and B<-b>, and memory
> checkpoint compression must be disabled.
>
> +=item B<-p>
> +
> +Use userspace COLO Proxy. This option must be used in conjunction
> +with B<-c>.
> +
> =back
>
> =item B<pause> I<domain-id>
> diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
> index 3924464..fce7fab 100644
> --- a/tools/libxl/libxl.h
> +++ b/tools/libxl/libxl.h
> @@ -870,6 +870,12 @@ typedef struct libxl__ctx libxl_ctx;
> */
> #define LIBXL_HAVE_REMUS 1
>
> +/*
> + * LIBXL_HAVE_COLO_USERSPACE_PROXY
> + * If this is defined, then libxl supports COLO userspace proxy.
> + */
> +#define LIBXL_HAVE_COLO_USERSPACE_PROXY 1
> +
Please add this macro in a separate patch. It shall be the last patch of
this series. That way we don't break bisection for libxl users.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side
2017-02-17 2:18 ` [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side Zhang Chen
@ 2017-02-20 15:55 ` Wei Liu
2017-02-21 2:57 ` Zhang Chen
0 siblings, 1 reply; 23+ messages in thread
From: Wei Liu @ 2017-02-20 15:55 UTC (permalink / raw)
To: Zhang Chen
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang,
Xen devel, Bian Naimeng
On Fri, Feb 17, 2017 at 10:18:24AM +0800, Zhang Chen wrote:
> In this patch we close kernel COLO-Proxy on primary side.
>
> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> ---
> tools/libxl/libxl_colo_proxy.c | 27 +++++++++++++++++++++++++++
> tools/libxl/libxl_colo_save.c | 9 +++++++--
> 2 files changed, 34 insertions(+), 2 deletions(-)
>
> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
> index 0983f42..dd902fc 100644
> --- a/tools/libxl/libxl_colo_proxy.c
> +++ b/tools/libxl/libxl_colo_proxy.c
> @@ -152,6 +152,10 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
>
> STATE_AO_GC(cps->ao);
>
> + /* If enable userspace proxy mode, we don't need setup kernel proxy */
> + if (cps->is_userspace_proxy)
> + return 0;
> +
> skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
> if (skfd < 0) {
> LOGD(ERROR, ao->domid, "can not create a netlink socket: %s", strerror(errno));
> @@ -222,6 +226,13 @@ out:
>
> void colo_proxy_teardown(libxl__colo_proxy_state *cps)
> {
> + /*
> + * If enable userspace proxy mode,
> + * we don't need teardown kernel proxy
> + */
> + if (cps->is_userspace_proxy)
> + return;
> +
> if (cps->sock_fd >= 0) {
> close(cps->sock_fd);
> cps->sock_fd = -1;
> @@ -232,6 +243,13 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>
> void colo_proxy_preresume(libxl__colo_proxy_state *cps)
> {
> + /*
> + * If enable userspace proxy mode,
> + * we don't need preresume kernel proxy
> + */
> + if (cps->is_userspace_proxy)
> + return;
> +
> colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
> /* TODO: need to handle if the call fails... */
> }
> @@ -262,6 +280,15 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
>
> STATE_AO_GC(cps->ao);
>
> + /*
> + * enable userspace proxy mode, tmp sleep.
> + * then we will add qemu API support this func.
> + */
> + if (cps->is_userspace_proxy) {
> + sleep(timeout_us / 1000000);
usleep is better.
But in general I don't think sleeping in libxl is a good idea.
What is the reason that you need to sleep here?
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 5/7] COLO-Proxy: Add primary userspace colo proxy start args
2017-02-17 2:18 ` [PATCH V3 5/7] COLO-Proxy: Add primary userspace colo proxy start args Zhang Chen
@ 2017-02-20 15:59 ` Wei Liu
2017-02-21 3:07 ` Zhang Chen
0 siblings, 1 reply; 23+ messages in thread
From: Wei Liu @ 2017-02-20 15:59 UTC (permalink / raw)
To: Zhang Chen
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang,
Xen devel, Bian Naimeng
On Fri, Feb 17, 2017 at 10:18:27AM +0800, Zhang Chen wrote:
> Qemu need this args to start userspace colo-proxy.
>
> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
[...]
> +
> +#undef APPEND_COLO_SOCK_SERVER
> +#undef APPEND_COLO_SOCK_CLIENT
I would move these a bit further up, right after the last site that uses
them.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 7/7] COLO-Proxy: Use socket to get checkpoint event.
2017-02-17 2:18 ` [PATCH V3 7/7] COLO-Proxy: Use socket to get checkpoint event Zhang Chen
@ 2017-02-20 16:07 ` Wei Liu
2017-02-21 6:24 ` Zhang Chen
0 siblings, 1 reply; 23+ messages in thread
From: Wei Liu @ 2017-02-20 16:07 UTC (permalink / raw)
To: Zhang Chen
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang,
Xen devel, Bian Naimeng
On Fri, Feb 17, 2017 at 10:18:29AM +0800, Zhang Chen wrote:
> We use kernel colo proxy's way to get the checkpoint event
> from qemu colo-compare.
> Qemu colo-compare need add a API to support this(I will add this in qemu).
>
> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> ---
> tools/libxl/libxl_colo.h | 2 +
> tools/libxl/libxl_colo_proxy.c | 85 +++++++++++++++++++++++++++++++++++++---
> tools/libxl/libxl_colo_restore.c | 11 ++++--
> tools/libxl/libxl_colo_save.c | 22 +++++++----
> tools/libxl/libxl_nic.c | 4 ++
> tools/libxl/libxl_types.idl | 4 +-
> tools/libxl/xl_cmdimpl.c | 4 ++
> 7 files changed, 114 insertions(+), 18 deletions(-)
>
> diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
> index 4746d8c..6c01b55 100644
> --- a/tools/libxl/libxl_colo.h
> +++ b/tools/libxl/libxl_colo.h
> @@ -69,6 +69,8 @@ struct libxl__colo_proxy_state {
> * False means use kernel colo proxy.
> */
> bool is_userspace_proxy;
> + const char *checkpoint_host;
> + const char *checkpoint_port;
> };
>
> struct libxl__colo_save_state {
> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
> index dd902fc..a6436b0 100644
> --- a/tools/libxl/libxl_colo_proxy.c
> +++ b/tools/libxl/libxl_colo_proxy.c
> @@ -18,9 +18,13 @@
> #include "libxl_internal.h"
>
> #include <netlink/netlink.h>
> +#include <arpa/inet.h>
> +#include <sys/socket.h>
> +#include <netinet/in.h>
>
> /* Consistent with the new COLO netlink channel in kernel side */
> #define NETLINK_COLO 28
> +#define COLO_DEFAULT_WAIT_TIME 500000
>
> enum colo_netlink_op {
> COLO_QUERY_CHECKPOINT = (NLMSG_MIN_TYPE + 1),
> @@ -76,6 +80,26 @@ static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
> return ret;
> }
>
> +static int colo_userspace_proxy_recv(libxl__colo_proxy_state *cps,
> + char *buff,
> + unsigned int timeout_us)
> +{
> + struct timeval tv;
> + int ret;
> +
> + STATE_AO_GC(cps->ao);
> +
> + if (timeout_us) {
> + tv.tv_sec = timeout_us / 1000000;
> + tv.tv_usec = timeout_us % 1000000;
> + setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
Please check the return value of setsockopt and handle error
appropriately. If you don't care about the error, please add a comment.
> + }
> +
> + ret = recv(cps->sock_fd, buff, sizeof(buff),0);
> +
> + return ret;
> +}
> +
> /* error: return -1, otherwise return 0 */
> static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
> unsigned int timeout_us)
> @@ -153,8 +177,45 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
> STATE_AO_GC(cps->ao);
>
> /* If enable userspace proxy mode, we don't need setup kernel proxy */
> - if (cps->is_userspace_proxy)
> + if (cps->is_userspace_proxy) {
> + struct sockaddr_in addr;
> + int port;
> + char recvbuff[1024];
> + char sendbuf[] = "COLO_USERSPACE_PROXY_INIT";
> +
> + memset(&addr, 0, sizeof(addr));
> + port = atoi(cps->checkpoint_port);
> + addr.sin_family = AF_INET;
> + addr.sin_port = htons(port);
> + addr.sin_addr.s_addr = inet_addr(cps->checkpoint_host);
> +
> + skfd = socket(AF_INET, SOCK_STREAM, 0);
> + if (skfd < 0) {
> + LOGD(ERROR, ao->domid, "can not create a TCP socket: %s",
> + strerror(errno));
> + goto out;
> + }
> +
> + cps->sock_fd = skfd;
> +
> + if (connect(skfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
> + LOGD(ERROR, ao->domid, "connect error");
> + goto out;
> + }
> +
> + ret = send(skfd, sendbuf, strlen(sendbuf),0);
> + if (ret < 0)
> + goto out;
> +
> + ret = colo_userspace_proxy_recv(cps, recvbuff, COLO_DEFAULT_WAIT_TIME);
> + if (ret < 0) {
> + LOGD(ERROR, ao->domid, "Can't recv msg from qemu colo-compare: %s",
> + strerror(errno));
> + goto out;
> + }
> +
> return 0;
> + }
>
> skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
> if (skfd < 0) {
> @@ -247,8 +308,11 @@ void colo_proxy_preresume(libxl__colo_proxy_state *cps)
> * If enable userspace proxy mode,
> * we don't need preresume kernel proxy
> */
> - if (cps->is_userspace_proxy)
> + if (cps->is_userspace_proxy) {
> + char sendbuf[] = "COLO_CHECKPOINT";
Please constify it.
> + send(cps->sock_fd, sendbuf, strlen(sendbuf),0);
Space before "0".
> return;
> + }
>
> colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
> /* TODO: need to handle if the call fails... */
> @@ -277,16 +341,25 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
> struct nlmsghdr *h;
> struct colo_msg *m;
> int ret = -1;
> + char recvbuff[1024];
>
> STATE_AO_GC(cps->ao);
>
> /*
> - * enable userspace proxy mode, tmp sleep.
> - * then we will add qemu API support this func.
> + * enable userspace proxy mode.
> + * Then we will add qemu API support for this func.
> */
> if (cps->is_userspace_proxy) {
> - sleep(timeout_us / 1000000);
> - return 0;
> + ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
> + if (ret <= 0)
> + return 0;
> +
> + if (!strcmp(recvbuff, "DO_CHECKPOINT")) {
> + return 1;
> + } else {
> + LOGD(ERROR, ao->domid, "receive qemu colo-compare checkpoint error");
> + return -1;
Please use goto style error handling if possible. Please refer to
CODING_STYLE file inside libxl directory.
> + }
> }
>
> size = colo_proxy_recv(cps, &buff, timeout_us);
> diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
> index c6d239a..065ea00 100644
> --- a/tools/libxl/libxl_colo_restore.c
> +++ b/tools/libxl/libxl_colo_restore.c
> @@ -613,7 +613,8 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
> }
> }
>
> - colo_proxy_preresume(&crs->cps);
> + if (!crs->cps.is_userspace_proxy)
> + colo_proxy_preresume(&crs->cps);
>
> colo_restore_resume_vm(egc, crcs);
>
> @@ -786,9 +787,11 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
> cds->ops = colo_restore_ops;
>
> crs->cps.ao = ao;
> - if (colo_proxy_setup(&crs->cps)) {
> - LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
> - goto out;
> + if (!crs->cps.is_userspace_proxy) {
> + if (colo_proxy_setup(&crs->cps)) {
> + LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
> + goto out;
> + }
> }
>
> if (init_device_subkind(cds))
> diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
> index 91e3fce..a0cfc5a 100644
> --- a/tools/libxl/libxl_colo_save.c
> +++ b/tools/libxl/libxl_colo_save.c
> @@ -86,6 +86,7 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
> libxl__checkpoint_devices_state *const cds = &dss->cds;
> libxl__srm_save_autogen_callbacks *const callbacks =
> &dss->sws.shs.callbacks.save.a;
> + libxl_device_nic *nics;
>
> STATE_AO_GC(dss->ao);
>
> @@ -110,24 +111,31 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
> css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
> libxl__xen_script_dir_path());
>
> - /* If enable userspace proxy mode, we don't need VIF */
> - if (css->cps.is_userspace_proxy)
> - cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
> - else
> - cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
> - (1 << LIBXL__DEVICE_KIND_VBD);
> -
> cds->ops = colo_ops;
> cds->callback = colo_save_setup_done;
> cds->ao = ao;
> cds->domid = dss->domid;
> cds->concrete_data = css;
>
> + /* If enable userspace proxy mode, we don't need VIF */
> + if (css->cps.is_userspace_proxy) {
> + cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
> +
> + /* Use this args we can connect to qemu colo-compare */
> + nics = libxl_device_nic_list(CTX, cds->domid, &cds->num_nics);
> + css->cps.checkpoint_host = nics->colo_checkpoint_host;
> + css->cps.checkpoint_port = nics->colo_checkpoint_port;
> + } else {
> + cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
> + (1 << LIBXL__DEVICE_KIND_VBD);
> + }
> +
> css->srs.ao = ao;
> css->srs.fd = css->recv_fd;
> css->srs.back_channel = true;
> libxl__stream_read_start(egc, &css->srs);
> css->cps.ao = ao;
> +
Stray blank line.
Wei.
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 3/7] tools/libxl: refactor do_domain_create()
2017-02-20 15:50 ` Wei Liu
@ 2017-02-21 2:48 ` Zhang Chen
0 siblings, 0 replies; 23+ messages in thread
From: Zhang Chen @ 2017-02-21 2:48 UTC (permalink / raw)
To: Wei Liu
Cc: Li Zhijian, eddie, Ian Jackson, Yang Hongyang, Xen devel, Bian Naimeng
On 02/20/2017 11:50 PM, Wei Liu wrote:
> On Fri, Feb 17, 2017 at 10:18:25AM +0800, Zhang Chen wrote:
>> We use params->colo_proxy_script to make do_domain_create()
>> doesn't take "colo_proxy_script" anymore.
>>
>> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
>> ---
> This patch is self-contained so:
>
> Acked-by: Wei Liu <wei.liu2@citrix.com>
>
> I will commit this soon. Please rebase your next version against
> staging.
OK
Thanks
Zhang Chen
>
> Wei.
>
>
> .
>
--
Thanks
Zhang Chen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 1/7] COLO-Proxy: Add remus command to open userspace proxy
2017-02-20 15:50 ` Wei Liu
@ 2017-02-21 2:49 ` Zhang Chen
0 siblings, 0 replies; 23+ messages in thread
From: Zhang Chen @ 2017-02-21 2:49 UTC (permalink / raw)
To: Wei Liu
Cc: Li Zhijian, eddie, Ian Jackson, Yang Hongyang, Xen devel, Bian Naimeng
On 02/20/2017 11:50 PM, Wei Liu wrote:
> On Fri, Feb 17, 2017 at 10:18:23AM +0800, Zhang Chen wrote:
>> Add remus '-p' to enable userspace colo proxy(in qemu).
>>
>> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
>> ---
>> docs/man/xl.pod.1.in | 5 +++++
>> tools/libxl/libxl.h | 6 ++++++
>> tools/libxl/libxl_colo.h | 5 +++++
>> tools/libxl/libxl_colo_save.c | 2 ++
>> tools/libxl/libxl_types.idl | 17 +++++++++--------
>> tools/libxl/xl_cmdimpl.c | 13 ++++++++++++-
>> tools/libxl/xl_cmdtable.c | 3 ++-
>> 7 files changed, 41 insertions(+), 10 deletions(-)
>>
>> diff --git a/docs/man/xl.pod.1.in b/docs/man/xl.pod.1.in
>> index 09c1faa..4260777 100644
>> --- a/docs/man/xl.pod.1.in
>> +++ b/docs/man/xl.pod.1.in
>> @@ -553,6 +553,11 @@ Disable disk replication. Requires enabling unsafe mode.
>> Enable COLO HA. This conflicts with B<-i> and B<-b>, and memory
>> checkpoint compression must be disabled.
>>
>> +=item B<-p>
>> +
>> +Use userspace COLO Proxy. This option must be used in conjunction
>> +with B<-c>.
>> +
>> =back
>>
>> =item B<pause> I<domain-id>
>> diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
>> index 3924464..fce7fab 100644
>> --- a/tools/libxl/libxl.h
>> +++ b/tools/libxl/libxl.h
>> @@ -870,6 +870,12 @@ typedef struct libxl__ctx libxl_ctx;
>> */
>> #define LIBXL_HAVE_REMUS 1
>>
>> +/*
>> + * LIBXL_HAVE_COLO_USERSPACE_PROXY
>> + * If this is defined, then libxl supports COLO userspace proxy.
>> + */
>> +#define LIBXL_HAVE_COLO_USERSPACE_PROXY 1
>> +
> Please add this macro in a separate patch. It shall be the last patch of
> this series. That way we don't break bisection for libxl users.
I got your point.
Thanks
Zhang Chen
>
> Wei.
>
>
> .
>
--
Thanks
Zhang Chen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side
2017-02-20 15:55 ` Wei Liu
@ 2017-02-21 2:57 ` Zhang Chen
2017-02-21 9:53 ` Wei Liu
0 siblings, 1 reply; 23+ messages in thread
From: Zhang Chen @ 2017-02-21 2:57 UTC (permalink / raw)
To: Wei Liu
Cc: Li Zhijian, eddie, Ian Jackson, Yang Hongyang, Xen devel, Bian Naimeng
On 02/20/2017 11:55 PM, Wei Liu wrote:
> On Fri, Feb 17, 2017 at 10:18:24AM +0800, Zhang Chen wrote:
>> In this patch we close kernel COLO-Proxy on primary side.
>>
>> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
>> ---
>> tools/libxl/libxl_colo_proxy.c | 27 +++++++++++++++++++++++++++
>> tools/libxl/libxl_colo_save.c | 9 +++++++--
>> 2 files changed, 34 insertions(+), 2 deletions(-)
>>
>> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
>> index 0983f42..dd902fc 100644
>> --- a/tools/libxl/libxl_colo_proxy.c
>> +++ b/tools/libxl/libxl_colo_proxy.c
>> @@ -152,6 +152,10 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
>>
>> STATE_AO_GC(cps->ao);
>>
>> + /* If enable userspace proxy mode, we don't need setup kernel proxy */
>> + if (cps->is_userspace_proxy)
>> + return 0;
>> +
>> skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
>> if (skfd < 0) {
>> LOGD(ERROR, ao->domid, "can not create a netlink socket: %s", strerror(errno));
>> @@ -222,6 +226,13 @@ out:
>>
>> void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>> {
>> + /*
>> + * If enable userspace proxy mode,
>> + * we don't need teardown kernel proxy
>> + */
>> + if (cps->is_userspace_proxy)
>> + return;
>> +
>> if (cps->sock_fd >= 0) {
>> close(cps->sock_fd);
>> cps->sock_fd = -1;
>> @@ -232,6 +243,13 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>>
>> void colo_proxy_preresume(libxl__colo_proxy_state *cps)
>> {
>> + /*
>> + * If enable userspace proxy mode,
>> + * we don't need preresume kernel proxy
>> + */
>> + if (cps->is_userspace_proxy)
>> + return;
>> +
>> colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
>> /* TODO: need to handle if the call fails... */
>> }
>> @@ -262,6 +280,15 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
>>
>> STATE_AO_GC(cps->ao);
>>
>> + /*
>> + * enable userspace proxy mode, tmp sleep.
>> + * then we will add qemu API support this func.
>> + */
>> + if (cps->is_userspace_proxy) {
>> + sleep(timeout_us / 1000000);
> usleep is better.
OK.
>
> But in general I don't think sleeping in libxl is a good idea.
> What is the reason that you need to sleep here?
In here we use this sleep to keep COLO period checkpoint,
We can not do checkpoint continuously, that will make performance poor.
After 7/7 we change this to
ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
Thanks
Zhang Chen
>
> Wei.
>
>
> .
>
--
Thanks
Zhang Chen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 5/7] COLO-Proxy: Add primary userspace colo proxy start args
2017-02-20 15:59 ` Wei Liu
@ 2017-02-21 3:07 ` Zhang Chen
2017-02-21 9:47 ` Wei Liu
0 siblings, 1 reply; 23+ messages in thread
From: Zhang Chen @ 2017-02-21 3:07 UTC (permalink / raw)
To: Wei Liu
Cc: Li Zhijian, eddie, Ian Jackson, Yang Hongyang, Xen devel, Bian Naimeng
On 02/20/2017 11:59 PM, Wei Liu wrote:
> On Fri, Feb 17, 2017 at 10:18:27AM +0800, Zhang Chen wrote:
>> Qemu need this args to start userspace colo-proxy.
>>
>> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> [...]
>> +
>> +#undef APPEND_COLO_SOCK_SERVER
>> +#undef APPEND_COLO_SOCK_CLIENT
> I would move these a bit further up, right after the last site that uses
> them.
How about move it to here?
} else {
if (!sdl && !vnc) {
flexarray_append(dm_args, "-nographic");
}
}
+#undef APPEND_COLO_SOCK_SERVER
+#undef APPEND_COLO_SOCK_CLIENT
if (state->saved_state) {
/* This file descriptor is meant to be used by QEMU */
*dm_state_fd = open(state->saved_state, O_RDONLY);
flexarray_append(dm_args, "-incoming");
flexarray_append(dm_args, GCSPRINTF("fd:%d",*dm_state_fd));
}
Thanks
Zhang Chen
>
> Wei.
>
>
> .
>
--
Thanks
Zhang Chen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 7/7] COLO-Proxy: Use socket to get checkpoint event.
2017-02-20 16:07 ` Wei Liu
@ 2017-02-21 6:24 ` Zhang Chen
0 siblings, 0 replies; 23+ messages in thread
From: Zhang Chen @ 2017-02-21 6:24 UTC (permalink / raw)
To: Wei Liu
Cc: zhangchen.fnst, Li Zhijian, eddie, Ian Jackson, Yang Hongyang,
Xen devel, Bian Naimeng
On 02/21/2017 12:07 AM, Wei Liu wrote:
> On Fri, Feb 17, 2017 at 10:18:29AM +0800, Zhang Chen wrote:
>> We use kernel colo proxy's way to get the checkpoint event
>> from qemu colo-compare.
>> Qemu colo-compare need add a API to support this(I will add this in qemu).
>>
>> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
>> ---
>> tools/libxl/libxl_colo.h | 2 +
>> tools/libxl/libxl_colo_proxy.c | 85 +++++++++++++++++++++++++++++++++++++---
>> tools/libxl/libxl_colo_restore.c | 11 ++++--
>> tools/libxl/libxl_colo_save.c | 22 +++++++----
>> tools/libxl/libxl_nic.c | 4 ++
>> tools/libxl/libxl_types.idl | 4 +-
>> tools/libxl/xl_cmdimpl.c | 4 ++
>> 7 files changed, 114 insertions(+), 18 deletions(-)
>>
>> diff --git a/tools/libxl/libxl_colo.h b/tools/libxl/libxl_colo.h
>> index 4746d8c..6c01b55 100644
>> --- a/tools/libxl/libxl_colo.h
>> +++ b/tools/libxl/libxl_colo.h
>> @@ -69,6 +69,8 @@ struct libxl__colo_proxy_state {
>> * False means use kernel colo proxy.
>> */
>> bool is_userspace_proxy;
>> + const char *checkpoint_host;
>> + const char *checkpoint_port;
>> };
>>
>> struct libxl__colo_save_state {
>> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
>> index dd902fc..a6436b0 100644
>> --- a/tools/libxl/libxl_colo_proxy.c
>> +++ b/tools/libxl/libxl_colo_proxy.c
>> @@ -18,9 +18,13 @@
>> #include "libxl_internal.h"
>>
>> #include <netlink/netlink.h>
>> +#include <arpa/inet.h>
>> +#include <sys/socket.h>
>> +#include <netinet/in.h>
>>
>> /* Consistent with the new COLO netlink channel in kernel side */
>> #define NETLINK_COLO 28
>> +#define COLO_DEFAULT_WAIT_TIME 500000
>>
>> enum colo_netlink_op {
>> COLO_QUERY_CHECKPOINT = (NLMSG_MIN_TYPE + 1),
>> @@ -76,6 +80,26 @@ static int colo_proxy_send(libxl__colo_proxy_state *cps, uint8_t *buff,
>> return ret;
>> }
>>
>> +static int colo_userspace_proxy_recv(libxl__colo_proxy_state *cps,
>> + char *buff,
>> + unsigned int timeout_us)
>> +{
>> + struct timeval tv;
>> + int ret;
>> +
>> + STATE_AO_GC(cps->ao);
>> +
>> + if (timeout_us) {
>> + tv.tv_sec = timeout_us / 1000000;
>> + tv.tv_usec = timeout_us % 1000000;
>> + setsockopt(cps->sock_fd, SOL_SOCKET, SO_RCVTIMEO, &tv, sizeof(tv));
> Please check the return value of setsockopt and handle error
> appropriately. If you don't care about the error, please add a comment.
OK.
>
>> + }
>> +
>> + ret = recv(cps->sock_fd, buff, sizeof(buff),0);
>> +
>> + return ret;
>> +}
>> +
>> /* error: return -1, otherwise return 0 */
>> static int64_t colo_proxy_recv(libxl__colo_proxy_state *cps, uint8_t **buff,
>> unsigned int timeout_us)
>> @@ -153,8 +177,45 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
>> STATE_AO_GC(cps->ao);
>>
>> /* If enable userspace proxy mode, we don't need setup kernel proxy */
>> - if (cps->is_userspace_proxy)
>> + if (cps->is_userspace_proxy) {
>> + struct sockaddr_in addr;
>> + int port;
>> + char recvbuff[1024];
>> + char sendbuf[] = "COLO_USERSPACE_PROXY_INIT";
>> +
>> + memset(&addr, 0, sizeof(addr));
>> + port = atoi(cps->checkpoint_port);
>> + addr.sin_family = AF_INET;
>> + addr.sin_port = htons(port);
>> + addr.sin_addr.s_addr = inet_addr(cps->checkpoint_host);
>> +
>> + skfd = socket(AF_INET, SOCK_STREAM, 0);
>> + if (skfd < 0) {
>> + LOGD(ERROR, ao->domid, "can not create a TCP socket: %s",
>> + strerror(errno));
>> + goto out;
>> + }
>> +
>> + cps->sock_fd = skfd;
>> +
>> + if (connect(skfd, (struct sockaddr *)&addr, sizeof(addr)) < 0) {
>> + LOGD(ERROR, ao->domid, "connect error");
>> + goto out;
>> + }
>> +
>> + ret = send(skfd, sendbuf, strlen(sendbuf),0);
>> + if (ret < 0)
>> + goto out;
>> +
>> + ret = colo_userspace_proxy_recv(cps, recvbuff, COLO_DEFAULT_WAIT_TIME);
>> + if (ret < 0) {
>> + LOGD(ERROR, ao->domid, "Can't recv msg from qemu colo-compare: %s",
>> + strerror(errno));
>> + goto out;
>> + }
>> +
>> return 0;
>> + }
>>
>> skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
>> if (skfd < 0) {
>> @@ -247,8 +308,11 @@ void colo_proxy_preresume(libxl__colo_proxy_state *cps)
>> * If enable userspace proxy mode,
>> * we don't need preresume kernel proxy
>> */
>> - if (cps->is_userspace_proxy)
>> + if (cps->is_userspace_proxy) {
>> + char sendbuf[] = "COLO_CHECKPOINT";
> Please constify it.
OK~~
>
>> + send(cps->sock_fd, sendbuf, strlen(sendbuf),0);
> Space before "0".
Good catch.
>
>> return;
>> + }
>>
>> colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
>> /* TODO: need to handle if the call fails... */
>> @@ -277,16 +341,25 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
>> struct nlmsghdr *h;
>> struct colo_msg *m;
>> int ret = -1;
>> + char recvbuff[1024];
>>
>> STATE_AO_GC(cps->ao);
>>
>> /*
>> - * enable userspace proxy mode, tmp sleep.
>> - * then we will add qemu API support this func.
>> + * enable userspace proxy mode.
>> + * Then we will add qemu API support for this func.
>> */
>> if (cps->is_userspace_proxy) {
>> - sleep(timeout_us / 1000000);
>> - return 0;
>> + ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
>> + if (ret <= 0)
>> + return 0;
>> +
>> + if (!strcmp(recvbuff, "DO_CHECKPOINT")) {
>> + return 1;
>> + } else {
>> + LOGD(ERROR, ao->domid, "receive qemu colo-compare checkpoint error");
>> + return -1;
> Please use goto style error handling if possible. Please refer to
> CODING_STYLE file inside libxl directory.
I will fix it in next version.
>
>> + }
>> }
>>
>> size = colo_proxy_recv(cps, &buff, timeout_us);
>> diff --git a/tools/libxl/libxl_colo_restore.c b/tools/libxl/libxl_colo_restore.c
>> index c6d239a..065ea00 100644
>> --- a/tools/libxl/libxl_colo_restore.c
>> +++ b/tools/libxl/libxl_colo_restore.c
>> @@ -613,7 +613,8 @@ static void colo_restore_preresume_cb(libxl__egc *egc,
>> }
>> }
>>
>> - colo_proxy_preresume(&crs->cps);
>> + if (!crs->cps.is_userspace_proxy)
>> + colo_proxy_preresume(&crs->cps);
>>
>> colo_restore_resume_vm(egc, crcs);
>>
>> @@ -786,9 +787,11 @@ static void colo_setup_checkpoint_devices(libxl__egc *egc,
>> cds->ops = colo_restore_ops;
>>
>> crs->cps.ao = ao;
>> - if (colo_proxy_setup(&crs->cps)) {
>> - LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
>> - goto out;
>> + if (!crs->cps.is_userspace_proxy) {
>> + if (colo_proxy_setup(&crs->cps)) {
>> + LOGD(ERROR, cds->domid, "COLO: failed to setup colo proxy for guest");
>> + goto out;
>> + }
>> }
>>
>> if (init_device_subkind(cds))
>> diff --git a/tools/libxl/libxl_colo_save.c b/tools/libxl/libxl_colo_save.c
>> index 91e3fce..a0cfc5a 100644
>> --- a/tools/libxl/libxl_colo_save.c
>> +++ b/tools/libxl/libxl_colo_save.c
>> @@ -86,6 +86,7 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
>> libxl__checkpoint_devices_state *const cds = &dss->cds;
>> libxl__srm_save_autogen_callbacks *const callbacks =
>> &dss->sws.shs.callbacks.save.a;
>> + libxl_device_nic *nics;
>>
>> STATE_AO_GC(dss->ao);
>>
>> @@ -110,24 +111,31 @@ void libxl__colo_save_setup(libxl__egc *egc, libxl__colo_save_state *css)
>> css->colo_proxy_script = GCSPRINTF("%s/colo-proxy-setup",
>> libxl__xen_script_dir_path());
>>
>> - /* If enable userspace proxy mode, we don't need VIF */
>> - if (css->cps.is_userspace_proxy)
>> - cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
>> - else
>> - cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
>> - (1 << LIBXL__DEVICE_KIND_VBD);
>> -
>> cds->ops = colo_ops;
>> cds->callback = colo_save_setup_done;
>> cds->ao = ao;
>> cds->domid = dss->domid;
>> cds->concrete_data = css;
>>
>> + /* If enable userspace proxy mode, we don't need VIF */
>> + if (css->cps.is_userspace_proxy) {
>> + cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VBD);
>> +
>> + /* Use this args we can connect to qemu colo-compare */
>> + nics = libxl_device_nic_list(CTX, cds->domid, &cds->num_nics);
>> + css->cps.checkpoint_host = nics->colo_checkpoint_host;
>> + css->cps.checkpoint_port = nics->colo_checkpoint_port;
>> + } else {
>> + cds->device_kind_flags = (1 << LIBXL__DEVICE_KIND_VIF) |
>> + (1 << LIBXL__DEVICE_KIND_VBD);
>> + }
>> +
>> css->srs.ao = ao;
>> css->srs.fd = css->recv_fd;
>> css->srs.back_channel = true;
>> libxl__stream_read_start(egc, &css->srs);
>> css->cps.ao = ao;
>> +
> Stray blank line.
I Will Remove it in next version.
Thanks
Zhang Chen
>
> Wei.
>
>
> .
>
--
Thanks
Zhang Chen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 5/7] COLO-Proxy: Add primary userspace colo proxy start args
2017-02-21 3:07 ` Zhang Chen
@ 2017-02-21 9:47 ` Wei Liu
0 siblings, 0 replies; 23+ messages in thread
From: Wei Liu @ 2017-02-21 9:47 UTC (permalink / raw)
To: Zhang Chen
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang,
Xen devel, Bian Naimeng
On Tue, Feb 21, 2017 at 11:07:00AM +0800, Zhang Chen wrote:
>
>
> On 02/20/2017 11:59 PM, Wei Liu wrote:
> > On Fri, Feb 17, 2017 at 10:18:27AM +0800, Zhang Chen wrote:
> > > Qemu need this args to start userspace colo-proxy.
> > >
> > > Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> > [...]
> > > +
> > > +#undef APPEND_COLO_SOCK_SERVER
> > > +#undef APPEND_COLO_SOCK_CLIENT
> > I would move these a bit further up, right after the last site that uses
> > them.
>
> How about move it to here?
>
> } else {
> if (!sdl && !vnc) {
> flexarray_append(dm_args, "-nographic");
> }
> }
>
> +#undef APPEND_COLO_SOCK_SERVER
> +#undef APPEND_COLO_SOCK_CLIENT
>
>
> if (state->saved_state) {
> /* This file descriptor is meant to be used by QEMU */
> *dm_state_fd = open(state->saved_state, O_RDONLY);
> flexarray_append(dm_args, "-incoming");
> flexarray_append(dm_args, GCSPRINTF("fd:%d",*dm_state_fd));
> }
>
I can't seem to find this context.
But, after thinking a bit more I think you can leave those in their
original position. That's just a trivial issue. I don't want to spend
too much time bikeshedding.
Wei.
>
> Thanks
> Zhang Chen
>
> >
> > Wei.
> >
> >
> > .
> >
>
> --
> Thanks
> Zhang Chen
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side
2017-02-21 2:57 ` Zhang Chen
@ 2017-02-21 9:53 ` Wei Liu
2017-02-21 11:03 ` Zhang Chen
0 siblings, 1 reply; 23+ messages in thread
From: Wei Liu @ 2017-02-21 9:53 UTC (permalink / raw)
To: Zhang Chen
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang,
Xen devel, Bian Naimeng
On Tue, Feb 21, 2017 at 10:57:46AM +0800, Zhang Chen wrote:
>
>
> On 02/20/2017 11:55 PM, Wei Liu wrote:
> > On Fri, Feb 17, 2017 at 10:18:24AM +0800, Zhang Chen wrote:
> > > In this patch we close kernel COLO-Proxy on primary side.
> > >
> > > Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> > > ---
> > > tools/libxl/libxl_colo_proxy.c | 27 +++++++++++++++++++++++++++
> > > tools/libxl/libxl_colo_save.c | 9 +++++++--
> > > 2 files changed, 34 insertions(+), 2 deletions(-)
> > >
> > > diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
> > > index 0983f42..dd902fc 100644
> > > --- a/tools/libxl/libxl_colo_proxy.c
> > > +++ b/tools/libxl/libxl_colo_proxy.c
> > > @@ -152,6 +152,10 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
> > > STATE_AO_GC(cps->ao);
> > > + /* If enable userspace proxy mode, we don't need setup kernel proxy */
> > > + if (cps->is_userspace_proxy)
> > > + return 0;
> > > +
> > > skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
> > > if (skfd < 0) {
> > > LOGD(ERROR, ao->domid, "can not create a netlink socket: %s", strerror(errno));
> > > @@ -222,6 +226,13 @@ out:
> > > void colo_proxy_teardown(libxl__colo_proxy_state *cps)
> > > {
> > > + /*
> > > + * If enable userspace proxy mode,
> > > + * we don't need teardown kernel proxy
> > > + */
> > > + if (cps->is_userspace_proxy)
> > > + return;
> > > +
> > > if (cps->sock_fd >= 0) {
> > > close(cps->sock_fd);
> > > cps->sock_fd = -1;
> > > @@ -232,6 +243,13 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
> > > void colo_proxy_preresume(libxl__colo_proxy_state *cps)
> > > {
> > > + /*
> > > + * If enable userspace proxy mode,
> > > + * we don't need preresume kernel proxy
> > > + */
> > > + if (cps->is_userspace_proxy)
> > > + return;
> > > +
> > > colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
> > > /* TODO: need to handle if the call fails... */
> > > }
> > > @@ -262,6 +280,15 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
> > > STATE_AO_GC(cps->ao);
> > > + /*
> > > + * enable userspace proxy mode, tmp sleep.
> > > + * then we will add qemu API support this func.
> > > + */
> > > + if (cps->is_userspace_proxy) {
> > > + sleep(timeout_us / 1000000);
> > usleep is better.
>
> OK.
>
> >
> > But in general I don't think sleeping in libxl is a good idea.
> > What is the reason that you need to sleep here?
>
> In here we use this sleep to keep COLO period checkpoint,
> We can not do checkpoint continuously, that will make performance poor.
> After 7/7 we change this to
> ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
>
I don't fully understand. Say, if I just use COLO with this sleep, will
it work?
> Thanks
> Zhang Chen
>
> >
> > Wei.
> >
> >
> > .
> >
>
> --
> Thanks
> Zhang Chen
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side
2017-02-21 9:53 ` Wei Liu
@ 2017-02-21 11:03 ` Zhang Chen
2017-02-21 11:18 ` Wei Liu
0 siblings, 1 reply; 23+ messages in thread
From: Zhang Chen @ 2017-02-21 11:03 UTC (permalink / raw)
To: Wei Liu
Cc: zhangchen.fnst, Li Zhijian, eddie, Ian Jackson, Yang Hongyang,
Xen devel, Bian Naimeng
On 02/21/2017 05:53 PM, Wei Liu wrote:
> On Tue, Feb 21, 2017 at 10:57:46AM +0800, Zhang Chen wrote:
>>
>> On 02/20/2017 11:55 PM, Wei Liu wrote:
>>> On Fri, Feb 17, 2017 at 10:18:24AM +0800, Zhang Chen wrote:
>>>> In this patch we close kernel COLO-Proxy on primary side.
>>>>
>>>> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
>>>> ---
>>>> tools/libxl/libxl_colo_proxy.c | 27 +++++++++++++++++++++++++++
>>>> tools/libxl/libxl_colo_save.c | 9 +++++++--
>>>> 2 files changed, 34 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
>>>> index 0983f42..dd902fc 100644
>>>> --- a/tools/libxl/libxl_colo_proxy.c
>>>> +++ b/tools/libxl/libxl_colo_proxy.c
>>>> @@ -152,6 +152,10 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
>>>> STATE_AO_GC(cps->ao);
>>>> + /* If enable userspace proxy mode, we don't need setup kernel proxy */
>>>> + if (cps->is_userspace_proxy)
>>>> + return 0;
>>>> +
>>>> skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
>>>> if (skfd < 0) {
>>>> LOGD(ERROR, ao->domid, "can not create a netlink socket: %s", strerror(errno));
>>>> @@ -222,6 +226,13 @@ out:
>>>> void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>>>> {
>>>> + /*
>>>> + * If enable userspace proxy mode,
>>>> + * we don't need teardown kernel proxy
>>>> + */
>>>> + if (cps->is_userspace_proxy)
>>>> + return;
>>>> +
>>>> if (cps->sock_fd >= 0) {
>>>> close(cps->sock_fd);
>>>> cps->sock_fd = -1;
>>>> @@ -232,6 +243,13 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>>>> void colo_proxy_preresume(libxl__colo_proxy_state *cps)
>>>> {
>>>> + /*
>>>> + * If enable userspace proxy mode,
>>>> + * we don't need preresume kernel proxy
>>>> + */
>>>> + if (cps->is_userspace_proxy)
>>>> + return;
>>>> +
>>>> colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
>>>> /* TODO: need to handle if the call fails... */
>>>> }
>>>> @@ -262,6 +280,15 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
>>>> STATE_AO_GC(cps->ao);
>>>> + /*
>>>> + * enable userspace proxy mode, tmp sleep.
>>>> + * then we will add qemu API support this func.
>>>> + */
>>>> + if (cps->is_userspace_proxy) {
>>>> + sleep(timeout_us / 1000000);
>>> usleep is better.
>> OK.
>>
>>> But in general I don't think sleeping in libxl is a good idea.
>>> What is the reason that you need to sleep here?
>> In here we use this sleep to keep COLO period checkpoint,
>> We can not do checkpoint continuously, that will make performance poor.
>> After 7/7 we change this to
>> ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
>>
> I don't fully understand. Say, if I just use COLO with this sleep, will
> it work?
Yes, but COLO just have period checkpoint.
for example, in this mode, we set 10 sec to do one checkpoint,
the guest's net packet will be buffered 0-10 sec before send to
client.because we must ensure primary guest status same with secondary
guest(need do a checkpoint), When we have colo-compare, if primary
guest's packet same
with secondary guest's packet we send packet to client immediately,
else have different packet we alse send packet to client immediately and
with a checkpoint notify to colo-frame. That's make colo performance greater
than period mode.
Thanks
Zhang Chen
>
>> Thanks
>> Zhang Chen
>>
>>> Wei.
>>>
>>>
>>> .
>>>
>> --
>> Thanks
>> Zhang Chen
>>
>>
>>
>
> .
>
--
Thanks
Zhang Chen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side
2017-02-21 11:03 ` Zhang Chen
@ 2017-02-21 11:18 ` Wei Liu
2017-02-21 13:12 ` Zhang Chen
0 siblings, 1 reply; 23+ messages in thread
From: Wei Liu @ 2017-02-21 11:18 UTC (permalink / raw)
To: Zhang Chen
Cc: Wei Liu, Li Zhijian, Ian Jackson, eddie, Yang Hongyang,
Xen devel, Bian Naimeng
On Tue, Feb 21, 2017 at 07:03:29PM +0800, Zhang Chen wrote:
>
>
> On 02/21/2017 05:53 PM, Wei Liu wrote:
> > On Tue, Feb 21, 2017 at 10:57:46AM +0800, Zhang Chen wrote:
> > >
> > > On 02/20/2017 11:55 PM, Wei Liu wrote:
> > > > On Fri, Feb 17, 2017 at 10:18:24AM +0800, Zhang Chen wrote:
> > > > > In this patch we close kernel COLO-Proxy on primary side.
> > > > >
> > > > > Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
> > > > > ---
> > > > > tools/libxl/libxl_colo_proxy.c | 27 +++++++++++++++++++++++++++
> > > > > tools/libxl/libxl_colo_save.c | 9 +++++++--
> > > > > 2 files changed, 34 insertions(+), 2 deletions(-)
> > > > >
> > > > > diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
> > > > > index 0983f42..dd902fc 100644
> > > > > --- a/tools/libxl/libxl_colo_proxy.c
> > > > > +++ b/tools/libxl/libxl_colo_proxy.c
> > > > > @@ -152,6 +152,10 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
> > > > > STATE_AO_GC(cps->ao);
> > > > > + /* If enable userspace proxy mode, we don't need setup kernel proxy */
> > > > > + if (cps->is_userspace_proxy)
> > > > > + return 0;
> > > > > +
> > > > > skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
> > > > > if (skfd < 0) {
> > > > > LOGD(ERROR, ao->domid, "can not create a netlink socket: %s", strerror(errno));
> > > > > @@ -222,6 +226,13 @@ out:
> > > > > void colo_proxy_teardown(libxl__colo_proxy_state *cps)
> > > > > {
> > > > > + /*
> > > > > + * If enable userspace proxy mode,
> > > > > + * we don't need teardown kernel proxy
> > > > > + */
> > > > > + if (cps->is_userspace_proxy)
> > > > > + return;
> > > > > +
> > > > > if (cps->sock_fd >= 0) {
> > > > > close(cps->sock_fd);
> > > > > cps->sock_fd = -1;
> > > > > @@ -232,6 +243,13 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
> > > > > void colo_proxy_preresume(libxl__colo_proxy_state *cps)
> > > > > {
> > > > > + /*
> > > > > + * If enable userspace proxy mode,
> > > > > + * we don't need preresume kernel proxy
> > > > > + */
> > > > > + if (cps->is_userspace_proxy)
> > > > > + return;
> > > > > +
> > > > > colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
> > > > > /* TODO: need to handle if the call fails... */
> > > > > }
> > > > > @@ -262,6 +280,15 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
> > > > > STATE_AO_GC(cps->ao);
> > > > > + /*
> > > > > + * enable userspace proxy mode, tmp sleep.
> > > > > + * then we will add qemu API support this func.
> > > > > + */
> > > > > + if (cps->is_userspace_proxy) {
> > > > > + sleep(timeout_us / 1000000);
> > > > usleep is better.
> > > OK.
> > >
> > > > But in general I don't think sleeping in libxl is a good idea.
> > > > What is the reason that you need to sleep here?
> > > In here we use this sleep to keep COLO period checkpoint,
> > > We can not do checkpoint continuously, that will make performance poor.
> > > After 7/7 we change this to
> > > ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
> > >
> > I don't fully understand. Say, if I just use COLO with this sleep, will
> > it work?
>
> Yes, but COLO just have period checkpoint.
> for example, in this mode, we set 10 sec to do one checkpoint,
> the guest's net packet will be buffered 0-10 sec before send to
> client.because we must ensure primary guest status same with secondary
> guest(need do a checkpoint), When we have colo-compare, if primary guest's
> packet same
> with secondary guest's packet we send packet to client immediately,
> else have different packet we alse send packet to client immediately and
> with a checkpoint notify to colo-frame. That's make colo performance greater
> than period mode.
>
OK, in short, this is for periodical checkpoint mode. I still think it
is a bad idea to invoke (u)sleep in libxl, but since you're going to
remove it anyway, there is no point in making things more complicated
than necessary.
But I want to suggest you improve the comment a bit.
1. Properly capitalise the sentence.
2. Replace "tmp sleep" with "sleeping temporarily for XXX mode".
3. Clarify the QEMU API part (why is that relevant to the sleep or
whatever comes next).
I don't like to nit-pick too much about language, but keep in mind that
other people may want to contribute to COLO project so it is important
to have comments as accurate and clear as possible.
Wei.
>
> Thanks
> Zhang Chen
>
> >
> > > Thanks
> > > Zhang Chen
> > >
> > > > Wei.
> > > >
> > > >
> > > > .
> > > >
> > > --
> > > Thanks
> > > Zhang Chen
> > >
> > >
> > >
> >
> > .
> >
>
> --
> Thanks
> Zhang Chen
>
>
>
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
* Re: [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side
2017-02-21 11:18 ` Wei Liu
@ 2017-02-21 13:12 ` Zhang Chen
0 siblings, 0 replies; 23+ messages in thread
From: Zhang Chen @ 2017-02-21 13:12 UTC (permalink / raw)
To: Wei Liu
Cc: zhangchen.fnst, Li Zhijian, eddie, Ian Jackson, Yang Hongyang,
Xen devel, Bian Naimeng
On 02/21/2017 07:18 PM, Wei Liu wrote:
> On Tue, Feb 21, 2017 at 07:03:29PM +0800, Zhang Chen wrote:
>>
>> On 02/21/2017 05:53 PM, Wei Liu wrote:
>>> On Tue, Feb 21, 2017 at 10:57:46AM +0800, Zhang Chen wrote:
>>>> On 02/20/2017 11:55 PM, Wei Liu wrote:
>>>>> On Fri, Feb 17, 2017 at 10:18:24AM +0800, Zhang Chen wrote:
>>>>>> In this patch we close kernel COLO-Proxy on primary side.
>>>>>>
>>>>>> Signed-off-by: Zhang Chen <zhangchen.fnst@cn.fujitsu.com>
>>>>>> ---
>>>>>> tools/libxl/libxl_colo_proxy.c | 27 +++++++++++++++++++++++++++
>>>>>> tools/libxl/libxl_colo_save.c | 9 +++++++--
>>>>>> 2 files changed, 34 insertions(+), 2 deletions(-)
>>>>>>
>>>>>> diff --git a/tools/libxl/libxl_colo_proxy.c b/tools/libxl/libxl_colo_proxy.c
>>>>>> index 0983f42..dd902fc 100644
>>>>>> --- a/tools/libxl/libxl_colo_proxy.c
>>>>>> +++ b/tools/libxl/libxl_colo_proxy.c
>>>>>> @@ -152,6 +152,10 @@ int colo_proxy_setup(libxl__colo_proxy_state *cps)
>>>>>> STATE_AO_GC(cps->ao);
>>>>>> + /* If enable userspace proxy mode, we don't need setup kernel proxy */
>>>>>> + if (cps->is_userspace_proxy)
>>>>>> + return 0;
>>>>>> +
>>>>>> skfd = socket(PF_NETLINK, SOCK_RAW, NETLINK_COLO);
>>>>>> if (skfd < 0) {
>>>>>> LOGD(ERROR, ao->domid, "can not create a netlink socket: %s", strerror(errno));
>>>>>> @@ -222,6 +226,13 @@ out:
>>>>>> void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>>>>>> {
>>>>>> + /*
>>>>>> + * If enable userspace proxy mode,
>>>>>> + * we don't need teardown kernel proxy
>>>>>> + */
>>>>>> + if (cps->is_userspace_proxy)
>>>>>> + return;
>>>>>> +
>>>>>> if (cps->sock_fd >= 0) {
>>>>>> close(cps->sock_fd);
>>>>>> cps->sock_fd = -1;
>>>>>> @@ -232,6 +243,13 @@ void colo_proxy_teardown(libxl__colo_proxy_state *cps)
>>>>>> void colo_proxy_preresume(libxl__colo_proxy_state *cps)
>>>>>> {
>>>>>> + /*
>>>>>> + * If enable userspace proxy mode,
>>>>>> + * we don't need preresume kernel proxy
>>>>>> + */
>>>>>> + if (cps->is_userspace_proxy)
>>>>>> + return;
>>>>>> +
>>>>>> colo_proxy_send(cps, NULL, 0, COLO_CHECKPOINT);
>>>>>> /* TODO: need to handle if the call fails... */
>>>>>> }
>>>>>> @@ -262,6 +280,15 @@ int colo_proxy_checkpoint(libxl__colo_proxy_state *cps,
>>>>>> STATE_AO_GC(cps->ao);
>>>>>> + /*
>>>>>> + * enable userspace proxy mode, tmp sleep.
>>>>>> + * then we will add qemu API support this func.
>>>>>> + */
>>>>>> + if (cps->is_userspace_proxy) {
>>>>>> + sleep(timeout_us / 1000000);
>>>>> usleep is better.
>>>> OK.
>>>>
>>>>> But in general I don't think sleeping in libxl is a good idea.
>>>>> What is the reason that you need to sleep here?
>>>> In here we use this sleep to keep COLO period checkpoint,
>>>> We can not do checkpoint continuously, that will make performance poor.
>>>> After 7/7 we change this to
>>>> ret = colo_userspace_proxy_recv(cps, recvbuff, timeout_us);
>>>>
>>> I don't fully understand. Say, if I just use COLO with this sleep, will
>>> it work?
>> Yes, but COLO just have period checkpoint.
>> for example, in this mode, we set 10 sec to do one checkpoint,
>> the guest's net packet will be buffered 0-10 sec before send to
>> client.because we must ensure primary guest status same with secondary
>> guest(need do a checkpoint), When we have colo-compare, if primary guest's
>> packet same
>> with secondary guest's packet we send packet to client immediately,
>> else have different packet we alse send packet to client immediately and
>> with a checkpoint notify to colo-frame. That's make colo performance greater
>> than period mode.
>>
> OK, in short, this is for periodical checkpoint mode. I still think it
> is a bad idea to invoke (u)sleep in libxl, but since you're going to
> remove it anyway, there is no point in making things more complicated
> than necessary.
>
> But I want to suggest you improve the comment a bit.
>
> 1. Properly capitalise the sentence.
> 2. Replace "tmp sleep" with "sleeping temporarily for XXX mode".
> 3. Clarify the QEMU API part (why is that relevant to the sleep or
> whatever comes next).
>
> I don't like to nit-pick too much about language, but keep in mind that
> other people may want to contribute to COLO project so it is important
> to have comments as accurate and clear as possible.
Make sense. I will add more comments in next version.
Thanks
Zhang Chen
>
> Wei.
>
>> Thanks
>> Zhang Chen
>>
>>>> Thanks
>>>> Zhang Chen
>>>>
>>>>> Wei.
>>>>>
>>>>>
>>>>> .
>>>>>
>>>> --
>>>> Thanks
>>>> Zhang Chen
>>>>
>>>>
>>>>
>>> .
>>>
>> --
>> Thanks
>> Zhang Chen
>>
>>
>>
>
> .
>
--
Thanks
Zhang Chen
_______________________________________________
Xen-devel mailing list
Xen-devel@lists.xen.org
https://lists.xen.org/xen-devel
^ permalink raw reply [flat|nested] 23+ messages in thread
end of thread, other threads:[~2017-02-21 13:13 UTC | newest]
Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-02-17 2:18 [PATCH V3 0/7] COLO-Proxy: Make Xen COLO use userspace colo-proxy Zhang Chen
2017-02-17 2:18 ` [PATCH V3 1/7] COLO-Proxy: Add remus command to open userspace proxy Zhang Chen
2017-02-20 15:50 ` Wei Liu
2017-02-21 2:49 ` Zhang Chen
2017-02-17 2:18 ` [PATCH V3 2/7] COLO-Proxy: Setup userspace colo-proxy on primary side Zhang Chen
2017-02-20 15:55 ` Wei Liu
2017-02-21 2:57 ` Zhang Chen
2017-02-21 9:53 ` Wei Liu
2017-02-21 11:03 ` Zhang Chen
2017-02-21 11:18 ` Wei Liu
2017-02-21 13:12 ` Zhang Chen
2017-02-17 2:18 ` [PATCH V3 3/7] tools/libxl: refactor do_domain_create() Zhang Chen
2017-02-20 15:50 ` Wei Liu
2017-02-21 2:48 ` Zhang Chen
2017-02-17 2:18 ` [PATCH V3 4/7] COLO-Proxy: Setup userspace colo-proxy on secondary side Zhang Chen
2017-02-17 2:18 ` [PATCH V3 5/7] COLO-Proxy: Add primary userspace colo proxy start args Zhang Chen
2017-02-20 15:59 ` Wei Liu
2017-02-21 3:07 ` Zhang Chen
2017-02-21 9:47 ` Wei Liu
2017-02-17 2:18 ` [PATCH V3 6/7] COLO-Proxy: Add secondary userspace colo-proxy " Zhang Chen
2017-02-17 2:18 ` [PATCH V3 7/7] COLO-Proxy: Use socket to get checkpoint event Zhang Chen
2017-02-20 16:07 ` Wei Liu
2017-02-21 6:24 ` Zhang Chen
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.