* [PATCH v4 01/29] bsd-sys-queue-h-seddery: Massage `offsetof'
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 02/29] tools/libxc: Always compile the compat qemu variables into xc_sr_context Andrew Cooper
` (28 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Ian Jackson
From: Ian Jackson <Ian.Jackson@eu.citrix.com>
For some reason BSD's queue.h uses `__offsetof'. It expects it to
work just like offsetof. So use offsetof.
Reported-by: Andrew Cooper <andrew.cooper3@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
---
tools/include/xen-external/bsd-sys-queue-h-seddery | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tools/include/xen-external/bsd-sys-queue-h-seddery b/tools/include/xen-external/bsd-sys-queue-h-seddery
index 7a957e3..3f8716d 100755
--- a/tools/include/xen-external/bsd-sys-queue-h-seddery
+++ b/tools/include/xen-external/bsd-sys-queue-h-seddery
@@ -69,4 +69,6 @@ s/\b struct \s+ type \b/type/xg;
s,^\#include.*sys/cdefs.*,/* $& */,xg;
+s,\b __offsetof \b ,offsetof,xg;
+
s/\b( NULL )/0/xg;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 02/29] tools/libxc: Always compile the compat qemu variables into xc_sr_context
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 01/29] bsd-sys-queue-h-seddery: Massage `offsetof' Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 03/29] tools/libxl: Introduce ROUNDUP() Andrew Cooper
` (27 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
This is safe (as the variables will simply be unused), and is required
for correct compilation when midway through untangling the libxc/libxl
interaction.
The #define is left in place to highlight that the variables can be
removed once the untangling is complete.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/xc_sr_common.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 565c5da..08c66db 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -307,10 +307,10 @@ struct xc_sr_context
void *context;
size_t contextsz;
-#ifdef XG_LIBXL_HVM_COMPAT
+/* #ifdef XG_LIBXL_HVM_COMPAT */
uint32_t qlen;
void *qbuf;
-#endif
+/* #endif */
} restore;
};
} x86_hvm;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 03/29] tools/libxl: Introduce ROUNDUP()
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 01/29] bsd-sys-queue-h-seddery: Massage `offsetof' Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 02/29] tools/libxc: Always compile the compat qemu variables into xc_sr_context Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 04/29] tools/libxl: Introduce libxl__kill() Andrew Cooper
` (26 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
This is the same as is used by libxc.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl_internal.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 5235d25..19fc425 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -110,6 +110,9 @@
#define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0]))
+#define ROUNDUP(_val, _order) \
+ (((unsigned long)(_val)+(1UL<<(_order))-1) & ~((1UL<<(_order))-1))
+
#define min(X, Y) ({ \
const typeof (X) _x = (X); \
const typeof (Y) _y = (Y); \
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 04/29] tools/libxl: Introduce libxl__kill()
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (2 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 03/29] tools/libxl: Introduce ROUNDUP() Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 05/29] tools/libxl: Stash all restore parameters in domain_create_state Andrew Cooper
` (25 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson
as a wrapper to kill(2), and use it in preference to sendsig in
libxl_save_callout.c.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
v3: Fix typo, add _hidden
Logically new in v2 - split out from a v1 change which was itself a
cherrypick-and-modify from the AO Abort series
---
tools/libxl/libxl_aoutils.c | 15 +++++++++++++++
tools/libxl/libxl_internal.h | 2 ++
tools/libxl/libxl_save_callout.c | 10 ++--------
3 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
index 0931eee..274ef39 100644
--- a/tools/libxl/libxl_aoutils.c
+++ b/tools/libxl/libxl_aoutils.c
@@ -621,3 +621,18 @@ bool libxl__async_exec_inuse(const libxl__async_exec_state *aes)
assert(time_inuse == child_inuse);
return child_inuse;
}
+
+void libxl__kill(libxl__gc *gc, pid_t pid, int sig, const char *what)
+{
+ int r = kill(pid, sig);
+ if (r) LOGE(WARN, "failed to kill() %s [%lu] (signal %d)",
+ what, (unsigned long)pid, sig);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 19fc425..7ccbf55 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2244,6 +2244,8 @@ struct libxl__async_exec_state {
int libxl__async_exec_start(libxl__async_exec_state *aes);
bool libxl__async_exec_inuse(const libxl__async_exec_state *aes);
+_hidden void libxl__kill(libxl__gc *gc, pid_t pid, int sig, const char *what);
+
/*----- device addition/removal -----*/
typedef struct libxl__ao_device libxl__ao_device;
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 087c2d5..b82a5c1 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -244,12 +244,6 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
libxl__carefd_close(childs_pipes[1]);
helper_failed(egc, shs, rc);;
}
-static void sendsig(libxl__gc *gc, libxl__save_helper_state *shs, int sig)
-{
- int r = kill(shs->child.pid, sig);
- if (r) LOGE(WARN, "failed to kill save/restore helper [%lu] (signal %d)",
- (unsigned long)shs->child.pid, sig);
-}
static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs,
int rc)
@@ -266,7 +260,7 @@ static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs,
return;
}
- sendsig(gc, shs, SIGKILL);
+ libxl__kill(gc, shs->child.pid, SIGKILL, "save/restore helper");
}
static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
@@ -282,7 +276,7 @@ static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
if (!shs->rc)
shs->rc = rc;
- sendsig(gc, shs, SIGTERM);
+ libxl__kill(gc, shs->child.pid, SIGTERM, "save/restore helper");
}
static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 05/29] tools/libxl: Stash all restore parameters in domain_create_state
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (3 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 04/29] tools/libxl: Introduce libxl__kill() Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 06/29] tools/libxl: Split libxl__domain_create_state.restore_fd in two Andrew Cooper
` (24 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Shortly more parameters will appear, and this saves unboxing each one.
libxl_domain_restore_params is mandatory for restore streams, and
ignored for plain creation. The old 'checkpointed_stream' was
incorrectly identified as a private parameter when it was infact
public.
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
Reviewed-by: Yang Hongyang <yanghy@cn.fujitsu.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
Since v1:
* Gate validity on restore_fd being valid.
---
tools/libxl/libxl_create.c | 13 +++++++------
tools/libxl/libxl_internal.h | 2 +-
tools/libxl/libxl_save_callout.c | 2 +-
3 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index b785ddd..61515da 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1512,8 +1512,8 @@ static void domain_create_cb(libxl__egc *egc,
int rc, uint32_t domid);
static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
- uint32_t *domid,
- int restore_fd, int checkpointed_stream,
+ uint32_t *domid, int restore_fd,
+ const libxl_domain_restore_params *params,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
@@ -1526,8 +1526,9 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
libxl_domain_config_init(&cdcs->dcs.guest_config_saved);
libxl_domain_config_copy(ctx, &cdcs->dcs.guest_config_saved, d_config);
cdcs->dcs.restore_fd = restore_fd;
+ if (restore_fd > -1)
+ cdcs->dcs.restore_params = *params;
cdcs->dcs.callback = domain_create_cb;
- cdcs->dcs.checkpointed_stream = checkpointed_stream;
libxl__ao_progress_gethow(&cdcs->dcs.aop_console_how, aop_console_how);
cdcs->domid_out = domid;
@@ -1553,7 +1554,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
- return do_domain_create(ctx, d_config, domid, -1, 0,
+ return do_domain_create(ctx, d_config, domid, -1, NULL,
ao_how, aop_console_how);
}
@@ -1563,8 +1564,8 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
- return do_domain_create(ctx, d_config, domid, restore_fd,
- params->checkpointed_stream, ao_how, aop_console_how);
+ return do_domain_create(ctx, d_config, domid, restore_fd, params,
+ ao_how, aop_console_how);
}
/*
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 7ccbf55..6428757 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3217,11 +3217,11 @@ struct libxl__domain_create_state {
libxl_domain_config *guest_config;
libxl_domain_config guest_config_saved; /* vanilla config */
int restore_fd;
+ libxl_domain_restore_params restore_params;
libxl__domain_create_cb *callback;
libxl_asyncprogress_how aop_console_how;
/* private to domain_create */
int guest_domid;
- int checkpointed_stream;
libxl__domain_build_state build_state;
libxl__bootloader_state bl;
libxl__stub_dm_spawn_state dmss;
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index b82a5c1..80aae1b 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -60,7 +60,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
state->store_domid, state->console_port,
state->console_domid,
hvm, pae, superpages,
- cbflags, dcs->checkpointed_stream,
+ cbflags, dcs->restore_params.checkpointed_stream,
};
dcs->shs.ao = ao;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 06/29] tools/libxl: Split libxl__domain_create_state.restore_fd in two
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (4 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 05/29] tools/libxl: Stash all restore parameters in domain_create_state Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 07/29] tools/libxl: Extra management APIs for the save helper Andrew Cooper
` (23 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
In a future patch, we shall support automatically converting a legacy
stream to a v2 stream, in which case libxc needs to read from a
different fd.
Simply overwriting restore_fd does not work; the two fd's have
different circumstances. The restore_fd needs to be returned to its
original state before libxl_domain_create_restore() returns, while in
the converted case, the fd needs allocating and deallocating
appropriately.
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
New in v2
---
tools/libxl/libxl_create.c | 2 +-
tools/libxl/libxl_internal.h | 2 +-
tools/libxl/libxl_save_callout.c | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 61515da..be13204 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1525,7 +1525,7 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
cdcs->dcs.guest_config = d_config;
libxl_domain_config_init(&cdcs->dcs.guest_config_saved);
libxl_domain_config_copy(ctx, &cdcs->dcs.guest_config_saved, d_config);
- cdcs->dcs.restore_fd = restore_fd;
+ cdcs->dcs.restore_fd = cdcs->dcs.libxc_fd = restore_fd;
if (restore_fd > -1)
cdcs->dcs.restore_params = *params;
cdcs->dcs.callback = domain_create_cb;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 6428757..e5599a3 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3216,7 +3216,7 @@ struct libxl__domain_create_state {
libxl__ao *ao;
libxl_domain_config *guest_config;
libxl_domain_config guest_config_saved; /* vanilla config */
- int restore_fd;
+ int restore_fd, libxc_fd;
libxl_domain_restore_params restore_params;
libxl__domain_create_cb *callback;
libxl_asyncprogress_how aop_console_how;
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 80aae1b..1136b79 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -48,7 +48,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
/* Convenience aliases */
const uint32_t domid = dcs->guest_domid;
- const int restore_fd = dcs->restore_fd;
+ const int restore_fd = dcs->libxc_fd;
libxl__domain_build_state *const state = &dcs->build_state;
unsigned cbflags = libxl__srm_callout_enumcallbacks_restore
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 07/29] tools/libxl: Extra management APIs for the save helper
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (5 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 06/29] tools/libxl: Split libxl__domain_create_state.restore_fd in two Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 13:23 ` Ian Jackson
2015-07-14 10:59 ` [PATCH v4 08/29] tools/libxl: Add save_helper_state pointers to libxl__xc_domain_{save, restore}() Andrew Cooper
` (22 subsequent siblings)
29 siblings, 1 reply; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
With migration v2, there are several moving parts needing to be
juggled at once. This requires the error handling logic to be able to
query the state of each moving part, possibly before they have been
started, and be able to cancel them.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v4: Don't force _init() to be mandatory
v3: Adjust helper_{stop,failed,done} to use libxl__save_helper_inuse()
v2: Add an _init() function which allows _inuse() to be safe to call even
before the save helper has started.
---
tools/libxl/libxl_internal.h | 9 +++++++++
tools/libxl/libxl_save_callout.c | 23 +++++++++++++++++------
2 files changed, 26 insertions(+), 6 deletions(-)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index e5599a3..8ce3d49 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3272,6 +3272,15 @@ _hidden void libxl__xc_domain_restore(libxl__egc *egc,
_hidden void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
int rc, int retval, int errnoval);
+_hidden void libxl__save_helper_init(libxl__save_helper_state *shs);
+_hidden void libxl__save_helper_abort(libxl__egc *egc,
+ libxl__save_helper_state *shs);
+
+static inline bool libxl__save_helper_inuse(const libxl__save_helper_state *shs)
+{
+ return libxl__ev_child_inuse(&shs->child);
+}
+
/* Each time the dm needs to be saved, we must call suspend and then save */
_hidden int libxl__domain_suspend_device_model(libxl__gc *gc,
libxl__domain_suspend_state *dss);
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 1136b79..c56b68c 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -146,6 +146,13 @@ void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
shs->egc = 0;
}
+void libxl__save_helper_init(libxl__save_helper_state *shs)
+{
+ libxl__ao_abortable_init(&shs->abrt);
+ libxl__ev_fd_init(&shs->readable);
+ libxl__ev_child_init(&shs->child);
+}
+
/*----- helper execution -----*/
static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
@@ -167,9 +174,7 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
shs->rc = 0;
shs->completed = 0;
shs->pipes[0] = shs->pipes[1] = 0;
- libxl__ao_abortable_init(&shs->abrt);
- libxl__ev_fd_init(&shs->readable);
- libxl__ev_child_init(&shs->child);
+ libxl__save_helper_init(shs);
shs->abrt.ao = shs->ao;
shs->abrt.callback = helper_stop;
@@ -255,7 +260,7 @@ static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs,
libxl__ev_fd_deregister(gc, &shs->readable);
- if (!libxl__ev_child_inuse(&shs->child)) {
+ if (!libxl__save_helper_inuse(shs)) {
helper_done(egc, shs);
return;
}
@@ -268,7 +273,7 @@ static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
libxl__save_helper_state *shs = CONTAINER_OF(abrt, *shs, abrt);
STATE_AO_GC(shs->ao);
- if (!libxl__ev_child_inuse(&shs->child)) {
+ if (!libxl__save_helper_inuse(shs)) {
helper_failed(egc, shs, rc);
return;
}
@@ -279,6 +284,12 @@ static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
libxl__kill(gc, shs->child.pid, SIGTERM, "save/restore helper");
}
+void libxl__save_helper_abort(libxl__egc *egc,
+ libxl__save_helper_state *shs)
+{
+ helper_stop(egc, &shs->abrt, ERROR_FAIL);
+}
+
static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
int fd, short events, short revents)
{
@@ -356,7 +367,7 @@ static void helper_done(libxl__egc *egc, libxl__save_helper_state *shs)
libxl__ev_fd_deregister(gc, &shs->readable);
libxl__carefd_close(shs->pipes[0]); shs->pipes[0] = 0;
libxl__carefd_close(shs->pipes[1]); shs->pipes[1] = 0;
- assert(!libxl__ev_child_inuse(&shs->child));
+ assert(!libxl__save_helper_inuse(shs));
if (shs->toolstack_data_file) fclose(shs->toolstack_data_file);
shs->egc = egc;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v4 07/29] tools/libxl: Extra management APIs for the save helper
2015-07-14 10:59 ` [PATCH v4 07/29] tools/libxl: Extra management APIs for the save helper Andrew Cooper
@ 2015-07-14 13:23 ` Ian Jackson
0 siblings, 0 replies; 39+ messages in thread
From: Ian Jackson @ 2015-07-14 13:23 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Xen-devel
Andrew Cooper writes ("[PATCH v4 07/29] tools/libxl: Extra management APIs for the save helper"):
> With migration v2, there are several moving parts needing to be
> juggled at once. This requires the error handling logic to be able to
> query the state of each moving part, possibly before they have been
> started, and be able to cancel them.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v4 08/29] tools/libxl: Add save_helper_state pointers to libxl__xc_domain_{save, restore}()
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (6 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 07/29] tools/libxl: Extra management APIs for the save helper Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 09/29] tools/libxl: Fix libxl__carefd_opened() to be more useful with an invalid fd Andrew Cooper
` (21 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Currently, libxl__xc_domain_{save,restore}() have specific knowledge
of where the libxl__save_helper_state lives inside a
libxl__domain_{create,save}_state object.
In later changes, the logical ownership of the
libxl__save_helper_state will change and will no longer be
d{c,s}s->shs.
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
New in v3
---
tools/libxl/libxl_create.c | 4 +--
tools/libxl/libxl_dom.c | 20 +++++++-------
tools/libxl/libxl_internal.h | 5 +++-
tools/libxl/libxl_save_callout.c | 54 ++++++++++++++++++++------------------
4 files changed, 44 insertions(+), 39 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index be13204..7f0ffc6 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -988,7 +988,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
rc = ERROR_INVAL;
goto out;
}
- libxl__xc_domain_restore(egc, dcs,
+ libxl__xc_domain_restore(egc, dcs, &dcs->shs,
hvm, pae, superpages);
return;
@@ -1000,7 +1000,7 @@ void libxl__srm_callout_callback_restore_results(unsigned long store_mfn,
unsigned long console_mfn, void *user)
{
libxl__save_helper_state *shs = user;
- libxl__domain_create_state *dcs = CONTAINER_OF(shs, *dcs, shs);
+ libxl__domain_create_state *dcs = shs->caller_state;
STATE_AO_GC(dcs->ao);
libxl__domain_build_state *const state = &dcs->build_state;
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 8642192..a685b77 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1121,7 +1121,7 @@ int libxl__toolstack_restore(uint32_t domid, const uint8_t *ptr,
uint32_t size, void *user)
{
libxl__save_helper_state *shs = user;
- libxl__domain_create_state *dcs = CONTAINER_OF(shs, *dcs, shs);
+ libxl__domain_create_state *dcs = shs->caller_state;
STATE_AO_GC(dcs->ao);
int ret;
uint32_t version = 0, bufsize;
@@ -1188,7 +1188,7 @@ static void logdirty_init(libxl__logdirty_switch *lds)
libxl__save_helper_state *shs)
{
libxl__egc *egc = shs->egc;
- libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+ libxl__domain_suspend_state *dss = shs->caller_state;
libxl__logdirty_switch *lds = &dss->logdirty;
STATE_AO_GC(dss->ao);
int rc;
@@ -1260,7 +1260,7 @@ static void logdirty_init(libxl__logdirty_switch *lds)
libxl__save_helper_state *shs)
{
libxl__egc *egc = shs->egc;
- libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+ libxl__domain_suspend_state *dss = shs->caller_state;
STATE_AO_GC(dss->ao);
int rc;
@@ -1279,7 +1279,7 @@ static void logdirty_init(libxl__logdirty_switch *lds)
{
libxl__save_helper_state *shs = user;
libxl__egc *egc = shs->egc;
- libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+ libxl__domain_suspend_state *dss = shs->caller_state;
STATE_AO_GC(dss->ao);
switch (libxl__device_model_version_running(gc, domid)) {
@@ -1832,7 +1832,7 @@ static void libxl__domain_suspend_callback(void *data)
{
libxl__save_helper_state *shs = data;
libxl__egc *egc = shs->egc;
- libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+ libxl__domain_suspend_state *dss = shs->caller_state;
dss->callback_common_done = domain_suspend_callback_common_done;
domain_suspend_callback_common(egc, dss);
@@ -1859,7 +1859,7 @@ static void libxl__remus_domain_suspend_callback(void *data)
{
libxl__save_helper_state *shs = data;
libxl__egc *egc = shs->egc;
- libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+ libxl__domain_suspend_state *dss = shs->caller_state;
dss->callback_common_done = remus_domain_suspend_callback_common_done;
domain_suspend_callback_common(egc, dss);
@@ -1902,7 +1902,7 @@ static void libxl__remus_domain_resume_callback(void *data)
{
libxl__save_helper_state *shs = data;
libxl__egc *egc = shs->egc;
- libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
+ libxl__domain_suspend_state *dss = shs->caller_state;
STATE_AO_GC(dss->ao);
libxl__remus_devices_state *const rds = &dss->rds;
@@ -1947,8 +1947,8 @@ static void remus_next_checkpoint(libxl__egc *egc, libxl__ev_time *ev,
static void libxl__remus_domain_checkpoint_callback(void *data)
{
libxl__save_helper_state *shs = data;
- libxl__domain_suspend_state *dss = CONTAINER_OF(shs, *dss, shs);
- libxl__egc *egc = dss->shs.egc;
+ libxl__domain_suspend_state *dss = shs->caller_state;
+ libxl__egc *egc = shs->egc;
STATE_AO_GC(dss->ao);
/* This would go into tailbuf. */
@@ -2117,7 +2117,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
callbacks->switch_qemu_logdirty = libxl__domain_suspend_common_switch_qemu_logdirty;
dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
- libxl__xc_domain_save(egc, dss);
+ libxl__xc_domain_save(egc, dss, &dss->shs);
return;
out:
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 8ce3d49..3f1fed8 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3241,7 +3241,9 @@ _hidden void libxl__domain_suspend(libxl__egc *egc,
/* calls libxl__xc_domain_suspend_done when done */
-_hidden void libxl__xc_domain_save(libxl__egc*, libxl__domain_suspend_state*);
+_hidden void libxl__xc_domain_save(libxl__egc *egc,
+ libxl__domain_suspend_state *dss,
+ libxl__save_helper_state *shs);
/* If rc==0 then retval is the return value from xc_domain_save
* and errnoval is the errno value it provided.
* If rc!=0, retval and errnoval are undefined. */
@@ -3265,6 +3267,7 @@ _hidden int libxl__toolstack_save(uint32_t domid, uint8_t **buf,
/* calls libxl__xc_domain_restore_done when done */
_hidden void libxl__xc_domain_restore(libxl__egc *egc,
libxl__domain_create_state *dcs,
+ libxl__save_helper_state *shs,
int hvm, int pae, int superpages);
/* If rc==0 then retval is the return value from xc_domain_save
* and errnoval is the errno value it provided.
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index c56b68c..a5e180c 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -42,6 +42,7 @@ static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
/*----- entrypoints -----*/
void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
+ libxl__save_helper_state *shs,
int hvm, int pae, int superpages)
{
STATE_AO_GC(dcs->ao);
@@ -51,8 +52,8 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
const int restore_fd = dcs->libxc_fd;
libxl__domain_build_state *const state = &dcs->build_state;
- unsigned cbflags = libxl__srm_callout_enumcallbacks_restore
- (&dcs->shs.callbacks.restore.a);
+ unsigned cbflags =
+ libxl__srm_callout_enumcallbacks_restore(&shs->callbacks.restore.a);
const unsigned long argnums[] = {
domid,
@@ -63,19 +64,20 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
cbflags, dcs->restore_params.checkpointed_stream,
};
- dcs->shs.ao = ao;
- dcs->shs.domid = domid;
- dcs->shs.recv_callback = libxl__srm_callout_received_restore;
- dcs->shs.completion_callback = libxl__xc_domain_restore_done;
- dcs->shs.caller_state = dcs;
- dcs->shs.need_results = 1;
- dcs->shs.toolstack_data_file = 0;
+ shs->ao = ao;
+ shs->domid = domid;
+ shs->recv_callback = libxl__srm_callout_received_restore;
+ shs->completion_callback = libxl__xc_domain_restore_done;
+ shs->caller_state = dcs;
+ shs->need_results = 1;
+ shs->toolstack_data_file = 0;
- run_helper(egc, &dcs->shs, "--restore-domain", restore_fd, 0,0,
+ run_helper(egc, shs, "--restore-domain", restore_fd, 0, 0,
argnums, ARRAY_SIZE(argnums));
}
-void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss)
+void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss,
+ libxl__save_helper_state *shs)
{
STATE_AO_GC(dss->ao);
int r, rc, toolstack_data_fd = -1;
@@ -84,21 +86,21 @@ void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss)
/* Resources we need to free */
uint8_t *toolstack_data_buf = 0;
- unsigned cbflags = libxl__srm_callout_enumcallbacks_save
- (&dss->shs.callbacks.save.a);
+ unsigned cbflags =
+ libxl__srm_callout_enumcallbacks_save(&shs->callbacks.save.a);
- if (dss->shs.callbacks.save.toolstack_save) {
- r = dss->shs.callbacks.save.toolstack_save
+ if (shs->callbacks.save.toolstack_save) {
+ r = shs->callbacks.save.toolstack_save
(dss->domid, &toolstack_data_buf, &toolstack_data_len, dss);
if (r) { rc = ERROR_FAIL; goto out; }
- dss->shs.toolstack_data_file = tmpfile();
- if (!dss->shs.toolstack_data_file) {
+ shs->toolstack_data_file = tmpfile();
+ if (!shs->toolstack_data_file) {
LOGE(ERROR, "cannot create toolstack data tmpfile");
rc = ERROR_FAIL;
goto out;
}
- toolstack_data_fd = fileno(dss->shs.toolstack_data_file);
+ toolstack_data_fd = fileno(shs->toolstack_data_file);
r = libxl_write_exactly(CTX, toolstack_data_fd,
toolstack_data_buf, toolstack_data_len,
@@ -116,23 +118,23 @@ void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss)
cbflags,
};
- dss->shs.ao = ao;
- dss->shs.domid = dss->domid;
- dss->shs.recv_callback = libxl__srm_callout_received_save;
- dss->shs.completion_callback = libxl__xc_domain_save_done;
- dss->shs.caller_state = dss;
- dss->shs.need_results = 0;
+ shs->ao = ao;
+ shs->domid = dss->domid;
+ shs->recv_callback = libxl__srm_callout_received_save;
+ shs->completion_callback = libxl__xc_domain_save_done;
+ shs->caller_state = dss;
+ shs->need_results = 0;
free(toolstack_data_buf);
- run_helper(egc, &dss->shs, "--save-domain", dss->fd,
+ run_helper(egc, shs, "--save-domain", dss->fd,
&toolstack_data_fd, 1,
argnums, ARRAY_SIZE(argnums));
return;
out:
free(toolstack_data_buf);
- if (dss->shs.toolstack_data_file) fclose(dss->shs.toolstack_data_file);
+ if (shs->toolstack_data_file) fclose(shs->toolstack_data_file);
libxl__xc_domain_save_done(egc, dss, rc, 0, 0);
}
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 09/29] tools/libxl: Fix libxl__carefd_opened() to be more useful with an invalid fd
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (7 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 08/29] tools/libxl: Add save_helper_state pointers to libxl__xc_domain_{save, restore}() Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 13:39 ` Ian Jackson
2015-07-14 10:59 ` [PATCH v4 10/29] tools/xl: Mandatory flag indicating the format of the migration stream Andrew Cooper
` (20 subsequent siblings)
29 siblings, 1 reply; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
In the case that fd is -1, preserve errno and don't attempt to set
CLOEXEC.
Note that the implementation can still fail, as it ignores fcntl()
errors and may not set CLOEXEC properly. Update the documentation
accordingly until it is fixed.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
Fixing the fnctl() error issue involves more TUITs than I currently
have.
New in v4
---
tools/libxl/libxl_fork.c | 5 ++++-
tools/libxl/libxl_internal.h | 2 +-
2 files changed, 5 insertions(+), 2 deletions(-)
diff --git a/tools/libxl/libxl_fork.c b/tools/libxl/libxl_fork.c
index 4486687..024c1e2 100644
--- a/tools/libxl/libxl_fork.c
+++ b/tools/libxl/libxl_fork.c
@@ -112,9 +112,12 @@ libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd)
libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd)
{
libxl__carefd *cf = 0;
+ int saved_errno = errno;
- cf = libxl__carefd_record(ctx, fd);
+ if (fd >= 0)
+ cf = libxl__carefd_record(ctx, fd);
libxl__carefd_unlock();
+ errno = saved_errno;
return cf;
}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 3f1fed8..5d3499e 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2189,7 +2189,7 @@ _hidden void libxl__ao_progress_report(libxl__egc *egc, libxl__ao *ao,
_hidden libxl__carefd *libxl__carefd_record(libxl_ctx *ctx, int fd);
/* Combines _record and _unlock in a single call. If fd==-1,
- * still does the unlock, but returns 0. Cannot fail. */
+ * still does the unlock, but returns 0. */
_hidden libxl__carefd *libxl__carefd_opened(libxl_ctx *ctx, int fd);
/* Works just like close(2). You may pass NULL, in which case it's
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v4 09/29] tools/libxl: Fix libxl__carefd_opened() to be more useful with an invalid fd
2015-07-14 10:59 ` [PATCH v4 09/29] tools/libxl: Fix libxl__carefd_opened() to be more useful with an invalid fd Andrew Cooper
@ 2015-07-14 13:39 ` Ian Jackson
2015-07-14 14:08 ` Ian Campbell
0 siblings, 1 reply; 39+ messages in thread
From: Ian Jackson @ 2015-07-14 13:39 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Campbell, Xen-devel
Andrew Cooper writes ("[PATCH v4 09/29] tools/libxl: Fix libxl__carefd_opened() to be more useful with an invalid fd"):
> In the case that fd is -1, preserve errno and don't attempt to set
> CLOEXEC.
>
> Note that the implementation can still fail, as it ignores fcntl()
> errors and may not set CLOEXEC properly. Update the documentation
> accordingly until it is fixed.
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
> Fixing the fnctl() error issue involves more TUITs than I currently
> have.
I think that if the fcntl set cloexec fails, we should probably abort
the program. Ian C, would that be OK with you ? If so then the
patch to fix this is trivial.
Otherwise all call sites need to grow error handling.
Ian.
^ permalink raw reply [flat|nested] 39+ messages in thread* Re: [PATCH v4 09/29] tools/libxl: Fix libxl__carefd_opened() to be more useful with an invalid fd
2015-07-14 13:39 ` Ian Jackson
@ 2015-07-14 14:08 ` Ian Campbell
0 siblings, 0 replies; 39+ messages in thread
From: Ian Campbell @ 2015-07-14 14:08 UTC (permalink / raw)
To: Ian Jackson; +Cc: Andrew Cooper, Wei Liu, Xen-devel
On Tue, 2015-07-14 at 14:39 +0100, Ian Jackson wrote:
> Andrew Cooper writes ("[PATCH v4 09/29] tools/libxl: Fix libxl__carefd_opened() to be more useful with an invalid fd"):
> > In the case that fd is -1, preserve errno and don't attempt to set
> > CLOEXEC.
> >
> > Note that the implementation can still fail, as it ignores fcntl()
> > errors and may not set CLOEXEC properly. Update the documentation
> > accordingly until it is fixed.
>
> Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
>
> > Fixing the fnctl() error issue involves more TUITs than I currently
> > have.
>
> I think that if the fcntl set cloexec fails, we should probably abort
> the program. Ian C, would that be OK with you ? If so then the
> patch to fix this is trivial.
http://pubs.opengroup.org/onlinepubs/9699919799/functions/fcntl.html
suggests the only relevant failures which can occur for setting cloexec
would be EBADF or EINVAL, both of which would be program errors in this
code path and not external factors or due to other threads.
So I think an abort would be ok in this instance.
Ian.
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v4 10/29] tools/xl: Mandatory flag indicating the format of the migration stream
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (8 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 09/29] tools/libxl: Fix libxl__carefd_opened() to be more useful with an invalid fd Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 11/29] docs: Libxl migration v2 stream specification Andrew Cooper
` (19 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Introduced at this point so the python stream conversion code has a
concrete ABI to use. Later when libxl itself starts supporting a v2
stream, it will be added to XL_MANDATORY_FLAG_ALL.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v2: Expand commit message
---
tools/libxl/xl_cmdimpl.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 971209c..26b1e7d 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -109,6 +109,7 @@
*/
#define XL_MANDATORY_FLAG_JSON (1U << 0) /* config data is in JSON format */
+#define XL_MANDATORY_FLAG_STREAMv2 (1U << 1) /* stream is v2 */
#define XL_MANDATORY_FLAG_ALL (XL_MANDATORY_FLAG_JSON)
struct save_file_header {
char magic[32]; /* savefileheader_magic */
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 11/29] docs: Libxl migration v2 stream specification
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (9 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 10/29] tools/xl: Mandatory flag indicating the format of the migration stream Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 12/29] tools/python: Libxc migration v2 infrastructure Andrew Cooper
` (18 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v3: Spelling fixes
---
docs/specs/libxl-migration-stream.pandoc | 205 ++++++++++++++++++++++++++++++
1 file changed, 205 insertions(+)
create mode 100644 docs/specs/libxl-migration-stream.pandoc
diff --git a/docs/specs/libxl-migration-stream.pandoc b/docs/specs/libxl-migration-stream.pandoc
new file mode 100644
index 0000000..4192950
--- /dev/null
+++ b/docs/specs/libxl-migration-stream.pandoc
@@ -0,0 +1,205 @@
+% LibXenLight Domain Image Format
+% Andrew Cooper <<andrew.cooper3@citrix.com>>
+% Draft B
+
+Introduction
+============
+
+For the purposes of this document, `xl` is used as a representation of any
+implementer of the `libxl` API. `xl` should be considered completely
+interchangeable with alternates, such as `libvirt` or `xenopsd-xl`.
+
+Purpose
+-------
+
+The _domain image format_ is the context of a running domain used for
+snapshots of a domain or for transferring domains between hosts during
+migration.
+
+There are a number of problems with the domain image format used in Xen 4.5
+and earlier (the _legacy format_)
+
+* There is no `libxl` context information. `xl` is required to send certain
+ pieces of `libxl` context itself.
+
+* The contents of the stream is passed directly through `libxl` to `libxc`.
+ The legacy `libxc` format contained some information which belonged at the
+ `libxl` level, resulting in awkward layer violation to return the
+ information back to `libxl`.
+
+* The legacy `libxc` format was inextensible, causing inextensibility in the
+ legacy `libxl` handling.
+
+This design addresses the above points, allowing for a completely
+self-contained, extensible stream with each layer responsible for its own
+appropriate information.
+
+
+Not Yet Included
+----------------
+
+The following features are not yet fully specified and will be
+included in a future draft.
+
+* Remus
+
+* ARM
+
+
+Overview
+========
+
+The image format consists of a _Header_, followed by 1 or more _Records_.
+Each record consists of a type and length field, followed by any type-specific
+data.
+
+\clearpage
+
+Header
+======
+
+The header identifies the stream as a `libxl` stream, including the version of
+this specification that it complies with.
+
+All fields in this header shall be in _big-endian_ byte order, regardless of
+the setting of the endianness bit.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+ | ident |
+ +-----------------------+-------------------------+
+ | version | options |
+ +-----------------------+-------------------------+
+
+--------------------------------------------------------------------
+Field Description
+----------- --------------------------------------------------------
+ident 0x4c6962786c466d74 ("LibxlFmt" in ASCII).
+
+version 0x00000002. The version of this specification.
+
+options bit 0: Endianness. 0 = little-endian, 1 = big-endian.
+
+ bit 1: Legacy Format. If set, this stream was created by
+ the legacy conversion tool.
+
+ bits 2-31: Reserved.
+--------------------------------------------------------------------
+
+The endianness shall be 0 (little-endian) for images generated on an
+i386, x86_64, or arm host.
+
+\clearpage
+
+
+Records
+=======
+
+A record has a record header, type specific data and a trailing footer. If
+`length` is not a multiple of 8, the body is padded with zeroes to align the
+end of the record on an 8 octet boundary.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-----------------------+-------------------------+
+ | type | body_length |
+ +-----------+-----------+-------------------------+
+ | body... |
+ ...
+ | | padding (0 to 7 octets) |
+ +-----------+-------------------------------------+
+
+--------------------------------------------------------------------
+Field Description
+----------- -------------------------------------------------------
+type 0x00000000: END
+
+ 0x00000001: LIBXC_CONTEXT
+
+ 0x00000002: XENSTORE_DATA
+
+ 0x00000003: EMULATOR_CONTEXT
+
+ 0x00000004 - 0x7FFFFFFF: Reserved for future _mandatory_
+ records.
+
+ 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
+ records.
+
+body_length Length in octets of the record body.
+
+body Content of the record.
+
+padding 0 to 7 octets of zeros to pad the whole record to a multiple
+ of 8 octets.
+--------------------------------------------------------------------
+
+\clearpage
+
+END
+----
+
+A end record marks the end of the image, and shall be the final record
+in the stream.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+
+The end record contains no fields; its body_length is 0.
+
+LIBXC\_CONTEXT
+--------------
+
+A libxc context record is a marker, indicating that the stream should be
+handed to `xc_domain_restore()`. `libxc` shall be responsible for reading its
+own image format from the stream.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+
+The libxc context record contains no fields; its body_length is 0[^1].
+
+
+[^1]: The sending side cannot calculate ahead of time how much data `libxc`
+might write into the stream, especially for live migration where the quantity
+of data is partially proportional to the elapsed time.
+
+XENSTORE\_DATA
+-------------
+
+A record containing xenstore key/value pairs of data.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+ | xenstore key/value pairs |
+ ...
+ +-------------------------------------------------+
+
+EMULATOR\_CONTEXT
+----------------
+
+A context blob for a specific emulator associated with the domain.
+
+ 0 1 2 3 4 5 6 7 octet
+ +------------------------+------------------------+
+ | emulator_id | index |
+ +------------------------+------------------------+
+ | emulator_ctx |
+ ...
+ +-------------------------------------------------+
+
+--------------------------------------------------------------------
+Field Description
+------------ ---------------------------------------------------
+emulator_id 0x00000000: Unknown (In the case of a legacy stream)
+
+ 0x00000001: Qemu Traditional
+
+ 0x00000002: Qemu Upstream
+
+ 0x00000003 - 0xFFFFFFFF: Reserved for future emulators.
+
+index Index of this emulator for the domain, if multiple
+ emulators are in use.
+
+emulator_ctx Emulator context blob.
+--------------------------------------------------------------------
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 12/29] tools/python: Libxc migration v2 infrastructure
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (10 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 11/29] docs: Libxl migration v2 stream specification Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 13/29] tools/python: Libxl " Andrew Cooper
` (17 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Contains:
* Python implementation of the libxc migration v2 records
* Verification code for spec compliance
* Unit tests
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/python/setup.py | 1 +
tools/python/xen/migration/libxc.py | 446 ++++++++++++++++++++++++++++++++++
tools/python/xen/migration/tests.py | 41 ++++
tools/python/xen/migration/verify.py | 37 +++
4 files changed, 525 insertions(+)
create mode 100644 tools/python/xen/migration/__init__.py
create mode 100644 tools/python/xen/migration/libxc.py
create mode 100644 tools/python/xen/migration/tests.py
create mode 100644 tools/python/xen/migration/verify.py
diff --git a/tools/python/setup.py b/tools/python/setup.py
index 439c429..5bf81be 100644
--- a/tools/python/setup.py
+++ b/tools/python/setup.py
@@ -43,6 +43,7 @@ setup(name = 'xen',
version = '3.0',
description = 'Xen',
packages = ['xen',
+ 'xen.migration',
'xen.lowlevel',
],
ext_package = "xen.lowlevel",
diff --git a/tools/python/xen/migration/__init__.py b/tools/python/xen/migration/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/tools/python/xen/migration/libxc.py b/tools/python/xen/migration/libxc.py
new file mode 100644
index 0000000..b0255ac
--- /dev/null
+++ b/tools/python/xen/migration/libxc.py
@@ -0,0 +1,446 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Libxc Migration v2 streams
+
+Record structures as per docs/specs/libxc-migration-stream.pandoc, and
+verification routines.
+"""
+
+import sys
+
+from struct import calcsize, unpack
+
+from xen.migration.verify import StreamError, RecordError, VerifyBase
+
+# Image Header
+IHDR_FORMAT = "!QIIHHI"
+
+IHDR_MARKER = 0xffffffffffffffff
+IHDR_IDENT = 0x58454E46 # "XENF" in ASCII
+IHDR_VERSION = 2
+
+IHDR_OPT_BIT_ENDIAN = 0
+IHDR_OPT_LE = (0 << IHDR_OPT_BIT_ENDIAN)
+IHDR_OPT_BE = (1 << IHDR_OPT_BIT_ENDIAN)
+
+IHDR_OPT_RESZ_MASK = 0xfffe
+
+# Domain Header
+DHDR_FORMAT = "IHHII"
+
+DHDR_TYPE_x86_pv = 0x00000001
+DHDR_TYPE_x86_hvm = 0x00000002
+DHDR_TYPE_x86_pvh = 0x00000003
+DHDR_TYPE_arm = 0x00000004
+
+dhdr_type_to_str = {
+ DHDR_TYPE_x86_pv : "x86 PV",
+ DHDR_TYPE_x86_hvm : "x86 HVM",
+ DHDR_TYPE_x86_pvh : "x86 PVH",
+ DHDR_TYPE_arm : "ARM",
+}
+
+# Records
+RH_FORMAT = "II"
+
+REC_TYPE_end = 0x00000000
+REC_TYPE_page_data = 0x00000001
+REC_TYPE_x86_pv_info = 0x00000002
+REC_TYPE_x86_pv_p2m_frames = 0x00000003
+REC_TYPE_x86_pv_vcpu_basic = 0x00000004
+REC_TYPE_x86_pv_vcpu_extended = 0x00000005
+REC_TYPE_x86_pv_vcpu_xsave = 0x00000006
+REC_TYPE_shared_info = 0x00000007
+REC_TYPE_tsc_info = 0x00000008
+REC_TYPE_hvm_context = 0x00000009
+REC_TYPE_hvm_params = 0x0000000a
+REC_TYPE_toolstack = 0x0000000b
+REC_TYPE_x86_pv_vcpu_msrs = 0x0000000c
+REC_TYPE_verify = 0x0000000d
+REC_TYPE_checkpoint = 0x0000000e
+
+rec_type_to_str = {
+ REC_TYPE_end : "End",
+ REC_TYPE_page_data : "Page data",
+ REC_TYPE_x86_pv_info : "x86 PV info",
+ REC_TYPE_x86_pv_p2m_frames : "x86 PV P2M frames",
+ REC_TYPE_x86_pv_vcpu_basic : "x86 PV vcpu basic",
+ REC_TYPE_x86_pv_vcpu_extended : "x86 PV vcpu extended",
+ REC_TYPE_x86_pv_vcpu_xsave : "x86 PV vcpu xsave",
+ REC_TYPE_shared_info : "Shared info",
+ REC_TYPE_tsc_info : "TSC info",
+ REC_TYPE_hvm_context : "HVM context",
+ REC_TYPE_hvm_params : "HVM params",
+ REC_TYPE_toolstack : "Toolstack",
+ REC_TYPE_x86_pv_vcpu_msrs : "x86 PV vcpu msrs",
+ REC_TYPE_verify : "Verify",
+ REC_TYPE_checkpoint : "Checkpoint",
+}
+
+# page_data
+PAGE_DATA_FORMAT = "II"
+PAGE_DATA_PFN_MASK = (1L << 52) - 1
+PAGE_DATA_PFN_RESZ_MASK = ((1L << 60) - 1) & ~((1L << 52) - 1)
+
+# flags from xen/public/domctl.h: XEN_DOMCTL_PFINFO_* shifted by 32 bits
+PAGE_DATA_TYPE_SHIFT = 60
+PAGE_DATA_TYPE_LTABTYPE_MASK = (0x7L << PAGE_DATA_TYPE_SHIFT)
+PAGE_DATA_TYPE_LTAB_MASK = (0xfL << PAGE_DATA_TYPE_SHIFT)
+PAGE_DATA_TYPE_LPINTAB = (0x8L << PAGE_DATA_TYPE_SHIFT) # Pinned pagetable
+
+PAGE_DATA_TYPE_NOTAB = (0x0L << PAGE_DATA_TYPE_SHIFT) # Regular page
+PAGE_DATA_TYPE_L1TAB = (0x1L << PAGE_DATA_TYPE_SHIFT) # L1 pagetable
+PAGE_DATA_TYPE_L2TAB = (0x2L << PAGE_DATA_TYPE_SHIFT) # L2 pagetable
+PAGE_DATA_TYPE_L3TAB = (0x3L << PAGE_DATA_TYPE_SHIFT) # L3 pagetable
+PAGE_DATA_TYPE_L4TAB = (0x4L << PAGE_DATA_TYPE_SHIFT) # L4 pagetable
+PAGE_DATA_TYPE_BROKEN = (0xdL << PAGE_DATA_TYPE_SHIFT) # Broken
+PAGE_DATA_TYPE_XALLOC = (0xeL << PAGE_DATA_TYPE_SHIFT) # Allocate-only
+PAGE_DATA_TYPE_XTAB = (0xfL << PAGE_DATA_TYPE_SHIFT) # Invalid
+
+# x86_pv_info
+X86_PV_INFO_FORMAT = "BBHI"
+
+X86_PV_P2M_FRAMES_FORMAT = "II"
+
+# x86_pv_vcpu_{basic,extended,xsave,msrs}
+X86_PV_VCPU_HDR_FORMAT = "II"
+
+# tsc_info
+TSC_INFO_FORMAT = "IIQII"
+
+# hvm_params
+HVM_PARAMS_ENTRY_FORMAT = "QQ"
+HVM_PARAMS_FORMAT = "II"
+
+class VerifyLibxc(VerifyBase):
+ """ Verify a Libxc v2 stream """
+
+ def __init__(self, info, read):
+ VerifyBase.__init__(self, info, read)
+
+ self.squashed_pagedata_records = 0
+
+
+ def verify(self):
+ """ Verity a libxc stream """
+
+ self.verify_ihdr()
+ self.verify_dhdr()
+
+ while self.verify_record() != REC_TYPE_end:
+ pass
+
+
+ def verify_ihdr(self):
+ """ Verify an Image Header """
+ marker, ident, version, options, res1, res2 = \
+ self.unpack_exact(IHDR_FORMAT)
+
+ if marker != IHDR_MARKER:
+ raise StreamError("Bad image marker: Expected 0x%x, got 0x%x"
+ % (IHDR_MARKER, marker))
+
+ if ident != IHDR_IDENT:
+ raise StreamError("Bad image id: Expected 0x%x, got 0x%x"
+ % (IHDR_IDENT, ident))
+
+ if version != IHDR_VERSION:
+ raise StreamError("Unknown image version: Expected %d, got %d"
+ % (IHDR_VERSION, version))
+
+ if options & IHDR_OPT_RESZ_MASK:
+ raise StreamError("Reserved bits set in image options field: 0x%x"
+ % (options & IHDR_OPT_RESZ_MASK))
+
+ if res1 != 0 or res2 != 0:
+ raise StreamError("Reserved bits set in image header: 0x%04x:0x%08x"
+ % (res1, res2))
+
+ if ( (sys.byteorder == "little") and
+ ((options & IHDR_OPT_BIT_ENDIAN) != IHDR_OPT_LE) ):
+ raise StreamError(
+ "Stream is not native endianess - unable to validate")
+
+ endian = ["little", "big"][options & IHDR_OPT_LE]
+ self.info("Libxc Image Header: %s endian" % (endian, ))
+
+
+ def verify_dhdr(self):
+ """ Verify a domain header """
+
+ gtype, page_shift, res1, major, minor = \
+ self.unpack_exact(DHDR_FORMAT)
+
+ if gtype not in dhdr_type_to_str:
+ raise StreamError("Unrecognised domain type 0x%x" % (gtype, ))
+
+ if res1 != 0:
+ raise StreamError("Reserved bits set in domain header 0x%04x"
+ % (res1, ))
+
+ if page_shift != 12:
+ raise StreamError("Page shift expected to be 12. Got %d"
+ % (page_shift, ))
+
+ if major == 0:
+ self.info("Domain Header: legacy converted %s"
+ % (dhdr_type_to_str[gtype], ))
+ else:
+ self.info("Domain Header: %s from Xen %d.%d"
+ % (dhdr_type_to_str[gtype], major, minor))
+
+
+ def verify_record(self):
+ """ Verify an individual record """
+
+ rtype, length = self.unpack_exact(RH_FORMAT)
+
+ if rtype not in rec_type_to_str:
+ raise StreamError("Unrecognised record type 0x%x" % (rtype, ))
+
+ contentsz = (length + 7) & ~7
+ content = self.rdexact(contentsz)
+
+ if rtype != REC_TYPE_page_data:
+
+ if self.squashed_pagedata_records > 0:
+ self.info("Squashed %d Page Data records together"
+ % (self.squashed_pagedata_records, ))
+ self.squashed_pagedata_records = 0
+
+ self.info("Libxc Record: %s, length %d"
+ % (rec_type_to_str[rtype], length))
+
+ else:
+ self.squashed_pagedata_records += 1
+
+ padding = content[length:]
+ if padding != "\x00" * len(padding):
+ raise StreamError("Padding containing non0 bytes found")
+
+ if rtype not in record_verifiers:
+ raise RuntimeError("No verification function for libxc record '%s'"
+ % rec_type_to_str[rtype])
+ else:
+ record_verifiers[rtype](self, content[:length])
+
+ return rtype
+
+
+ def verify_record_end(self, content):
+ """ End record """
+
+ if len(content) != 0:
+ raise RecordError("End record with non-zero length")
+
+
+ def verify_record_page_data(self, content):
+ """ Page Data record """
+ minsz = calcsize(PAGE_DATA_FORMAT)
+
+ if len(content) <= minsz:
+ raise RecordError("PAGE_DATA record must be at least %d bytes long"
+ % (minsz, ))
+
+ count, res1 = unpack(PAGE_DATA_FORMAT, content[:minsz])
+
+ if res1 != 0:
+ raise StreamError("Reserved bits set in PAGE_DATA record 0x%04x"
+ % (res1, ))
+
+ pfnsz = count * 8
+ if (len(content) - minsz) < pfnsz:
+ raise RecordError("PAGE_DATA record must contain a pfn record for "
+ "each count")
+
+ pfns = list(unpack("=%dQ" % (count,), content[minsz:minsz + pfnsz]))
+
+ nr_pages = 0
+ for idx, pfn in enumerate(pfns):
+
+ if pfn & PAGE_DATA_PFN_RESZ_MASK:
+ raise RecordError("Reserved bits set in pfn[%d]: 0x%016x",
+ idx, pfn & PAGE_DATA_PFN_RESZ_MASK)
+
+ if pfn >> PAGE_DATA_TYPE_SHIFT in (5, 6, 7, 8):
+ raise RecordError("Invalid type value in pfn[%d]: 0x%016x",
+ idx, pfn & PAGE_DATA_TYPE_LTAB_MASK)
+
+ # We expect page data for each normal page or pagetable
+ if PAGE_DATA_TYPE_NOTAB <= (pfn & PAGE_DATA_TYPE_LTABTYPE_MASK) \
+ <= PAGE_DATA_TYPE_L4TAB:
+ nr_pages += 1
+
+ pagesz = nr_pages * 4096
+ if len(content) != minsz + pfnsz + pagesz:
+ raise RecordError("Expected %u + %u + %u, got %u"
+ % (minsz, pfnsz, pagesz, len(content)))
+
+
+ def verify_record_x86_pv_info(self, content):
+ """ x86 PV Info record """
+
+ expectedsz = calcsize(X86_PV_INFO_FORMAT)
+ if len(content) != expectedsz:
+ raise RecordError("x86_pv_info: expected length of %d, got %d"
+ % (expectedsz, len(content)))
+
+ width, levels, res1, res2 = unpack(X86_PV_INFO_FORMAT, content)
+
+ if width not in (4, 8):
+ raise RecordError("Expected width of 4 or 8, got %d" % (width, ))
+
+ if levels not in (3, 4):
+ raise RecordError("Expected levels of 3 or 4, got %d" % (levels, ))
+
+ if res1 != 0 or res2 != 0:
+ raise StreamError("Reserved bits set in X86_PV_INFO: 0x%04x 0x%08x"
+ % (res1, res2))
+
+ bitness = {4:32, 8:64}[width]
+ self.info(" %sbit guest, %d levels of pagetables" % (bitness, levels))
+
+
+ def verify_record_x86_pv_p2m_frames(self, content):
+ """ x86 PV p2m frames record """
+
+ if len(content) % 8 != 0:
+ raise RecordError("Length expected to be a multiple of 8, not %d"
+ % (len(content), ))
+
+ start, end = unpack("=II", content[:8])
+ self.info(" Start pfn 0x%x, End 0x%x" % (start, end))
+
+
+ def verify_record_x86_pv_vcpu_generic(self, content, name):
+ """ Generic for all REC_TYPE_x86_pv_vcpu_{basic,extended,xsave,msrs} """
+ minsz = calcsize(X86_PV_VCPU_HDR_FORMAT)
+
+ if len(content) <= minsz:
+ raise RecordError("X86_PV_VCPU_%s record length must be at least %d"
+ " bytes long" % (name, minsz))
+
+ vcpuid, res1 = unpack(X86_PV_VCPU_HDR_FORMAT, content[:minsz])
+
+ if res1 != 0:
+ raise StreamError(
+ "Reserved bits set in x86_pv_vcpu_%s record 0x%04x"
+ % (name, res1))
+
+ self.info(" vcpu%d %s context, %d bytes"
+ % (vcpuid, name, len(content) - minsz))
+
+
+ def verify_record_shared_info(self, content):
+ """ shared info record """
+
+ if len(content) != 4096:
+ raise RecordError("Length expected to be 4906 bytes, not %d"
+ % (len(content), ))
+
+
+ def verify_record_tsc_info(self, content):
+ """ tsc info record """
+
+ sz = calcsize(TSC_INFO_FORMAT)
+
+ if len(content) != sz:
+ raise RecordError("Length should be %u bytes" % (sz, ))
+
+ mode, khz, nsec, incarn, res1 = unpack(TSC_INFO_FORMAT, content)
+
+ if res1 != 0:
+ raise StreamError("Reserved bits set in TSC_INFO: 0x%08x"
+ % (res1, ))
+
+ self.info(" Mode %u, %u kHz, %u ns, incarnation %d"
+ % (mode, khz, nsec, incarn))
+
+
+ def verify_record_hvm_context(self, content):
+ """ hvm context record """
+
+ if len(content) == 0:
+ raise RecordError("Zero length HVM context")
+
+
+ def verify_record_hvm_params(self, content):
+ """ hvm params record """
+
+ sz = calcsize(HVM_PARAMS_FORMAT)
+
+ if len(content) < sz:
+ raise RecordError("Length should be at least %u bytes" % (sz, ))
+
+ count, rsvd = unpack(HVM_PARAMS_FORMAT, content[:sz])
+
+ if rsvd != 0:
+ raise RecordError("Reserved field not zero (0x%04x)" % (rsvd, ))
+
+ sz += count * calcsize(HVM_PARAMS_ENTRY_FORMAT)
+
+ if len(content) != sz:
+ raise RecordError("Length should be %u bytes" % (sz, ))
+
+
+ def verify_record_toolstack(self, _):
+ """ toolstack record """
+ raise DeprecationWarning("Found Toolstack record in stream")
+
+
+ def verify_record_verify(self, content):
+ """ verify record """
+
+ if len(content) != 0:
+ raise RecordError("Verify record with non-zero length")
+
+
+ def verify_record_checkpoint(self, content):
+ """ checkpoint record """
+
+ if len(content) != 0:
+ raise RecordError("Checkpoint record with non-zero length")
+
+
+record_verifiers = {
+ REC_TYPE_end:
+ VerifyLibxc.verify_record_end,
+ REC_TYPE_page_data:
+ VerifyLibxc.verify_record_page_data,
+
+ REC_TYPE_x86_pv_info:
+ VerifyLibxc.verify_record_x86_pv_info,
+ REC_TYPE_x86_pv_p2m_frames:
+ VerifyLibxc.verify_record_x86_pv_p2m_frames,
+
+ REC_TYPE_x86_pv_vcpu_basic:
+ lambda s, x:
+ VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "basic"),
+ REC_TYPE_x86_pv_vcpu_extended:
+ lambda s, x:
+ VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "extended"),
+ REC_TYPE_x86_pv_vcpu_xsave:
+ lambda s, x:
+ VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "xsave"),
+ REC_TYPE_x86_pv_vcpu_msrs:
+ lambda s, x:
+ VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"),
+
+ REC_TYPE_shared_info:
+ VerifyLibxc.verify_record_shared_info,
+ REC_TYPE_tsc_info:
+ VerifyLibxc.verify_record_tsc_info,
+
+ REC_TYPE_hvm_context:
+ VerifyLibxc.verify_record_hvm_context,
+ REC_TYPE_hvm_params:
+ VerifyLibxc.verify_record_hvm_params,
+ REC_TYPE_toolstack:
+ VerifyLibxc.verify_record_toolstack,
+ REC_TYPE_verify:
+ VerifyLibxc.verify_record_verify,
+ REC_TYPE_checkpoint:
+ VerifyLibxc.verify_record_checkpoint,
+ }
diff --git a/tools/python/xen/migration/tests.py b/tools/python/xen/migration/tests.py
new file mode 100644
index 0000000..3e97268
--- /dev/null
+++ b/tools/python/xen/migration/tests.py
@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Unit tests for migration v2 streams
+"""
+
+import unittest
+
+from struct import calcsize
+
+from xen.migration import libxc, libxl
+
+class TestLibxc(unittest.TestCase):
+
+ def test_format_sizes(self):
+
+ for fmt, sz in ( (libxc.IHDR_FORMAT, 24),
+ (libxc.DHDR_FORMAT, 16),
+ (libxc.RH_FORMAT, 8),
+
+ (libxc.PAGE_DATA_FORMAT, 8),
+ (libxc.X86_PV_INFO_FORMAT, 8),
+ (libxc.X86_PV_P2M_FRAMES_FORMAT, 8),
+ (libxc.X86_PV_VCPU_HDR_FORMAT, 8),
+ (libxc.TSC_INFO_FORMAT, 24),
+ (libxc.HVM_PARAMS_ENTRY_FORMAT, 16),
+ (libxc.HVM_PARAMS_FORMAT, 8),
+ ):
+ self.assertEqual(calcsize(fmt), sz)
+
+
+def test_suite():
+ suite = unittest.TestSuite()
+
+ suite.addTest(unittest.makeSuite(TestLibxc))
+
+ return suite
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tools/python/xen/migration/verify.py b/tools/python/xen/migration/verify.py
new file mode 100644
index 0000000..7a42dbf
--- /dev/null
+++ b/tools/python/xen/migration/verify.py
@@ -0,0 +1,37 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Common verification infrastructure for v2 streams
+"""
+
+from struct import calcsize, unpack
+
+class StreamError(StandardError):
+ """Error with the stream"""
+ pass
+
+class RecordError(StandardError):
+ """Error with a record in the stream"""
+ pass
+
+
+class VerifyBase(object):
+
+ def __init__(self, info, read):
+
+ self.info = info
+ self.read = read
+
+ def rdexact(self, nr_bytes):
+ """Read exactly nr_bytes from the stream"""
+ _ = self.read(nr_bytes)
+ if len(_) != nr_bytes:
+ raise IOError("Stream truncated")
+ return _
+
+ def unpack_exact(self, fmt):
+ """Unpack a struct format string from the stream"""
+ sz = calcsize(fmt)
+ return unpack(fmt, self.rdexact(sz))
+
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 13/29] tools/python: Libxl migration v2 infrastructure
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (11 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 12/29] tools/python: Libxc migration v2 infrastructure Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 14/29] tools/python: Other migration infrastructure Andrew Cooper
` (16 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Contains:
* Python implementation of the libxl migration v2 records
* Verification code for spec compliance
* Unit tests
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/python/xen/migration/libxl.py | 188 +++++++++++++++++++++++++++++++++++
tools/python/xen/migration/tests.py | 13 +++
2 files changed, 201 insertions(+)
create mode 100644 tools/python/xen/migration/libxl.py
diff --git a/tools/python/xen/migration/libxl.py b/tools/python/xen/migration/libxl.py
new file mode 100644
index 0000000..4e1f4f8
--- /dev/null
+++ b/tools/python/xen/migration/libxl.py
@@ -0,0 +1,188 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Libxl Migration v2 streams
+
+Record structures as per docs/specs/libxl-migration-stream.pandoc, and
+verification routines.
+"""
+
+import sys
+
+from struct import calcsize, unpack
+from xen.migration.verify import StreamError, RecordError, VerifyBase
+from xen.migration.libxc import VerifyLibxc
+
+# Header
+HDR_FORMAT = "!QII"
+
+HDR_IDENT = 0x4c6962786c466d74 # "LibxlFmt" in ASCII
+HDR_VERSION = 2
+
+HDR_OPT_BIT_ENDIAN = 0
+HDR_OPT_BIT_LEGACY = 1
+
+HDR_OPT_LE = (0 << HDR_OPT_BIT_ENDIAN)
+HDR_OPT_BE = (1 << HDR_OPT_BIT_ENDIAN)
+HDR_OPT_LEGACY = (1 << HDR_OPT_BIT_LEGACY)
+
+HDR_OPT_RESZ_MASK = 0xfffc
+
+# Records
+RH_FORMAT = "II"
+
+REC_TYPE_end = 0x00000000
+REC_TYPE_libxc_context = 0x00000001
+REC_TYPE_xenstore_data = 0x00000002
+REC_TYPE_emulator_context = 0x00000003
+
+rec_type_to_str = {
+ REC_TYPE_end : "End",
+ REC_TYPE_libxc_context : "Libxc context",
+ REC_TYPE_xenstore_data : "Xenstore data",
+ REC_TYPE_emulator_context : "Emulator context",
+}
+
+# emulator_context
+EMULATOR_CONTEXT_FORMAT = "II"
+
+EMULATOR_ID_unknown = 0x00000000
+EMULATOR_ID_qemu_trad = 0x00000001
+EMULATOR_ID_qemu_upstream = 0x00000002
+
+emulator_id_to_str = {
+ EMULATOR_ID_unknown : "Unknown",
+ EMULATOR_ID_qemu_trad : "Qemu Traditional",
+ EMULATOR_ID_qemu_upstream : "Qemu Upstream",
+}
+
+
+#
+# libxl format
+#
+
+LIBXL_QEMU_SIGNATURE = "DeviceModelRecord0002"
+LIBXL_QEMU_RECORD_HDR = "=%dsI" % (len(LIBXL_QEMU_SIGNATURE), )
+
+class VerifyLibxl(VerifyBase):
+ """ Verify a Libxl v2 stream """
+
+ def __init__(self, info, read):
+ VerifyBase.__init__(self, info, read)
+
+
+ def verify(self):
+ """ Verity a libxl stream """
+
+ self.verify_hdr()
+
+ while self.verify_record() != REC_TYPE_end:
+ pass
+
+
+ def verify_hdr(self):
+ """ Verify a Header """
+ ident, version, options = self.unpack_exact(HDR_FORMAT)
+
+ if ident != HDR_IDENT:
+ raise StreamError("Bad image id: Expected 0x%x, got 0x%x"
+ % (HDR_IDENT, ident))
+
+ if version != HDR_VERSION:
+ raise StreamError("Unknown image version: Expected %d, got %d"
+ % (HDR_VERSION, version))
+
+ if options & HDR_OPT_RESZ_MASK:
+ raise StreamError("Reserved bits set in image options field: 0x%x"
+ % (options & HDR_OPT_RESZ_MASK))
+
+ if ( (sys.byteorder == "little") and
+ ((options & HDR_OPT_BIT_ENDIAN) != HDR_OPT_LE) ):
+ raise StreamError(
+ "Stream is not native endianess - unable to validate")
+
+ endian = ["little", "big"][options & HDR_OPT_LE]
+
+ if options & HDR_OPT_LEGACY:
+ self.info("Libxl Header: %s endian, legacy converted" % (endian, ))
+ else:
+ self.info("Libxl Header: %s endian" % (endian, ))
+
+
+ def verify_record(self):
+ """ Verify an individual record """
+ rtype, length = self.unpack_exact(RH_FORMAT)
+
+ if rtype not in rec_type_to_str:
+ raise StreamError("Unrecognised record type %x" % (rtype, ))
+
+ self.info("Libxl Record: %s, length %d"
+ % (rec_type_to_str[rtype], length))
+
+ contentsz = (length + 7) & ~7
+ content = self.rdexact(contentsz)
+
+ padding = content[length:]
+ if padding != "\x00" * len(padding):
+ raise StreamError("Padding containing non0 bytes found")
+
+ if rtype not in record_verifiers:
+ raise RuntimeError("No verification function for libxl record '%s'"
+ % rec_type_to_str[rtype])
+ else:
+ record_verifiers[rtype](self, content[:length])
+
+ return rtype
+
+
+ def verify_record_end(self, content):
+ """ End record """
+
+ if len(content) != 0:
+ raise RecordError("End record with non-zero length")
+
+
+ def verify_record_libxc_context(self, content):
+ """ Libxc context record """
+
+ if len(content) != 0:
+ raise RecordError("Libxc context record with non-zero length")
+
+ # Verify the libxc stream, as we can't seek forwards through it
+ VerifyLibxc(self.info, self.read).verify()
+
+
+ def verify_record_xenstore_data(self, content):
+ """ Xenstore Data record """
+
+ if len(content) == 0:
+ raise RecordError("Xenstore data record with zero length")
+
+
+ def verify_record_emulator_context(self, content):
+ """ Emulator Context record """
+ minsz = calcsize(EMULATOR_CONTEXT_FORMAT)
+
+ if len(content) < minsz:
+ raise RecordError("Length must be at least %d bytes, got %d"
+ % (minsz, len(content)))
+
+ emu_id, emu_idx = unpack(EMULATOR_CONTEXT_FORMAT, content[:minsz])
+
+ if emu_id not in emulator_id_to_str:
+ raise RecordError("Unrecognised emulator id 0x%x" % (emu_id, ))
+
+ self.info(" Index %d, type %s" % (emu_idx, emulator_id_to_str[emu_id]))
+
+
+record_verifiers = {
+ REC_TYPE_end:
+ VerifyLibxl.verify_record_end,
+ REC_TYPE_libxc_context:
+ VerifyLibxl.verify_record_libxc_context,
+ REC_TYPE_xenstore_data:
+ VerifyLibxl.verify_record_xenstore_data,
+ REC_TYPE_emulator_context:
+ VerifyLibxl.verify_record_emulator_context,
+}
diff --git a/tools/python/xen/migration/tests.py b/tools/python/xen/migration/tests.py
index 3e97268..91044cd 100644
--- a/tools/python/xen/migration/tests.py
+++ b/tools/python/xen/migration/tests.py
@@ -30,10 +30,23 @@ class TestLibxc(unittest.TestCase):
self.assertEqual(calcsize(fmt), sz)
+class TestLibxl(unittest.TestCase):
+
+ def test_format_sizes(self):
+
+ for fmt, sz in ( (libxl.HDR_FORMAT, 16),
+ (libxl.RH_FORMAT, 8),
+
+ (libxl.EMULATOR_CONTEXT_FORMAT, 8),
+ ):
+ self.assertEqual(calcsize(fmt), sz)
+
+
def test_suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(TestLibxc))
+ suite.addTest(unittest.makeSuite(TestLibxl))
return suite
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 14/29] tools/python: Other migration infrastructure
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (12 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 13/29] tools/python: Libxl " Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 15/29] tools/python: Verification utility for v2 stream spec compliance Andrew Cooper
` (15 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Contains:
* Reverse-engineered notes of the legacy format from xg_save_restore.h
* Python implementation of the legacy format
* Public HVM Params used in the legacy stream
* XL header format
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
New in v2 - removes various many magic numbers from subsequent scripts
---
tools/python/xen/migration/legacy.py | 279 ++++++++++++++++++++++++++++++++++
tools/python/xen/migration/public.py | 21 +++
tools/python/xen/migration/xl.py | 12 ++
3 files changed, 312 insertions(+)
create mode 100644 tools/python/xen/migration/legacy.py
create mode 100644 tools/python/xen/migration/public.py
create mode 100644 tools/python/xen/migration/xl.py
diff --git a/tools/python/xen/migration/legacy.py b/tools/python/xen/migration/legacy.py
new file mode 100644
index 0000000..2f2240a
--- /dev/null
+++ b/tools/python/xen/migration/legacy.py
@@ -0,0 +1,279 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Libxc legacy migration streams
+
+Documentation and record structures for legacy migration
+"""
+
+"""
+SAVE/RESTORE/MIGRATE PROTOCOL
+=============================
+
+The general form of a stream of chunks is a header followed by a
+body consisting of a variable number of chunks (terminated by a
+chunk with type 0) followed by a trailer.
+
+For a rolling/checkpoint (e.g. remus) migration then the body and
+trailer phases can be repeated until an external event
+(e.g. failure) causes the process to terminate and commit to the
+most recent complete checkpoint.
+
+HEADER
+------
+
+unsigned long : p2m_size
+
+extended-info (PV-only, optional):
+
+ If first unsigned long == ~0UL then extended info is present,
+ otherwise unsigned long is part of p2m. Note that p2m_size above
+ does not include the length of the extended info.
+
+ extended-info:
+
+ unsigned long : signature == ~0UL
+ uint32_t : number of bytes remaining in extended-info
+
+ 1 or more extended-info blocks of form:
+ char[4] : block identifier
+ uint32_t : block data size
+ bytes : block data
+
+ defined extended-info blocks:
+ "vcpu" : VCPU context info containing vcpu_guest_context_t.
+ The precise variant of the context structure
+ (e.g. 32 vs 64 bit) is distinguished by
+ the block size.
+ "extv" : Presence indicates use of extended VCPU context in
+ tail, data size is 0.
+
+p2m (PV-only):
+
+ consists of p2m_size bytes comprising an array of xen_pfn_t sized entries.
+
+BODY PHASE - Format A (for live migration or Remus without compression)
+----------
+
+A series of chunks with a common header:
+ int : chunk type
+
+If the chunk type is +ve then chunk contains guest memory data, and the
+type contains the number of pages in the batch:
+
+ unsigned long[] : PFN array, length == number of pages in batch
+ Each entry consists of XEN_DOMCTL_PFINFO_*
+ in bits 31-28 and the PFN number in bits 27-0.
+ page data : PAGE_SIZE bytes for each page marked present in PFN
+ array
+
+If the chunk type is -ve then chunk consists of one of a number of
+metadata types. See definitions of XC_SAVE_ID_* below.
+
+If chunk type is 0 then body phase is complete.
+
+
+BODY PHASE - Format B (for Remus with compression)
+----------
+
+A series of chunks with a common header:
+ int : chunk type
+
+If the chunk type is +ve then chunk contains array of PFNs corresponding
+to guest memory and type contains the number of PFNs in the batch:
+
+ unsigned long[] : PFN array, length == number of pages in batch
+ Each entry consists of XEN_DOMCTL_PFINFO_*
+ in bits 31-28 and the PFN number in bits 27-0.
+
+If the chunk type is -ve then chunk consists of one of a number of
+metadata types. See definitions of XC_SAVE_ID_* below.
+
+If the chunk type is -ve and equals XC_SAVE_ID_COMPRESSED_DATA, then the
+chunk consists of compressed page data, in the following format:
+
+ unsigned long : Size of the compressed chunk to follow
+ compressed data : variable length data of size indicated above.
+ This chunk consists of compressed page data.
+ The number of pages in one chunk depends on
+ the amount of space available in the sender's
+ output buffer.
+
+Format of compressed data:
+ compressed_data = <deltas>*
+ delta = <marker, run*>
+ marker = (RUNFLAG|SKIPFLAG) bitwise-or RUNLEN [1 byte marker]
+ RUNFLAG = 0
+ SKIPFLAG = 1 << 7
+ RUNLEN = 7-bit unsigned value indicating number of WORDS in the run
+ run = string of bytes of length sizeof(WORD) * RUNLEN
+
+ If marker contains RUNFLAG, then RUNLEN * sizeof(WORD) bytes of data following
+ the marker is copied into the target page at the appropriate offset indicated by
+ the offset_ptr
+ If marker contains SKIPFLAG, then the offset_ptr is advanced
+ by RUNLEN * sizeof(WORD).
+
+If chunk type is 0 then body phase is complete.
+
+There can be one or more chunks with type XC_SAVE_ID_COMPRESSED_DATA,
+containing compressed pages. The compressed chunks are collated to form
+one single compressed chunk for the entire iteration. The number of pages
+present in this final compressed chunk will be equal to the total number
+of valid PFNs specified by the +ve chunks.
+
+At the sender side, compressed pages are inserted into the output stream
+in the same order as they would have been if compression logic was absent.
+
+Until last iteration, the BODY is sent in Format A, to maintain live
+migration compatibility with receivers of older Xen versions.
+At the last iteration, if Remus compression was enabled, the sender sends
+a trigger, XC_SAVE_ID_ENABLE_COMPRESSION to tell the receiver to parse the
+BODY in Format B from the next iteration onwards.
+
+An example sequence of chunks received in Format B:
+ +16 +ve chunk
+ unsigned long[16] PFN array
+ +100 +ve chunk
+ unsigned long[100] PFN array
+ +50 +ve chunk
+ unsigned long[50] PFN array
+
+ XC_SAVE_ID_COMPRESSED_DATA TAG
+ N Length of compressed data
+ N bytes of DATA Decompresses to 166 pages
+
+ XC_SAVE_ID_* other xc save chunks
+ 0 END BODY TAG
+
+Corner case with checkpoint compression:
+ At sender side, after pausing the domain, dirty pages are usually
+ copied out to a temporary buffer. After the domain is resumed,
+ compression is done and the compressed chunk(s) are sent, followed by
+ other XC_SAVE_ID_* chunks.
+ If the temporary buffer gets full while scanning for dirty pages,
+ the sender stops buffering of dirty pages, compresses the temporary
+ buffer and sends the compressed data with XC_SAVE_ID_COMPRESSED_DATA.
+ The sender then resumes the buffering of dirty pages and continues
+ scanning for the dirty pages.
+ For e.g., assume that the temporary buffer can hold 4096 pages and
+ there are 5000 dirty pages. The following is the sequence of chunks
+ that the receiver will see:
+
+ +1024 +ve chunk
+ unsigned long[1024] PFN array
+ +1024 +ve chunk
+ unsigned long[1024] PFN array
+ +1024 +ve chunk
+ unsigned long[1024] PFN array
+ +1024 +ve chunk
+ unsigned long[1024] PFN array
+
+ XC_SAVE_ID_COMPRESSED_DATA TAG
+ N Length of compressed data
+ N bytes of DATA Decompresses to 4096 pages
+
+ +4 +ve chunk
+ unsigned long[4] PFN array
+
+ XC_SAVE_ID_COMPRESSED_DATA TAG
+ M Length of compressed data
+ M bytes of DATA Decompresses to 4 pages
+
+ XC_SAVE_ID_* other xc save chunks
+ 0 END BODY TAG
+
+ In other words, XC_SAVE_ID_COMPRESSED_DATA can be interleaved with
+ +ve chunks arbitrarily. But at the receiver end, the following condition
+ always holds true until the end of BODY PHASE:
+ num(PFN entries +ve chunks) >= num(pages received in compressed form)
+
+TAIL PHASE
+----------
+
+Content differs for PV and HVM guests.
+
+HVM TAIL:
+
+ "Magic" pages:
+ uint64_t : I/O req PFN
+ uint64_t : Buffered I/O req PFN
+ uint64_t : Store PFN
+ Xen HVM Context:
+ uint32_t : Length of context in bytes
+ bytes : Context data
+ Qemu context:
+ char[21] : Signature:
+ "QemuDeviceModelRecord" : Read Qemu save data until EOF
+ "DeviceModelRecord0002" : uint32_t length field followed by that many
+ bytes of Qemu save data
+ "RemusDeviceModelState" : Currently the same as "DeviceModelRecord0002".
+
+PV TAIL:
+
+ Unmapped PFN list : list of all the PFNs that were not in map at the close
+ unsigned int : Number of unmapped pages
+ unsigned long[] : PFNs of unmapped pages
+
+ VCPU context data : A series of VCPU records, one per present VCPU
+ Maximum and present map supplied in XC_SAVE_ID_VCPUINFO
+ bytes: : VCPU context structure. Size is determined by size
+ provided in extended-info header
+ bytes[128] : Extended VCPU context (present IFF "extv" block
+ present in extended-info header)
+
+ Shared Info Page : 4096 bytes of shared info page
+"""
+
+CHUNK_end = 0
+CHUNK_enable_verify_mode = -1
+CHUNK_vcpu_info = -2
+CHUNK_hvm_ident_pt = -3
+CHUNK_hvm_vm86_tss = -4
+CHUNK_tmem = -5
+CHUNK_tmem_extra = -6
+CHUNK_tsc_info = -7
+CHUNK_hvm_console_pfn = -8
+CHUNK_last_checkpoint = -9
+CHUNK_hvm_acpi_ioports_location = -10
+CHUNK_hvm_viridian = -11
+CHUNK_compressed_data = -12
+CHUNK_enable_compression = -13
+CHUNK_hvm_generation_id_addr = -14
+CHUNK_hvm_paging_ring_pfn = -15
+CHUNK_hvm_monitor_ring_pfn = -16
+CHUNK_hvm_sharing_ring_pfn = -17
+CHUNK_toolstack = -18
+CHUNK_hvm_ioreq_server_pfn = -19
+CHUNK_hvm_nr_ioreq_server_pages = -20
+
+chunk_type_to_str = {
+ CHUNK_end : "end",
+ CHUNK_enable_verify_mode : "enable_verify_mode",
+ CHUNK_vcpu_info : "vcpu_info",
+ CHUNK_hvm_ident_pt : "hvm_ident_pt",
+ CHUNK_hvm_vm86_tss : "hvm_vm86_tss",
+ CHUNK_tmem : "tmem",
+ CHUNK_tmem_extra : "tmem_extra",
+ CHUNK_tsc_info : "tsc_info",
+ CHUNK_hvm_console_pfn : "hvm_console_pfn",
+ CHUNK_last_checkpoint : "last_checkpoint",
+ CHUNK_hvm_acpi_ioports_location : "hvm_acpi_ioports_location",
+ CHUNK_hvm_viridian : "hvm_viridian",
+ CHUNK_compressed_data : "compressed_data",
+ CHUNK_enable_compression : "enable_compression",
+ CHUNK_hvm_generation_id_addr : "hvm_generation_id_addr",
+ CHUNK_hvm_paging_ring_pfn : "hvm_paging_ring_pfn",
+ CHUNK_hvm_monitor_ring_pfn : "hvm_monitor_ring_pfn",
+ CHUNK_hvm_sharing_ring_pfn : "hvm_sharing_ring_pfn",
+ CHUNK_toolstack : "toolstack",
+ CHUNK_hvm_ioreq_server_pfn : "hvm_ioreq_server_pfn",
+ CHUNK_hvm_nr_ioreq_server_pages : "hvm_nr_ioreq_server_pages",
+}
+
+# Up to 1024 pages (4MB) at a time
+MAX_BATCH = 1024
+
+# Maximum #VCPUs currently supported for save/restore
+MAX_VCPU_ID = 4095
diff --git a/tools/python/xen/migration/public.py b/tools/python/xen/migration/public.py
new file mode 100644
index 0000000..fab2f84
--- /dev/null
+++ b/tools/python/xen/migration/public.py
@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Xen public ABI constants, used in migration
+"""
+
+HVM_PARAM_STORE_PFN = 1
+HVM_PARAM_IOREQ_PFN = 5
+HVM_PARAM_BUFIOREQ_PFN = 6
+HVM_PARAM_VIRIDIAN = 9
+HVM_PARAM_IDENT_PT = 12
+HVM_PARAM_VM86_TSS = 15
+HVM_PARAM_CONSOLE_PFN = 17
+HVM_PARAM_ACPI_IOPORTS_LOCATION = 19
+HVM_PARAM_PAGING_RING_PFN = 27
+HVM_PARAM_MONITOR_RING_PFN = 28
+HVM_PARAM_SHARING_RING_PFN = 29
+HVM_PARAM_IOREQ_SERVER_PFN = 32
+HVM_PARAM_NR_IOREQ_SERVER_PAGES = 33
+HVM_PARAM_VM_GENERATION_ID_ADDR = 34
diff --git a/tools/python/xen/migration/xl.py b/tools/python/xen/migration/xl.py
new file mode 100644
index 0000000..978e744
--- /dev/null
+++ b/tools/python/xen/migration/xl.py
@@ -0,0 +1,12 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+XL migration stream format
+"""
+
+MAGIC = "Xen saved domain, xl format\n \0 \r"
+
+HEADER_FORMAT = "=IIII"
+
+MANDATORY_FLAG_STREAMV2 = 2
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 15/29] tools/python: Verification utility for v2 stream spec compliance
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (13 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 14/29] tools/python: Other migration infrastructure Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 16/29] tools/python: Conversion utility for legacy migration streams Andrew Cooper
` (14 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
This is exceedingly useful for development, but not of practical use being
installed into a production dom0.
---
tools/python/scripts/verify-stream-v2 | 174 +++++++++++++++++++++++++++++++++
1 file changed, 174 insertions(+)
create mode 100755 tools/python/scripts/verify-stream-v2
diff --git a/tools/python/scripts/verify-stream-v2 b/tools/python/scripts/verify-stream-v2
new file mode 100755
index 0000000..3daf257
--- /dev/null
+++ b/tools/python/scripts/verify-stream-v2
@@ -0,0 +1,174 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+""" Verify a v2 format migration stream """
+
+import sys
+import struct
+import os, os.path
+import syslog
+import traceback
+
+from xen.migration.verify import StreamError, RecordError
+from xen.migration.libxc import VerifyLibxc
+from xen.migration.libxl import VerifyLibxl
+
+fin = None # Input file/fd
+log_to_syslog = False # Boolean - Log to syslog instead of stdout/err?
+verbose = False # Boolean - Summarise stream contents
+quiet = False # Boolean - Suppress error printing
+
+def info(msg):
+ """Info message, routed to appropriate destination"""
+ if not quiet and verbose:
+ if log_to_syslog:
+ for line in msg.split("\n"):
+ syslog.syslog(syslog.LOG_INFO, line)
+ else:
+ print msg
+
+def err(msg):
+ """Error message, routed to appropriate destination"""
+ if not quiet:
+ if log_to_syslog:
+ for line in msg.split("\n"):
+ syslog.syslog(syslog.LOG_ERR, line)
+ print >> sys.stderr, msg
+
+def stream_read(_ = None):
+ """Read from input"""
+ return fin.read(_)
+
+def rdexact(nr_bytes):
+ """Read exactly nr_bytes from fin"""
+ _ = stream_read(nr_bytes)
+ if len(_) != nr_bytes:
+ raise IOError("Stream truncated")
+ return _
+
+def unpack_exact(fmt):
+ """Unpack a format from fin"""
+ sz = struct.calcsize(fmt)
+ return struct.unpack(fmt, rdexact(sz))
+
+
+def skip_xl_header():
+ """Skip over an xl header in the stream"""
+
+ hdr = rdexact(32)
+ if hdr != "Xen saved domain, xl format\n \0 \r":
+ raise StreamError("No xl header")
+
+ _, mflags, _, optlen = unpack_exact("=IIII")
+ _ = rdexact(optlen)
+
+ info("Processed xl header")
+
+ if mflags & 2: # XL_MANDATORY_FLAG_STREAMv2
+ return "libxl"
+ else:
+ return "libxc"
+
+def read_stream(fmt):
+ """ Read an entire stream """
+
+ try:
+ if fmt == "xl":
+ fmt = skip_xl_header()
+
+ if fmt == "libxc":
+ VerifyLibxc(info, stream_read).verify()
+ else:
+ VerifyLibxl(info, stream_read).verify()
+
+ except (IOError, StreamError, RecordError):
+ err("Stream Error:")
+ err(traceback.format_exc())
+ return 1
+
+ except StandardError:
+ err("Script Error:")
+ err(traceback.format_exc())
+ err("Please fix me")
+ return 2
+
+ return 0
+
+def open_file_or_fd(val, mode, buffering):
+ """
+ If 'val' looks like a decimal integer, open it as an fd. If not, try to
+ open it as a regular file.
+ """
+
+ fd = -1
+ try:
+ # Does it look like an integer?
+ try:
+ fd = int(val, 10)
+ except ValueError:
+ pass
+
+ # Try to open it...
+ if fd != -1:
+ return os.fdopen(fd, mode, buffering)
+ else:
+ return open(val, mode, buffering)
+
+ except StandardError, e:
+ if fd != -1:
+ err("Unable to open fd %d: %s: %s" %
+ (fd, e.__class__.__name__, e))
+ else:
+ err("Unable to open file '%s': %s: %s" %
+ (val, e.__class__.__name__, e))
+
+ raise SystemExit(2)
+
+def main():
+ """ main """
+ from optparse import OptionParser
+ global fin, quiet, verbose
+
+ # Change stdout to be line-buffered.
+ sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1)
+
+ parser = OptionParser(usage = "%prog [options]",
+ description =
+ "Verify a stream according to the v2 spec")
+
+ # Optional options
+ parser.add_option("-i", "--in", dest = "fin", metavar = "<FD or FILE>",
+ default = "0",
+ help = "Stream to verify (defaults to stdin)")
+ parser.add_option("-v", "--verbose", action = "store_true", default = False,
+ help = "Summarise stream contents")
+ parser.add_option("-q", "--quiet", action = "store_true", default = False,
+ help = "Suppress all logging/errors")
+ parser.add_option("-f", "--format", dest = "format",
+ metavar = "<libxc|libxl|xl>", default = "libxc",
+ choices = ["libxc", "libxl", "xl"],
+ help = "Format of the incoming stream (defaults to libxc)")
+ parser.add_option("--syslog", action = "store_true", default = False,
+ help = "Log to syslog instead of stdout")
+
+ opts, _ = parser.parse_args()
+
+ if opts.syslog:
+ global log_to_syslog
+
+ syslog.openlog("verify-stream-v2", syslog.LOG_PID)
+ log_to_syslog = True
+
+ verbose = opts.verbose
+ quiet = opts.quiet
+ fin = open_file_or_fd(opts.fin, "rb", 0)
+
+ return read_stream(opts.format)
+
+if __name__ == "__main__":
+ try:
+ sys.exit(main())
+ except SystemExit, e:
+ sys.exit(e.code)
+ except KeyboardInterrupt:
+ sys.exit(2)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 16/29] tools/python: Conversion utility for legacy migration streams
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (14 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 15/29] tools/python: Verification utility for v2 stream spec compliance Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 17/29] tools/libxl: Migration v2 stream format Andrew Cooper
` (13 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
This utility will take a legacy stream as in input, and produce a v2
stream as an output. It is exec()'d by libxl to provide backwards
compatibility.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/python/Makefile | 4 +
tools/python/scripts/convert-legacy-stream | 678 ++++++++++++++++++++++++++++
2 files changed, 682 insertions(+)
create mode 100755 tools/python/scripts/convert-legacy-stream
diff --git a/tools/python/Makefile b/tools/python/Makefile
index e933be8..df942a7 100644
--- a/tools/python/Makefile
+++ b/tools/python/Makefile
@@ -17,9 +17,13 @@ build: genwrap.py $(XEN_ROOT)/tools/libxl/libxl_types.idl \
.PHONY: install
install:
+ $(INSTALL_DIR) $(DESTDIR)$(PRIVATE_BINDIR)
+
CC="$(CC)" CFLAGS="$(PY_CFLAGS)" $(PYTHON) setup.py install \
$(PYTHON_PREFIX_ARG) --root="$(DESTDIR)" --force
+ $(INSTALL_PROG) scripts/convert-legacy-stream $(DESTDIR)$(PRIVATE_BINDIR)
+
.PHONY: test
test:
export LD_LIBRARY_PATH=$$(readlink -f ../libxc):$$(readlink -f ../xenstore); $(PYTHON) test.py -b -u
diff --git a/tools/python/scripts/convert-legacy-stream b/tools/python/scripts/convert-legacy-stream
new file mode 100755
index 0000000..d54fa22
--- /dev/null
+++ b/tools/python/scripts/convert-legacy-stream
@@ -0,0 +1,678 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Convert a legacy migration stream to a v2 stream.
+"""
+
+import sys
+import os, os.path
+import syslog
+import traceback
+
+from struct import calcsize, unpack, pack
+
+from xen.migration import legacy, public, libxc, libxl, xl
+
+__version__ = 1
+
+fin = None # Input file/fd
+fout = None # Output file/fd
+twidth = 0 # Legacy toolstack bitness (32 or 64)
+pv = None # Boolean (pv or hvm)
+qemu = True # Boolean - process qemu record?
+log_to_syslog = False # Boolean - Log to syslog instead of stdout/err?
+verbose = False # Boolean - Summarise stream contents
+
+def stream_read(_ = None):
+ """Read from the input"""
+ return fin.read(_)
+
+def stream_write(_):
+ """Write to the output"""
+ return fout.write(_)
+
+def info(msg):
+ """Info message, routed to appropriate destination"""
+ if verbose:
+ if log_to_syslog:
+ for line in msg.split("\n"):
+ syslog.syslog(syslog.LOG_INFO, line)
+ else:
+ print msg
+
+def err(msg):
+ """Error message, routed to appropriate destination"""
+ if log_to_syslog:
+ for line in msg.split("\n"):
+ syslog.syslog(syslog.LOG_ERR, line)
+ print >> sys.stderr, msg
+
+class StreamError(StandardError):
+ """Error with the incoming migration stream"""
+ pass
+
+class VM(object):
+ """Container of VM parameters"""
+
+ def __init__(self, fmt):
+ # Common
+ self.p2m_size = 0
+
+ # PV
+ self.max_vcpu_id = 0
+ self.online_vcpu_map = []
+ self.width = 0
+ self.levels = 0
+ self.basic_len = 0
+ self.extd = False
+ self.xsave_len = 0
+
+ # libxl
+ self.libxl = fmt == "libxl"
+ self.xenstore = [] # Deferred "toolstack" records
+
+def write_libxc_ihdr():
+ stream_write(pack(libxc.IHDR_FORMAT,
+ libxc.IHDR_MARKER, # Marker
+ libxc.IHDR_IDENT, # Ident
+ libxc.IHDR_VERSION, # Version
+ libxc.IHDR_OPT_LE, # Options
+ 0, 0)) # Reserved
+
+def write_libxc_dhdr():
+ if pv:
+ dtype = libxc.DHDR_TYPE_x86_pv
+ else:
+ dtype = libxc.DHDR_TYPE_x86_hvm
+
+ stream_write(pack(libxc.DHDR_FORMAT,
+ dtype, # Type
+ 12, # Page size
+ 0, # Reserved
+ 0, # Xen major (converted)
+ __version__)) # Xen minor (converted)
+
+def write_libxl_hdr():
+ stream_write(pack(libxl.HDR_FORMAT,
+ libxl.HDR_IDENT, # Ident
+ libxl.HDR_VERSION, # Version 2
+ libxl.HDR_OPT_LE | # Options
+ libxl.HDR_OPT_LEGACY # Little Endian and Legacy
+ ))
+
+def write_record(rt, *argl):
+ alldata = ''.join(argl)
+ length = len(alldata)
+
+ record = pack(libxc.RH_FORMAT, rt, length) + alldata
+ plen = (8 - (length & 7)) & 7
+ record += '\x00' * plen
+
+ stream_write(record)
+
+def write_libxc_pv_info(vm):
+ write_record(libxc.REC_TYPE_x86_pv_info,
+ pack(libxc.X86_PV_INFO_FORMAT,
+ vm.width, vm.levels, 0, 0))
+
+def write_libxc_pv_p2m_frames(vm, pfns):
+ write_record(libxc.REC_TYPE_x86_pv_p2m_frames,
+ pack(libxc.X86_PV_P2M_FRAMES_FORMAT,
+ 0, vm.p2m_size - 1),
+ pack("Q" * len(pfns), *pfns))
+
+def write_libxc_pv_vcpu_basic(vcpu_id, data):
+ write_record(libxc.REC_TYPE_x86_pv_vcpu_basic,
+ pack(libxc.X86_PV_VCPU_HDR_FORMAT, vcpu_id, 0), data)
+
+def write_libxc_pv_vcpu_extd(vcpu_id, data):
+ write_record(libxc.REC_TYPE_x86_pv_vcpu_extended,
+ pack(libxc.X86_PV_VCPU_HDR_FORMAT, vcpu_id, 0), data)
+
+def write_libxc_pv_vcpu_xsave(vcpu_id, data):
+ write_record(libxc.REC_TYPE_x86_pv_vcpu_xsave,
+ pack(libxc.X86_PV_VCPU_HDR_FORMAT, vcpu_id, 0), data)
+
+def write_page_data(pfns, pages):
+ if fout is None: # Save copying 1M buffers around for no reason
+ return
+
+ new_pfns = [(((x & 0xf0000000) << 32) | (x & 0x0fffffff)) for x in pfns]
+
+ # Optimise the needless buffer copying in write_record()
+ stream_write(pack(libxc.RH_FORMAT,
+ libxc.REC_TYPE_page_data,
+ 8 + (len(new_pfns) * 8) + len(pages)))
+ stream_write(pack(libxc.PAGE_DATA_FORMAT, len(new_pfns), 0))
+ stream_write(pack("Q" * len(new_pfns), *new_pfns))
+ stream_write(pages)
+
+def write_libxc_tsc_info(mode, khz, nsec, incarn):
+ write_record(libxc.REC_TYPE_tsc_info,
+ pack(libxc.TSC_INFO_FORMAT,
+ mode, khz, nsec, incarn, 0))
+
+def write_libxc_hvm_params(params):
+ if pv:
+ raise StreamError("HVM-only param in PV stream")
+ elif len(params) % 2:
+ raise RuntimeError("Expected even length list of hvm parameters")
+
+ write_record(libxc.REC_TYPE_hvm_params,
+ pack(libxc.HVM_PARAMS_FORMAT, len(params) / 2, 0),
+ pack("Q" * len(params), *params))
+
+def write_libxl_end():
+ write_record(libxl.REC_TYPE_end, "")
+
+def write_libxl_libxc_context():
+ write_record(libxl.REC_TYPE_libxc_context, "")
+
+def write_libxl_xenstore_data(data):
+ write_record(libxl.REC_TYPE_xenstore_data, data)
+
+def write_libxl_emulator_context(blob):
+ write_record(libxl.REC_TYPE_emulator_context,
+ pack(libxl.EMULATOR_CONTEXT_FORMAT,
+ libxl.EMULATOR_ID_unknown, 0) + blob)
+
+def rdexact(nr_bytes):
+ """Read exactly nr_bytes from fin"""
+ _ = stream_read(nr_bytes)
+ if len(_) != nr_bytes:
+ raise IOError("Stream truncated")
+ return _
+
+def unpack_exact(fmt):
+ """Unpack a format from fin"""
+ sz = calcsize(fmt)
+ return unpack(fmt, rdexact(sz))
+
+def unpack_ulongs(nr_ulongs):
+ if twidth == 32:
+ return unpack_exact("I" * nr_ulongs)
+ else:
+ return unpack_exact("Q" * nr_ulongs)
+
+def read_pv_extended_info(vm):
+
+ marker, = unpack_ulongs(1)
+
+ if twidth == 32:
+ expected = 0xffffffff
+ else:
+ expected = 0xffffffffffffffff
+
+ if marker != expected:
+ raise StreamError("Unexpected extended info marker 0x%x" % (marker, ))
+
+ total_length, = unpack_exact("I")
+ so_far = 0
+
+ info("Extended Info: length 0x%x" % (total_length, ))
+
+ while so_far < total_length:
+
+ blkid, datasz = unpack_exact("=4sI")
+ so_far += 8
+
+ info(" Record type: %s, size 0x%x" % (blkid, datasz))
+
+ data = rdexact(datasz)
+ so_far += datasz
+
+ # Eww, but this is how it is done :(
+ if blkid == "vcpu":
+
+ vm.basic_len = datasz
+
+ if datasz == 0x1430:
+ vm.width = 8
+ vm.levels = 4
+ info(" 64bit domain, 4 levels")
+ elif datasz == 0xaf0:
+ vm.width = 4
+ vm.levels = 3
+ info(" 32bit domain, 3 levels")
+ else:
+ raise StreamError("Unable to determine guest width/level")
+
+ write_libxc_pv_info(vm)
+
+ elif blkid == "extv":
+ vm.extd = True
+
+ elif blkid == "xcnt":
+ vm.xsave_len, = unpack("I", data[:4])
+ info("xcnt sz 0x%x" % (vm.xsave_len, ))
+
+ else:
+ raise StreamError("Unrecognised extended block")
+
+
+ if so_far != total_length:
+ raise StreamError("Overshot Extended Info size by %d bytes"
+ % (so_far - total_length,))
+
+def read_pv_p2m_frames(vm):
+ fpp = 4096 / vm.width
+ p2m_frame_len = (vm.p2m_size - 1) / fpp + 1
+
+ info("P2M frames: fpp %d, p2m_frame_len %d" % (fpp, p2m_frame_len))
+ write_libxc_pv_p2m_frames(vm, unpack_ulongs(p2m_frame_len))
+
+def read_pv_tail(vm):
+
+ nr_unmapped_pfns, = unpack_exact("I")
+
+ if nr_unmapped_pfns != 0:
+ # "Unmapped" pfns are bogus
+ _ = unpack_ulongs(nr_unmapped_pfns)
+ info("discarding %d bogus 'unmapped pfns'" % (nr_unmapped_pfns, ))
+
+ for vcpu_id in vm.online_vcpu_map:
+
+ basic = rdexact(vm.basic_len)
+ info("Got VCPU basic (size 0x%x)" % (vm.basic_len, ))
+ write_libxc_pv_vcpu_basic(vcpu_id, basic)
+
+ if vm.extd:
+ extd = rdexact(128)
+ info("Got VCPU extd (size 0x%x)" % (128, ))
+ write_libxc_pv_vcpu_extd(vcpu_id, extd)
+
+ if vm.xsave_len:
+ mask, size = unpack_exact("QQ")
+ assert vm.xsave_len - 16 == size
+
+ xsave = rdexact(size)
+ info("Got VCPU xsave (mask 0x%x, size 0x%x)" % (mask, size))
+ write_libxc_pv_vcpu_xsave(vcpu_id, xsave)
+
+ shinfo = rdexact(4096)
+ info("Got shinfo")
+
+ write_record(libxc.REC_TYPE_shared_info, shinfo)
+ write_record(libxc.REC_TYPE_end, "")
+
+
+def read_chunks(vm):
+
+ hvm_params = []
+
+ while True:
+
+ marker, = unpack_exact("=i")
+ if marker <= 0:
+ info("Chunk: %d - %s" %
+ (marker, legacy.chunk_type_to_str.get(marker, "unknown")))
+
+ if marker == legacy.CHUNK_end:
+ info(" End")
+
+ if hvm_params:
+ write_libxc_hvm_params(hvm_params)
+
+ return
+
+ elif marker > 0:
+
+ if marker > legacy.MAX_BATCH:
+ raise StreamError("Page batch (%d) exceeded MAX_BATCH (%d)"
+ % (marker, legacy.MAX_BATCH))
+ pfns = unpack_ulongs(marker)
+
+ # xc_domain_save() leaves many XEN_DOMCTL_PFINFO_XTAB records for
+ # sequences of pfns it cant map. Drop these.
+ pfns = [ x for x in pfns if x != 0xf0000000 ]
+
+ if len(set(pfns)) != len(pfns):
+ raise StreamError("Duplicate pfns in batch")
+
+ nr_pages = len([x for x in pfns if (x & 0xf0000000) < 0xd0000000])
+ pages = rdexact(nr_pages * 4096)
+
+ write_page_data(pfns, pages)
+
+ elif marker == legacy.CHUNK_enable_verify_mode:
+ # For debugging purposes only. Will not be seen in real migration
+ raise RuntimeError("Unable to convert a debug stream")
+
+ elif marker == legacy.CHUNK_vcpu_info:
+ max_id, = unpack_exact("i")
+
+ if max_id > legacy.MAX_VCPU_ID:
+ raise StreamError("Vcpu max_id out of range: %d > %d"
+ % (max_id, legacy.MAX_VCPU_ID))
+
+ vm.max_vcpu_id = max_id
+ bitmap = unpack_exact("Q" * ((max_id/64) + 1))
+
+ for idx, word in enumerate(bitmap):
+ bit_idx = 0
+
+ while word > 0:
+ if word & 1:
+ vm.online_vcpu_map.append((idx * 64) + bit_idx)
+
+ bit_idx += 1
+ word >>= 1
+
+ info(" Vcpu info: max_id %d, online map %s"
+ % (vm.max_vcpu_id, vm.online_vcpu_map))
+
+ elif marker == legacy.CHUNK_hvm_ident_pt:
+ _, ident_pt = unpack_exact("=IQ")
+ info(" EPT Identity Pagetable: 0x%x" % (ident_pt, ))
+ hvm_params.extend([public.HVM_PARAM_IDENT_PT, ident_pt])
+
+ elif marker == legacy.CHUNK_hvm_vm86_tss:
+ _, vm86_tss = unpack_exact("=IQ")
+ info(" VM86 TSS: 0x%x" % (vm86_tss, ))
+ hvm_params.extend([public.HVM_PARAM_VM86_TSS, vm86_tss])
+
+ elif marker == legacy.CHUNK_tmem:
+ raise RuntimeError("todo")
+
+ elif marker == legacy.CHUNK_tmem_extra:
+ raise RuntimeError("todo")
+
+ elif marker == legacy.CHUNK_tsc_info:
+ mode, nsec, khz, incarn = unpack_exact("=IQII")
+ info(" TSC_INFO: mode %s, %d ns, %d khz, %d incarn"
+ % (mode, nsec, khz, incarn))
+ write_libxc_tsc_info(mode, khz, nsec, incarn)
+
+ elif marker == legacy.CHUNK_hvm_console_pfn:
+ _, console_pfn = unpack_exact("=IQ")
+ info(" Console pfn: 0x%x" % (console_pfn, ))
+ hvm_params.extend([public.HVM_PARAM_CONSOLE_PFN, console_pfn])
+
+ elif marker == legacy.CHUNK_last_checkpoint:
+ info(" Last Checkpoint")
+ # Nothing to do
+
+ elif marker == legacy.CHUNK_hvm_acpi_ioports_location:
+ _, loc = unpack_exact("=IQ")
+ info(" ACPI ioport location: 0x%x" % (loc, ))
+ hvm_params.extend([public.HVM_PARAM_ACPI_IOPORTS_LOCATION, loc])
+
+ elif marker == legacy.CHUNK_hvm_viridian:
+ _, loc = unpack_exact("=IQ")
+ info(" Viridian location: 0x%x" % (loc, ))
+ hvm_params.extend([public.HVM_PARAM_VIRIDIAN, loc])
+
+ elif marker == legacy.CHUNK_compressed_data:
+ sz, = unpack_exact("I")
+ data = rdexact(sz)
+ info(" Compressed Data: sz 0x%x" % (sz, ))
+ raise RuntimeError("todo")
+
+ elif marker == legacy.CHUNK_enable_compression:
+ raise RuntimeError("todo")
+
+ elif marker == legacy.CHUNK_hvm_generation_id_addr:
+ _, genid_loc = unpack_exact("=IQ")
+ info(" Generation ID Address: 0x%x" % (genid_loc, ))
+ hvm_params.extend(
+ [public.HVM_PARAM_VM_GENERATION_ID_ADDR, genid_loc])
+
+ elif marker == legacy.CHUNK_hvm_paging_ring_pfn:
+ _, pfn = unpack_exact("=IQ")
+ info(" Paging ring pfn: 0x%x" % (pfn, ))
+ hvm_params.extend([public.HVM_PARAM_PAGING_RING_PFN, pfn])
+
+ elif marker == legacy.CHUNK_hvm_monitor_ring_pfn:
+ _, pfn = unpack_exact("=IQ")
+ info(" Monitor ring pfn: 0x%x" % (pfn, ))
+ hvm_params.extend([public.HVM_PARAM_MONITOR_RING_PFN, pfn])
+
+ elif marker == legacy.CHUNK_hvm_sharing_ring_pfn:
+ _, pfn = unpack_exact("=IQ")
+ info(" Sharing ring pfn: 0x%x" % (pfn, ))
+ hvm_params.extend([public.HVM_PARAM_SHARING_RING_PFN, pfn])
+
+ elif marker == legacy.CHUNK_toolstack:
+ sz, = unpack_exact("I")
+
+ if sz:
+ data = rdexact(sz)
+ info(" Toolstack Data: sz 0x%x" % (sz, ))
+
+ if vm.libxl:
+ vm.xenstore.append(data)
+ else:
+ info(" Discarding")
+
+ elif marker == legacy.CHUNK_hvm_ioreq_server_pfn:
+ _, pfn = unpack_exact("=IQ")
+ info(" IOREQ server pfn: 0x%x" % (pfn, ))
+ hvm_params.extend([public.HVM_PARAM_IOREQ_SERVER_PFN, pfn])
+
+ elif marker == legacy.CHUNK_hvm_nr_ioreq_server_pages:
+ _, nr_pages = unpack_exact("=IQ")
+ info(" IOREQ server pages: %d" % (nr_pages, ))
+ hvm_params.extend(
+ [public.HVM_PARAM_NR_IOREQ_SERVER_PAGES, nr_pages])
+
+ else:
+ raise StreamError("Unrecognised chunk %d" % (marker,))
+
+def read_hvm_tail(vm):
+
+ io, bufio, store = unpack_exact("QQQ")
+ info("Magic pfns: 0x%x 0x%x 0x%x" % (io, bufio, store))
+ write_libxc_hvm_params([public.HVM_PARAM_IOREQ_PFN, io,
+ public.HVM_PARAM_BUFIOREQ_PFN, bufio,
+ public.HVM_PARAM_STORE_PFN, store])
+
+ blobsz, = unpack_exact("I")
+ info("Got HVM Context (0x%x bytes)" % (blobsz, ))
+ blob = rdexact(blobsz)
+
+ write_record(libxc.REC_TYPE_hvm_context, blob)
+ write_record(libxc.REC_TYPE_end, "")
+
+
+
+def read_qemu(vm):
+
+ rawsig = rdexact(21)
+ sig, = unpack("21s", rawsig)
+ info("Qemu signature: %s" % (sig, ))
+
+ if sig == "DeviceModelRecord0002":
+ rawsz = rdexact(4)
+ sz, = unpack("I", rawsz)
+ qdata = rdexact(sz)
+
+ if vm.libxl:
+ write_libxl_emulator_context(qdata)
+ else:
+ stream_write(rawsig)
+ stream_write(rawsz)
+ stream_write(qdata)
+
+ else:
+ raise RuntimeError("Unrecognised Qemu sig '%s'" % (sig, ))
+
+
+def skip_xl_header(fmt):
+ """Skip over an xl header in the stream"""
+
+ hdr = rdexact(len(xl.MAGIC))
+ if hdr != xl.MAGIC:
+ raise StreamError("No xl header")
+
+ byteorder, mflags, oflags, optlen = unpack_exact(xl.HEADER_FORMAT)
+
+ if fmt == "libxl":
+ mflags |= xl.MANDATORY_FLAG_STREAMV2
+
+ opts = pack(xl.HEADER_FORMAT, byteorder, mflags, oflags, optlen)
+
+ optdata = rdexact(optlen)
+
+ info("Processed xl header")
+
+ stream_write(hdr)
+ stream_write(opts)
+ stream_write(optdata)
+
+def read_legacy_stream(vm):
+
+ try:
+ vm.p2m_size, = unpack_ulongs(1)
+ info("P2M Size: 0x%x" % (vm.p2m_size,))
+
+ if vm.libxl:
+ write_libxl_hdr()
+ write_libxl_libxc_context()
+
+ write_libxc_ihdr()
+ write_libxc_dhdr()
+
+ if pv:
+ read_pv_extended_info(vm)
+ read_pv_p2m_frames(vm)
+
+ read_chunks(vm)
+
+ if pv:
+ read_pv_tail(vm)
+ else:
+ read_hvm_tail(vm)
+
+ if vm.libxl:
+ for rec in vm.xenstore:
+ write_libxl_xenstore_data(rec)
+
+ if not pv and (vm.libxl or qemu):
+ read_qemu(vm)
+
+ if vm.libxl:
+ write_libxl_end()
+
+ except (IOError, StreamError):
+ err("Stream Error:")
+ err(traceback.format_exc())
+ return 1
+
+ except RuntimeError:
+ err("Script Error:")
+ err(traceback.format_exc())
+ err("Please fix me")
+ return 2
+ return 0
+
+def open_file_or_fd(val, mode):
+ """
+ If 'val' looks like a decimal integer, open it as an fd. If not, try to
+ open it as a regular file.
+ """
+
+ fd = -1
+ try:
+ # Does it look like an integer?
+ try:
+ fd = int(val, 10)
+ except ValueError:
+ pass
+
+ # Try to open it...
+ if fd != -1:
+ return os.fdopen(fd, mode, 0)
+ else:
+ return open(val, mode, 0)
+
+ except StandardError, e:
+ if fd != -1:
+ err("Unable to open fd %d: %s: %s" %
+ (fd, e.__class__.__name__, e))
+ else:
+ err("Unable to open file '%s': %s: %s" %
+ (val, e.__class__.__name__, e))
+
+ raise SystemExit(1)
+
+
+def main():
+ from optparse import OptionParser
+ global fin, fout, twidth, pv, qemu, verbose
+
+ # Change stdout to be line-buffered.
+ sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1)
+
+ parser = OptionParser(version = __version__,
+ usage = ("%prog [options] -i INPUT -o OUTPUT"
+ " -w WIDTH -g GUEST"),
+ description =
+ "Convert a legacy stream to a v2 stream")
+
+ # Required options
+ parser.add_option("-i", "--in", dest = "fin", metavar = "<FD or FILE>",
+ help = "Legacy input to convert")
+ parser.add_option("-o", "--out", dest = "fout", metavar = "<FD or FILE>",
+ help = "v2 format output")
+ parser.add_option("-w", "--width", dest = "twidth",
+ metavar = "<32/64>", choices = ["32", "64"],
+ help = "Legacy toolstack bitness")
+ parser.add_option("-g", "--guest-type", dest = "gtype",
+ metavar = "<pv/hvm>", choices = ["pv", "hvm"],
+ help = "Type of guest in stream")
+
+ # Optional options
+ parser.add_option("-f", "--format", dest = "format",
+ metavar = "<libxc|libxl>", default = "libxc",
+ choices = ["libxc", "libxl"],
+ help = "Desired format of the outgoing stream " \
+ "(defaults to libxc)")
+ parser.add_option("-v", "--verbose", action = "store_true", default = False,
+ help = "Summarise stream contents")
+ parser.add_option("-x", "--xl", action = "store_true", default = False,
+ help = ("Is an `xl` header present in the stream?"
+ " (default no)"))
+ parser.add_option("--skip-qemu", action = "store_true", default = False,
+ help = ("Skip processing of the qemu tail?"
+ " (default no)"))
+ parser.add_option("--syslog", action = "store_true", default = False,
+ help = "Log to syslog instead of stdout")
+
+ opts, _ = parser.parse_args()
+
+ if (opts.fin is None or opts.fout is None or
+ opts.twidth is None or opts.gtype is None):
+
+ parser.print_help(sys.stderr)
+ raise SystemExit(1)
+
+ if opts.syslog:
+ global log_to_syslog
+
+ syslog.openlog("convert-legacy-stream", syslog.LOG_PID)
+ log_to_syslog = True
+
+ fin = open_file_or_fd(opts.fin, "rb")
+ fout = open_file_or_fd(opts.fout, "wb")
+ twidth = int(opts.twidth)
+ pv = opts.gtype == "pv"
+ verbose = opts.verbose
+ if opts.skip_qemu:
+ qemu = False
+
+ if opts.xl:
+ skip_xl_header(opts.format)
+
+ rc = read_legacy_stream(VM(opts.format))
+ fout.close()
+
+ return rc
+
+if __name__ == "__main__":
+ try:
+ sys.exit(main())
+ except SystemExit, e:
+ sys.exit(e.code)
+ except KeyboardInterrupt:
+ sys.exit(1)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 17/29] tools/libxl: Migration v2 stream format
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (15 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 16/29] tools/python: Conversion utility for legacy migration streams Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 18/29] tools/libxl: Infrastructure for reading a libxl migration v2 stream Andrew Cooper
` (12 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Ross Lagerwall, Ian Jackson, Wei Liu, Andrew Cooper
From: Ross Lagerwall <ross.lagerwall@citrix.com>
C structures describing the Libxl migration v2 stream format
Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v2: Move into libxl__ namespace
---
tools/libxl/libxl_sr_stream_format.h | 57 ++++++++++++++++++++++++++++++++++
1 file changed, 57 insertions(+)
create mode 100644 tools/libxl/libxl_sr_stream_format.h
diff --git a/tools/libxl/libxl_sr_stream_format.h b/tools/libxl/libxl_sr_stream_format.h
new file mode 100644
index 0000000..f4f790b
--- /dev/null
+++ b/tools/libxl/libxl_sr_stream_format.h
@@ -0,0 +1,57 @@
+#ifndef LIBXL__SR_STREAM_FORMAT_H
+#define LIBXL__SR_STREAM_FORMAT_H
+
+/*
+ * C structures for the Migration v2 stream format.
+ * See docs/specs/libxl-migration-stream.pandoc
+ */
+
+#include <stdint.h>
+
+typedef struct libxl__sr_hdr
+{
+ uint64_t ident;
+ uint32_t version;
+ uint32_t options;
+} libxl__sr_hdr;
+
+#define RESTORE_STREAM_IDENT 0x4c6962786c466d74UL
+#define RESTORE_STREAM_VERSION 0x00000002U
+
+#define RESTORE_OPT_BIG_ENDIAN (1u << 0)
+#define RESTORE_OPT_LEGACY (1u << 1)
+
+
+typedef struct libxl__sr_rec_hdr
+{
+ uint32_t type;
+ uint32_t length;
+} libxl__sr_rec_hdr;
+
+/* All records must be aligned up to an 8 octet boundary */
+#define REC_ALIGN_ORDER 3U
+
+#define REC_TYPE_END 0x00000000U
+#define REC_TYPE_LIBXC_CONTEXT 0x00000001U
+#define REC_TYPE_XENSTORE_DATA 0x00000002U
+#define REC_TYPE_EMULATOR_CONTEXT 0x00000003U
+
+typedef struct libxl__sr_emulator_hdr
+{
+ uint32_t id;
+ uint32_t index;
+} libxl__sr_emulator_hdr;
+
+#define EMULATOR_UNKNOWN 0x00000000U
+#define EMULATOR_QEMU_TRADITIONAL 0x00000001U
+#define EMULATOR_QEMU_UPSTREAM 0x00000002U
+
+#endif /* LIBXL__SR_STREAM_FORMAT_H */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 18/29] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (16 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 17/29] tools/libxl: Migration v2 stream format Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 13:32 ` Ian Jackson
2015-07-14 10:59 ` [PATCH v4 19/29] tools/libxl: Infrastructure to convert a legacy stream Andrew Cooper
` (11 subsequent siblings)
29 siblings, 1 reply; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel
Cc: Ross Lagerwall, Wei Liu, Ian Jackson, Ian Campbell, Andrew Cooper
From: Ross Lagerwall <ross.lagerwall@citrix.com>
This contains the event machinery and state machines to read an act on
a non-checkpointed migration v2 stream (with the exception of the
xc_domain_restore() handling which is spliced later in a bisectable
way).
It also contains some boilerplate to help support checkpointed
streams, which shall be introduced in a later patch.
Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v4:
* Don't make _init() mandatory
* Don't unilaterally clobber rc from datacopier callback
* Comment adjustments
* More assertions
v3: More descriptions of the internal state, shuffle function order, add
an _init() call, condense error handling
---
tools/libxl/Makefile | 1 +
tools/libxl/libxl_internal.h | 53 ++++
tools/libxl/libxl_stream_read.c | 564 +++++++++++++++++++++++++++++++++++++++
3 files changed, 618 insertions(+)
create mode 100644 tools/libxl/libxl_stream_read.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index cc9c152..c71c5fe 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -94,6 +94,7 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
libxl_internal.o libxl_utils.o libxl_uuid.o \
libxl_json.o libxl_aoutils.o libxl_numa.o libxl_vnuma.o \
+ libxl_stream_read.o \
libxl_save_callout.o _libxl_save_msgs_callout.o \
libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
LIBXL_OBJS += libxl_genid.o
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 5d3499e..9397d53 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -19,6 +19,8 @@
#include "libxl_osdeps.h" /* must come before any other headers */
+#include "libxl_sr_stream_format.h"
+
#include <assert.h>
#include <dirent.h>
#include <errno.h>
@@ -3211,6 +3213,57 @@ typedef void libxl__domain_create_cb(libxl__egc *egc,
libxl__domain_create_state*,
int rc, uint32_t domid);
+/* State for manipulating a libxl migration v2 stream */
+typedef struct libxl__stream_read_state libxl__stream_read_state;
+
+typedef struct libxl__sr_record_buf {
+ /* private to stream read helper */
+ LIBXL_STAILQ_ENTRY(struct libxl__sr_record_buf) entry;
+ libxl__sr_rec_hdr hdr;
+ void *body; /* iff hdr.length != 0 */
+} libxl__sr_record_buf;
+
+struct libxl__stream_read_state {
+ /* filled by the user */
+ libxl__ao *ao;
+ libxl__domain_create_state *dcs;
+ int fd;
+ void (*completion_callback)(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int rc);
+ /* Private */
+ int rc;
+ bool running;
+
+ /* Main stream-reading data. */
+ libxl__datacopier_state dc; /* Only used when reading a record */
+ libxl__sr_hdr hdr;
+ LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue; /* NOGC */
+ enum {
+ SRS_PHASE_NORMAL,
+ } phase;
+ bool recursion_guard;
+
+ /* Only used while actively reading a record from the stream. */
+ libxl__sr_record_buf *incoming_record; /* NOGC */
+
+ /* Both only used when processing an EMULATOR record. */
+ libxl__datacopier_state emu_dc;
+ libxl__carefd *emu_carefd;
+};
+
+_hidden void libxl__stream_read_init(libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_start(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_abort(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
+static inline bool
+libxl__stream_read_inuse(const libxl__stream_read_state *stream)
+{
+ return stream->running;
+}
+
+
struct libxl__domain_create_state {
/* filled in by user */
libxl__ao *ao;
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
new file mode 100644
index 0000000..2eb4ff5
--- /dev/null
+++ b/tools/libxl/libxl_stream_read.c
@@ -0,0 +1,564 @@
+/*
+ * Copyright (C) 2015 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/*
+ * Infrastructure for reading and acting on the contents of a libxl
+ * migration stream. There are a lot of moving parts here.
+ *
+ * The logic revolves around two actions; reading another record from
+ * the stream, and processing the records. The stream_continue()
+ * function is responsible for choosing the next action to perform.
+ *
+ * The exact order of reading and processing is controlled by 'phase'.
+ * All complete records are held in the record_queue before being
+ * processed, and all records will be processed in queue order.
+ *
+ * Internal states:
+ * running phase in_ record incoming
+ * checkpoint _queue _record
+ *
+ * Undefined undef undef undef undef undef
+ * Idle false undef false 0 0
+ * Active true NORMAL false 0/1 0/partial
+ *
+ * While reading data from the stream, 'dc' is active and a callback
+ * is expected. Most actions in process_record() start a callback of
+ * their own. Those which don't return out and stream_continue() sets
+ * up the next action.
+ *
+ * PHASE_NORMAL:
+ * This phase is used for regular migration or resume from file.
+ * Records are read one at time and immediately processed. (The
+ * record queue will not contain more than a single record.)
+ *
+ * Note:
+ * Record buffers are not allocated from a GC; they are allocated
+ * and tracked manually. This is to avoid OOM with Remus where the
+ * AO lives for the lifetime of the process. Per-checkpoint AO's
+ * might be an avenue to explore.
+ *
+ * Entry points from outside:
+ * - libxl__stream_read_init()
+ * - Initialises state. Must be called once before _start()
+ * - libxl__stream_read_start()
+ * - Starts reading records from the stream, and acting on them.
+ *
+ * There are several chains of event:
+ *
+ * 1) Starting a stream follows:
+ * - libxl__stream_read_start()
+ * - stream_header_done()
+ * - stream_continue()
+ *
+ * 2) Reading a record follows:
+ * - stream_continue()
+ * - record_header_done()
+ * - record_body_done()
+ * - stream_continue()
+ *
+ * 3) Processing a record had several chains to follow, depending on
+ * the record in question.
+ * 3a) "Simple" record:
+ * - process_record()
+ * - stream_continue()
+ * 3b) LIBXC record:
+ * - process_record()
+ * - libxl__xc_domain_restore()
+ * - libxl__xc_domain_restore_done()
+ * - stream_continue()
+ * 3c) EMULATOR record:
+ * - process_record()
+ * - stream_write_emulator()
+ * - stream_write_emulator_done()
+ * - stream_continue()
+ */
+
+/* Success/error/cleanup handling. */
+static void stream_complete(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
+static void stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+
+/* Event chain for first iteration, from _start(). */
+static void stream_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+static void stream_continue(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+static void setup_read_record(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+static void record_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+static void record_body_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+static bool process_record(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+
+/* Event chain for processing an emulator blob. */
+static void write_emulator_blob(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ libxl__sr_record_buf *rec);
+static void write_emulator_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+
+/*----- Helpers -----*/
+
+/* Helper to set up reading some data from the stream. */
+static int setup_read(libxl__stream_read_state *stream,
+ const char *what, void *ptr, size_t nr_bytes,
+ libxl__datacopier_callback cb)
+{
+ libxl__datacopier_state *dc = &stream->dc;
+
+ dc->readwhat = what;
+ dc->readbuf = ptr;
+ dc->bytes_to_read = nr_bytes;
+ dc->used = 0;
+ dc->callback = cb;
+
+ return libxl__datacopier_start(dc);
+}
+
+static void free_record(libxl__sr_record_buf *rec)
+{
+ if (rec) {
+ free(rec->body);
+ free(rec);
+ }
+}
+
+/*----- Entrypoints -----*/
+
+void libxl__stream_read_init(libxl__stream_read_state *stream)
+{
+ stream->rc = 0;
+ stream->running = false;
+ FILLZERO(stream->dc);
+ FILLZERO(stream->hdr);
+ LIBXL_STAILQ_INIT(&stream->record_queue);
+ stream->phase = SRS_PHASE_NORMAL;
+ stream->recursion_guard = false;
+ stream->incoming_record = NULL;
+ FILLZERO(stream->emu_dc);
+ stream->emu_carefd = NULL;
+}
+
+void libxl__stream_read_start(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ libxl__datacopier_state *dc = &stream->dc;
+ int rc = 0;
+
+ libxl__stream_read_init(stream);
+
+ stream->running = true;
+ stream->phase = SRS_PHASE_NORMAL;
+
+ dc->ao = stream->ao;
+ dc->readfd = stream->fd;
+ dc->writefd = -1;
+
+ /* Start reading the stream header. */
+ rc = setup_read(stream, "stream header",
+ &stream->hdr, sizeof(stream->hdr),
+ stream_header_done);
+ if (rc)
+ goto err;
+
+ assert(!rc);
+ return;
+
+ err:
+ assert(rc);
+ stream_complete(egc, stream, rc);
+}
+
+void libxl__stream_read_abort(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc)
+{
+ assert(rc);
+
+ if (stream->running)
+ stream_complete(egc, stream, rc);
+}
+
+/*----- Event logic -----*/
+
+static void stream_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
+ libxl__sr_hdr *hdr = &stream->hdr;
+ STATE_AO_GC(dc->ao);
+
+ if (rc)
+ goto err;
+
+ hdr->ident = be64toh(hdr->ident);
+ hdr->version = be32toh(hdr->version);
+ hdr->options = be32toh(hdr->options);
+
+ if (hdr->ident != RESTORE_STREAM_IDENT) {
+ rc = ERROR_FAIL;
+ LOG(ERROR,
+ "Invalid ident: expected 0x%016"PRIx64", got 0x%016"PRIx64,
+ RESTORE_STREAM_IDENT, hdr->ident);
+ goto err;
+ }
+ if (hdr->version != RESTORE_STREAM_VERSION) {
+ rc = ERROR_FAIL;
+ LOG(ERROR, "Unexpected Version: expected %"PRIu32", got %"PRIu32,
+ RESTORE_STREAM_VERSION, hdr->version);
+ goto err;
+ }
+ if (hdr->options & RESTORE_OPT_BIG_ENDIAN) {
+ rc = ERROR_FAIL;
+ LOG(ERROR, "Unable to handle big endian streams");
+ goto err;
+ }
+
+ LOG(DEBUG, "Stream v%"PRIu32"%s", hdr->version,
+ hdr->options & RESTORE_OPT_LEGACY ? " (from legacy)" : "");
+
+ stream_continue(egc, stream);
+ return;
+
+ err:
+ assert(rc);
+ stream_complete(egc, stream, rc);
+}
+
+static void stream_continue(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ STATE_AO_GC(stream->ao);
+
+ /*
+ * Must not mutually recurse with process_record().
+ *
+ * For records whose processing function is synchronous
+ * (e.g. TOOLSTACK), process_record() does not start another async
+ * operation, and a further operation should be started.
+ *
+ * A naive solution, which would function in general, would be for
+ * process_record() to call stream_continue(). However, this
+ * would allow the content of the stream to cause mutual
+ * recursion, and possibly for us to fall off our stack.
+ *
+ * Instead, process_record() indicates with its return value
+ * whether a further operation needs to start, and the
+ * recursion_guard is in place to catch any code paths which get
+ * this wrong.
+ */
+ assert(stream->recursion_guard == false);
+ stream->recursion_guard = true;
+
+ switch (stream->phase) {
+ case SRS_PHASE_NORMAL:
+ /*
+ * Normal phase (regular migration or restore from file):
+ *
+ * logically:
+ * do { read_record(); process_record(); } while ( not END );
+ *
+ * Alternate between reading a record from the stream, and
+ * processing the record. There should never be two records
+ * in the queue.
+ */
+ if (LIBXL_STAILQ_EMPTY(&stream->record_queue))
+ setup_read_record(egc, stream);
+ else {
+ if (process_record(egc, stream))
+ setup_read_record(egc, stream);
+
+ /*
+ * process_record() had better have consumed the one and
+ * only record in the queue.
+ */
+ assert(LIBXL_STAILQ_EMPTY(&stream->record_queue));
+ }
+ break;
+
+ default:
+ abort();
+ }
+
+ assert(stream->recursion_guard == true);
+ stream->recursion_guard = false;
+}
+
+static void setup_read_record(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ libxl__sr_record_buf *rec = NULL;
+ STATE_AO_GC(stream->ao);
+ int rc;
+
+ assert(stream->incoming_record == NULL);
+ stream->incoming_record = rec = libxl__zalloc(NOGC, sizeof(*rec));
+
+ rc = setup_read(stream, "record header",
+ &rec->hdr, sizeof(rec->hdr),
+ record_header_done);
+ if (rc)
+ goto err;
+ return;
+
+ err:
+ assert(rc);
+ stream_complete(egc, stream, rc);
+}
+
+static void record_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
+ libxl__sr_record_buf *rec = stream->incoming_record;
+ STATE_AO_GC(dc->ao);
+
+ if (rc)
+ goto err;
+
+ /* No body? All done. */
+ if (rec->hdr.length == 0) {
+ record_body_done(egc, dc, 0, 0, 0);
+ return;
+ }
+
+ size_t bytes_to_read = ROUNDUP(rec->hdr.length, REC_ALIGN_ORDER);
+ rec->body = libxl__malloc(NOGC, bytes_to_read);
+
+ rc = setup_read(stream, "record body",
+ rec->body, bytes_to_read,
+ record_body_done);
+ if (rc)
+ goto err;
+ return;
+
+ err:
+ assert(rc);
+ stream_complete(egc, stream, rc);
+}
+
+static void record_body_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
+ libxl__sr_record_buf *rec = stream->incoming_record;
+ STATE_AO_GC(dc->ao);
+
+ if (rc)
+ goto err;
+
+ LIBXL_STAILQ_INSERT_TAIL(&stream->record_queue, rec, entry);
+ stream->incoming_record = NULL;
+
+ stream_continue(egc, stream);
+ return;
+
+ err:
+ assert(rc);
+ stream_complete(egc, stream, rc);
+}
+
+/*
+ * Returns a boolean indicating whether a further action should be set
+ * up by the caller. This is needed to prevent mutual recursion with
+ * stream_continue().
+ *
+ * It is a bug for this function to ever call stream_continue() or
+ * setup_read_record().
+ */
+static bool process_record(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ STATE_AO_GC(stream->ao);
+ libxl__domain_create_state *dcs = stream->dcs;
+ libxl__sr_record_buf *rec;
+ bool further_action_needed = false;
+ int rc = 0;
+
+ /* Pop a record from the head of the queue. */
+ assert(!LIBXL_STAILQ_EMPTY(&stream->record_queue));
+ rec = LIBXL_STAILQ_FIRST(&stream->record_queue);
+ LIBXL_STAILQ_REMOVE_HEAD(&stream->record_queue, entry);
+
+ LOG(DEBUG, "Record: %u, length %u", rec->hdr.type, rec->hdr.length);
+
+ switch (rec->hdr.type) {
+
+ case REC_TYPE_END:
+ stream_complete(egc, stream, 0);
+ break;
+
+ case REC_TYPE_XENSTORE_DATA:
+ rc = libxl__toolstack_restore(dcs->guest_domid, rec->body,
+ rec->hdr.length, &dcs->shs);
+ if (rc)
+ goto err;
+
+ /*
+ * libxl__toolstack_restore() is a synchronous function.
+ * Request that our caller queues another action for us.
+ */
+ further_action_needed = true;
+ break;
+
+ case REC_TYPE_EMULATOR_CONTEXT:
+ write_emulator_blob(egc, stream, rec);
+ break;
+
+ default:
+ LOG(ERROR, "Unrecognised record 0x%08x", rec->hdr.type);
+ rc = ERROR_FAIL;
+ goto err;
+ }
+
+ assert(!rc);
+ free_record(rec);
+ return further_action_needed;
+
+ err:
+ assert(rc);
+ free_record(rec);
+ stream_complete(egc, stream, rc);
+ return false;
+}
+
+static void write_emulator_blob(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ libxl__sr_record_buf *rec)
+{
+ libxl__domain_create_state *dcs = stream->dcs;
+ libxl__datacopier_state *dc = &stream->emu_dc;
+ libxl__sr_emulator_hdr *emu_hdr;
+ STATE_AO_GC(stream->ao);
+ char path[256];
+ int rc = 0, writefd = -1;
+
+ if (rec->hdr.length < sizeof(*emu_hdr)) {
+ rc = ERROR_FAIL;
+ LOG(ERROR, "Emulator record too short to contain header");
+ goto err;
+ }
+ emu_hdr = rec->body;
+
+ sprintf(path, XC_DEVICE_MODEL_RESTORE_FILE".%u", dcs->guest_domid);
+
+ assert(stream->emu_carefd == NULL);
+ libxl__carefd_begin();
+ writefd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
+ stream->emu_carefd = libxl__carefd_opened(CTX, writefd);
+
+ if (writefd == -1) {
+ rc = ERROR_FAIL;
+ LOGE(ERROR, "unable to open %s", path);
+ goto err;
+ }
+
+ FILLZERO(*dc);
+ dc->ao = stream->ao;
+ dc->writewhat = "qemu save file";
+ dc->writefd = writefd;
+ dc->maxsz = -1;
+ dc->callback = write_emulator_done;
+
+ rc = libxl__datacopier_start(dc);
+ if (rc)
+ goto err;
+
+ libxl__datacopier_prefixdata(egc, dc,
+ rec->body + sizeof(*emu_hdr),
+ rec->hdr.length - sizeof(*emu_hdr));
+ return;
+
+ err:
+ assert(rc);
+ stream_complete(egc, stream, rc);
+}
+
+static void write_emulator_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, emu_dc);
+ STATE_AO_GC(dc->ao);
+
+ libxl__carefd_close(stream->emu_carefd);
+ stream->emu_carefd = NULL;
+
+ if (rc)
+ goto err;
+
+ stream_continue(egc, stream);
+ return;
+
+ err:
+ assert(rc);
+ stream_complete(egc, stream, rc);
+}
+
+/*----- Success/error/cleanup handling. -----*/
+
+static void stream_complete(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc)
+{
+ assert(stream->running);
+
+ if (!stream->rc)
+ stream->rc = rc;
+ stream_done(egc, stream);
+}
+
+static void stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ libxl__sr_record_buf *rec, *trec;
+
+ assert(stream->running);
+ stream->running = false;
+
+ if (stream->incoming_record)
+ free_record(stream->incoming_record);
+
+ if (stream->emu_carefd)
+ libxl__carefd_close(stream->emu_carefd);
+
+ /* The record queue had better be empty if the stream believes
+ * itself to have been successful. */
+ assert(LIBXL_STAILQ_EMPTY(&stream->record_queue) || stream->rc);
+
+ LIBXL_STAILQ_FOREACH_SAFE(rec, &stream->record_queue, entry, trec)
+ free_record(rec);
+
+ stream->completion_callback(egc, stream, stream->rc);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 19/29] tools/libxl: Infrastructure to convert a legacy stream
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (17 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 18/29] tools/libxl: Infrastructure for reading a libxl migration v2 stream Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 13:33 ` Ian Jackson
2015-07-14 10:59 ` [PATCH v4 20/29] tools/libxl: Convert a legacy stream if needed Andrew Cooper
` (10 subsequent siblings)
29 siblings, 1 reply; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
Provide a thin wrapper around exec()ing the python conversion utility,
and a stub implementation for cases where conversion is not wanted
(i.e. not x86).
One complication is that the caller of this interface needs to assume
ownership of the output fd, to prevent it being closed while still in
use in a datacopier.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v4: Formatting tweaks, description of the width field.
v3: Trimmed down massively from v2, provide libxl_no_convert_callout.c
---
tools/libxl/Makefile | 6 ++
tools/libxl/libxl_convert_callout.c | 171 ++++++++++++++++++++++++++++++++
tools/libxl/libxl_internal.h | 30 ++++++
tools/libxl/libxl_no_convert_callout.c | 35 +++++++
4 files changed, 242 insertions(+)
create mode 100644 tools/libxl/libxl_convert_callout.c
create mode 100644 tools/libxl/libxl_no_convert_callout.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index c71c5fe..68fe1c1 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -56,6 +56,12 @@ else
LIBXL_OBJS-y += libxl_nonetbuffer.o
endif
+ifeq ($(CONFIG_X86),y)
+LIBXL_OBJS-y += libxl_convert_callout.o
+else
+LIBXL_OBJS-y += libxl_no_convert_callout.o
+endif
+
LIBXL_OBJS-y += libxl_remus_device.o libxl_remus_disk_drbd.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
diff --git a/tools/libxl/libxl_convert_callout.c b/tools/libxl/libxl_convert_callout.c
new file mode 100644
index 0000000..65b4df9
--- /dev/null
+++ b/tools/libxl/libxl_convert_callout.c
@@ -0,0 +1,171 @@
+/*
+ * Copyright (C) 2014 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h"
+
+#include "libxl_internal.h"
+
+/*
+ * Infrastructure for converting a legacy migration stream into a
+ * libxl v2 stream.
+ *
+ * This is done by fork()ing the python conversion script, which takes
+ * in a legacy stream, and puts out a suitably-formatted v2 stream.
+ */
+
+static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
+ pid_t pid, int status);
+static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc);
+static void helper_done(libxl__egc *egc,
+ libxl__conversion_helper_state *chs);
+
+/*----- Entrypoints -----*/
+
+void libxl__conversion_helper_init(libxl__conversion_helper_state *chs)
+{
+ chs->v2_carefd = NULL;
+ chs->rc = 0;
+ libxl__ao_abortable_init(&chs->abrt);
+ libxl__ev_child_init(&chs->child);
+}
+
+int libxl__convert_legacy_stream(libxl__egc *egc,
+ libxl__conversion_helper_state *chs)
+{
+ STATE_AO_GC(chs->ao);
+ libxl__carefd *child_in = NULL, *child_out = NULL;
+ int rc = 0;
+
+ chs->abrt.ao = chs->ao;
+ chs->abrt.callback = helper_stop;
+ rc = libxl__ao_abortable_register(&chs->abrt);
+ if (rc) goto err;
+
+ libxl__carefd_begin();
+ int fds[2];
+ if (libxl_pipe(CTX, fds)) {
+ rc = ERROR_FAIL;
+ libxl__carefd_unlock();
+ goto err;
+ }
+ child_out = libxl__carefd_record(CTX, fds[0]);
+ child_in = libxl__carefd_record(CTX, fds[1]);
+ libxl__carefd_unlock();
+
+ pid_t pid = libxl__ev_child_fork(gc, &chs->child, helper_exited);
+ if (!pid) {
+ char * const args[] =
+ {
+ getenv("LIBXL_CONVERT_HELPER") ?:
+ LIBEXEC_BIN "/convert-legacy-stream",
+ "--in", GCSPRINTF("%d", chs->legacy_fd),
+ "--out", GCSPRINTF("%d", fds[1]),
+ /*
+ * The width calculation is an assumption for the common
+ * case. The conversion script needs to know the width of
+ * the toolstack which saved the legacy stream.
+ *
+ * In the overwhelming majority of cases, the width of the
+ * saving toolstack will be the same as our current
+ * width. To avoid extending the libxl API with a
+ * parameter intended to disappear shortly, this option
+ * has not been exposed to the caller.
+ *
+ * If more complicated conversion is required, the
+ * conversion script can be instantiated manually, which
+ * will bypass all of this conversion logic.
+ */
+ "--width", sizeof(unsigned long) == 8 ? "64" : "32",
+
+ "--guest", chs->hvm ? "hvm" : "pv",
+ "--format", "libxl",
+ /* "--verbose", */
+ NULL,
+ };
+
+ libxl_fd_set_cloexec(CTX, chs->legacy_fd, 0);
+ libxl_fd_set_cloexec(CTX, libxl__carefd_fd(child_in), 0);
+
+ libxl__exec(gc,
+ -1, -1, -1,
+ args[0], args, NULL);
+ }
+
+ libxl__carefd_close(child_in);
+ chs->v2_carefd = child_out;
+
+ assert(!rc);
+ return rc;
+
+ err:
+ libxl__ao_abortable_deregister(&chs->abrt);
+ assert(rc);
+ return rc;
+}
+
+void libxl__conversion_helper_abort(libxl__egc *egc,
+ libxl__conversion_helper_state *chs,
+ int rc)
+{
+ STATE_AO_GC(chs->ao);
+ assert(rc);
+
+ if (libxl__conversion_helper_inuse(chs)) {
+
+ if (!chs->rc)
+ chs->rc = rc;
+
+ libxl__kill(gc, chs->child.pid, SIGTERM, "conversion helper");
+ }
+}
+
+/*----- State handling -----*/
+
+static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
+{
+ libxl__conversion_helper_state *chs = CONTAINER_OF(abrt, *chs, abrt);
+ STATE_AO_GC(chs->ao);
+
+ libxl__conversion_helper_abort(egc, chs, rc);
+}
+
+static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
+ pid_t pid, int status)
+{
+ libxl__conversion_helper_state *chs = CONTAINER_OF(ch, *chs, child);
+ STATE_AO_GC(chs->ao);
+
+ if (status) {
+ libxl_report_child_exitstatus(
+ CTX, chs->rc ? XTL_DEBUG : XTL_ERROR,
+ "conversion helper", pid, status);
+
+ if (!chs->rc)
+ chs->rc = ERROR_FAIL;
+ }
+
+ helper_done(egc, chs);
+}
+
+static void helper_done(libxl__egc *egc,
+ libxl__conversion_helper_state *chs)
+{
+ STATE_AO_GC(chs->ao);
+
+ assert(!libxl__conversion_helper_inuse(chs));
+
+ libxl__ao_abortable_deregister(&chs->abrt);
+
+ chs->completion_callback(egc, chs, chs->rc);
+}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 9397d53..97755fe 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2917,6 +2917,36 @@ _hidden void libxl__remus_devices_commit(libxl__egc *egc,
libxl__remus_devices_state *rds);
_hidden int libxl__netbuffer_enabled(libxl__gc *gc);
+/*----- Legacy conversion helper -----*/
+typedef struct libxl__conversion_helper_state libxl__conversion_helper_state;
+
+struct libxl__conversion_helper_state {
+ /* public */
+ libxl__ao *ao;
+ int legacy_fd;
+ bool hvm; /* pv or hvm domain? */
+ libxl__carefd *v2_carefd; /* Filled by successful call to
+ * libxl__convert_legacy_stream(). Caller
+ * assumes ownership of the fd. */
+ void (*completion_callback)(
+ libxl__egc *egc, libxl__conversion_helper_state *chs, int rc);
+ /* private */
+ int rc;
+ libxl__ao_abortable abrt;
+ libxl__ev_child child;
+};
+
+_hidden void libxl__conversion_helper_init
+ (libxl__conversion_helper_state *chs);
+_hidden int libxl__convert_legacy_stream(libxl__egc *egc,
+ libxl__conversion_helper_state *chs);
+_hidden void libxl__conversion_helper_abort(libxl__egc *egc,
+ libxl__conversion_helper_state *chs, int rc);
+static inline bool libxl__conversion_helper_inuse
+ (const libxl__conversion_helper_state *chs)
+{ return libxl__ev_child_inuse(&chs->child); }
+
+
/*----- Domain suspend (save) state structure -----*/
typedef struct libxl__domain_suspend_state libxl__domain_suspend_state;
diff --git a/tools/libxl/libxl_no_convert_callout.c b/tools/libxl/libxl_no_convert_callout.c
new file mode 100644
index 0000000..6ba4d92
--- /dev/null
+++ b/tools/libxl/libxl_no_convert_callout.c
@@ -0,0 +1,35 @@
+/*
+ * Copyright (C) 2015 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h"
+
+#include "libxl_internal.h"
+
+void libxl__conversion_helper_init(libxl__conversion_helper_state *chs)
+{
+ libxl__ev_child_init(&chs->child);
+}
+
+int libxl__convert_legacy_stream(libxl__egc *egc,
+ libxl__conversion_helper_state *chs)
+{
+ return ERROR_FAIL;
+}
+
+void libxl__conversion_helper_abort(libxl__egc *egc,
+ libxl__conversion_helper_state *chs,
+ int rc)
+{
+ /* no op */
+}
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 20/29] tools/libxl: Convert a legacy stream if needed
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (18 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 19/29] tools/libxl: Infrastructure to convert a legacy stream Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 13:37 ` Ian Jackson
2015-07-14 10:59 ` [PATCH v4 21/29] tools/libxc+libxl+xl: Restore v2 streams Andrew Cooper
` (9 subsequent siblings)
29 siblings, 1 reply; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
For backwards compatibility, a legacy stream needs converting before
it can be read by the v2 stream logic.
This causes the v2 stream logic to need to juggle two parallel tasks.
check_all_finished() is introduced for the purpose of joining the
tasks in both success and error cases.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v4: Alter logic layout in check_all_finished()
v3: Simplified greatly from v2. No practical change.
---
tools/libxl/libxl_internal.h | 2 ++
tools/libxl/libxl_stream_read.c | 66 +++++++++++++++++++++++++++++++++++++++
2 files changed, 68 insertions(+)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 97755fe..b65019b 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3258,12 +3258,14 @@ struct libxl__stream_read_state {
libxl__ao *ao;
libxl__domain_create_state *dcs;
int fd;
+ bool legacy;
void (*completion_callback)(libxl__egc *egc,
libxl__stream_read_state *srs,
int rc);
/* Private */
int rc;
bool running;
+ libxl__conversion_helper_state chs;
/* Main stream-reading data. */
libxl__datacopier_state dc; /* Only used when reading a record */
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index 2eb4ff5..4b677f6 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -86,6 +86,10 @@
* - stream_write_emulator()
* - stream_write_emulator_done()
* - stream_continue()
+ *
+ * Depending on the contents of the stream, there are likely to be several
+ * parallel tasks being managed. check_all_finished() is used to join all
+ * tasks in both success and error cases.
*/
/* Success/error/cleanup handling. */
@@ -93,6 +97,10 @@ static void stream_complete(libxl__egc *egc,
libxl__stream_read_state *stream, int rc);
static void stream_done(libxl__egc *egc,
libxl__stream_read_state *stream);
+static void conversion_done(libxl__egc *egc,
+ libxl__conversion_helper_state *chs, int rc);
+static void check_all_finished(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
/* Event chain for first iteration, from _start(). */
static void stream_header_done(libxl__egc *egc,
@@ -151,6 +159,7 @@ void libxl__stream_read_init(libxl__stream_read_state *stream)
{
stream->rc = 0;
stream->running = false;
+ libxl__conversion_helper_init(&stream->chs);
FILLZERO(stream->dc);
FILLZERO(stream->hdr);
LIBXL_STAILQ_INIT(&stream->record_queue);
@@ -165,6 +174,7 @@ void libxl__stream_read_start(libxl__egc *egc,
libxl__stream_read_state *stream)
{
libxl__datacopier_state *dc = &stream->dc;
+ STATE_AO_GC(stream->ao);
int rc = 0;
libxl__stream_read_init(stream);
@@ -172,6 +182,29 @@ void libxl__stream_read_start(libxl__egc *egc,
stream->running = true;
stream->phase = SRS_PHASE_NORMAL;
+ if (stream->legacy) {
+ /* Convert the legacy stream. */
+ libxl__conversion_helper_state *chs = &stream->chs;
+
+ chs->ao = stream->ao;
+ chs->legacy_fd = stream->fd;
+ chs->hvm =
+ (stream->dcs->guest_config->b_info.type == LIBXL_DOMAIN_TYPE_HVM);
+ chs->completion_callback = conversion_done;
+
+ rc = libxl__convert_legacy_stream(egc, &stream->chs);
+
+ if (rc) {
+ LOG(ERROR, "Failed to start the legacy stream conversion helper");
+ goto err;
+ }
+
+ assert(stream->chs.v2_carefd);
+ stream->fd = libxl__carefd_fd(stream->chs.v2_carefd);
+ stream->dcs->libxc_fd = stream->fd;
+ }
+ /* stream->fd is now a v2 stream. */
+
dc->ao = stream->ao;
dc->readfd = stream->fd;
dc->writefd = -1;
@@ -545,6 +578,10 @@ static void stream_done(libxl__egc *egc,
if (stream->emu_carefd)
libxl__carefd_close(stream->emu_carefd);
+ /* If we started a conversion helper, we took ownership of its carefd. */
+ if (stream->chs.v2_carefd)
+ libxl__carefd_close(stream->chs.v2_carefd);
+
/* The record queue had better be empty if the stream believes
* itself to have been successful. */
assert(LIBXL_STAILQ_EMPTY(&stream->record_queue) || stream->rc);
@@ -552,6 +589,35 @@ static void stream_done(libxl__egc *egc,
LIBXL_STAILQ_FOREACH_SAFE(rec, &stream->record_queue, entry, trec)
free_record(rec);
+ check_all_finished(egc, stream, stream->rc);
+}
+
+static void conversion_done(libxl__egc *egc,
+ libxl__conversion_helper_state *chs, int rc)
+{
+ libxl__stream_read_state *stream = CONTAINER_OF(chs, *stream, chs);
+
+ check_all_finished(egc, stream, rc);
+}
+
+static void check_all_finished(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc)
+{
+ STATE_AO_GC(stream->ao);
+
+ if (!stream->rc && rc) {
+ /* First reported failure. Tear everything down. */
+ stream->rc = rc;
+
+ libxl__stream_read_abort(egc, stream, rc);
+ libxl__conversion_helper_abort(egc, &stream->chs, rc);
+ }
+
+ /* Don't fire the callback until all our parallel tasks have stopped. */
+ if (libxl__stream_read_inuse(stream) ||
+ libxl__conversion_helper_inuse(&stream->chs))
+ return;
+
stream->completion_callback(egc, stream, stream->rc);
}
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v4 20/29] tools/libxl: Convert a legacy stream if needed
2015-07-14 10:59 ` [PATCH v4 20/29] tools/libxl: Convert a legacy stream if needed Andrew Cooper
@ 2015-07-14 13:37 ` Ian Jackson
0 siblings, 0 replies; 39+ messages in thread
From: Ian Jackson @ 2015-07-14 13:37 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Campbell, Xen-devel
Andrew Cooper writes ("[PATCH v4 20/29] tools/libxl: Convert a legacy stream if needed"):
> For backwards compatibility, a legacy stream needs converting before
> it can be read by the v2 stream logic.
>
> This causes the v2 stream logic to need to juggle two parallel tasks.
> check_all_finished() is introduced for the purpose of joining the
> tasks in both success and error cases.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Jackson <ian.jackson@eu.citrix.com>
> + /* If we started a conversion helper, we took ownership of its carefd. */
> + if (stream->chs.v2_carefd)
> + libxl__carefd_close(stream->chs.v2_carefd);
I have just spotted that this if is not needed because
libxl__carefd_close(NULL) is a no-op (and I have just checked that the
implementation matches the documentation!)
If you need to respin for some other reason you might want to adjust
this, but I won't mind if you don't. Otherwise, you or I might want
to propose a cleanup patch for this later.
Note that this is not the only occurrence.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 39+ messages in thread
* [PATCH v4 21/29] tools/libxc+libxl+xl: Restore v2 streams
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (19 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 20/29] tools/libxl: Convert a legacy stream if needed Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 22/29] tools/libxl: Infrastructure for writing a v2 stream Andrew Cooper
` (8 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Wei Liu, Ian Campbell
This is a complicated set of changes which must be done together for
bisectability.
* libxl-save-helper is updated to unconditionally use libxc migration
v2.
* libxl compatibility workarounds in libxc are disabled for restore
operations.
* libxl__stream_read_start() is logically spliced into the event
location where libxl__xc_domain_restore() used to reside.
* Ownership of the save_helper_state moves to stream_read_state.
The parameters 'hvm', 'pae', and 'superpages' were previously
superfluous, and are completely unused in migration
v2. callbacks->toolstack_restore is handled via a migration v2 record
now, rather than via a callback from libxc.
NB: this change breaks Remus. Further untangling needs to happen
before Remus will function.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v3:
* Simplify from v2.
* Alter the ownership of save_helper_state
v2:
* Drop "legacy_width" from the IDL
* Gain a LIBXL_HAVE_ to signify support of migration v2 streams
---
tools/libxc/Makefile | 4 ++--
tools/libxl/libxl.h | 17 ++++++++++++++
tools/libxl/libxl_create.c | 49 +++++++++++++--------------------------
tools/libxl/libxl_internal.h | 3 ++-
tools/libxl/libxl_save_helper.c | 2 +-
tools/libxl/libxl_stream_read.c | 35 +++++++++++++++++++++++++++-
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl_cmdimpl.c | 7 +++++-
8 files changed, 79 insertions(+), 39 deletions(-)
diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
index b659df4..2cd0b1a 100644
--- a/tools/libxc/Makefile
+++ b/tools/libxc/Makefile
@@ -64,8 +64,8 @@ GUEST_SRCS-$(CONFIG_X86) += xc_sr_save_x86_hvm.c
GUEST_SRCS-y += xc_sr_restore.c
GUEST_SRCS-y += xc_sr_save.c
GUEST_SRCS-y += xc_offline_page.c xc_compression.c
-$(patsubst %.c,%.o,$(GUEST_SRCS-y)): CFLAGS += -DXG_LIBXL_HVM_COMPAT
-$(patsubst %.c,%.opic,$(GUEST_SRCS-y)): CFLAGS += -DXG_LIBXL_HVM_COMPAT
+xc_sr_save_x86_hvm.o: CFLAGS += -DXG_LIBXL_HVM_COMPAT
+xc_sr_save_x86_hvm.opic: CFLAGS += -DXG_LIBXL_HVM_COMPAT
else
GUEST_SRCS-y += xc_nomigrate.c
endif
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index e9d63c9..74b0829 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -807,6 +807,23 @@
*/
#define LIBXL_HAVE_SOCKET_BITMAP_ALLOC 1
+/*
+ * LIBXL_HAVE_SRM_V2
+ *
+ * If this is defined, then the libxl_domain_create_restore() interface takes
+ * a "stream_version" parameter and supports a value of 2.
+ */
+#define LIBXL_HAVE_SRM_V2 1
+
+/*
+ * LIBXL_HAVE_SRM_V1
+ *
+ * In the case that LIBXL_HAVE_SRM_V2 is set, LIBXL_HAVE_SRM_V1
+ * indicates that libxl_domain_create_restore() can handle a "stream_version"
+ * parameter of 1, and convert the stream format automatically.
+ */
+#define LIBXL_HAVE_SRM_V1 1
+
typedef char **libxl_string_list;
void libxl_string_list_dispose(libxl_string_list *sl);
int libxl_string_list_length(const libxl_string_list *sl);
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 7f0ffc6..c51d64f 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -704,6 +704,10 @@ static void domcreate_attach_dtdev(libxl__egc *egc,
static void domcreate_console_available(libxl__egc *egc,
libxl__domain_create_state *dcs);
+static void domcreate_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int ret);
+
static void domcreate_rebuild_done(libxl__egc *egc,
libxl__domain_create_state *dcs,
int ret);
@@ -933,11 +937,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
/* convenience aliases */
const uint32_t domid = dcs->guest_domid;
libxl_domain_config *const d_config = dcs->guest_config;
- libxl_domain_build_info *const info = &d_config->b_info;
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
- libxl__srm_restore_autogen_callbacks *const callbacks =
- &dcs->shs.callbacks.restore.a;
if (rc) {
domcreate_rebuild_done(egc, dcs, rc);
@@ -970,30 +971,17 @@ static void domcreate_bootloader_done(libxl__egc *egc,
if (rc)
goto out;
- /* read signature */
- int hvm, pae, superpages;
- switch (info->type) {
- case LIBXL_DOMAIN_TYPE_HVM:
- hvm = 1;
- superpages = 1;
- pae = libxl_defbool_val(info->u.hvm.pae);
- callbacks->toolstack_restore = libxl__toolstack_restore;
- break;
- case LIBXL_DOMAIN_TYPE_PV:
- hvm = 0;
- superpages = 0;
- pae = 1;
- break;
- default:
- rc = ERROR_INVAL;
- goto out;
- }
- libxl__xc_domain_restore(egc, dcs, &dcs->shs,
- hvm, pae, superpages);
+ dcs->srs.ao = ao;
+ dcs->srs.dcs = dcs;
+ dcs->srs.fd = restore_fd;
+ dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
+ dcs->srs.completion_callback = domcreate_stream_done;
+
+ libxl__stream_read_start(egc, &dcs->srs);
return;
out:
- libxl__xc_domain_restore_done(egc, dcs, rc, 0, 0);
+ domcreate_stream_done(egc, &dcs->srs, rc);
}
void libxl__srm_callout_callback_restore_results(unsigned long store_mfn,
@@ -1009,10 +997,11 @@ void libxl__srm_callout_callback_restore_results(unsigned long store_mfn,
shs->need_results = 0;
}
-void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
- int ret, int retval, int errnoval)
+static void domcreate_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int ret)
{
- libxl__domain_create_state *dcs = dcs_void;
+ libxl__domain_create_state *dcs = srs->dcs;
STATE_AO_GC(dcs->ao);
libxl_ctx *ctx = libxl__gc_owner(gc);
char **vments = NULL, **localents = NULL;
@@ -1029,12 +1018,6 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
if (ret)
goto out;
- if (retval) {
- LOGEV(ERROR, errnoval, "restoring domain");
- ret = ERROR_FAIL;
- goto out;
- }
-
gettimeofday(&start_time, NULL);
switch (info->type) {
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index b65019b..0a13c7f 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3265,6 +3265,7 @@ struct libxl__stream_read_state {
/* Private */
int rc;
bool running;
+ libxl__save_helper_state shs;
libxl__conversion_helper_state chs;
/* Main stream-reading data. */
@@ -3312,7 +3313,7 @@ struct libxl__domain_create_state {
libxl__stub_dm_spawn_state dmss;
/* If we're not doing stubdom, we use only dmss.dm,
* for the non-stubdom device model. */
- libxl__save_helper_state shs;
+ libxl__stream_read_state srs;
/* necessary if the domain creation failed and we have to destroy it */
libxl__domain_destroy_state dds;
libxl__multidev multidev;
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index 14675ae..efbe2eb 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -313,7 +313,7 @@ int main(int argc, char **argv)
startup("restore");
setup_signals(SIG_DFL);
- r = xc_domain_restore(xch, io_fd, dom, store_evtchn, &store_mfn,
+ r = xc_domain_restore2(xch, io_fd, dom, store_evtchn, &store_mfn,
store_domid, console_evtchn, &console_mfn,
console_domid, hvm, pae, superpages,
checkpointed,
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index 4b677f6..3051a2b 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -159,6 +159,7 @@ void libxl__stream_read_init(libxl__stream_read_state *stream)
{
stream->rc = 0;
stream->running = false;
+ libxl__save_helper_init(&stream->shs);
libxl__conversion_helper_init(&stream->chs);
FILLZERO(stream->dc);
FILLZERO(stream->hdr);
@@ -445,9 +446,13 @@ static bool process_record(libxl__egc *egc,
stream_complete(egc, stream, 0);
break;
+ case REC_TYPE_LIBXC_CONTEXT:
+ libxl__xc_domain_restore(egc, dcs, &stream->shs, 0, 0, 0);
+ break;
+
case REC_TYPE_XENSTORE_DATA:
rc = libxl__toolstack_restore(dcs->guest_domid, rec->body,
- rec->hdr.length, &dcs->shs);
+ rec->hdr.length, &stream->shs);
if (rc)
goto err;
@@ -592,6 +597,32 @@ static void stream_done(libxl__egc *egc,
check_all_finished(egc, stream, stream->rc);
}
+void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
+ int rc, int retval, int errnoval)
+{
+ libxl__domain_create_state *dcs = dcs_void;
+ libxl__stream_read_state *stream = &dcs->srs;
+ STATE_AO_GC(dcs->ao);
+
+ if (rc)
+ goto err;
+
+ if (retval) {
+ LOGEV(ERROR, errnoval, "restoring domain");
+ rc = ERROR_FAIL;
+ goto err;
+ }
+
+ /*
+ * Libxc has indicated that it is done with the stream. Resume reading
+ * libxl records from it.
+ */
+ stream_continue(egc, stream);
+
+ err:
+ check_all_finished(egc, stream, rc);
+}
+
static void conversion_done(libxl__egc *egc,
libxl__conversion_helper_state *chs, int rc)
{
@@ -610,11 +641,13 @@ static void check_all_finished(libxl__egc *egc,
stream->rc = rc;
libxl__stream_read_abort(egc, stream, rc);
+ libxl__save_helper_abort(egc, &stream->shs);
libxl__conversion_helper_abort(egc, &stream->chs, rc);
}
/* Don't fire the callback until all our parallel tasks have stopped. */
if (libxl__stream_read_inuse(stream) ||
+ libxl__save_helper_inuse(&stream->shs) ||
libxl__conversion_helper_inuse(&stream->chs))
return;
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 8dacf8d..bc0c4ef 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -355,6 +355,7 @@ libxl_domain_create_info = Struct("domain_create_info",[
libxl_domain_restore_params = Struct("domain_restore_params", [
("checkpointed_stream", integer),
+ ("stream_version", uint32, {'init_val': '1'}),
])
libxl_domain_sched_params = Struct("domain_sched_params",[
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 26b1e7d..71a22b8 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -110,7 +110,9 @@
#define XL_MANDATORY_FLAG_JSON (1U << 0) /* config data is in JSON format */
#define XL_MANDATORY_FLAG_STREAMv2 (1U << 1) /* stream is v2 */
-#define XL_MANDATORY_FLAG_ALL (XL_MANDATORY_FLAG_JSON)
+#define XL_MANDATORY_FLAG_ALL (XL_MANDATORY_FLAG_JSON | \
+ XL_MANDATORY_FLAG_STREAMv2)
+
struct save_file_header {
char magic[32]; /* savefileheader_magic */
/* All uint32_ts are in domain's byte order. */
@@ -2757,6 +2759,9 @@ static uint32_t create_domain(struct domain_create *dom_info)
libxl_domain_restore_params_init(¶ms);
params.checkpointed_stream = dom_info->checkpointed_stream;
+ params.stream_version =
+ (hdr.mandatory_flags & XL_MANDATORY_FLAG_STREAMv2) ? 2 : 1;
+
ret = libxl_domain_create_restore(ctx, &d_config,
&domid, restore_fd,
¶ms,
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 22/29] tools/libxl: Infrastructure for writing a v2 stream
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (20 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 21/29] tools/libxc+libxl+xl: Restore v2 streams Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 13:40 ` Ian Jackson
2015-07-14 10:59 ` [PATCH v4 23/29] tools/libxc+libxl+xl: Save v2 streams Andrew Cooper
` (7 subsequent siblings)
29 siblings, 1 reply; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel
Cc: Ross Lagerwall, Wei Liu, Ian Jackson, Ian Campbell, Andrew Cooper
From: Ross Lagerwall <ross.lagerwall@citrix.com>
This contains the event machinery and state machines to write
non-checkpointed migration v2 stream (with the exception of the
xc_domain_save() handling which is spliced later in a bisectable way).
Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v4: Don't make _init() mandatory, use FILLZERO(), adjust stream_success()
v3: Similar changes to the read side, following the read side review.
---
tools/libxl/Makefile | 2 +-
tools/libxl/libxl_internal.h | 39 ++++
tools/libxl/libxl_stream_write.c | 445 ++++++++++++++++++++++++++++++++++++++
3 files changed, 485 insertions(+), 1 deletion(-)
create mode 100644 tools/libxl/libxl_stream_write.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 68fe1c1..0150ec7 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -100,7 +100,7 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
libxl_internal.o libxl_utils.o libxl_uuid.o \
libxl_json.o libxl_aoutils.o libxl_numa.o libxl_vnuma.o \
- libxl_stream_read.o \
+ libxl_stream_read.o libxl_stream_write.o \
libxl_save_callout.o _libxl_save_msgs_callout.o \
libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
LIBXL_OBJS += libxl_genid.o
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 0a13c7f..9ddb774 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2956,6 +2956,45 @@ typedef void libxl__domain_suspend_cb(libxl__egc*,
typedef void libxl__save_device_model_cb(libxl__egc*,
libxl__domain_suspend_state*, int rc);
+/* State for writing a libxl migration v2 stream */
+typedef struct libxl__stream_write_state libxl__stream_write_state;
+typedef void (*sws_record_done_cb)(libxl__egc *egc,
+ libxl__stream_write_state *sws);
+struct libxl__stream_write_state {
+ /* filled by the user */
+ libxl__ao *ao;
+ libxl__domain_suspend_state *dss;
+ int fd;
+ void (*completion_callback)(libxl__egc *egc,
+ libxl__stream_write_state *sws,
+ int rc);
+ /* Private */
+ int rc;
+ bool running;
+
+ /* Main stream-writing data. */
+ libxl__datacopier_state dc;
+ sws_record_done_cb record_done_callback;
+
+ /* Only used when constructing an EMULATOR record. */
+ libxl__datacopier_state emu_dc;
+ libxl__carefd *emu_carefd;
+ libxl__sr_rec_hdr emu_rec_hdr;
+ void *emu_body;
+};
+
+_hidden void libxl__stream_write_init(libxl__stream_write_state *stream);
+_hidden void libxl__stream_write_start(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+_hidden void libxl__stream_write_abort(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc);
+static inline bool
+libxl__stream_write_inuse(const libxl__stream_write_state *stream)
+{
+ return stream->running;
+}
+
typedef struct libxl__logdirty_switch {
const char *cmd;
const char *cmd_path;
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
new file mode 100644
index 0000000..a324d30
--- /dev/null
+++ b/tools/libxl/libxl_stream_write.c
@@ -0,0 +1,445 @@
+/*
+ * Copyright (C) 2015 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/*
+ * Infrastructure for writing a domain to a libxl migration v2 stream.
+ *
+ * Entry points from outside:
+ * - libxl__stream_write_start()
+ * - Start writing a stream from the start.
+ *
+ * In normal operation, there are two tasks running at once; this
+ * stream processing, and the libxl-save-helper. check_all_finished()
+ * is used to join all the tasks in both success and error cases.
+ *
+ * Nomenclature for event callbacks:
+ * - $FOO_done(): Completion callback for $FOO
+ * - write_$FOO(): Set up the datacopier to write a $FOO
+ * - $BAR_header(): A $BAR record header only
+ * - $BAR_record(): A complete $BAR record with header and content
+ *
+ * The main loop for a plain VM writes:
+ * - Stream header
+ * - Libxc record
+ * - Toolstack record
+ * - if (hvm), Qemu record
+ * - End record
+ */
+
+/* Success/error/cleanup handling. */
+static void stream_success(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void stream_complete(libxl__egc *egc,
+ libxl__stream_write_state *stream, int rc);
+static void stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void check_all_finished(libxl__egc *egc,
+ libxl__stream_write_state *stream, int rc);
+
+/* Event chain for a plain VM. */
+static void stream_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+static void libxc_header_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+/* libxl__xc_domain_save_done() lives here, event-order wise. */
+static void write_toolstack_record(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void toolstack_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void write_emulator_record(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void emulator_read_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+static void emulator_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void write_end_record(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+
+/*----- Helpers -----*/
+
+static void write_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+
+/* Helper to set up reading some data from the stream. */
+static void setup_write(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ const char *what,
+ libxl__sr_rec_hdr *hdr, void *body,
+ sws_record_done_cb cb)
+{
+ static const uint8_t zero_padding[1U << REC_ALIGN_ORDER] = { 0 };
+
+ libxl__datacopier_state *dc = &stream->dc;
+ int rc;
+
+ assert(stream->record_done_callback == NULL);
+
+ dc->writewhat = what;
+ dc->used = 0;
+ dc->callback = write_done;
+ rc = libxl__datacopier_start(dc);
+
+ if (rc) {
+ stream_complete(egc, stream, rc);
+ return;
+ }
+
+ size_t padsz = ROUNDUP(hdr->length, REC_ALIGN_ORDER) - hdr->length;
+
+ /* Insert header */
+ libxl__datacopier_prefixdata(egc, dc, hdr, sizeof(*hdr));
+
+ /* Optional body */
+ if (body)
+ libxl__datacopier_prefixdata(egc, dc, body, hdr->length);
+
+ /* Any required padding */
+ if (padsz > 0)
+ libxl__datacopier_prefixdata(egc, dc,
+ zero_padding, padsz);
+ stream->record_done_callback = cb;
+}
+
+static void write_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, dc);
+ STATE_AO_GC(stream->ao);
+ sws_record_done_cb cb = stream->record_done_callback;
+
+ stream->record_done_callback = NULL;
+
+ if (onwrite || errnoval)
+ stream_complete(egc, stream, rc ?: ERROR_FAIL);
+ else
+ cb(egc, stream);
+}
+
+/*----- Entrypoints -----*/
+
+void libxl__stream_write_init(libxl__stream_write_state *stream)
+{
+ stream->rc = 0;
+ stream->running = false;
+ FILLZERO(stream->dc);
+ stream->record_done_callback = NULL;
+ FILLZERO(stream->emu_dc);
+ stream->emu_carefd = NULL;
+ FILLZERO(stream->emu_rec_hdr);
+ stream->emu_body = NULL;
+}
+
+void libxl__stream_write_start(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ libxl__datacopier_state *dc = &stream->dc;
+ STATE_AO_GC(stream->ao);
+ struct libxl__sr_hdr hdr;
+ int rc = 0;
+
+ libxl__stream_write_init(stream);
+
+ stream->running = true;
+
+ dc->ao = ao;
+ dc->readfd = -1;
+ dc->writewhat = "save/migration stream";
+ dc->writefd = stream->fd;
+ dc->maxsz = -1;
+ dc->callback = stream_header_done;
+
+ rc = libxl__datacopier_start(dc);
+ if (rc)
+ goto err;
+
+ FILLZERO(hdr);
+ hdr.ident = htobe64(RESTORE_STREAM_IDENT);
+ hdr.version = htobe32(RESTORE_STREAM_VERSION);
+ hdr.options = htobe32(0);
+
+ libxl__datacopier_prefixdata(egc, dc, &hdr, sizeof(hdr));
+ return;
+
+ err:
+ assert(rc);
+ stream_complete(egc, stream, rc);
+}
+
+void libxl__stream_write_abort(libxl__egc *egc,
+ libxl__stream_write_state *stream, int rc)
+{
+ assert(rc);
+
+ if (stream->running)
+ stream_complete(egc, stream, rc);
+}
+
+/*----- Event logic -----*/
+
+static void stream_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, dc);
+ STATE_AO_GC(stream->ao);
+ struct libxl__sr_rec_hdr rec;
+
+ if (rc || errnoval) {
+ stream_complete(egc, stream, rc ?: ERROR_FAIL);
+ return;
+ }
+
+ FILLZERO(rec);
+ rec.type = REC_TYPE_LIBXC_CONTEXT;
+
+ setup_write(egc, stream, "libxc header",
+ &rec, NULL, libxc_header_done);
+}
+
+static void libxc_header_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ libxl__xc_domain_save(egc, stream->dss, &stream->dss->shs);
+}
+
+static void __attribute__((unused))
+write_toolstack_record(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ libxl__domain_suspend_state *dss = stream->dss;
+ STATE_AO_GC(stream->ao);
+ struct libxl__sr_rec_hdr rec;
+ int rc;
+ uint8_t *toolstack_buf = NULL; /* We must free this. */
+ uint32_t toolstack_len;
+
+ rc = libxl__toolstack_save(dss->domid, &toolstack_buf,
+ &toolstack_len, dss);
+ if (rc)
+ goto err;
+
+ FILLZERO(rec);
+ rec.type = REC_TYPE_XENSTORE_DATA;
+ rec.length = toolstack_len;
+
+ setup_write(egc, stream, "toolstack record",
+ &rec, toolstack_buf,
+ toolstack_record_done);
+
+ free(toolstack_buf);
+ return;
+
+ err:
+ assert(rc);
+ free(toolstack_buf);
+ stream_complete(egc, stream, rc);
+}
+
+static void toolstack_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ libxl__domain_suspend_state *dss = stream->dss;
+
+ if (dss->type == LIBXL_DOMAIN_TYPE_HVM)
+ write_emulator_record(egc, stream);
+ else
+ write_end_record(egc, stream);
+}
+
+static void write_emulator_record(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ libxl__domain_suspend_state *dss = stream->dss;
+ libxl__datacopier_state *dc = &stream->emu_dc;
+ STATE_AO_GC(stream->ao);
+ struct libxl__sr_rec_hdr *rec = &stream->emu_rec_hdr;
+ struct libxl__sr_emulator_hdr *ehdr = NULL;
+ struct stat st;
+ int rc;
+
+ assert(dss->type == LIBXL_DOMAIN_TYPE_HVM);
+
+ /* Convenience aliases */
+ const char *const filename = dss->dm_savefile;
+ const uint32_t domid = dss->domid;
+
+ libxl__carefd_begin();
+ int readfd = open(filename, O_RDONLY);
+ stream->emu_carefd = libxl__carefd_opened(CTX, readfd);
+ if (readfd == -1) {
+ rc = ERROR_FAIL;
+ LOGE(ERROR, "unable to open %s", filename);
+ goto err;
+ }
+
+ if (fstat(readfd, &st)) {
+ rc = ERROR_FAIL;
+ LOGE(ERROR, "unable to fstat %s", filename);
+ goto err;
+ }
+
+ if (!S_ISREG(st.st_mode)) {
+ rc = ERROR_FAIL;
+ LOG(ERROR, "%s is not a plain file!", filename);
+ goto err;
+ }
+
+ rec->type = REC_TYPE_EMULATOR_CONTEXT;
+ rec->length = st.st_size + sizeof(*ehdr);
+ stream->emu_body = ehdr = libxl__malloc(NOGC, rec->length);
+
+ switch(libxl__device_model_version_running(gc, domid)) {
+ case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
+ ehdr->id = EMULATOR_QEMU_TRADITIONAL;
+ break;
+
+ case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
+ ehdr->id = EMULATOR_QEMU_UPSTREAM;
+ break;
+
+ default:
+ rc = ERROR_FAIL;
+ goto err;
+ }
+ ehdr->index = 0;
+
+ FILLZERO(*dc);
+ dc->ao = stream->ao;
+ dc->readwhat = "qemu save file";
+ dc->readfd = readfd;
+ dc->maxsz = -1;
+ dc->readbuf = stream->emu_body + sizeof(*ehdr);
+ dc->bytes_to_read = rec->length - sizeof(*ehdr);
+ dc->callback = emulator_read_done;
+
+ rc = libxl__datacopier_start(dc);
+ if (rc)
+ goto err;
+
+ return;
+
+ err:
+ assert(rc);
+ stream_complete(egc, stream, rc);
+}
+
+static void emulator_read_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, emu_dc);
+ STATE_AO_GC(stream->ao);
+
+ if (rc || onwrite || errnoval) {
+ stream_complete(egc, stream, rc ?: ERROR_FAIL);
+ return;
+ }
+
+ libxl__carefd_close(stream->emu_carefd);
+ stream->emu_carefd = NULL;
+
+ setup_write(egc, stream, "emulator record",
+ &stream->emu_rec_hdr, stream->emu_body,
+ emulator_record_done);
+}
+
+static void emulator_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ free(stream->emu_body);
+ stream->emu_body = NULL;
+
+ write_end_record(egc, stream);
+}
+
+static void write_end_record(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ struct libxl__sr_rec_hdr rec;
+
+ FILLZERO(rec);
+ rec.type = REC_TYPE_END;
+
+ setup_write(egc, stream, "end record",
+ &rec, NULL, stream_success);
+}
+
+/*----- Success/error/cleanup handling. -----*/
+
+static void stream_success(libxl__egc *egc, libxl__stream_write_state *stream)
+{
+ stream_complete(egc, stream, 0);
+}
+
+static void stream_complete(libxl__egc *egc,
+ libxl__stream_write_state *stream, int rc)
+{
+ assert(stream->running);
+
+ if (!stream->rc)
+ stream->rc = rc;
+ stream_done(egc, stream);
+}
+
+static void stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ assert(stream->running);
+ stream->running = false;
+
+ if (stream->emu_carefd)
+ libxl__carefd_close(stream->emu_carefd);
+ free(stream->emu_body);
+
+ check_all_finished(egc, stream, stream->rc);
+}
+
+static void check_all_finished(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc)
+{
+ libxl__domain_suspend_state *dss = stream->dss;
+ STATE_AO_GC(stream->ao);
+
+ if (!stream->rc && rc) {
+ /* First reported failure. Tear everything down. */
+ stream->rc = rc;
+
+ libxl__stream_write_abort(egc, stream, rc);
+ libxl__save_helper_abort(egc, &dss->shs);
+ }
+
+ /* Don't fire the callback until all our parallel tasks have stopped. */
+ if (libxl__stream_write_inuse(stream) ||
+ libxl__save_helper_inuse(&dss->shs))
+ return;
+
+ stream->completion_callback(egc, stream, stream->rc);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 23/29] tools/libxc+libxl+xl: Save v2 streams
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (21 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 22/29] tools/libxl: Infrastructure for writing a v2 stream Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 24/29] docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams Andrew Cooper
` (6 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Wei Liu, Ian Campbell
This is a complicated set of changes which must be done together for
bisectability.
* libxl-save-helper is updated to unconditionally use libxc migration v2.
* libxl compatibility workarounds in libxc are disabled for save operations.
* libxl__stream_write_start() is logically spliced into the event location
where libxl__xc_domain_save() used to reside.
* Ownership of the helper state moves into stream_write_state.
* xl is updated to indicate that the stream is now v2
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v3: Many small changes, follwing similar review to the restore side
---
tools/libxc/Makefile | 2 --
tools/libxl/libxl.h | 2 ++
tools/libxl/libxl_dom.c | 69 ++++++++++++--------------------------
tools/libxl/libxl_internal.h | 3 +-
tools/libxl/libxl_save_helper.c | 2 +-
tools/libxl/libxl_stream_write.c | 40 ++++++++++++++++++----
tools/libxl/xl_cmdimpl.c | 1 +
7 files changed, 61 insertions(+), 58 deletions(-)
diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
index 2cd0b1a..1aec848 100644
--- a/tools/libxc/Makefile
+++ b/tools/libxc/Makefile
@@ -64,8 +64,6 @@ GUEST_SRCS-$(CONFIG_X86) += xc_sr_save_x86_hvm.c
GUEST_SRCS-y += xc_sr_restore.c
GUEST_SRCS-y += xc_sr_save.c
GUEST_SRCS-y += xc_offline_page.c xc_compression.c
-xc_sr_save_x86_hvm.o: CFLAGS += -DXG_LIBXL_HVM_COMPAT
-xc_sr_save_x86_hvm.opic: CFLAGS += -DXG_LIBXL_HVM_COMPAT
else
GUEST_SRCS-y += xc_nomigrate.c
endif
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index 74b0829..5a7308d 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -812,6 +812,8 @@
*
* If this is defined, then the libxl_domain_create_restore() interface takes
* a "stream_version" parameter and supports a value of 2.
+ *
+ * libxl_domain_suspend() will produce a v2 stream.
*/
#define LIBXL_HAVE_SRM_V2 1
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index a685b77..cda3a7a 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1153,6 +1153,8 @@ int libxl__toolstack_restore(uint32_t domid, const uint8_t *ptr,
/*==================== Domain suspend (save) ====================*/
+static void stream_done(libxl__egc *egc,
+ libxl__stream_write_state *sws, int rc);
static void domain_suspend_done(libxl__egc *egc,
libxl__domain_suspend_state *dss, int rc);
static void domain_suspend_callback_common_done(libxl__egc *egc,
@@ -1379,7 +1381,7 @@ static void switch_logdirty_done(libxl__egc *egc,
} else {
broke = 0;
}
- libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, broke);
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, broke);
}
/*----- callbacks, called by xc_domain_save -----*/
@@ -1842,7 +1844,7 @@ static void domain_suspend_callback_common_done(libxl__egc *egc,
libxl__domain_suspend_state *dss, int rc)
{
dss->rc = rc;
- libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, !rc);
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, !rc);
}
/*----- remus callbacks -----*/
@@ -1878,7 +1880,7 @@ static void remus_domain_suspend_callback_common_done(libxl__egc *egc,
out:
dss->rc = rc;
- libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, !rc);
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, !rc);
}
static void remus_devices_postsuspend_cb(libxl__egc *egc,
@@ -1895,7 +1897,7 @@ static void remus_devices_postsuspend_cb(libxl__egc *egc,
out:
if (rc)
dss->rc = rc;
- libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, !rc);
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, !rc);
}
static void libxl__remus_domain_resume_callback(void *data)
@@ -1930,7 +1932,7 @@ static void remus_devices_preresume_cb(libxl__egc *egc,
out:
if (rc)
dss->rc = rc;
- libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, !rc);
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, !rc);
}
/*----- remus asynchronous checkpoint callback -----*/
@@ -1978,7 +1980,7 @@ static void remus_checkpoint_dm_saved(libxl__egc *egc,
return;
out:
- libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 0);
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
}
static void remus_devices_commit_cb(libxl__egc *egc,
@@ -2013,7 +2015,7 @@ static void remus_devices_commit_cb(libxl__egc *egc,
return;
out:
- libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, 0);
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, 0);
}
static void remus_next_checkpoint(libxl__egc *egc, libxl__ev_time *ev,
@@ -2034,7 +2036,7 @@ static void remus_next_checkpoint(libxl__egc *egc, libxl__ev_time *ev,
if (rc)
dss->rc = rc;
- libxl__xc_domain_saverestore_async_callback_done(egc, &dss->shs, !rc);
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dss->sws.shs, !rc);
}
/*----- main code for suspending, in order of execution -----*/
@@ -2052,7 +2054,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
const int debug = dss->debug;
const libxl_domain_remus_info *const r_info = dss->remus;
libxl__srm_save_autogen_callbacks *const callbacks =
- &dss->shs.callbacks.save.a;
+ &dss->sws.shs.callbacks.save.a;
dss->rc = 0;
logdirty_init(&dss->logdirty);
@@ -2115,53 +2117,24 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
callbacks->suspend = libxl__domain_suspend_callback;
callbacks->switch_qemu_logdirty = libxl__domain_suspend_common_switch_qemu_logdirty;
- dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
+ dss->sws.shs.callbacks.save.toolstack_save = libxl__toolstack_save;
- libxl__xc_domain_save(egc, dss, &dss->shs);
+ dss->sws.ao = dss->ao;
+ dss->sws.dss = dss;
+ dss->sws.fd = dss->fd;
+ dss->sws.completion_callback = stream_done;
+
+ libxl__stream_write_start(egc, &dss->sws);
return;
out:
domain_suspend_done(egc, dss, rc);
}
-void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void,
- int rc, int retval, int errnoval)
+static void stream_done(libxl__egc *egc,
+ libxl__stream_write_state *sws, int rc)
{
- libxl__domain_suspend_state *dss = dss_void;
- STATE_AO_GC(dss->ao);
-
- /* Convenience aliases */
- const libxl_domain_type type = dss->type;
-
- if (rc)
- goto out;
-
- if (retval) {
- LOGEV(ERROR, errnoval, "saving domain: %s",
- dss->guest_responded ?
- "domain responded to suspend request" :
- "domain did not respond to suspend request");
- if ( !dss->guest_responded )
- rc = ERROR_GUEST_TIMEDOUT;
- else if (dss->rc)
- rc = dss->rc;
- else
- rc = ERROR_FAIL;
- goto out;
- }
-
- if (type == LIBXL_DOMAIN_TYPE_HVM) {
- rc = libxl__domain_suspend_device_model(gc, dss);
- if (rc) goto out;
-
- libxl__domain_save_device_model(egc, dss, domain_suspend_done);
- return;
- }
-
- rc = 0;
-
-out:
- domain_suspend_done(egc, dss, rc);
+ domain_suspend_done(egc, sws->dss, rc);
}
static void save_device_model_datacopier_done(libxl__egc *egc,
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 9ddb774..0697ba6 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2971,6 +2971,7 @@ struct libxl__stream_write_state {
/* Private */
int rc;
bool running;
+ libxl__save_helper_state shs;
/* Main stream-writing data. */
libxl__datacopier_state dc;
@@ -3028,7 +3029,7 @@ struct libxl__domain_suspend_state {
libxl__remus_devices_state rds;
libxl__ev_time checkpoint_timeout; /* used for Remus checkpoint */
int interval; /* checkpoint interval (for Remus) */
- libxl__save_helper_state shs;
+ libxl__stream_write_state sws;
libxl__logdirty_switch logdirty;
void (*callback_common_done)(libxl__egc*,
struct libxl__domain_suspend_state*, int ok);
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index efbe2eb..f196786 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -286,7 +286,7 @@ int main(int argc, char **argv)
startup("save");
setup_signals(save_signal_handler);
- r = xc_domain_save(xch, io_fd, dom, max_iters, max_factor, flags,
+ r = xc_domain_save2(xch, io_fd, dom, max_iters, max_factor, flags,
&helper_save_callbacks, hvm);
complete(r);
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
index a324d30..d1343e0 100644
--- a/tools/libxl/libxl_stream_write.c
+++ b/tools/libxl/libxl_stream_write.c
@@ -218,11 +218,40 @@ static void stream_header_done(libxl__egc *egc,
static void libxc_header_done(libxl__egc *egc,
libxl__stream_write_state *stream)
{
- libxl__xc_domain_save(egc, stream->dss, &stream->dss->shs);
+ libxl__xc_domain_save(egc, stream->dss, &stream->shs);
}
-static void __attribute__((unused))
-write_toolstack_record(libxl__egc *egc,
+void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void,
+ int rc, int retval, int errnoval)
+{
+ libxl__domain_suspend_state *dss = dss_void;
+ libxl__stream_write_state *stream = &dss->sws;
+ STATE_AO_GC(dss->ao);
+
+ if (rc)
+ goto err;
+
+ if (retval) {
+ LOGEV(ERROR, errnoval, "saving domain: %s",
+ dss->guest_responded ?
+ "domain responded to suspend request" :
+ "domain did not respond to suspend request");
+ if (!dss->guest_responded)
+ rc = ERROR_GUEST_TIMEDOUT;
+ else if (dss->rc)
+ rc = dss->rc;
+ else
+ rc = ERROR_FAIL;
+ goto err;
+ }
+
+ write_toolstack_record(egc, stream);
+
+ err:
+ check_all_finished(egc, stream, rc);
+}
+
+static void write_toolstack_record(libxl__egc *egc,
libxl__stream_write_state *stream)
{
libxl__domain_suspend_state *dss = stream->dss;
@@ -417,7 +446,6 @@ static void check_all_finished(libxl__egc *egc,
libxl__stream_write_state *stream,
int rc)
{
- libxl__domain_suspend_state *dss = stream->dss;
STATE_AO_GC(stream->ao);
if (!stream->rc && rc) {
@@ -425,12 +453,12 @@ static void check_all_finished(libxl__egc *egc,
stream->rc = rc;
libxl__stream_write_abort(egc, stream, rc);
- libxl__save_helper_abort(egc, &dss->shs);
+ libxl__save_helper_abort(egc, &stream->shs);
}
/* Don't fire the callback until all our parallel tasks have stopped. */
if (libxl__stream_write_inuse(stream) ||
- libxl__save_helper_inuse(&dss->shs))
+ libxl__save_helper_inuse(&stream->shs))
return;
stream->completion_callback(egc, stream, stream->rc);
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 71a22b8..13e154d 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -3922,6 +3922,7 @@ static void save_domain_core_writeconfig(int fd, const char *source,
memset(&hdr, 0, sizeof(hdr));
memcpy(hdr.magic, savefileheader_magic, sizeof(hdr.magic));
hdr.byteorder = SAVEFILE_BYTEORDER_VALUE;
+ hdr.mandatory_flags = XL_MANDATORY_FLAG_STREAMv2;
optdata_begin= 0;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 24/29] docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (22 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 23/29] tools/libxc+libxl+xl: Save v2 streams Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 25/29] tools/libxl: Write checkpoint records into the stream Andrew Cooper
` (5 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
In a remus scenario, libxc will write a CHECKPOINT record, then hand
ownership of the fd to libxl. Libxl then writes any records required
and finishes with a CHECKPOINT_END record, then hands ownership of the
fd back to libxc.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
docs/specs/libxl-migration-stream.pandoc | 14 +++++++++++++-
tools/libxl/libxl_sr_stream_format.h | 1 +
tools/python/xen/migration/libxl.py | 11 +++++++++++
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/docs/specs/libxl-migration-stream.pandoc b/docs/specs/libxl-migration-stream.pandoc
index 4192950..c24a434 100644
--- a/docs/specs/libxl-migration-stream.pandoc
+++ b/docs/specs/libxl-migration-stream.pandoc
@@ -119,7 +119,9 @@ type 0x00000000: END
0x00000003: EMULATOR_CONTEXT
- 0x00000004 - 0x7FFFFFFF: Reserved for future _mandatory_
+ 0x00000004: CHECKPOINT_END
+
+ 0x00000005 - 0x7FFFFFFF: Reserved for future _mandatory_
records.
0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -203,3 +205,13 @@ index Index of this emulator for the domain, if multiple
emulator_ctx Emulator context blob.
--------------------------------------------------------------------
+
+CHECKPOINT\_END
+---------------
+
+A checkpoint end record marks the end of a checkpoint in the image.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+
+The end record contains no fields; its body_length is 0.
diff --git a/tools/libxl/libxl_sr_stream_format.h b/tools/libxl/libxl_sr_stream_format.h
index f4f790b..3f3c497 100644
--- a/tools/libxl/libxl_sr_stream_format.h
+++ b/tools/libxl/libxl_sr_stream_format.h
@@ -35,6 +35,7 @@
#define REC_TYPE_LIBXC_CONTEXT 0x00000001U
#define REC_TYPE_XENSTORE_DATA 0x00000002U
#define REC_TYPE_EMULATOR_CONTEXT 0x00000003U
+#define REC_TYPE_CHECKPOINT_END 0x00000004U
typedef struct libxl__sr_emulator_hdr
{
diff --git a/tools/python/xen/migration/libxl.py b/tools/python/xen/migration/libxl.py
index 4e1f4f8..415502e 100644
--- a/tools/python/xen/migration/libxl.py
+++ b/tools/python/xen/migration/libxl.py
@@ -36,12 +36,14 @@ REC_TYPE_end = 0x00000000
REC_TYPE_libxc_context = 0x00000001
REC_TYPE_xenstore_data = 0x00000002
REC_TYPE_emulator_context = 0x00000003
+REC_TYPE_checkpoint_end = 0x00000004
rec_type_to_str = {
REC_TYPE_end : "End",
REC_TYPE_libxc_context : "Libxc context",
REC_TYPE_xenstore_data : "Xenstore data",
REC_TYPE_emulator_context : "Emulator context",
+ REC_TYPE_checkpoint_end : "Checkpoint end",
}
# emulator_context
@@ -176,6 +178,13 @@ class VerifyLibxl(VerifyBase):
self.info(" Index %d, type %s" % (emu_idx, emulator_id_to_str[emu_id]))
+ def verify_record_checkpoint_end(self, content):
+ """ Checkpoint end record """
+
+ if len(content) != 0:
+ raise RecordError("Checkpoint end record with non-zero length")
+
+
record_verifiers = {
REC_TYPE_end:
VerifyLibxl.verify_record_end,
@@ -185,4 +194,6 @@ record_verifiers = {
VerifyLibxl.verify_record_xenstore_data,
REC_TYPE_emulator_context:
VerifyLibxl.verify_record_emulator_context,
+ REC_TYPE_checkpoint_end:
+ VerifyLibxl.verify_record_checkpoint_end,
}
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 25/29] tools/libxl: Write checkpoint records into the stream
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (23 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 24/29] docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 26/29] tools/libx{c, l}: Introduce restore_callbacks.checkpoint() Andrew Cooper
` (4 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
when signalled to do so by libxl__remus_domain_checkpoint_callback()
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v3: Corrections to comments
This patch has changed substantially in v2 as a result of changes earlier in
the series. No behavioural difference from v1.
---
tools/libxl/libxl_dom.c | 18 ++++-----
tools/libxl/libxl_internal.h | 7 ++++
tools/libxl/libxl_stream_write.c | 81 ++++++++++++++++++++++++++++++++++++--
3 files changed, 93 insertions(+), 13 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index cda3a7a..927a10e 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1937,8 +1937,8 @@ static void remus_devices_preresume_cb(libxl__egc *egc,
/*----- remus asynchronous checkpoint callback -----*/
-static void remus_checkpoint_dm_saved(libxl__egc *egc,
- libxl__domain_suspend_state *dss, int rc);
+static void remus_checkpoint_stream_written(
+ libxl__egc *egc, libxl__stream_write_state *sws, int rc);
static void remus_devices_commit_cb(libxl__egc *egc,
libxl__remus_devices_state *rds,
int rc);
@@ -1953,17 +1953,14 @@ static void libxl__remus_domain_checkpoint_callback(void *data)
libxl__egc *egc = shs->egc;
STATE_AO_GC(dss->ao);
- /* This would go into tailbuf. */
- if (dss->hvm) {
- libxl__domain_save_device_model(egc, dss, remus_checkpoint_dm_saved);
- } else {
- remus_checkpoint_dm_saved(egc, dss, 0);
- }
+ libxl__stream_write_start_checkpoint(egc, &dss->sws);
}
-static void remus_checkpoint_dm_saved(libxl__egc *egc,
- libxl__domain_suspend_state *dss, int rc)
+static void remus_checkpoint_stream_written(
+ libxl__egc *egc, libxl__stream_write_state *sws, int rc)
{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(sws, *dss, sws);
+
/* Convenience aliases */
libxl__remus_devices_state *const rds = &dss->rds;
@@ -2113,6 +2110,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
callbacks->suspend = libxl__remus_domain_suspend_callback;
callbacks->postcopy = libxl__remus_domain_resume_callback;
callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;
+ dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
} else
callbacks->suspend = libxl__domain_suspend_callback;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 0697ba6..9961a53 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2968,9 +2968,13 @@ struct libxl__stream_write_state {
void (*completion_callback)(libxl__egc *egc,
libxl__stream_write_state *sws,
int rc);
+ void (*checkpoint_callback)(libxl__egc *egc,
+ libxl__stream_write_state *sws,
+ int rc);
/* Private */
int rc;
bool running;
+ bool in_checkpoint;
libxl__save_helper_state shs;
/* Main stream-writing data. */
@@ -2987,6 +2991,9 @@ struct libxl__stream_write_state {
_hidden void libxl__stream_write_init(libxl__stream_write_state *stream);
_hidden void libxl__stream_write_start(libxl__egc *egc,
libxl__stream_write_state *stream);
+_hidden void
+libxl__stream_write_start_checkpoint(libxl__egc *egc,
+ libxl__stream_write_state *stream);
_hidden void libxl__stream_write_abort(libxl__egc *egc,
libxl__stream_write_state *stream,
int rc);
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
index d1343e0..944a87b 100644
--- a/tools/libxl/libxl_stream_write.c
+++ b/tools/libxl/libxl_stream_write.c
@@ -22,6 +22,8 @@
* Entry points from outside:
* - libxl__stream_write_start()
* - Start writing a stream from the start.
+ * - libxl__stream_write_start_checkpoint()
+ * - Write the records which form a checkpoint into a stream.
*
* In normal operation, there are two tasks running at once; this
* stream processing, and the libxl-save-helper. check_all_finished()
@@ -39,6 +41,12 @@
* - Toolstack record
* - if (hvm), Qemu record
* - End record
+ *
+ * For checkpointed stream, there is a second loop which is triggered by a
+ * save-helper checkpoint callback. It writes:
+ * - Toolstack record
+ * - if (hvm), Qemu record
+ * - Checkpoint end record
*/
/* Success/error/cleanup handling. */
@@ -48,6 +56,9 @@ static void stream_complete(libxl__egc *egc,
libxl__stream_write_state *stream, int rc);
static void stream_done(libxl__egc *egc,
libxl__stream_write_state *stream);
+static void checkpoint_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc);
static void check_all_finished(libxl__egc *egc,
libxl__stream_write_state *stream, int rc);
@@ -72,6 +83,12 @@ static void emulator_record_done(libxl__egc *egc,
static void write_end_record(libxl__egc *egc,
libxl__stream_write_state *stream);
+/* Event chain unique to checkpointed streams. */
+static void write_checkpoint_end_record(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void checkpoint_end_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+
/*----- Helpers -----*/
static void write_done(libxl__egc *egc,
@@ -140,6 +157,7 @@ void libxl__stream_write_init(libxl__stream_write_state *stream)
{
stream->rc = 0;
stream->running = false;
+ stream->in_checkpoint = false;
FILLZERO(stream->dc);
stream->record_done_callback = NULL;
FILLZERO(stream->emu_dc);
@@ -184,6 +202,16 @@ void libxl__stream_write_start(libxl__egc *egc,
stream_complete(egc, stream, rc);
}
+void libxl__stream_write_start_checkpoint(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ assert(stream->running);
+ assert(!stream->in_checkpoint);
+ stream->in_checkpoint = true;
+
+ write_toolstack_record(egc, stream);
+}
+
void libxl__stream_write_abort(libxl__egc *egc,
libxl__stream_write_state *stream, int rc)
{
@@ -290,8 +318,12 @@ static void toolstack_record_done(libxl__egc *egc,
if (dss->type == LIBXL_DOMAIN_TYPE_HVM)
write_emulator_record(egc, stream);
- else
- write_end_record(egc, stream);
+ else {
+ if (stream->in_checkpoint)
+ write_checkpoint_end_record(egc, stream);
+ else
+ write_end_record(egc, stream);
+ }
}
static void write_emulator_record(libxl__egc *egc,
@@ -397,7 +429,10 @@ static void emulator_record_done(libxl__egc *egc,
free(stream->emu_body);
stream->emu_body = NULL;
- write_end_record(egc, stream);
+ if (stream->in_checkpoint)
+ write_checkpoint_end_record(egc, stream);
+ else
+ write_end_record(egc, stream);
}
static void write_end_record(libxl__egc *egc,
@@ -412,6 +447,24 @@ static void write_end_record(libxl__egc *egc,
&rec, NULL, stream_success);
}
+static void write_checkpoint_end_record(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ struct libxl__sr_rec_hdr rec;
+
+ FILLZERO(rec);
+ rec.type = REC_TYPE_CHECKPOINT_END;
+
+ setup_write(egc, stream, "checkpoint end record",
+ &rec, NULL, checkpoint_end_record_done);
+}
+
+static void checkpoint_end_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ checkpoint_done(egc, stream, 0);
+}
+
/*----- Success/error/cleanup handling. -----*/
static void stream_success(libxl__egc *egc, libxl__stream_write_state *stream)
@@ -424,6 +477,18 @@ static void stream_complete(libxl__egc *egc,
{
assert(stream->running);
+ if (stream->in_checkpoint) {
+ assert(rc);
+
+ /*
+ * If an error is encountered while in a checkpoint, pass it
+ * back to libxc. The failure will come back around to us via
+ * libxl__xc_domain_save_done()
+ */
+ checkpoint_done(egc, stream, rc);
+ return;
+ }
+
if (!stream->rc)
stream->rc = rc;
stream_done(egc, stream);
@@ -442,6 +507,16 @@ static void stream_done(libxl__egc *egc,
check_all_finished(egc, stream, stream->rc);
}
+static void checkpoint_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc)
+{
+ assert(stream->in_checkpoint);
+
+ stream->in_checkpoint = false;
+ stream->checkpoint_callback(egc, stream, rc);
+}
+
static void check_all_finished(libxl__egc *egc,
libxl__stream_write_state *stream,
int rc)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 26/29] tools/libx{c, l}: Introduce restore_callbacks.checkpoint()
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (24 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 25/29] tools/libxl: Write checkpoint records into the stream Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 27/29] tools/libxl: Handle checkpoint records in a libxl migration v2 stream Andrew Cooper
` (3 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel
Cc: Andrew Cooper, Yang Hongyang, Ian Jackson, Wei Liu, Wen Congyang
And call it when a checkpoint record is found in the libxc stream.
Some parts of this patch have been based on patches from the COLO
series.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v3: Named constants for the API
v2: Borrow sufficient fragments from several COLO patches to get
BROKEN_CHANNEL and checkpoint failover to function.
---
tools/libxc/include/xenguest.h | 7 +++++
tools/libxc/xc_sr_common.h | 7 +++--
tools/libxc/xc_sr_restore.c | 53 ++++++++++++++++++++++++++----------
tools/libxl/libxl_save_msgs_gen.pl | 2 +-
4 files changed, 51 insertions(+), 18 deletions(-)
diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index 7581263..e95af54 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -102,6 +102,13 @@ struct restore_callbacks {
int (*toolstack_restore)(uint32_t domid, const uint8_t *buf,
uint32_t size, void* data);
+ /* A checkpoint record has been found in the stream.
+ * returns: */
+#define XGR_CHECKPOINT_ERROR 0 /* Terminate processing */
+#define XGR_CHECKPOINT_SUCCESS 1 /* Continue reading more data from the stream */
+#define XGR_CHECKPOINT_FAILOVER 2 /* Failover and resume VM */
+ int (*checkpoint)(void* data);
+
/* to be provided as the last argument to each callback function */
void* data;
};
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 08c66db..1f4d4e4 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -130,10 +130,13 @@ struct xc_sr_restore_ops
* Process an individual record from the stream. The caller shall take
* care of processing common records (e.g. END, PAGE_DATA).
*
- * @return 0 for success, -1 for failure, or the sentinel value
- * RECORD_NOT_PROCESSED.
+ * @return 0 for success, -1 for failure, or the following sentinels:
+ * - RECORD_NOT_PROCESSED
+ * - BROKEN_CHANNEL: under Remus/COLO, this means master may be dead, and
+ * a failover is needed.
*/
#define RECORD_NOT_PROCESSED 1
+#define BROKEN_CHANNEL 2
int (*process_record)(struct xc_sr_context *ctx, struct xc_sr_record *rec);
/**
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index 9e27dba..18ba411 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -1,5 +1,7 @@
#include <arpa/inet.h>
+#include <assert.h>
+
#include "xc_sr_common.h"
/*
@@ -472,7 +474,7 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec)
static int handle_checkpoint(struct xc_sr_context *ctx)
{
xc_interface *xch = ctx->xch;
- int rc = 0;
+ int rc = 0, ret;
unsigned i;
if ( !ctx->restore.checkpointed )
@@ -482,6 +484,21 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
goto err;
}
+ ret = ctx->restore.callbacks->checkpoint(ctx->restore.callbacks->data);
+ switch ( ret )
+ {
+ case XGR_CHECKPOINT_SUCCESS:
+ break;
+
+ case XGR_CHECKPOINT_FAILOVER:
+ rc = BROKEN_CHANNEL;
+ goto err;
+
+ default: /* Other fatal error */
+ rc = -1;
+ goto err;
+ }
+
if ( ctx->restore.buffer_all_records )
{
IPRINTF("All records buffered");
@@ -560,19 +577,6 @@ static int process_record(struct xc_sr_context *ctx, struct xc_sr_record *rec)
free(rec->data);
rec->data = NULL;
- if ( rc == RECORD_NOT_PROCESSED )
- {
- if ( rec->type & REC_TYPE_OPTIONAL )
- DPRINTF("Ignoring optional record %#x (%s)",
- rec->type, rec_type_to_str(rec->type));
- else
- {
- ERROR("Mandatory record %#x (%s) not handled",
- rec->type, rec_type_to_str(rec->type));
- rc = -1;
- }
- }
-
return rc;
}
@@ -678,7 +682,22 @@ static int restore(struct xc_sr_context *ctx)
else
{
rc = process_record(ctx, &rec);
- if ( rc )
+ if ( rc == RECORD_NOT_PROCESSED )
+ {
+ if ( rec.type & REC_TYPE_OPTIONAL )
+ DPRINTF("Ignoring optional record %#x (%s)",
+ rec.type, rec_type_to_str(rec.type));
+ else
+ {
+ ERROR("Mandatory record %#x (%s) not handled",
+ rec.type, rec_type_to_str(rec.type));
+ rc = -1;
+ goto err;
+ }
+ }
+ else if ( rc == BROKEN_CHANNEL )
+ goto remus_failover;
+ else if ( rc )
goto err;
}
@@ -735,6 +754,10 @@ int xc_domain_restore2(xc_interface *xch, int io_fd, uint32_t dom,
ctx.restore.checkpointed = checkpointed_stream;
ctx.restore.callbacks = callbacks;
+ /* Sanity checks for callbacks. */
+ if ( checkpointed_stream )
+ assert(callbacks->checkpoint);
+
IPRINTF("In experimental %s", __func__);
DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
", checkpointed_stream %d", io_fd, dom, hvm, pae,
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index 6b4b65e..825d5cc 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -25,7 +25,7 @@ our @msgs = (
'unsigned long', 'total'] ],
[ 3, 'scxA', "suspend", [] ],
[ 4, 'scxA', "postcopy", [] ],
- [ 5, 'scxA', "checkpoint", [] ],
+ [ 5, 'srcxA', "checkpoint", [] ],
[ 6, 'scxA', "switch_qemu_logdirty", [qw(int domid
unsigned enable)] ],
# toolstack_save done entirely `by hand'
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 27/29] tools/libxl: Handle checkpoint records in a libxl migration v2 stream
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (25 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 26/29] tools/libx{c, l}: Introduce restore_callbacks.checkpoint() Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 28/29] tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc Andrew Cooper
` (2 subsequent siblings)
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
This is the final bit of untangling for Remus.
When libxc issues a checkpoint callback, start reading and buffering
all libxl records from the stream. Once a CHECKPOINT_END record is
encountered, start processing all records.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v3: Simplify, use named constants for API
---
tools/libxl/libxl_create.c | 25 ++++++++
tools/libxl/libxl_internal.h | 8 +++
tools/libxl/libxl_stream_read.c | 123 +++++++++++++++++++++++++++++++++++++++
3 files changed, 156 insertions(+)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index c51d64f..5b4d333 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -672,6 +672,27 @@ static int store_libxl_entry(libxl__gc *gc, uint32_t domid,
libxl_device_model_version_to_string(b_info->device_model_version));
}
+/*----- remus asynchronous checkpoint callback -----*/
+
+static void remus_checkpoint_stream_done(
+ libxl__egc *egc, libxl__stream_read_state *srs, int rc);
+
+static void libxl__remus_domain_checkpoint_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__domain_create_state *dcs = shs->caller_state;
+ libxl__egc *egc = shs->egc;
+ STATE_AO_GC(dcs->ao);
+
+ libxl__stream_read_start_checkpoint(egc, &dcs->srs);
+}
+
+static void remus_checkpoint_stream_done(
+ libxl__egc *egc, libxl__stream_read_state *stream, int rc)
+{
+ libxl__xc_domain_saverestore_async_callback_done(egc, &stream->shs, rc);
+}
+
/*----- main domain creation -----*/
/* We have a linear control flow; only one event callback is
@@ -939,6 +960,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
libxl_domain_config *const d_config = dcs->guest_config;
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
+ libxl__srm_restore_autogen_callbacks *const callbacks =
+ &dcs->srs.shs.callbacks.restore.a;
if (rc) {
domcreate_rebuild_done(egc, dcs, rc);
@@ -966,6 +989,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
}
/* Restore */
+ callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;
rc = libxl__build_pre(gc, domid, d_config, state);
if (rc)
@@ -976,6 +1000,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
dcs->srs.fd = restore_fd;
dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
dcs->srs.completion_callback = domcreate_stream_done;
+ dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
libxl__stream_read_start(egc, &dcs->srs);
return;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 9961a53..7aa4f16 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3309,9 +3309,13 @@ struct libxl__stream_read_state {
void (*completion_callback)(libxl__egc *egc,
libxl__stream_read_state *srs,
int rc);
+ void (*checkpoint_callback)(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int rc);
/* Private */
int rc;
bool running;
+ bool in_checkpoint;
libxl__save_helper_state shs;
libxl__conversion_helper_state chs;
@@ -3321,6 +3325,8 @@ struct libxl__stream_read_state {
LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue; /* NOGC */
enum {
SRS_PHASE_NORMAL,
+ SRS_PHASE_BUFFERING,
+ SRS_PHASE_UNBUFFERING,
} phase;
bool recursion_guard;
@@ -3335,6 +3341,8 @@ struct libxl__stream_read_state {
_hidden void libxl__stream_read_init(libxl__stream_read_state *stream);
_hidden void libxl__stream_read_start(libxl__egc *egc,
libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_start_checkpoint(libxl__egc *egc,
+ libxl__stream_read_state *stream);
_hidden void libxl__stream_read_abort(libxl__egc *egc,
libxl__stream_read_state *stream, int rc);
static inline bool
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index 3051a2b..2d17403 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -35,6 +35,8 @@
* Undefined undef undef undef undef undef
* Idle false undef false 0 0
* Active true NORMAL false 0/1 0/partial
+ * Active true BUFFERING true any 0/partial
+ * Active true UNBUFFERING true any 0
*
* While reading data from the stream, 'dc' is active and a callback
* is expected. Most actions in process_record() start a callback of
@@ -46,6 +48,15 @@
* Records are read one at time and immediately processed. (The
* record queue will not contain more than a single record.)
*
+ * PHASE_BUFFERING:
+ * This phase is used in checkpointed streams, when libxc signals
+ * the presence of a checkpoint in the stream. Records are read and
+ * buffered until a CHECKPOINT_END record has been read.
+ *
+ * PHASE_UNBUFFERING:
+ * Once a CHECKPOINT_END record has been read, all buffered records
+ * are processed.
+ *
* Note:
* Record buffers are not allocated from a GC; they are allocated
* and tracked manually. This is to avoid OOM with Remus where the
@@ -57,6 +68,9 @@
* - Initialises state. Must be called once before _start()
* - libxl__stream_read_start()
* - Starts reading records from the stream, and acting on them.
+ * - libxl__stream_read_start_checkpoint()
+ * - Starts buffering records at a checkpoint. Must be called on
+ * a running stream.
*
* There are several chains of event:
*
@@ -95,6 +109,8 @@
/* Success/error/cleanup handling. */
static void stream_complete(libxl__egc *egc,
libxl__stream_read_state *stream, int rc);
+static void checkpoint_done(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
static void stream_done(libxl__egc *egc,
libxl__stream_read_state *stream);
static void conversion_done(libxl__egc *egc,
@@ -159,6 +175,7 @@ void libxl__stream_read_init(libxl__stream_read_state *stream)
{
stream->rc = 0;
stream->running = false;
+ stream->in_checkpoint = false;
libxl__save_helper_init(&stream->shs);
libxl__conversion_helper_init(&stream->chs);
FILLZERO(stream->dc);
@@ -225,6 +242,22 @@ void libxl__stream_read_start(libxl__egc *egc,
stream_complete(egc, stream, rc);
}
+void libxl__stream_read_start_checkpoint(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ assert(stream->running);
+ assert(!stream->in_checkpoint);
+
+ stream->in_checkpoint = true;
+ stream->phase = SRS_PHASE_BUFFERING;
+
+ /*
+ * Libxc has handed control of the fd to us. Start reading some
+ * libxl records out of it.
+ */
+ stream_continue(egc, stream);
+}
+
void libxl__stream_read_abort(libxl__egc *egc,
libxl__stream_read_state *stream, int rc)
{
@@ -332,6 +365,54 @@ static void stream_continue(libxl__egc *egc,
}
break;
+ case SRS_PHASE_BUFFERING: {
+ /*
+ * Buffering phase (checkpointed streams only):
+ *
+ * logically:
+ * do { read_record(); } while ( not CHECKPOINT_END );
+ *
+ * Read and buffer all records from the stream until a
+ * CHECKPOINT_END record is encountered. We need to peek at
+ * the tail to spot the CHECKPOINT_END record, and switch to
+ * the unbuffering phase.
+ */
+ libxl__sr_record_buf *rec = LIBXL_STAILQ_LAST(
+ &stream->record_queue, libxl__sr_record_buf, entry);
+
+ assert(stream->in_checkpoint);
+
+ if (!rec || (rec->hdr.type != REC_TYPE_CHECKPOINT_END)) {
+ setup_read_record(egc, stream);
+ break;
+ }
+
+ /*
+ * There are now some number of buffered records, with a
+ * CHECKPOINT_END at the end. Start processing them all.
+ */
+ stream->phase = SRS_PHASE_UNBUFFERING;
+ }
+ /* FALLTHROUGH */
+ case SRS_PHASE_UNBUFFERING:
+ /*
+ * Unbuffering phase (checkpointed streams only):
+ *
+ * logically:
+ * do { process_record(); } while ( not CHECKPOINT_END );
+ *
+ * Process all records collected during the buffering phase.
+ */
+ assert(stream->in_checkpoint);
+
+ while (process_record(egc, stream))
+ ; /*
+ * Nothing! process_record() helpfully tells us if no specific
+ * futher actions have been set up, in which case we want to go
+ * ahead and process the next record.
+ */
+ break;
+
default:
abort();
}
@@ -467,6 +548,15 @@ static bool process_record(libxl__egc *egc,
write_emulator_blob(egc, stream, rec);
break;
+ case REC_TYPE_CHECKPOINT_END:
+ if (!stream->in_checkpoint) {
+ LOG(ERROR, "Unexpected CHECKPOINT_END record in stream");
+ rc = ERROR_FAIL;
+ goto err;
+ }
+ checkpoint_done(egc, stream, 0);
+ break;
+
default:
LOG(ERROR, "Unrecognised record 0x%08x", rec->hdr.type);
rc = ERROR_FAIL;
@@ -564,17 +654,50 @@ static void stream_complete(libxl__egc *egc,
{
assert(stream->running);
+ if (stream->in_checkpoint) {
+ assert(rc);
+
+ /*
+ * If an error is encountered while in a checkpoint, pass it
+ * back to libxc. The failure will come back around to us via
+ * libxl__xc_domain_restore_done()
+ */
+ checkpoint_done(egc, stream, rc);
+ return;
+ }
+
if (!stream->rc)
stream->rc = rc;
stream_done(egc, stream);
}
+static void checkpoint_done(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc)
+{
+ int ret;
+
+ assert(stream->in_checkpoint);
+
+ if (rc == 0)
+ ret = XGR_CHECKPOINT_SUCCESS;
+ else if (stream->phase == SRS_PHASE_BUFFERING)
+ ret = XGR_CHECKPOINT_FAILOVER;
+ else
+ ret = XGR_CHECKPOINT_ERROR;
+
+ stream->checkpoint_callback(egc, stream, ret);
+
+ stream->in_checkpoint = false;
+ stream->phase = SRS_PHASE_NORMAL;
+}
+
static void stream_done(libxl__egc *egc,
libxl__stream_read_state *stream)
{
libxl__sr_record_buf *rec, *trec;
assert(stream->running);
+ assert(!stream->in_checkpoint);
stream->running = false;
if (stream->incoming_record)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 28/29] tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (26 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 27/29] tools/libxl: Handle checkpoint records in a libxl migration v2 stream Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-14 10:59 ` [PATCH v4 29/29] tools/libxl: Drop all knowledge of toolstack callbacks Andrew Cooper
2015-07-15 10:21 ` [PATCH v4 00/27] Libxl migration v2 Wei Liu
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Libxl has now been fully adjusted not to need it.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/xc_sr_common.h | 5 --
tools/libxc/xc_sr_restore.c | 18 -----
tools/libxc/xc_sr_restore_x86_hvm.c | 124 -----------------------------------
tools/libxc/xc_sr_save_x86_hvm.c | 36 ----------
4 files changed, 183 deletions(-)
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 1f4d4e4..64f6082 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -309,11 +309,6 @@ struct xc_sr_context
/* HVM context blob. */
void *context;
size_t contextsz;
-
-/* #ifdef XG_LIBXL_HVM_COMPAT */
- uint32_t qlen;
- void *qbuf;
-/* #endif */
} restore;
};
} x86_hvm;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index 18ba411..bf1ee15 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -627,9 +627,6 @@ static void cleanup(struct xc_sr_context *ctx)
PERROR("Failed to clean up");
}
-#ifdef XG_LIBXL_HVM_COMPAT
-extern int read_qemu(struct xc_sr_context *ctx);
-#endif
/*
* Restore a domain.
*/
@@ -656,21 +653,6 @@ static int restore(struct xc_sr_context *ctx)
goto err;
}
-#ifdef XG_LIBXL_HVM_COMPAT
- if ( ctx->dominfo.hvm &&
- (rec.type == REC_TYPE_END || rec.type == REC_TYPE_CHECKPOINT) )
- {
- rc = read_qemu(ctx);
- if ( rc )
- {
- if ( ctx->restore.buffer_all_records )
- goto remus_failover;
- else
- goto err;
- }
- }
-#endif
-
if ( ctx->restore.buffer_all_records &&
rec.type != REC_TYPE_END &&
rec.type != REC_TYPE_CHECKPOINT )
diff --git a/tools/libxc/xc_sr_restore_x86_hvm.c b/tools/libxc/xc_sr_restore_x86_hvm.c
index 6f5af0e..49d22c7 100644
--- a/tools/libxc/xc_sr_restore_x86_hvm.c
+++ b/tools/libxc/xc_sr_restore_x86_hvm.c
@@ -3,24 +3,6 @@
#include "xc_sr_common_x86.h"
-#ifdef XG_LIBXL_HVM_COMPAT
-static int handle_toolstack(struct xc_sr_context *ctx, struct xc_sr_record *rec)
-{
- xc_interface *xch = ctx->xch;
- int rc;
-
- if ( !ctx->restore.callbacks || !ctx->restore.callbacks->toolstack_restore )
- return 0;
-
- rc = ctx->restore.callbacks->toolstack_restore(
- ctx->domid, rec->data, rec->length, ctx->restore.callbacks->data);
-
- if ( rc < 0 )
- PERROR("restoring toolstack");
- return rc;
-}
-#endif
-
/*
* Process an HVM_CONTEXT record from the stream.
*/
@@ -93,98 +75,6 @@ static int handle_hvm_params(struct xc_sr_context *ctx,
return 0;
}
-#ifdef XG_LIBXL_HVM_COMPAT
-int read_qemu(struct xc_sr_context *ctx);
-int read_qemu(struct xc_sr_context *ctx)
-{
- xc_interface *xch = ctx->xch;
- char qemusig[21];
- uint32_t qlen;
- void *qbuf = NULL;
- int rc = -1;
-
- if ( read_exact(ctx->fd, qemusig, sizeof(qemusig)) )
- {
- PERROR("Error reading QEMU signature");
- goto out;
- }
-
- if ( !memcmp(qemusig, "DeviceModelRecord0002", sizeof(qemusig)) )
- {
- if ( read_exact(ctx->fd, &qlen, sizeof(qlen)) )
- {
- PERROR("Error reading QEMU record length");
- goto out;
- }
-
- qbuf = malloc(qlen);
- if ( !qbuf )
- {
- PERROR("no memory for device model state");
- goto out;
- }
-
- if ( read_exact(ctx->fd, qbuf, qlen) )
- {
- PERROR("Error reading device model state");
- goto out;
- }
- }
- else
- {
- ERROR("Invalid device model state signature '%*.*s'",
- (int)sizeof(qemusig), (int)sizeof(qemusig), qemusig);
- goto out;
- }
-
- /* With Remus, this could be read many times */
- if ( ctx->x86_hvm.restore.qbuf )
- free(ctx->x86_hvm.restore.qbuf);
- ctx->x86_hvm.restore.qbuf = qbuf;
- ctx->x86_hvm.restore.qlen = qlen;
- rc = 0;
-
-out:
- if (rc)
- free(qbuf);
- return rc;
-}
-
-static int handle_qemu(struct xc_sr_context *ctx)
-{
- xc_interface *xch = ctx->xch;
- char path[256];
- uint32_t qlen = ctx->x86_hvm.restore.qlen;
- void *qbuf = ctx->x86_hvm.restore.qbuf;
- int rc = -1;
- FILE *fp = NULL;
-
- sprintf(path, XC_DEVICE_MODEL_RESTORE_FILE".%u", ctx->domid);
- fp = fopen(path, "wb");
- if ( !fp )
- {
- PERROR("Failed to open '%s' for writing", path);
- goto out;
- }
-
- DPRINTF("Writing %u bytes of QEMU data", qlen);
- if ( fwrite(qbuf, 1, qlen, fp) != qlen )
- {
- PERROR("Failed to write %u bytes of QEMU data", qlen);
- goto out;
- }
-
- rc = 0;
-
- out:
- if ( fp )
- fclose(fp);
- free(qbuf);
-
- return rc;
-}
-#endif
-
/* restore_ops function. */
static bool x86_hvm_pfn_is_valid(const struct xc_sr_context *ctx, xen_pfn_t pfn)
{
@@ -260,11 +150,6 @@ static int x86_hvm_process_record(struct xc_sr_context *ctx,
case REC_TYPE_HVM_PARAMS:
return handle_hvm_params(ctx, rec);
-#ifdef XG_LIBXL_HVM_COMPAT
- case REC_TYPE_TOOLSTACK:
- return handle_toolstack(ctx, rec);
-#endif
-
default:
return RECORD_NOT_PROCESSED;
}
@@ -314,15 +199,6 @@ static int x86_hvm_stream_complete(struct xc_sr_context *ctx)
return rc;
}
-#ifdef XG_LIBXL_HVM_COMPAT
- rc = handle_qemu(ctx);
- if ( rc )
- {
- ERROR("Failed to dump qemu");
- return rc;
- }
-#endif
-
return rc;
}
diff --git a/tools/libxc/xc_sr_save_x86_hvm.c b/tools/libxc/xc_sr_save_x86_hvm.c
index f4604db..cdee774 100644
--- a/tools/libxc/xc_sr_save_x86_hvm.c
+++ b/tools/libxc/xc_sr_save_x86_hvm.c
@@ -118,36 +118,6 @@ static int write_hvm_params(struct xc_sr_context *ctx)
return rc;
}
-#ifdef XG_LIBXL_HVM_COMPAT
-static int write_toolstack(struct xc_sr_context *ctx)
-{
- xc_interface *xch = ctx->xch;
- struct xc_sr_record rec = {
- .type = REC_TYPE_TOOLSTACK,
- .length = 0,
- };
- uint8_t *buf;
- uint32_t len;
- int rc;
-
- if ( !ctx->save.callbacks || !ctx->save.callbacks->toolstack_save )
- return 0;
-
- if ( ctx->save.callbacks->toolstack_save(
- ctx->domid, &buf, &len, ctx->save.callbacks->data) < 0 )
- {
- PERROR("Error calling toolstack_save");
- return -1;
- }
-
- rc = write_split_record(ctx, &rec, buf, len);
- if ( rc < 0 )
- PERROR("Error writing TOOLSTACK record");
- free(buf);
- return rc;
-}
-#endif
-
static xen_pfn_t x86_hvm_pfn_to_gfn(const struct xc_sr_context *ctx,
xen_pfn_t pfn)
{
@@ -199,12 +169,6 @@ static int x86_hvm_end_of_checkpoint(struct xc_sr_context *ctx)
if ( rc )
return rc;
-#ifdef XG_LIBXL_HVM_COMPAT
- rc = write_toolstack(ctx);
- if ( rc )
- return rc;
-#endif
-
/* Write the HVM_CONTEXT record. */
rc = write_hvm_context(ctx);
if ( rc )
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* [PATCH v4 29/29] tools/libxl: Drop all knowledge of toolstack callbacks
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (27 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 28/29] tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc Andrew Cooper
@ 2015-07-14 10:59 ` Andrew Cooper
2015-07-15 10:21 ` [PATCH v4 00/27] Libxl migration v2 Wei Liu
29 siblings, 0 replies; 39+ messages in thread
From: Andrew Cooper @ 2015-07-14 10:59 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Libxl has now been fully adjusted not to need them.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl_dom.c | 1 -
tools/libxl/libxl_internal.h | 2 --
tools/libxl/libxl_save_callout.c | 39 +-----------------------------------
tools/libxl/libxl_save_helper.c | 29 ---------------------------
tools/libxl/libxl_save_msgs_gen.pl | 7 ++-----
5 files changed, 3 insertions(+), 75 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 927a10e..81adb3d 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -2115,7 +2115,6 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
callbacks->suspend = libxl__domain_suspend_callback;
callbacks->switch_qemu_logdirty = libxl__domain_suspend_common_switch_qemu_logdirty;
- dss->sws.shs.callbacks.save.toolstack_save = libxl__toolstack_save;
dss->sws.ao = dss->ao;
dss->sws.dss = dss;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 7aa4f16..d9deaad 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2699,8 +2699,6 @@ _hidden void libxl__datacopier_prefixdata(libxl__egc*, libxl__datacopier_state*,
typedef struct libxl__srm_save_callbacks {
libxl__srm_save_autogen_callbacks a;
- int (*toolstack_save)(uint32_t domid, uint8_t **buf,
- uint32_t *len, void *data);
} libxl__srm_save_callbacks;
typedef struct libxl__srm_restore_callbacks {
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index a5e180c..f2ce868 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -80,41 +80,12 @@ void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss,
libxl__save_helper_state *shs)
{
STATE_AO_GC(dss->ao);
- int r, rc, toolstack_data_fd = -1;
- uint32_t toolstack_data_len = 0;
-
- /* Resources we need to free */
- uint8_t *toolstack_data_buf = 0;
unsigned cbflags =
libxl__srm_callout_enumcallbacks_save(&shs->callbacks.save.a);
- if (shs->callbacks.save.toolstack_save) {
- r = shs->callbacks.save.toolstack_save
- (dss->domid, &toolstack_data_buf, &toolstack_data_len, dss);
- if (r) { rc = ERROR_FAIL; goto out; }
-
- shs->toolstack_data_file = tmpfile();
- if (!shs->toolstack_data_file) {
- LOGE(ERROR, "cannot create toolstack data tmpfile");
- rc = ERROR_FAIL;
- goto out;
- }
- toolstack_data_fd = fileno(shs->toolstack_data_file);
-
- r = libxl_write_exactly(CTX, toolstack_data_fd,
- toolstack_data_buf, toolstack_data_len,
- "toolstack data tmpfile", 0);
- if (r) { rc = ERROR_FAIL; goto out; }
-
- /* file position must be reset before passing to libxl-save-helper. */
- r = lseek(toolstack_data_fd, 0, SEEK_SET);
- if (r) { rc = ERROR_FAIL; goto out; }
- }
-
const unsigned long argnums[] = {
dss->domid, 0, 0, dss->xcflags, dss->hvm,
- toolstack_data_fd, toolstack_data_len,
cbflags,
};
@@ -125,18 +96,10 @@ void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss,
shs->caller_state = dss;
shs->need_results = 0;
- free(toolstack_data_buf);
-
run_helper(egc, shs, "--save-domain", dss->fd,
- &toolstack_data_fd, 1,
+ NULL, 0,
argnums, ARRAY_SIZE(argnums));
return;
-
- out:
- free(toolstack_data_buf);
- if (shs->toolstack_data_file) fclose(shs->toolstack_data_file);
-
- libxl__xc_domain_save_done(egc, dss, rc, 0, 0);
}
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index f196786..1622bb7 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -213,32 +213,8 @@ int helper_getreply(void *user)
/*----- other callbacks -----*/
-static int toolstack_save_fd;
-static uint32_t toolstack_save_len;
static struct save_callbacks helper_save_callbacks;
-static int toolstack_save_cb(uint32_t domid, uint8_t **buf,
- uint32_t *len, void *data)
-{
- int r;
-
- assert(toolstack_save_fd > 0);
-
- /* This is a hack for remus */
- if (helper_save_callbacks.checkpoint) {
- r = lseek(toolstack_save_fd, 0, SEEK_SET);
- if (r) fail(errno,"rewind toolstack data tmpfile");
- }
-
- *buf = xmalloc(toolstack_save_len);
- r = read_exactly(toolstack_save_fd, *buf, toolstack_save_len);
- if (r<0) fail(errno,"read toolstack data");
- if (r==0) fail(0,"read toolstack data eof");
-
- *len = toolstack_save_len;
- return 0;
-}
-
static void startup(const char *op) {
xtl_log(&logger,XTL_DEBUG,0,program,"starting %s",op);
@@ -273,14 +249,9 @@ int main(int argc, char **argv)
uint32_t max_factor = strtoul(NEXTARG,0,10);
uint32_t flags = strtoul(NEXTARG,0,10);
int hvm = atoi(NEXTARG);
- toolstack_save_fd = atoi(NEXTARG);
- toolstack_save_len = strtoul(NEXTARG,0,10);
unsigned cbflags = strtoul(NEXTARG,0,10);
assert(!*++argv);
- if (toolstack_save_fd >= 0)
- helper_save_callbacks.toolstack_save = toolstack_save_cb;
-
helper_setcallbacks_save(&helper_save_callbacks, cbflags);
startup("save");
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index 825d5cc..d6d2967 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -28,12 +28,9 @@ our @msgs = (
[ 5, 'srcxA', "checkpoint", [] ],
[ 6, 'scxA', "switch_qemu_logdirty", [qw(int domid
unsigned enable)] ],
- # toolstack_save done entirely `by hand'
- [ 7, 'rcxW', "toolstack_restore", [qw(uint32_t domid
- BLOCK tsdata)] ],
- [ 8, 'r', "restore_results", ['unsigned long', 'store_mfn',
+ [ 7, 'r', "restore_results", ['unsigned long', 'store_mfn',
'unsigned long', 'console_mfn'] ],
- [ 9, 'srW', "complete", [qw(int retval
+ [ 8, 'srW', "complete", [qw(int retval
int errnoval)] ],
);
--
1.7.10.4
^ permalink raw reply related [flat|nested] 39+ messages in thread* Re: [PATCH v4 00/27] Libxl migration v2
2015-07-14 10:59 [PATCH v4 00/27] Libxl migration v2 Andrew Cooper
` (28 preceding siblings ...)
2015-07-14 10:59 ` [PATCH v4 29/29] tools/libxl: Drop all knowledge of toolstack callbacks Andrew Cooper
@ 2015-07-15 10:21 ` Wei Liu
2015-07-15 10:25 ` Ian Jackson
29 siblings, 1 reply; 39+ messages in thread
From: Wei Liu @ 2015-07-15 10:21 UTC (permalink / raw)
To: Andrew Cooper
Cc: Wei Liu, Ian Campbell, Wen Congyang, Ian Jackson, Xen-devel,
Yang Hongyang
On Tue, Jul 14, 2015 at 11:59:15AM +0100, Andrew Cooper wrote:
> This series adds support for the libxl migration v2 stream, and untangles the
> existing layering violations of the toolstack and qemu records.
>
> It can be found on the branch "libxl-migv2-v4"
> git://xenbits.xen.org/people/andrewcoop/xen.git
> http://xenbits.xen.org/git-http/people/andrewcoop/xen.git
>
> No major changes over v3, but a lot of minor changes following review.
>
> At the end of the series, legacy migration is no longer used.
>
> The Remus code is untested by me. All other combinations of
> suspend/migrate/resume have been tested with PV and HVM guests (qemu-trad and
> qemu-upstream), including automatic legacy -> v2 inline conversion.
>
> Anyway, thoughts/comments welcome. Please test!
>
> ~Andrew
>
> Summary of Acks/Modified/New from v2
>
> A bsd-sys-queue-h-seddery: Massage `offsetof'
> A tools/libxc: Always compile the compat qemu variables into xc_sr_context
> A tools/libxl: Introduce ROUNDUP()
> A tools/libxl: Introduce libxl__kill()
> A tools/libxl: Stash all restore parameters in domain_create_state
> A tools/libxl: Split libxl__domain_create_state.restore_fd in two
> AM tools/libxl: Extra management APIs for the save helper
> A tools/libxl: Add save_helper_state pointers to libxl__xc_domain_{save,restore}()
> N tools/libxl: Fix libxl__carefd_opened() to be more useful with an invalid fd
> A tools/xl: Mandatory flag indicating the format of the migration stream
> A docs: Libxl migration v2 stream specification
> A tools/python: Libxc migration v2 infrastructure
> A tools/python: Libxl migration v2 infrastructure
> A tools/python: Other migration infrastructure
> A tools/python: Verification utility for v2 stream spec compliance
> A tools/python: Conversion utility for legacy migration streams
> A tools/libxl: Migration v2 stream format
> M tools/libxl: Infrastructure for reading a libxl migration v2 stream
> M tools/libxl: Infrastructure to convert a legacy stream
> M tools/libxl: Convert a legacy stream if needed
> AM tools/libxc+libxl+xl: Restore v2 streams
> M tools/libxl: Infrastructure for writing a v2 stream
> AM tools/libxc+libxl+xl: Save v2 streams
> A docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams
> AM tools/libxl: Write checkpoint records into the stream
> A tools/libx{c,l}: Introduce restore_callbacks.checkpoint()
> AM tools/libxl: Handle checkpoint records in a libxl migration v2 stream
> A tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc
> A tools/libxl: Drop all knowledge of toolstack callbacks
This series:
Release-acked-by: Wei Liu <wei.liu2@citrix.com>
^ permalink raw reply [flat|nested] 39+ messages in thread