* [PATCH v2 00/27] Libxl migration v2
@ 2015-07-09 18:26 Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 01/27] bsd-sys-queue-h-seddery: Massage `offsetof' Andrew Cooper
` (27 more replies)
0 siblings, 28 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel
Cc: Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper, Ian Jackson,
Yang Hongyang
This series adds support for the libxl migration v2 stream, and untangles the
existing layering violations of the toolstack and qemu records.
It can be found on the branch "libxl-migv2-v2"
git://xenbits.xen.org/people/andrewcoop/xen.git
http://xenbits.xen.org/git-http/people/andrewcoop/xen.git
Major changes in v2 are being rebased over the libxl AO-abort series, and a
redesign of the internal logic to support Remus/COLO buffering and failover.
At the end of the series, legacy migration is no longer used.
The Remus code is untested by me. All other combinations of
suspend/migrate/resume have been tested with PV and HVM guests (qemu-trad and
qemu-upstream), including 32 -> 64 bit migration (which was the underlying bug
causing us to write migration v2 in the first place).
Anyway, thoughts/comments welcome. Please test!
~Andrew
Summary of Acks/Modified/New from v1
N bsd-sys-queue-h-seddery: Massage `offsetof'
A tools/libxc: Always compile the compat qemu variables into xc_sr_context
A tools/libxl: Introduce ROUNDUP()
N tools/libxl: Introduce libxl__kill()
AM tools/libxl: Stash all restore parameters in domain_create_state
N tools/libxl: Split libxl__domain_create_state.restore_fd in two
M tools/libxl: Extra management APIs for the save helper
AM tools/xl: Mandatory flag indicating the format of the migration stream
docs: Libxl migration v2 stream specification
AM tools/python: Libxc migration v2 infrastructure
AM tools/python: Libxl migration v2 infrastructure
N tools/python: Other migration infrastructure
AM tools/python: Verification utility for v2 stream spec compliance
AM tools/python: Conversion utility for legacy migration streams
M tools/libxl: Migration v2 stream format
M tools/libxl: Infrastructure for reading a libxl migration v2 stream
M tools/libxl: Support converting a legacy stream to a v2 stream
M tools/libxl: Convert a legacy stream if needed
M tools/libxc+libxl+xl: Restore v2 streams
M tools/libxl: Infrastructure for writing a v2 stream
M tools/libxc+libxl+xl: Save v2 streams
AM docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams
M tools/libxl: Write checkpoint records into the stream
M tools/libx{c,l}: Introduce restore_callbacks.checkpoint()
M tools/libxl: Handle checkpoint records in a libxl migration v2 stream
A tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc
A tools/libxl: Drop all knowledge of toolstack callbacks
Andrew Cooper (23):
tools/libxc: Always compile the compat qemu variables into xc_sr_context
tools/libxl: Introduce ROUNDUP()
tools/libxl: Introduce libxl__kill()
tools/libxl: Stash all restore parameters in domain_create_state
tools/libxl: Split libxl__domain_create_state.restore_fd in two
tools/libxl: Extra management APIs for the save helper
tools/xl: Mandatory flag indicating the format of the migration stream
docs: Libxl migration v2 stream specification
tools/python: Libxc migration v2 infrastructure
tools/python: Libxl migration v2 infrastructure
tools/python: Other migration infrastructure
tools/python: Verification utility for v2 stream spec compliance
tools/python: Conversion utility for legacy migration streams
tools/libxl: Support converting a legacy stream to a v2 stream
tools/libxl: Convert a legacy stream if needed
tools/libxc+libxl+xl: Restore v2 streams
tools/libxc+libxl+xl: Save v2 streams
docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams
tools/libxl: Write checkpoint records into the stream
tools/libx{c,l}: Introduce restore_callbacks.checkpoint()
tools/libxl: Handle checkpoint records in a libxl migration v2 stream
tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc
tools/libxl: Drop all knowledge of toolstack callbacks
Ian Jackson (1):
bsd-sys-queue-h-seddery: Massage `offsetof'
Ross Lagerwall (3):
tools/libxl: Migration v2 stream format
tools/libxl: Infrastructure for reading a libxl migration v2 stream
tools/libxl: Infrastructure for writing a v2 stream
docs/specs/libxl-migration-stream.pandoc | 217 ++++++
tools/include/xen-external/bsd-sys-queue-h-seddery | 2 +
tools/libxc/Makefile | 2 -
tools/libxc/include/xenguest.h | 9 +
tools/libxc/xc_sr_common.h | 12 +-
tools/libxc/xc_sr_restore.c | 71 +-
tools/libxc/xc_sr_restore_x86_hvm.c | 124 ----
tools/libxc/xc_sr_save_x86_hvm.c | 36 -
tools/libxl/Makefile | 2 +
tools/libxl/libxl.h | 19 +
tools/libxl/libxl_aoutils.c | 15 +
tools/libxl/libxl_convert_callout.c | 172 +++++
tools/libxl/libxl_create.c | 86 ++-
tools/libxl/libxl_dom.c | 65 +-
tools/libxl/libxl_internal.h | 192 ++++-
tools/libxl/libxl_save_callout.c | 70 +-
tools/libxl/libxl_save_helper.c | 33 +-
tools/libxl/libxl_save_msgs_gen.pl | 9 +-
tools/libxl/libxl_sr_stream_format.h | 58 ++
tools/libxl/libxl_stream_read.c | 731 ++++++++++++++++++++
tools/libxl/libxl_stream_write.c | 554 +++++++++++++++
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl_cmdimpl.c | 9 +-
tools/python/Makefile | 4 +
tools/python/scripts/convert-legacy-stream | 678 ++++++++++++++++++
tools/python/scripts/verify-stream-v2 | 174 +++++
tools/python/setup.py | 1 +
tools/python/xen/migration/legacy.py | 279 ++++++++
tools/python/xen/migration/libxc.py | 446 ++++++++++++
tools/python/xen/migration/libxl.py | 199 ++++++
tools/python/xen/migration/public.py | 21 +
tools/python/xen/migration/tests.py | 54 ++
tools/python/xen/migration/verify.py | 37 +
tools/python/xen/migration/xl.py | 12 +
34 files changed, 4014 insertions(+), 380 deletions(-)
create mode 100644 docs/specs/libxl-migration-stream.pandoc
create mode 100644 tools/libxl/libxl_convert_callout.c
create mode 100644 tools/libxl/libxl_sr_stream_format.h
create mode 100644 tools/libxl/libxl_stream_read.c
create mode 100644 tools/libxl/libxl_stream_write.c
create mode 100755 tools/python/scripts/convert-legacy-stream
create mode 100755 tools/python/scripts/verify-stream-v2
create mode 100644 tools/python/xen/migration/__init__.py
create mode 100644 tools/python/xen/migration/legacy.py
create mode 100644 tools/python/xen/migration/libxc.py
create mode 100644 tools/python/xen/migration/libxl.py
create mode 100644 tools/python/xen/migration/public.py
create mode 100644 tools/python/xen/migration/tests.py
create mode 100644 tools/python/xen/migration/verify.py
create mode 100644 tools/python/xen/migration/xl.py
--
1.7.10.4
^ permalink raw reply [flat|nested] 65+ messages in thread
* [PATCH v2 01/27] bsd-sys-queue-h-seddery: Massage `offsetof'
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 9:32 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 02/27] tools/libxc: Always compile the compat qemu variables into xc_sr_context Andrew Cooper
` (26 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Ian Jackson
From: Ian Jackson <Ian.Jackson@eu.citrix.com>
For some reason BSD's queue.h uses `__offsetof'. It expects it to
work just like offsetof. So use offsetof.
Reported-by: Andrew Cooper <andrew.cooper3@citrix.com>
Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
---
tools/include/xen-external/bsd-sys-queue-h-seddery | 2 ++
1 file changed, 2 insertions(+)
diff --git a/tools/include/xen-external/bsd-sys-queue-h-seddery b/tools/include/xen-external/bsd-sys-queue-h-seddery
index 7a957e3..3f8716d 100755
--- a/tools/include/xen-external/bsd-sys-queue-h-seddery
+++ b/tools/include/xen-external/bsd-sys-queue-h-seddery
@@ -69,4 +69,6 @@ s/\b struct \s+ type \b/type/xg;
s,^\#include.*sys/cdefs.*,/* $& */,xg;
+s,\b __offsetof \b ,offsetof,xg;
+
s/\b( NULL )/0/xg;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 02/27] tools/libxc: Always compile the compat qemu variables into xc_sr_context
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 01/27] bsd-sys-queue-h-seddery: Massage `offsetof' Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 03/27] tools/libxl: Introduce ROUNDUP() Andrew Cooper
` (25 subsequent siblings)
27 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
This is safe (as the variables will simply be unused), and is required for
correct compilation when midway through untangling the libxc/libxl
interaction.
The #define is left in place to highlight that the variables can be removed
once the untangling is complete.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/xc_sr_common.h | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 565c5da..08c66db 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -307,10 +307,10 @@ struct xc_sr_context
void *context;
size_t contextsz;
-#ifdef XG_LIBXL_HVM_COMPAT
+/* #ifdef XG_LIBXL_HVM_COMPAT */
uint32_t qlen;
void *qbuf;
-#endif
+/* #endif */
} restore;
};
} x86_hvm;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 03/27] tools/libxl: Introduce ROUNDUP()
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 01/27] bsd-sys-queue-h-seddery: Massage `offsetof' Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 02/27] tools/libxc: Always compile the compat qemu variables into xc_sr_context Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 04/27] tools/libxl: Introduce libxl__kill() Andrew Cooper
` (24 subsequent siblings)
27 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
This is the same as is used by libxc.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl_internal.h | 3 +++
1 file changed, 3 insertions(+)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 5235d25..19fc425 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -110,6 +110,9 @@
#define ARRAY_SIZE(a) (sizeof(a) / sizeof(a[0]))
+#define ROUNDUP(_val, _order) \
+ (((unsigned long)(_val)+(1UL<<(_order))-1) & ~((1UL<<(_order))-1))
+
#define min(X, Y) ({ \
const typeof (X) _x = (X); \
const typeof (Y) _y = (Y); \
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 04/27] tools/libxl: Introduce libxl__kill()
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (2 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 03/27] tools/libxl: Introduce ROUNDUP() Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 1:34 ` Yang Hongyang
2015-07-10 9:08 ` Wei Liu
2015-07-09 18:26 ` [PATCH v2 05/27] tools/libxl: Stash all restore parameters in domain_create_state Andrew Cooper
` (23 subsequent siblings)
27 siblings, 2 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
as a wrapper to kill(2), and use it in preference to sendig in
libxl_save_callout.c.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
Logically new in v2 - split out from a v1 change which was itself a
cherrypick-and-modify from the AO Abort series
---
tools/libxl/libxl_aoutils.c | 15 +++++++++++++++
tools/libxl/libxl_internal.h | 2 ++
tools/libxl/libxl_save_callout.c | 10 ++--------
3 files changed, 19 insertions(+), 8 deletions(-)
diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
index 0931eee..274ef39 100644
--- a/tools/libxl/libxl_aoutils.c
+++ b/tools/libxl/libxl_aoutils.c
@@ -621,3 +621,18 @@ bool libxl__async_exec_inuse(const libxl__async_exec_state *aes)
assert(time_inuse == child_inuse);
return child_inuse;
}
+
+void libxl__kill(libxl__gc *gc, pid_t pid, int sig, const char *what)
+{
+ int r = kill(pid, sig);
+ if (r) LOGE(WARN, "failed to kill() %s [%lu] (signal %d)",
+ what, (unsigned long)pid, sig);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 19fc425..9147de1 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2244,6 +2244,8 @@ struct libxl__async_exec_state {
int libxl__async_exec_start(libxl__async_exec_state *aes);
bool libxl__async_exec_inuse(const libxl__async_exec_state *aes);
+void libxl__kill(libxl__gc *gc, pid_t pid, int sig, const char *what);
+
/*----- device addition/removal -----*/
typedef struct libxl__ao_device libxl__ao_device;
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 087c2d5..b82a5c1 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -244,12 +244,6 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
libxl__carefd_close(childs_pipes[1]);
helper_failed(egc, shs, rc);;
}
-static void sendsig(libxl__gc *gc, libxl__save_helper_state *shs, int sig)
-{
- int r = kill(shs->child.pid, sig);
- if (r) LOGE(WARN, "failed to kill save/restore helper [%lu] (signal %d)",
- (unsigned long)shs->child.pid, sig);
-}
static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs,
int rc)
@@ -266,7 +260,7 @@ static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs,
return;
}
- sendsig(gc, shs, SIGKILL);
+ libxl__kill(gc, shs->child.pid, SIGKILL, "save/restore helper");
}
static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
@@ -282,7 +276,7 @@ static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
if (!shs->rc)
shs->rc = rc;
- sendsig(gc, shs, SIGTERM);
+ libxl__kill(gc, shs->child.pid, SIGTERM, "save/restore helper");
}
static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 05/27] tools/libxl: Stash all restore parameters in domain_create_state
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (3 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 04/27] tools/libxl: Introduce libxl__kill() Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 06/27] tools/libxl: Split libxl__domain_create_state.restore_fd in two Andrew Cooper
` (22 subsequent siblings)
27 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Shortly more parameters will appear, and this saves unboxing each one.
libxl_domain_restore_params is mandatory for restore streams, and ignored for
plain creation. The old 'checkpointed_stream' was incorrectly identified as a
private parameter when it was infact public.
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
Reviewed-by: Yang Hongyang <yanghy@cn.fujitsu.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
Since v1:
* Gate validity on restore_fd being valid.
---
tools/libxl/libxl_create.c | 13 +++++++------
tools/libxl/libxl_internal.h | 2 +-
tools/libxl/libxl_save_callout.c | 2 +-
3 files changed, 9 insertions(+), 8 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index b785ddd..61515da 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1512,8 +1512,8 @@ static void domain_create_cb(libxl__egc *egc,
int rc, uint32_t domid);
static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
- uint32_t *domid,
- int restore_fd, int checkpointed_stream,
+ uint32_t *domid, int restore_fd,
+ const libxl_domain_restore_params *params,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
@@ -1526,8 +1526,9 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
libxl_domain_config_init(&cdcs->dcs.guest_config_saved);
libxl_domain_config_copy(ctx, &cdcs->dcs.guest_config_saved, d_config);
cdcs->dcs.restore_fd = restore_fd;
+ if (restore_fd > -1)
+ cdcs->dcs.restore_params = *params;
cdcs->dcs.callback = domain_create_cb;
- cdcs->dcs.checkpointed_stream = checkpointed_stream;
libxl__ao_progress_gethow(&cdcs->dcs.aop_console_how, aop_console_how);
cdcs->domid_out = domid;
@@ -1553,7 +1554,7 @@ int libxl_domain_create_new(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
- return do_domain_create(ctx, d_config, domid, -1, 0,
+ return do_domain_create(ctx, d_config, domid, -1, NULL,
ao_how, aop_console_how);
}
@@ -1563,8 +1564,8 @@ int libxl_domain_create_restore(libxl_ctx *ctx, libxl_domain_config *d_config,
const libxl_asyncop_how *ao_how,
const libxl_asyncprogress_how *aop_console_how)
{
- return do_domain_create(ctx, d_config, domid, restore_fd,
- params->checkpointed_stream, ao_how, aop_console_how);
+ return do_domain_create(ctx, d_config, domid, restore_fd, params,
+ ao_how, aop_console_how);
}
/*
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 9147de1..5ab945a 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3217,11 +3217,11 @@ struct libxl__domain_create_state {
libxl_domain_config *guest_config;
libxl_domain_config guest_config_saved; /* vanilla config */
int restore_fd;
+ libxl_domain_restore_params restore_params;
libxl__domain_create_cb *callback;
libxl_asyncprogress_how aop_console_how;
/* private to domain_create */
int guest_domid;
- int checkpointed_stream;
libxl__domain_build_state build_state;
libxl__bootloader_state bl;
libxl__stub_dm_spawn_state dmss;
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index b82a5c1..80aae1b 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -60,7 +60,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
state->store_domid, state->console_port,
state->console_domid,
hvm, pae, superpages,
- cbflags, dcs->checkpointed_stream,
+ cbflags, dcs->restore_params.checkpointed_stream,
};
dcs->shs.ao = ao;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 06/27] tools/libxl: Split libxl__domain_create_state.restore_fd in two
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (4 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 05/27] tools/libxl: Stash all restore parameters in domain_create_state Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 9:37 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 07/27] tools/libxl: Extra management APIs for the save helper Andrew Cooper
` (21 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
In a future patch, we shall support automatically converting a legacy stream
to a v2 stream, in which case libxc needs to read from a different fd.
Simply overwriting restore_fd does not work; the two fd's have different
circumstances. The restore_fd needs to be returned to its origial state
before libxl_domain_create_restore() returns, while in the converted case, the
fd needs allocating and deallocating appropriately.
No functional change.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
New in v2
---
tools/libxl/libxl_create.c | 2 +-
tools/libxl/libxl_internal.h | 2 +-
tools/libxl/libxl_save_callout.c | 2 +-
3 files changed, 3 insertions(+), 3 deletions(-)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 61515da..be13204 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -1525,7 +1525,7 @@ static int do_domain_create(libxl_ctx *ctx, libxl_domain_config *d_config,
cdcs->dcs.guest_config = d_config;
libxl_domain_config_init(&cdcs->dcs.guest_config_saved);
libxl_domain_config_copy(ctx, &cdcs->dcs.guest_config_saved, d_config);
- cdcs->dcs.restore_fd = restore_fd;
+ cdcs->dcs.restore_fd = cdcs->dcs.libxc_fd = restore_fd;
if (restore_fd > -1)
cdcs->dcs.restore_params = *params;
cdcs->dcs.callback = domain_create_cb;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 5ab945a..588cfb8 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3216,7 +3216,7 @@ struct libxl__domain_create_state {
libxl__ao *ao;
libxl_domain_config *guest_config;
libxl_domain_config guest_config_saved; /* vanilla config */
- int restore_fd;
+ int restore_fd, libxc_fd;
libxl_domain_restore_params restore_params;
libxl__domain_create_cb *callback;
libxl_asyncprogress_how aop_console_how;
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 80aae1b..1136b79 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -48,7 +48,7 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
/* Convenience aliases */
const uint32_t domid = dcs->guest_domid;
- const int restore_fd = dcs->restore_fd;
+ const int restore_fd = dcs->libxc_fd;
libxl__domain_build_state *const state = &dcs->build_state;
unsigned cbflags = libxl__srm_callout_enumcallbacks_restore
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 07/27] tools/libxl: Extra management APIs for the save helper
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (5 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 06/27] tools/libxl: Split libxl__domain_create_state.restore_fd in two Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 9:41 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 08/27] tools/xl: Mandatory flag indicating the format of the migration stream Andrew Cooper
` (20 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
With migration v2, there are several moving parts needing to be juggled at
once. This requires the error handling logic to be able to query the state of
each moving part, possibly before they have been started, and be able to
cancel them.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
Since v1:
* Add an _init() function which allows _inuse() to be safe to call even
before the save helper has started.
---
tools/libxl/libxl_internal.h | 9 +++++++++
tools/libxl/libxl_save_callout.c | 17 ++++++++++++++---
2 files changed, 23 insertions(+), 3 deletions(-)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 588cfb8..4bd6ea1 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3272,6 +3272,15 @@ _hidden void libxl__xc_domain_restore(libxl__egc *egc,
_hidden void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
int rc, int retval, int errnoval);
+_hidden void libxl__save_helper_init(libxl__save_helper_state *shs);
+_hidden void libxl__save_helper_abort(libxl__egc *egc,
+ libxl__save_helper_state *shs);
+
+static inline bool libxl__save_helper_inuse(const libxl__save_helper_state *shs)
+{
+ return libxl__ev_child_inuse(&shs->child);
+}
+
/* Each time the dm needs to be saved, we must call suspend and then save */
_hidden int libxl__domain_suspend_device_model(libxl__gc *gc,
libxl__domain_suspend_state *dss);
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index 1136b79..cd18cd2 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -146,6 +146,13 @@ void libxl__xc_domain_saverestore_async_callback_done(libxl__egc *egc,
shs->egc = 0;
}
+void libxl__save_helper_init(libxl__save_helper_state *shs)
+{
+ libxl__ao_abortable_init(&shs->abrt);
+ libxl__ev_fd_init(&shs->readable);
+ libxl__ev_child_init(&shs->child);
+}
+
/*----- helper execution -----*/
static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
@@ -167,9 +174,7 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
shs->rc = 0;
shs->completed = 0;
shs->pipes[0] = shs->pipes[1] = 0;
- libxl__ao_abortable_init(&shs->abrt);
- libxl__ev_fd_init(&shs->readable);
- libxl__ev_child_init(&shs->child);
+ libxl__save_helper_init(shs);
shs->abrt.ao = shs->ao;
shs->abrt.callback = helper_stop;
@@ -279,6 +284,12 @@ static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
libxl__kill(gc, shs->child.pid, SIGTERM, "save/restore helper");
}
+void libxl__save_helper_abort(libxl__egc *egc,
+ libxl__save_helper_state *shs)
+{
+ helper_stop(egc, &shs->abrt, ERROR_FAIL);
+}
+
static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
int fd, short events, short revents)
{
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 08/27] tools/xl: Mandatory flag indicating the format of the migration stream
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (6 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 07/27] tools/libxl: Extra management APIs for the save helper Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 09/27] docs: Libxl migration v2 stream specification Andrew Cooper
` (19 subsequent siblings)
27 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Introduced at this point so the python stream conversion code has a concrete
ABI to use. Later when libxl itself starts supporting a v2 stream, it will be
added to XL_MANDATORY_FLAG_ALL.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v2: Expand commit message
---
tools/libxl/xl_cmdimpl.c | 1 +
1 file changed, 1 insertion(+)
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 971209c..26b1e7d 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -109,6 +109,7 @@
*/
#define XL_MANDATORY_FLAG_JSON (1U << 0) /* config data is in JSON format */
+#define XL_MANDATORY_FLAG_STREAMv2 (1U << 1) /* stream is v2 */
#define XL_MANDATORY_FLAG_ALL (XL_MANDATORY_FLAG_JSON)
struct save_file_header {
char magic[32]; /* savefileheader_magic */
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 09/27] docs: Libxl migration v2 stream specification
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (7 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 08/27] tools/xl: Mandatory flag indicating the format of the migration stream Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 9:46 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 10/27] tools/python: Libxc migration v2 infrastructure Andrew Cooper
` (18 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
docs/specs/libxl-migration-stream.pandoc | 205 ++++++++++++++++++++++++++++++
1 file changed, 205 insertions(+)
create mode 100644 docs/specs/libxl-migration-stream.pandoc
diff --git a/docs/specs/libxl-migration-stream.pandoc b/docs/specs/libxl-migration-stream.pandoc
new file mode 100644
index 0000000..7235317
--- /dev/null
+++ b/docs/specs/libxl-migration-stream.pandoc
@@ -0,0 +1,205 @@
+% LibXenLight Domain Image Format
+% Andrew Cooper <<andrew.cooper3@citrix.com>>
+% Draft B
+
+Introduction
+============
+
+For the purposes of this document, `xl` is used as a representation of any
+implementer of the `libxl` API. `xl` should be considered completely
+interchangeable with alternates, such as `libvirt` or `xenopsd-xl`.
+
+Purpose
+-------
+
+The _domain image format_ is the context of a running domain used for
+snapshots of a domain or for transferring domains between hosts during
+migration.
+
+There are a number of problems with the domain image format used in Xen 4.5
+and earlier (the _legacy format_)
+
+* There is no `libxl` context information. `xl` is required to send certain
+ pieces of `libxl` context itself.
+
+* The contents of the stream is passed directly through `libxl` to `libxc`.
+ The legacy `libxc` format contained some information which belonged at the
+ `libxl` level, resulting in awkward layer violation to return the
+ information back to `libxl`.
+
+* The legacy `libxc` format was inextensible, causing inextensibility in the
+ legacy `libxl` handling.
+
+This design addresses the above points, allowing for a completely
+self-contained, extensible stream with each layer responsibile for its own
+appropriate information.
+
+
+Not Yet Included
+----------------
+
+The following features are not yet fully specified and will be
+included in a future draft.
+
+* Remus
+
+* ARM
+
+
+Overview
+========
+
+The image format consists of a _Header_, followed by 1 or more _Records_.
+Each record consists of a type and length field, followed by any type-specific
+data.
+
+\clearpage
+
+Header
+======
+
+The header identifies the stream as a `libxl` stream, including the version of
+this specification that it complies with.
+
+All fields in this header shall be in _big-endian_ byte order, regardless of
+the setting of the endianness bit.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+ | ident |
+ +-----------------------+-------------------------+
+ | version | options |
+ +-----------------------+-------------------------+
+
+--------------------------------------------------------------------
+Field Description
+----------- --------------------------------------------------------
+ident 0x4c6962786c466d74 ("LibxlFmt" in ASCII).
+
+version 0x00000002. The version of this specification.
+
+options bit 0: Endianness. 0 = little-endian, 1 = big-endian.
+
+ bit 1: Legacy Format. If set, this stream was created by
+ the legacy conversion tool.
+
+ bits 2-31: Reserved.
+--------------------------------------------------------------------
+
+The endianness shall be 0 (little-endian) for images generated on an
+i386, x86_64, or arm host.
+
+\clearpage
+
+
+Records
+=======
+
+A record has a record header, type specific data and a trailing footer. If
+`length` is not a multiple of 8, the body is padded with zeroes to align the
+end of the record on an 8 octet boundary.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-----------------------+-------------------------+
+ | type | body_length |
+ +-----------+-----------+-------------------------+
+ | body... |
+ ...
+ | | padding (0 to 7 octets) |
+ +-----------+-------------------------------------+
+
+--------------------------------------------------------------------
+Field Description
+----------- -------------------------------------------------------
+type 0x00000000: END
+
+ 0x00000001: LIBXC_CONTEXT
+
+ 0x00000002: XENSTORE_DATA
+
+ 0x00000003: EMULATOR_CONTEXT
+
+ 0x00000004 - 0x7FFFFFFF: Reserved for future _mandatory_
+ records.
+
+ 0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
+ records.
+
+body_length Length in octets of the record body.
+
+body Content of the record.
+
+padding 0 to 7 octets of zeros to pad the whole record to a multiple
+ of 8 octets.
+--------------------------------------------------------------------
+
+\clearpage
+
+END
+----
+
+A end record marks the end of the image, and shall be the final record
+in the stream.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+
+The end record contains no fields; its body_length is 0.
+
+LIBXC\_CONTEXT
+--------------
+
+A libxc context record is a marker, indicating that the stream should be
+handed to `xc_domain_restore()`. `libxc` shall be resonsible for reading its
+own image format from the stream.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+
+The libxc context record contains no fields; its body_length is 0[^1].
+
+
+[^1]: The sending side cannot calculate ahead of time how much data `libxc`
+might write into the stream, especially for live migration where the quantity
+of data is partially proportional to the elapsed time.
+
+XENSTORE\_DATA
+-------------
+
+A record containing xenstore key/value pairs of data.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+ | xenstore key/value pairs |
+ ...
+ +-------------------------------------------------+
+
+EMULATOR\_CONTEXT
+----------------
+
+A context blob for a specific emulator associated with the domain.
+
+ 0 1 2 3 4 5 6 7 octet
+ +------------------------+------------------------+
+ | emulator_id | index |
+ +------------------------+------------------------+
+ | emulator_ctx |
+ ...
+ +-------------------------------------------------+
+
+--------------------------------------------------------------------
+Field Description
+------------ ---------------------------------------------------
+emulator_id 0x00000000: Unknown (In the case of a legacy stream)
+
+ 0x00000001: Qemu Traditional
+
+ 0x00000002: Qemu Upstream
+
+ 0x00000003 - 0xFFFFFFFF: Reserved for future emulators.
+
+index Index of this emulator for the domain, if multiple
+ emulators are in use.
+
+emulator_ctx Emulator context blob.
+--------------------------------------------------------------------
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 10/27] tools/python: Libxc migration v2 infrastructure
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (8 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 09/27] docs: Libxl migration v2 stream specification Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 11/27] tools/python: Libxl " Andrew Cooper
` (17 subsequent siblings)
27 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Contains:
* Python implementation of the libxc migration v2 records
* Verification code for spec compliance
* Unit tests
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/python/setup.py | 1 +
tools/python/xen/migration/libxc.py | 446 ++++++++++++++++++++++++++++++++++
tools/python/xen/migration/tests.py | 41 ++++
tools/python/xen/migration/verify.py | 37 +++
4 files changed, 525 insertions(+)
create mode 100644 tools/python/xen/migration/__init__.py
create mode 100644 tools/python/xen/migration/libxc.py
create mode 100644 tools/python/xen/migration/tests.py
create mode 100644 tools/python/xen/migration/verify.py
diff --git a/tools/python/setup.py b/tools/python/setup.py
index 439c429..5bf81be 100644
--- a/tools/python/setup.py
+++ b/tools/python/setup.py
@@ -43,6 +43,7 @@ setup(name = 'xen',
version = '3.0',
description = 'Xen',
packages = ['xen',
+ 'xen.migration',
'xen.lowlevel',
],
ext_package = "xen.lowlevel",
diff --git a/tools/python/xen/migration/__init__.py b/tools/python/xen/migration/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/tools/python/xen/migration/libxc.py b/tools/python/xen/migration/libxc.py
new file mode 100644
index 0000000..b0255ac
--- /dev/null
+++ b/tools/python/xen/migration/libxc.py
@@ -0,0 +1,446 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Libxc Migration v2 streams
+
+Record structures as per docs/specs/libxc-migration-stream.pandoc, and
+verification routines.
+"""
+
+import sys
+
+from struct import calcsize, unpack
+
+from xen.migration.verify import StreamError, RecordError, VerifyBase
+
+# Image Header
+IHDR_FORMAT = "!QIIHHI"
+
+IHDR_MARKER = 0xffffffffffffffff
+IHDR_IDENT = 0x58454E46 # "XENF" in ASCII
+IHDR_VERSION = 2
+
+IHDR_OPT_BIT_ENDIAN = 0
+IHDR_OPT_LE = (0 << IHDR_OPT_BIT_ENDIAN)
+IHDR_OPT_BE = (1 << IHDR_OPT_BIT_ENDIAN)
+
+IHDR_OPT_RESZ_MASK = 0xfffe
+
+# Domain Header
+DHDR_FORMAT = "IHHII"
+
+DHDR_TYPE_x86_pv = 0x00000001
+DHDR_TYPE_x86_hvm = 0x00000002
+DHDR_TYPE_x86_pvh = 0x00000003
+DHDR_TYPE_arm = 0x00000004
+
+dhdr_type_to_str = {
+ DHDR_TYPE_x86_pv : "x86 PV",
+ DHDR_TYPE_x86_hvm : "x86 HVM",
+ DHDR_TYPE_x86_pvh : "x86 PVH",
+ DHDR_TYPE_arm : "ARM",
+}
+
+# Records
+RH_FORMAT = "II"
+
+REC_TYPE_end = 0x00000000
+REC_TYPE_page_data = 0x00000001
+REC_TYPE_x86_pv_info = 0x00000002
+REC_TYPE_x86_pv_p2m_frames = 0x00000003
+REC_TYPE_x86_pv_vcpu_basic = 0x00000004
+REC_TYPE_x86_pv_vcpu_extended = 0x00000005
+REC_TYPE_x86_pv_vcpu_xsave = 0x00000006
+REC_TYPE_shared_info = 0x00000007
+REC_TYPE_tsc_info = 0x00000008
+REC_TYPE_hvm_context = 0x00000009
+REC_TYPE_hvm_params = 0x0000000a
+REC_TYPE_toolstack = 0x0000000b
+REC_TYPE_x86_pv_vcpu_msrs = 0x0000000c
+REC_TYPE_verify = 0x0000000d
+REC_TYPE_checkpoint = 0x0000000e
+
+rec_type_to_str = {
+ REC_TYPE_end : "End",
+ REC_TYPE_page_data : "Page data",
+ REC_TYPE_x86_pv_info : "x86 PV info",
+ REC_TYPE_x86_pv_p2m_frames : "x86 PV P2M frames",
+ REC_TYPE_x86_pv_vcpu_basic : "x86 PV vcpu basic",
+ REC_TYPE_x86_pv_vcpu_extended : "x86 PV vcpu extended",
+ REC_TYPE_x86_pv_vcpu_xsave : "x86 PV vcpu xsave",
+ REC_TYPE_shared_info : "Shared info",
+ REC_TYPE_tsc_info : "TSC info",
+ REC_TYPE_hvm_context : "HVM context",
+ REC_TYPE_hvm_params : "HVM params",
+ REC_TYPE_toolstack : "Toolstack",
+ REC_TYPE_x86_pv_vcpu_msrs : "x86 PV vcpu msrs",
+ REC_TYPE_verify : "Verify",
+ REC_TYPE_checkpoint : "Checkpoint",
+}
+
+# page_data
+PAGE_DATA_FORMAT = "II"
+PAGE_DATA_PFN_MASK = (1L << 52) - 1
+PAGE_DATA_PFN_RESZ_MASK = ((1L << 60) - 1) & ~((1L << 52) - 1)
+
+# flags from xen/public/domctl.h: XEN_DOMCTL_PFINFO_* shifted by 32 bits
+PAGE_DATA_TYPE_SHIFT = 60
+PAGE_DATA_TYPE_LTABTYPE_MASK = (0x7L << PAGE_DATA_TYPE_SHIFT)
+PAGE_DATA_TYPE_LTAB_MASK = (0xfL << PAGE_DATA_TYPE_SHIFT)
+PAGE_DATA_TYPE_LPINTAB = (0x8L << PAGE_DATA_TYPE_SHIFT) # Pinned pagetable
+
+PAGE_DATA_TYPE_NOTAB = (0x0L << PAGE_DATA_TYPE_SHIFT) # Regular page
+PAGE_DATA_TYPE_L1TAB = (0x1L << PAGE_DATA_TYPE_SHIFT) # L1 pagetable
+PAGE_DATA_TYPE_L2TAB = (0x2L << PAGE_DATA_TYPE_SHIFT) # L2 pagetable
+PAGE_DATA_TYPE_L3TAB = (0x3L << PAGE_DATA_TYPE_SHIFT) # L3 pagetable
+PAGE_DATA_TYPE_L4TAB = (0x4L << PAGE_DATA_TYPE_SHIFT) # L4 pagetable
+PAGE_DATA_TYPE_BROKEN = (0xdL << PAGE_DATA_TYPE_SHIFT) # Broken
+PAGE_DATA_TYPE_XALLOC = (0xeL << PAGE_DATA_TYPE_SHIFT) # Allocate-only
+PAGE_DATA_TYPE_XTAB = (0xfL << PAGE_DATA_TYPE_SHIFT) # Invalid
+
+# x86_pv_info
+X86_PV_INFO_FORMAT = "BBHI"
+
+X86_PV_P2M_FRAMES_FORMAT = "II"
+
+# x86_pv_vcpu_{basic,extended,xsave,msrs}
+X86_PV_VCPU_HDR_FORMAT = "II"
+
+# tsc_info
+TSC_INFO_FORMAT = "IIQII"
+
+# hvm_params
+HVM_PARAMS_ENTRY_FORMAT = "QQ"
+HVM_PARAMS_FORMAT = "II"
+
+class VerifyLibxc(VerifyBase):
+ """ Verify a Libxc v2 stream """
+
+ def __init__(self, info, read):
+ VerifyBase.__init__(self, info, read)
+
+ self.squashed_pagedata_records = 0
+
+
+ def verify(self):
+ """ Verity a libxc stream """
+
+ self.verify_ihdr()
+ self.verify_dhdr()
+
+ while self.verify_record() != REC_TYPE_end:
+ pass
+
+
+ def verify_ihdr(self):
+ """ Verify an Image Header """
+ marker, ident, version, options, res1, res2 = \
+ self.unpack_exact(IHDR_FORMAT)
+
+ if marker != IHDR_MARKER:
+ raise StreamError("Bad image marker: Expected 0x%x, got 0x%x"
+ % (IHDR_MARKER, marker))
+
+ if ident != IHDR_IDENT:
+ raise StreamError("Bad image id: Expected 0x%x, got 0x%x"
+ % (IHDR_IDENT, ident))
+
+ if version != IHDR_VERSION:
+ raise StreamError("Unknown image version: Expected %d, got %d"
+ % (IHDR_VERSION, version))
+
+ if options & IHDR_OPT_RESZ_MASK:
+ raise StreamError("Reserved bits set in image options field: 0x%x"
+ % (options & IHDR_OPT_RESZ_MASK))
+
+ if res1 != 0 or res2 != 0:
+ raise StreamError("Reserved bits set in image header: 0x%04x:0x%08x"
+ % (res1, res2))
+
+ if ( (sys.byteorder == "little") and
+ ((options & IHDR_OPT_BIT_ENDIAN) != IHDR_OPT_LE) ):
+ raise StreamError(
+ "Stream is not native endianess - unable to validate")
+
+ endian = ["little", "big"][options & IHDR_OPT_LE]
+ self.info("Libxc Image Header: %s endian" % (endian, ))
+
+
+ def verify_dhdr(self):
+ """ Verify a domain header """
+
+ gtype, page_shift, res1, major, minor = \
+ self.unpack_exact(DHDR_FORMAT)
+
+ if gtype not in dhdr_type_to_str:
+ raise StreamError("Unrecognised domain type 0x%x" % (gtype, ))
+
+ if res1 != 0:
+ raise StreamError("Reserved bits set in domain header 0x%04x"
+ % (res1, ))
+
+ if page_shift != 12:
+ raise StreamError("Page shift expected to be 12. Got %d"
+ % (page_shift, ))
+
+ if major == 0:
+ self.info("Domain Header: legacy converted %s"
+ % (dhdr_type_to_str[gtype], ))
+ else:
+ self.info("Domain Header: %s from Xen %d.%d"
+ % (dhdr_type_to_str[gtype], major, minor))
+
+
+ def verify_record(self):
+ """ Verify an individual record """
+
+ rtype, length = self.unpack_exact(RH_FORMAT)
+
+ if rtype not in rec_type_to_str:
+ raise StreamError("Unrecognised record type 0x%x" % (rtype, ))
+
+ contentsz = (length + 7) & ~7
+ content = self.rdexact(contentsz)
+
+ if rtype != REC_TYPE_page_data:
+
+ if self.squashed_pagedata_records > 0:
+ self.info("Squashed %d Page Data records together"
+ % (self.squashed_pagedata_records, ))
+ self.squashed_pagedata_records = 0
+
+ self.info("Libxc Record: %s, length %d"
+ % (rec_type_to_str[rtype], length))
+
+ else:
+ self.squashed_pagedata_records += 1
+
+ padding = content[length:]
+ if padding != "\x00" * len(padding):
+ raise StreamError("Padding containing non0 bytes found")
+
+ if rtype not in record_verifiers:
+ raise RuntimeError("No verification function for libxc record '%s'"
+ % rec_type_to_str[rtype])
+ else:
+ record_verifiers[rtype](self, content[:length])
+
+ return rtype
+
+
+ def verify_record_end(self, content):
+ """ End record """
+
+ if len(content) != 0:
+ raise RecordError("End record with non-zero length")
+
+
+ def verify_record_page_data(self, content):
+ """ Page Data record """
+ minsz = calcsize(PAGE_DATA_FORMAT)
+
+ if len(content) <= minsz:
+ raise RecordError("PAGE_DATA record must be at least %d bytes long"
+ % (minsz, ))
+
+ count, res1 = unpack(PAGE_DATA_FORMAT, content[:minsz])
+
+ if res1 != 0:
+ raise StreamError("Reserved bits set in PAGE_DATA record 0x%04x"
+ % (res1, ))
+
+ pfnsz = count * 8
+ if (len(content) - minsz) < pfnsz:
+ raise RecordError("PAGE_DATA record must contain a pfn record for "
+ "each count")
+
+ pfns = list(unpack("=%dQ" % (count,), content[minsz:minsz + pfnsz]))
+
+ nr_pages = 0
+ for idx, pfn in enumerate(pfns):
+
+ if pfn & PAGE_DATA_PFN_RESZ_MASK:
+ raise RecordError("Reserved bits set in pfn[%d]: 0x%016x",
+ idx, pfn & PAGE_DATA_PFN_RESZ_MASK)
+
+ if pfn >> PAGE_DATA_TYPE_SHIFT in (5, 6, 7, 8):
+ raise RecordError("Invalid type value in pfn[%d]: 0x%016x",
+ idx, pfn & PAGE_DATA_TYPE_LTAB_MASK)
+
+ # We expect page data for each normal page or pagetable
+ if PAGE_DATA_TYPE_NOTAB <= (pfn & PAGE_DATA_TYPE_LTABTYPE_MASK) \
+ <= PAGE_DATA_TYPE_L4TAB:
+ nr_pages += 1
+
+ pagesz = nr_pages * 4096
+ if len(content) != minsz + pfnsz + pagesz:
+ raise RecordError("Expected %u + %u + %u, got %u"
+ % (minsz, pfnsz, pagesz, len(content)))
+
+
+ def verify_record_x86_pv_info(self, content):
+ """ x86 PV Info record """
+
+ expectedsz = calcsize(X86_PV_INFO_FORMAT)
+ if len(content) != expectedsz:
+ raise RecordError("x86_pv_info: expected length of %d, got %d"
+ % (expectedsz, len(content)))
+
+ width, levels, res1, res2 = unpack(X86_PV_INFO_FORMAT, content)
+
+ if width not in (4, 8):
+ raise RecordError("Expected width of 4 or 8, got %d" % (width, ))
+
+ if levels not in (3, 4):
+ raise RecordError("Expected levels of 3 or 4, got %d" % (levels, ))
+
+ if res1 != 0 or res2 != 0:
+ raise StreamError("Reserved bits set in X86_PV_INFO: 0x%04x 0x%08x"
+ % (res1, res2))
+
+ bitness = {4:32, 8:64}[width]
+ self.info(" %sbit guest, %d levels of pagetables" % (bitness, levels))
+
+
+ def verify_record_x86_pv_p2m_frames(self, content):
+ """ x86 PV p2m frames record """
+
+ if len(content) % 8 != 0:
+ raise RecordError("Length expected to be a multiple of 8, not %d"
+ % (len(content), ))
+
+ start, end = unpack("=II", content[:8])
+ self.info(" Start pfn 0x%x, End 0x%x" % (start, end))
+
+
+ def verify_record_x86_pv_vcpu_generic(self, content, name):
+ """ Generic for all REC_TYPE_x86_pv_vcpu_{basic,extended,xsave,msrs} """
+ minsz = calcsize(X86_PV_VCPU_HDR_FORMAT)
+
+ if len(content) <= minsz:
+ raise RecordError("X86_PV_VCPU_%s record length must be at least %d"
+ " bytes long" % (name, minsz))
+
+ vcpuid, res1 = unpack(X86_PV_VCPU_HDR_FORMAT, content[:minsz])
+
+ if res1 != 0:
+ raise StreamError(
+ "Reserved bits set in x86_pv_vcpu_%s record 0x%04x"
+ % (name, res1))
+
+ self.info(" vcpu%d %s context, %d bytes"
+ % (vcpuid, name, len(content) - minsz))
+
+
+ def verify_record_shared_info(self, content):
+ """ shared info record """
+
+ if len(content) != 4096:
+ raise RecordError("Length expected to be 4906 bytes, not %d"
+ % (len(content), ))
+
+
+ def verify_record_tsc_info(self, content):
+ """ tsc info record """
+
+ sz = calcsize(TSC_INFO_FORMAT)
+
+ if len(content) != sz:
+ raise RecordError("Length should be %u bytes" % (sz, ))
+
+ mode, khz, nsec, incarn, res1 = unpack(TSC_INFO_FORMAT, content)
+
+ if res1 != 0:
+ raise StreamError("Reserved bits set in TSC_INFO: 0x%08x"
+ % (res1, ))
+
+ self.info(" Mode %u, %u kHz, %u ns, incarnation %d"
+ % (mode, khz, nsec, incarn))
+
+
+ def verify_record_hvm_context(self, content):
+ """ hvm context record """
+
+ if len(content) == 0:
+ raise RecordError("Zero length HVM context")
+
+
+ def verify_record_hvm_params(self, content):
+ """ hvm params record """
+
+ sz = calcsize(HVM_PARAMS_FORMAT)
+
+ if len(content) < sz:
+ raise RecordError("Length should be at least %u bytes" % (sz, ))
+
+ count, rsvd = unpack(HVM_PARAMS_FORMAT, content[:sz])
+
+ if rsvd != 0:
+ raise RecordError("Reserved field not zero (0x%04x)" % (rsvd, ))
+
+ sz += count * calcsize(HVM_PARAMS_ENTRY_FORMAT)
+
+ if len(content) != sz:
+ raise RecordError("Length should be %u bytes" % (sz, ))
+
+
+ def verify_record_toolstack(self, _):
+ """ toolstack record """
+ raise DeprecationWarning("Found Toolstack record in stream")
+
+
+ def verify_record_verify(self, content):
+ """ verify record """
+
+ if len(content) != 0:
+ raise RecordError("Verify record with non-zero length")
+
+
+ def verify_record_checkpoint(self, content):
+ """ checkpoint record """
+
+ if len(content) != 0:
+ raise RecordError("Checkpoint record with non-zero length")
+
+
+record_verifiers = {
+ REC_TYPE_end:
+ VerifyLibxc.verify_record_end,
+ REC_TYPE_page_data:
+ VerifyLibxc.verify_record_page_data,
+
+ REC_TYPE_x86_pv_info:
+ VerifyLibxc.verify_record_x86_pv_info,
+ REC_TYPE_x86_pv_p2m_frames:
+ VerifyLibxc.verify_record_x86_pv_p2m_frames,
+
+ REC_TYPE_x86_pv_vcpu_basic:
+ lambda s, x:
+ VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "basic"),
+ REC_TYPE_x86_pv_vcpu_extended:
+ lambda s, x:
+ VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "extended"),
+ REC_TYPE_x86_pv_vcpu_xsave:
+ lambda s, x:
+ VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "xsave"),
+ REC_TYPE_x86_pv_vcpu_msrs:
+ lambda s, x:
+ VerifyLibxc.verify_record_x86_pv_vcpu_generic(s, x, "msrs"),
+
+ REC_TYPE_shared_info:
+ VerifyLibxc.verify_record_shared_info,
+ REC_TYPE_tsc_info:
+ VerifyLibxc.verify_record_tsc_info,
+
+ REC_TYPE_hvm_context:
+ VerifyLibxc.verify_record_hvm_context,
+ REC_TYPE_hvm_params:
+ VerifyLibxc.verify_record_hvm_params,
+ REC_TYPE_toolstack:
+ VerifyLibxc.verify_record_toolstack,
+ REC_TYPE_verify:
+ VerifyLibxc.verify_record_verify,
+ REC_TYPE_checkpoint:
+ VerifyLibxc.verify_record_checkpoint,
+ }
diff --git a/tools/python/xen/migration/tests.py b/tools/python/xen/migration/tests.py
new file mode 100644
index 0000000..3e97268
--- /dev/null
+++ b/tools/python/xen/migration/tests.py
@@ -0,0 +1,41 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Unit tests for migration v2 streams
+"""
+
+import unittest
+
+from struct import calcsize
+
+from xen.migration import libxc, libxl
+
+class TestLibxc(unittest.TestCase):
+
+ def test_format_sizes(self):
+
+ for fmt, sz in ( (libxc.IHDR_FORMAT, 24),
+ (libxc.DHDR_FORMAT, 16),
+ (libxc.RH_FORMAT, 8),
+
+ (libxc.PAGE_DATA_FORMAT, 8),
+ (libxc.X86_PV_INFO_FORMAT, 8),
+ (libxc.X86_PV_P2M_FRAMES_FORMAT, 8),
+ (libxc.X86_PV_VCPU_HDR_FORMAT, 8),
+ (libxc.TSC_INFO_FORMAT, 24),
+ (libxc.HVM_PARAMS_ENTRY_FORMAT, 16),
+ (libxc.HVM_PARAMS_FORMAT, 8),
+ ):
+ self.assertEqual(calcsize(fmt), sz)
+
+
+def test_suite():
+ suite = unittest.TestSuite()
+
+ suite.addTest(unittest.makeSuite(TestLibxc))
+
+ return suite
+
+if __name__ == "__main__":
+ unittest.main()
diff --git a/tools/python/xen/migration/verify.py b/tools/python/xen/migration/verify.py
new file mode 100644
index 0000000..7a42dbf
--- /dev/null
+++ b/tools/python/xen/migration/verify.py
@@ -0,0 +1,37 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Common verification infrastructure for v2 streams
+"""
+
+from struct import calcsize, unpack
+
+class StreamError(StandardError):
+ """Error with the stream"""
+ pass
+
+class RecordError(StandardError):
+ """Error with a record in the stream"""
+ pass
+
+
+class VerifyBase(object):
+
+ def __init__(self, info, read):
+
+ self.info = info
+ self.read = read
+
+ def rdexact(self, nr_bytes):
+ """Read exactly nr_bytes from the stream"""
+ _ = self.read(nr_bytes)
+ if len(_) != nr_bytes:
+ raise IOError("Stream truncated")
+ return _
+
+ def unpack_exact(self, fmt):
+ """Unpack a struct format string from the stream"""
+ sz = calcsize(fmt)
+ return unpack(fmt, self.rdexact(sz))
+
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 11/27] tools/python: Libxl migration v2 infrastructure
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (9 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 10/27] tools/python: Libxc migration v2 infrastructure Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 12/27] tools/python: Other migration infrastructure Andrew Cooper
` (16 subsequent siblings)
27 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Contains:
* Python implementation of the libxl migration v2 records
* Verification code for spec compliance
* Unit tests
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/python/xen/migration/libxl.py | 188 +++++++++++++++++++++++++++++++++++
tools/python/xen/migration/tests.py | 13 +++
2 files changed, 201 insertions(+)
create mode 100644 tools/python/xen/migration/libxl.py
diff --git a/tools/python/xen/migration/libxl.py b/tools/python/xen/migration/libxl.py
new file mode 100644
index 0000000..4e1f4f8
--- /dev/null
+++ b/tools/python/xen/migration/libxl.py
@@ -0,0 +1,188 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Libxl Migration v2 streams
+
+Record structures as per docs/specs/libxl-migration-stream.pandoc, and
+verification routines.
+"""
+
+import sys
+
+from struct import calcsize, unpack
+from xen.migration.verify import StreamError, RecordError, VerifyBase
+from xen.migration.libxc import VerifyLibxc
+
+# Header
+HDR_FORMAT = "!QII"
+
+HDR_IDENT = 0x4c6962786c466d74 # "LibxlFmt" in ASCII
+HDR_VERSION = 2
+
+HDR_OPT_BIT_ENDIAN = 0
+HDR_OPT_BIT_LEGACY = 1
+
+HDR_OPT_LE = (0 << HDR_OPT_BIT_ENDIAN)
+HDR_OPT_BE = (1 << HDR_OPT_BIT_ENDIAN)
+HDR_OPT_LEGACY = (1 << HDR_OPT_BIT_LEGACY)
+
+HDR_OPT_RESZ_MASK = 0xfffc
+
+# Records
+RH_FORMAT = "II"
+
+REC_TYPE_end = 0x00000000
+REC_TYPE_libxc_context = 0x00000001
+REC_TYPE_xenstore_data = 0x00000002
+REC_TYPE_emulator_context = 0x00000003
+
+rec_type_to_str = {
+ REC_TYPE_end : "End",
+ REC_TYPE_libxc_context : "Libxc context",
+ REC_TYPE_xenstore_data : "Xenstore data",
+ REC_TYPE_emulator_context : "Emulator context",
+}
+
+# emulator_context
+EMULATOR_CONTEXT_FORMAT = "II"
+
+EMULATOR_ID_unknown = 0x00000000
+EMULATOR_ID_qemu_trad = 0x00000001
+EMULATOR_ID_qemu_upstream = 0x00000002
+
+emulator_id_to_str = {
+ EMULATOR_ID_unknown : "Unknown",
+ EMULATOR_ID_qemu_trad : "Qemu Traditional",
+ EMULATOR_ID_qemu_upstream : "Qemu Upstream",
+}
+
+
+#
+# libxl format
+#
+
+LIBXL_QEMU_SIGNATURE = "DeviceModelRecord0002"
+LIBXL_QEMU_RECORD_HDR = "=%dsI" % (len(LIBXL_QEMU_SIGNATURE), )
+
+class VerifyLibxl(VerifyBase):
+ """ Verify a Libxl v2 stream """
+
+ def __init__(self, info, read):
+ VerifyBase.__init__(self, info, read)
+
+
+ def verify(self):
+ """ Verity a libxl stream """
+
+ self.verify_hdr()
+
+ while self.verify_record() != REC_TYPE_end:
+ pass
+
+
+ def verify_hdr(self):
+ """ Verify a Header """
+ ident, version, options = self.unpack_exact(HDR_FORMAT)
+
+ if ident != HDR_IDENT:
+ raise StreamError("Bad image id: Expected 0x%x, got 0x%x"
+ % (HDR_IDENT, ident))
+
+ if version != HDR_VERSION:
+ raise StreamError("Unknown image version: Expected %d, got %d"
+ % (HDR_VERSION, version))
+
+ if options & HDR_OPT_RESZ_MASK:
+ raise StreamError("Reserved bits set in image options field: 0x%x"
+ % (options & HDR_OPT_RESZ_MASK))
+
+ if ( (sys.byteorder == "little") and
+ ((options & HDR_OPT_BIT_ENDIAN) != HDR_OPT_LE) ):
+ raise StreamError(
+ "Stream is not native endianess - unable to validate")
+
+ endian = ["little", "big"][options & HDR_OPT_LE]
+
+ if options & HDR_OPT_LEGACY:
+ self.info("Libxl Header: %s endian, legacy converted" % (endian, ))
+ else:
+ self.info("Libxl Header: %s endian" % (endian, ))
+
+
+ def verify_record(self):
+ """ Verify an individual record """
+ rtype, length = self.unpack_exact(RH_FORMAT)
+
+ if rtype not in rec_type_to_str:
+ raise StreamError("Unrecognised record type %x" % (rtype, ))
+
+ self.info("Libxl Record: %s, length %d"
+ % (rec_type_to_str[rtype], length))
+
+ contentsz = (length + 7) & ~7
+ content = self.rdexact(contentsz)
+
+ padding = content[length:]
+ if padding != "\x00" * len(padding):
+ raise StreamError("Padding containing non0 bytes found")
+
+ if rtype not in record_verifiers:
+ raise RuntimeError("No verification function for libxl record '%s'"
+ % rec_type_to_str[rtype])
+ else:
+ record_verifiers[rtype](self, content[:length])
+
+ return rtype
+
+
+ def verify_record_end(self, content):
+ """ End record """
+
+ if len(content) != 0:
+ raise RecordError("End record with non-zero length")
+
+
+ def verify_record_libxc_context(self, content):
+ """ Libxc context record """
+
+ if len(content) != 0:
+ raise RecordError("Libxc context record with non-zero length")
+
+ # Verify the libxc stream, as we can't seek forwards through it
+ VerifyLibxc(self.info, self.read).verify()
+
+
+ def verify_record_xenstore_data(self, content):
+ """ Xenstore Data record """
+
+ if len(content) == 0:
+ raise RecordError("Xenstore data record with zero length")
+
+
+ def verify_record_emulator_context(self, content):
+ """ Emulator Context record """
+ minsz = calcsize(EMULATOR_CONTEXT_FORMAT)
+
+ if len(content) < minsz:
+ raise RecordError("Length must be at least %d bytes, got %d"
+ % (minsz, len(content)))
+
+ emu_id, emu_idx = unpack(EMULATOR_CONTEXT_FORMAT, content[:minsz])
+
+ if emu_id not in emulator_id_to_str:
+ raise RecordError("Unrecognised emulator id 0x%x" % (emu_id, ))
+
+ self.info(" Index %d, type %s" % (emu_idx, emulator_id_to_str[emu_id]))
+
+
+record_verifiers = {
+ REC_TYPE_end:
+ VerifyLibxl.verify_record_end,
+ REC_TYPE_libxc_context:
+ VerifyLibxl.verify_record_libxc_context,
+ REC_TYPE_xenstore_data:
+ VerifyLibxl.verify_record_xenstore_data,
+ REC_TYPE_emulator_context:
+ VerifyLibxl.verify_record_emulator_context,
+}
diff --git a/tools/python/xen/migration/tests.py b/tools/python/xen/migration/tests.py
index 3e97268..91044cd 100644
--- a/tools/python/xen/migration/tests.py
+++ b/tools/python/xen/migration/tests.py
@@ -30,10 +30,23 @@ class TestLibxc(unittest.TestCase):
self.assertEqual(calcsize(fmt), sz)
+class TestLibxl(unittest.TestCase):
+
+ def test_format_sizes(self):
+
+ for fmt, sz in ( (libxl.HDR_FORMAT, 16),
+ (libxl.RH_FORMAT, 8),
+
+ (libxl.EMULATOR_CONTEXT_FORMAT, 8),
+ ):
+ self.assertEqual(calcsize(fmt), sz)
+
+
def test_suite():
suite = unittest.TestSuite()
suite.addTest(unittest.makeSuite(TestLibxc))
+ suite.addTest(unittest.makeSuite(TestLibxl))
return suite
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 12/27] tools/python: Other migration infrastructure
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (10 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 11/27] tools/python: Libxl " Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 9:48 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 13/27] tools/python: Verification utility for v2 stream spec compliance Andrew Cooper
` (15 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
Contains:
* Reverse-engineered notes of the legacy format from xg_save_restore.h
* Python implementation of the legacy format
* Public HVM Params used in the legacy stream
* XL header format
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
New in v2 - removes various many magic numbers from subsequent scripts
---
tools/python/xen/migration/legacy.py | 279 ++++++++++++++++++++++++++++++++++
tools/python/xen/migration/public.py | 21 +++
tools/python/xen/migration/xl.py | 12 ++
3 files changed, 312 insertions(+)
create mode 100644 tools/python/xen/migration/legacy.py
create mode 100644 tools/python/xen/migration/public.py
create mode 100644 tools/python/xen/migration/xl.py
diff --git a/tools/python/xen/migration/legacy.py b/tools/python/xen/migration/legacy.py
new file mode 100644
index 0000000..2f2240a
--- /dev/null
+++ b/tools/python/xen/migration/legacy.py
@@ -0,0 +1,279 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Libxc legacy migration streams
+
+Documentation and record structures for legacy migration
+"""
+
+"""
+SAVE/RESTORE/MIGRATE PROTOCOL
+=============================
+
+The general form of a stream of chunks is a header followed by a
+body consisting of a variable number of chunks (terminated by a
+chunk with type 0) followed by a trailer.
+
+For a rolling/checkpoint (e.g. remus) migration then the body and
+trailer phases can be repeated until an external event
+(e.g. failure) causes the process to terminate and commit to the
+most recent complete checkpoint.
+
+HEADER
+------
+
+unsigned long : p2m_size
+
+extended-info (PV-only, optional):
+
+ If first unsigned long == ~0UL then extended info is present,
+ otherwise unsigned long is part of p2m. Note that p2m_size above
+ does not include the length of the extended info.
+
+ extended-info:
+
+ unsigned long : signature == ~0UL
+ uint32_t : number of bytes remaining in extended-info
+
+ 1 or more extended-info blocks of form:
+ char[4] : block identifier
+ uint32_t : block data size
+ bytes : block data
+
+ defined extended-info blocks:
+ "vcpu" : VCPU context info containing vcpu_guest_context_t.
+ The precise variant of the context structure
+ (e.g. 32 vs 64 bit) is distinguished by
+ the block size.
+ "extv" : Presence indicates use of extended VCPU context in
+ tail, data size is 0.
+
+p2m (PV-only):
+
+ consists of p2m_size bytes comprising an array of xen_pfn_t sized entries.
+
+BODY PHASE - Format A (for live migration or Remus without compression)
+----------
+
+A series of chunks with a common header:
+ int : chunk type
+
+If the chunk type is +ve then chunk contains guest memory data, and the
+type contains the number of pages in the batch:
+
+ unsigned long[] : PFN array, length == number of pages in batch
+ Each entry consists of XEN_DOMCTL_PFINFO_*
+ in bits 31-28 and the PFN number in bits 27-0.
+ page data : PAGE_SIZE bytes for each page marked present in PFN
+ array
+
+If the chunk type is -ve then chunk consists of one of a number of
+metadata types. See definitions of XC_SAVE_ID_* below.
+
+If chunk type is 0 then body phase is complete.
+
+
+BODY PHASE - Format B (for Remus with compression)
+----------
+
+A series of chunks with a common header:
+ int : chunk type
+
+If the chunk type is +ve then chunk contains array of PFNs corresponding
+to guest memory and type contains the number of PFNs in the batch:
+
+ unsigned long[] : PFN array, length == number of pages in batch
+ Each entry consists of XEN_DOMCTL_PFINFO_*
+ in bits 31-28 and the PFN number in bits 27-0.
+
+If the chunk type is -ve then chunk consists of one of a number of
+metadata types. See definitions of XC_SAVE_ID_* below.
+
+If the chunk type is -ve and equals XC_SAVE_ID_COMPRESSED_DATA, then the
+chunk consists of compressed page data, in the following format:
+
+ unsigned long : Size of the compressed chunk to follow
+ compressed data : variable length data of size indicated above.
+ This chunk consists of compressed page data.
+ The number of pages in one chunk depends on
+ the amount of space available in the sender's
+ output buffer.
+
+Format of compressed data:
+ compressed_data = <deltas>*
+ delta = <marker, run*>
+ marker = (RUNFLAG|SKIPFLAG) bitwise-or RUNLEN [1 byte marker]
+ RUNFLAG = 0
+ SKIPFLAG = 1 << 7
+ RUNLEN = 7-bit unsigned value indicating number of WORDS in the run
+ run = string of bytes of length sizeof(WORD) * RUNLEN
+
+ If marker contains RUNFLAG, then RUNLEN * sizeof(WORD) bytes of data following
+ the marker is copied into the target page at the appropriate offset indicated by
+ the offset_ptr
+ If marker contains SKIPFLAG, then the offset_ptr is advanced
+ by RUNLEN * sizeof(WORD).
+
+If chunk type is 0 then body phase is complete.
+
+There can be one or more chunks with type XC_SAVE_ID_COMPRESSED_DATA,
+containing compressed pages. The compressed chunks are collated to form
+one single compressed chunk for the entire iteration. The number of pages
+present in this final compressed chunk will be equal to the total number
+of valid PFNs specified by the +ve chunks.
+
+At the sender side, compressed pages are inserted into the output stream
+in the same order as they would have been if compression logic was absent.
+
+Until last iteration, the BODY is sent in Format A, to maintain live
+migration compatibility with receivers of older Xen versions.
+At the last iteration, if Remus compression was enabled, the sender sends
+a trigger, XC_SAVE_ID_ENABLE_COMPRESSION to tell the receiver to parse the
+BODY in Format B from the next iteration onwards.
+
+An example sequence of chunks received in Format B:
+ +16 +ve chunk
+ unsigned long[16] PFN array
+ +100 +ve chunk
+ unsigned long[100] PFN array
+ +50 +ve chunk
+ unsigned long[50] PFN array
+
+ XC_SAVE_ID_COMPRESSED_DATA TAG
+ N Length of compressed data
+ N bytes of DATA Decompresses to 166 pages
+
+ XC_SAVE_ID_* other xc save chunks
+ 0 END BODY TAG
+
+Corner case with checkpoint compression:
+ At sender side, after pausing the domain, dirty pages are usually
+ copied out to a temporary buffer. After the domain is resumed,
+ compression is done and the compressed chunk(s) are sent, followed by
+ other XC_SAVE_ID_* chunks.
+ If the temporary buffer gets full while scanning for dirty pages,
+ the sender stops buffering of dirty pages, compresses the temporary
+ buffer and sends the compressed data with XC_SAVE_ID_COMPRESSED_DATA.
+ The sender then resumes the buffering of dirty pages and continues
+ scanning for the dirty pages.
+ For e.g., assume that the temporary buffer can hold 4096 pages and
+ there are 5000 dirty pages. The following is the sequence of chunks
+ that the receiver will see:
+
+ +1024 +ve chunk
+ unsigned long[1024] PFN array
+ +1024 +ve chunk
+ unsigned long[1024] PFN array
+ +1024 +ve chunk
+ unsigned long[1024] PFN array
+ +1024 +ve chunk
+ unsigned long[1024] PFN array
+
+ XC_SAVE_ID_COMPRESSED_DATA TAG
+ N Length of compressed data
+ N bytes of DATA Decompresses to 4096 pages
+
+ +4 +ve chunk
+ unsigned long[4] PFN array
+
+ XC_SAVE_ID_COMPRESSED_DATA TAG
+ M Length of compressed data
+ M bytes of DATA Decompresses to 4 pages
+
+ XC_SAVE_ID_* other xc save chunks
+ 0 END BODY TAG
+
+ In other words, XC_SAVE_ID_COMPRESSED_DATA can be interleaved with
+ +ve chunks arbitrarily. But at the receiver end, the following condition
+ always holds true until the end of BODY PHASE:
+ num(PFN entries +ve chunks) >= num(pages received in compressed form)
+
+TAIL PHASE
+----------
+
+Content differs for PV and HVM guests.
+
+HVM TAIL:
+
+ "Magic" pages:
+ uint64_t : I/O req PFN
+ uint64_t : Buffered I/O req PFN
+ uint64_t : Store PFN
+ Xen HVM Context:
+ uint32_t : Length of context in bytes
+ bytes : Context data
+ Qemu context:
+ char[21] : Signature:
+ "QemuDeviceModelRecord" : Read Qemu save data until EOF
+ "DeviceModelRecord0002" : uint32_t length field followed by that many
+ bytes of Qemu save data
+ "RemusDeviceModelState" : Currently the same as "DeviceModelRecord0002".
+
+PV TAIL:
+
+ Unmapped PFN list : list of all the PFNs that were not in map at the close
+ unsigned int : Number of unmapped pages
+ unsigned long[] : PFNs of unmapped pages
+
+ VCPU context data : A series of VCPU records, one per present VCPU
+ Maximum and present map supplied in XC_SAVE_ID_VCPUINFO
+ bytes: : VCPU context structure. Size is determined by size
+ provided in extended-info header
+ bytes[128] : Extended VCPU context (present IFF "extv" block
+ present in extended-info header)
+
+ Shared Info Page : 4096 bytes of shared info page
+"""
+
+CHUNK_end = 0
+CHUNK_enable_verify_mode = -1
+CHUNK_vcpu_info = -2
+CHUNK_hvm_ident_pt = -3
+CHUNK_hvm_vm86_tss = -4
+CHUNK_tmem = -5
+CHUNK_tmem_extra = -6
+CHUNK_tsc_info = -7
+CHUNK_hvm_console_pfn = -8
+CHUNK_last_checkpoint = -9
+CHUNK_hvm_acpi_ioports_location = -10
+CHUNK_hvm_viridian = -11
+CHUNK_compressed_data = -12
+CHUNK_enable_compression = -13
+CHUNK_hvm_generation_id_addr = -14
+CHUNK_hvm_paging_ring_pfn = -15
+CHUNK_hvm_monitor_ring_pfn = -16
+CHUNK_hvm_sharing_ring_pfn = -17
+CHUNK_toolstack = -18
+CHUNK_hvm_ioreq_server_pfn = -19
+CHUNK_hvm_nr_ioreq_server_pages = -20
+
+chunk_type_to_str = {
+ CHUNK_end : "end",
+ CHUNK_enable_verify_mode : "enable_verify_mode",
+ CHUNK_vcpu_info : "vcpu_info",
+ CHUNK_hvm_ident_pt : "hvm_ident_pt",
+ CHUNK_hvm_vm86_tss : "hvm_vm86_tss",
+ CHUNK_tmem : "tmem",
+ CHUNK_tmem_extra : "tmem_extra",
+ CHUNK_tsc_info : "tsc_info",
+ CHUNK_hvm_console_pfn : "hvm_console_pfn",
+ CHUNK_last_checkpoint : "last_checkpoint",
+ CHUNK_hvm_acpi_ioports_location : "hvm_acpi_ioports_location",
+ CHUNK_hvm_viridian : "hvm_viridian",
+ CHUNK_compressed_data : "compressed_data",
+ CHUNK_enable_compression : "enable_compression",
+ CHUNK_hvm_generation_id_addr : "hvm_generation_id_addr",
+ CHUNK_hvm_paging_ring_pfn : "hvm_paging_ring_pfn",
+ CHUNK_hvm_monitor_ring_pfn : "hvm_monitor_ring_pfn",
+ CHUNK_hvm_sharing_ring_pfn : "hvm_sharing_ring_pfn",
+ CHUNK_toolstack : "toolstack",
+ CHUNK_hvm_ioreq_server_pfn : "hvm_ioreq_server_pfn",
+ CHUNK_hvm_nr_ioreq_server_pages : "hvm_nr_ioreq_server_pages",
+}
+
+# Up to 1024 pages (4MB) at a time
+MAX_BATCH = 1024
+
+# Maximum #VCPUs currently supported for save/restore
+MAX_VCPU_ID = 4095
diff --git a/tools/python/xen/migration/public.py b/tools/python/xen/migration/public.py
new file mode 100644
index 0000000..fab2f84
--- /dev/null
+++ b/tools/python/xen/migration/public.py
@@ -0,0 +1,21 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Xen public ABI constants, used in migration
+"""
+
+HVM_PARAM_STORE_PFN = 1
+HVM_PARAM_IOREQ_PFN = 5
+HVM_PARAM_BUFIOREQ_PFN = 6
+HVM_PARAM_VIRIDIAN = 9
+HVM_PARAM_IDENT_PT = 12
+HVM_PARAM_VM86_TSS = 15
+HVM_PARAM_CONSOLE_PFN = 17
+HVM_PARAM_ACPI_IOPORTS_LOCATION = 19
+HVM_PARAM_PAGING_RING_PFN = 27
+HVM_PARAM_MONITOR_RING_PFN = 28
+HVM_PARAM_SHARING_RING_PFN = 29
+HVM_PARAM_IOREQ_SERVER_PFN = 32
+HVM_PARAM_NR_IOREQ_SERVER_PAGES = 33
+HVM_PARAM_VM_GENERATION_ID_ADDR = 34
diff --git a/tools/python/xen/migration/xl.py b/tools/python/xen/migration/xl.py
new file mode 100644
index 0000000..978e744
--- /dev/null
+++ b/tools/python/xen/migration/xl.py
@@ -0,0 +1,12 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+XL migration stream format
+"""
+
+MAGIC = "Xen saved domain, xl format\n \0 \r"
+
+HEADER_FORMAT = "=IIII"
+
+MANDATORY_FLAG_STREAMV2 = 2
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 13/27] tools/python: Verification utility for v2 stream spec compliance
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (11 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 12/27] tools/python: Other migration infrastructure Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 14/27] tools/python: Conversion utility for legacy migration streams Andrew Cooper
` (14 subsequent siblings)
27 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
This is exceedingly useful for development, but not of practical use being
installed into a production dom0.
---
tools/python/scripts/verify-stream-v2 | 174 +++++++++++++++++++++++++++++++++
1 file changed, 174 insertions(+)
create mode 100755 tools/python/scripts/verify-stream-v2
diff --git a/tools/python/scripts/verify-stream-v2 b/tools/python/scripts/verify-stream-v2
new file mode 100755
index 0000000..3daf257
--- /dev/null
+++ b/tools/python/scripts/verify-stream-v2
@@ -0,0 +1,174 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+""" Verify a v2 format migration stream """
+
+import sys
+import struct
+import os, os.path
+import syslog
+import traceback
+
+from xen.migration.verify import StreamError, RecordError
+from xen.migration.libxc import VerifyLibxc
+from xen.migration.libxl import VerifyLibxl
+
+fin = None # Input file/fd
+log_to_syslog = False # Boolean - Log to syslog instead of stdout/err?
+verbose = False # Boolean - Summarise stream contents
+quiet = False # Boolean - Suppress error printing
+
+def info(msg):
+ """Info message, routed to appropriate destination"""
+ if not quiet and verbose:
+ if log_to_syslog:
+ for line in msg.split("\n"):
+ syslog.syslog(syslog.LOG_INFO, line)
+ else:
+ print msg
+
+def err(msg):
+ """Error message, routed to appropriate destination"""
+ if not quiet:
+ if log_to_syslog:
+ for line in msg.split("\n"):
+ syslog.syslog(syslog.LOG_ERR, line)
+ print >> sys.stderr, msg
+
+def stream_read(_ = None):
+ """Read from input"""
+ return fin.read(_)
+
+def rdexact(nr_bytes):
+ """Read exactly nr_bytes from fin"""
+ _ = stream_read(nr_bytes)
+ if len(_) != nr_bytes:
+ raise IOError("Stream truncated")
+ return _
+
+def unpack_exact(fmt):
+ """Unpack a format from fin"""
+ sz = struct.calcsize(fmt)
+ return struct.unpack(fmt, rdexact(sz))
+
+
+def skip_xl_header():
+ """Skip over an xl header in the stream"""
+
+ hdr = rdexact(32)
+ if hdr != "Xen saved domain, xl format\n \0 \r":
+ raise StreamError("No xl header")
+
+ _, mflags, _, optlen = unpack_exact("=IIII")
+ _ = rdexact(optlen)
+
+ info("Processed xl header")
+
+ if mflags & 2: # XL_MANDATORY_FLAG_STREAMv2
+ return "libxl"
+ else:
+ return "libxc"
+
+def read_stream(fmt):
+ """ Read an entire stream """
+
+ try:
+ if fmt == "xl":
+ fmt = skip_xl_header()
+
+ if fmt == "libxc":
+ VerifyLibxc(info, stream_read).verify()
+ else:
+ VerifyLibxl(info, stream_read).verify()
+
+ except (IOError, StreamError, RecordError):
+ err("Stream Error:")
+ err(traceback.format_exc())
+ return 1
+
+ except StandardError:
+ err("Script Error:")
+ err(traceback.format_exc())
+ err("Please fix me")
+ return 2
+
+ return 0
+
+def open_file_or_fd(val, mode, buffering):
+ """
+ If 'val' looks like a decimal integer, open it as an fd. If not, try to
+ open it as a regular file.
+ """
+
+ fd = -1
+ try:
+ # Does it look like an integer?
+ try:
+ fd = int(val, 10)
+ except ValueError:
+ pass
+
+ # Try to open it...
+ if fd != -1:
+ return os.fdopen(fd, mode, buffering)
+ else:
+ return open(val, mode, buffering)
+
+ except StandardError, e:
+ if fd != -1:
+ err("Unable to open fd %d: %s: %s" %
+ (fd, e.__class__.__name__, e))
+ else:
+ err("Unable to open file '%s': %s: %s" %
+ (val, e.__class__.__name__, e))
+
+ raise SystemExit(2)
+
+def main():
+ """ main """
+ from optparse import OptionParser
+ global fin, quiet, verbose
+
+ # Change stdout to be line-buffered.
+ sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1)
+
+ parser = OptionParser(usage = "%prog [options]",
+ description =
+ "Verify a stream according to the v2 spec")
+
+ # Optional options
+ parser.add_option("-i", "--in", dest = "fin", metavar = "<FD or FILE>",
+ default = "0",
+ help = "Stream to verify (defaults to stdin)")
+ parser.add_option("-v", "--verbose", action = "store_true", default = False,
+ help = "Summarise stream contents")
+ parser.add_option("-q", "--quiet", action = "store_true", default = False,
+ help = "Suppress all logging/errors")
+ parser.add_option("-f", "--format", dest = "format",
+ metavar = "<libxc|libxl|xl>", default = "libxc",
+ choices = ["libxc", "libxl", "xl"],
+ help = "Format of the incoming stream (defaults to libxc)")
+ parser.add_option("--syslog", action = "store_true", default = False,
+ help = "Log to syslog instead of stdout")
+
+ opts, _ = parser.parse_args()
+
+ if opts.syslog:
+ global log_to_syslog
+
+ syslog.openlog("verify-stream-v2", syslog.LOG_PID)
+ log_to_syslog = True
+
+ verbose = opts.verbose
+ quiet = opts.quiet
+ fin = open_file_or_fd(opts.fin, "rb", 0)
+
+ return read_stream(opts.format)
+
+if __name__ == "__main__":
+ try:
+ sys.exit(main())
+ except SystemExit, e:
+ sys.exit(e.code)
+ except KeyboardInterrupt:
+ sys.exit(2)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 14/27] tools/python: Conversion utility for legacy migration streams
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (12 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 13/27] tools/python: Verification utility for v2 stream spec compliance Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 15/27] tools/libxl: Migration v2 stream format Andrew Cooper
` (13 subsequent siblings)
27 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
This utility will take a legacy stream as in input, and produce a v2 stream as
an output. It is exec()'d by libxl to provide backwards compatibility.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/python/Makefile | 4 +
tools/python/scripts/convert-legacy-stream | 678 ++++++++++++++++++++++++++++
2 files changed, 682 insertions(+)
create mode 100755 tools/python/scripts/convert-legacy-stream
diff --git a/tools/python/Makefile b/tools/python/Makefile
index e933be8..df942a7 100644
--- a/tools/python/Makefile
+++ b/tools/python/Makefile
@@ -17,9 +17,13 @@ build: genwrap.py $(XEN_ROOT)/tools/libxl/libxl_types.idl \
.PHONY: install
install:
+ $(INSTALL_DIR) $(DESTDIR)$(PRIVATE_BINDIR)
+
CC="$(CC)" CFLAGS="$(PY_CFLAGS)" $(PYTHON) setup.py install \
$(PYTHON_PREFIX_ARG) --root="$(DESTDIR)" --force
+ $(INSTALL_PROG) scripts/convert-legacy-stream $(DESTDIR)$(PRIVATE_BINDIR)
+
.PHONY: test
test:
export LD_LIBRARY_PATH=$$(readlink -f ../libxc):$$(readlink -f ../xenstore); $(PYTHON) test.py -b -u
diff --git a/tools/python/scripts/convert-legacy-stream b/tools/python/scripts/convert-legacy-stream
new file mode 100755
index 0000000..d54fa22
--- /dev/null
+++ b/tools/python/scripts/convert-legacy-stream
@@ -0,0 +1,678 @@
+#!/usr/bin/env python
+# -*- coding: utf-8 -*-
+
+"""
+Convert a legacy migration stream to a v2 stream.
+"""
+
+import sys
+import os, os.path
+import syslog
+import traceback
+
+from struct import calcsize, unpack, pack
+
+from xen.migration import legacy, public, libxc, libxl, xl
+
+__version__ = 1
+
+fin = None # Input file/fd
+fout = None # Output file/fd
+twidth = 0 # Legacy toolstack bitness (32 or 64)
+pv = None # Boolean (pv or hvm)
+qemu = True # Boolean - process qemu record?
+log_to_syslog = False # Boolean - Log to syslog instead of stdout/err?
+verbose = False # Boolean - Summarise stream contents
+
+def stream_read(_ = None):
+ """Read from the input"""
+ return fin.read(_)
+
+def stream_write(_):
+ """Write to the output"""
+ return fout.write(_)
+
+def info(msg):
+ """Info message, routed to appropriate destination"""
+ if verbose:
+ if log_to_syslog:
+ for line in msg.split("\n"):
+ syslog.syslog(syslog.LOG_INFO, line)
+ else:
+ print msg
+
+def err(msg):
+ """Error message, routed to appropriate destination"""
+ if log_to_syslog:
+ for line in msg.split("\n"):
+ syslog.syslog(syslog.LOG_ERR, line)
+ print >> sys.stderr, msg
+
+class StreamError(StandardError):
+ """Error with the incoming migration stream"""
+ pass
+
+class VM(object):
+ """Container of VM parameters"""
+
+ def __init__(self, fmt):
+ # Common
+ self.p2m_size = 0
+
+ # PV
+ self.max_vcpu_id = 0
+ self.online_vcpu_map = []
+ self.width = 0
+ self.levels = 0
+ self.basic_len = 0
+ self.extd = False
+ self.xsave_len = 0
+
+ # libxl
+ self.libxl = fmt == "libxl"
+ self.xenstore = [] # Deferred "toolstack" records
+
+def write_libxc_ihdr():
+ stream_write(pack(libxc.IHDR_FORMAT,
+ libxc.IHDR_MARKER, # Marker
+ libxc.IHDR_IDENT, # Ident
+ libxc.IHDR_VERSION, # Version
+ libxc.IHDR_OPT_LE, # Options
+ 0, 0)) # Reserved
+
+def write_libxc_dhdr():
+ if pv:
+ dtype = libxc.DHDR_TYPE_x86_pv
+ else:
+ dtype = libxc.DHDR_TYPE_x86_hvm
+
+ stream_write(pack(libxc.DHDR_FORMAT,
+ dtype, # Type
+ 12, # Page size
+ 0, # Reserved
+ 0, # Xen major (converted)
+ __version__)) # Xen minor (converted)
+
+def write_libxl_hdr():
+ stream_write(pack(libxl.HDR_FORMAT,
+ libxl.HDR_IDENT, # Ident
+ libxl.HDR_VERSION, # Version 2
+ libxl.HDR_OPT_LE | # Options
+ libxl.HDR_OPT_LEGACY # Little Endian and Legacy
+ ))
+
+def write_record(rt, *argl):
+ alldata = ''.join(argl)
+ length = len(alldata)
+
+ record = pack(libxc.RH_FORMAT, rt, length) + alldata
+ plen = (8 - (length & 7)) & 7
+ record += '\x00' * plen
+
+ stream_write(record)
+
+def write_libxc_pv_info(vm):
+ write_record(libxc.REC_TYPE_x86_pv_info,
+ pack(libxc.X86_PV_INFO_FORMAT,
+ vm.width, vm.levels, 0, 0))
+
+def write_libxc_pv_p2m_frames(vm, pfns):
+ write_record(libxc.REC_TYPE_x86_pv_p2m_frames,
+ pack(libxc.X86_PV_P2M_FRAMES_FORMAT,
+ 0, vm.p2m_size - 1),
+ pack("Q" * len(pfns), *pfns))
+
+def write_libxc_pv_vcpu_basic(vcpu_id, data):
+ write_record(libxc.REC_TYPE_x86_pv_vcpu_basic,
+ pack(libxc.X86_PV_VCPU_HDR_FORMAT, vcpu_id, 0), data)
+
+def write_libxc_pv_vcpu_extd(vcpu_id, data):
+ write_record(libxc.REC_TYPE_x86_pv_vcpu_extended,
+ pack(libxc.X86_PV_VCPU_HDR_FORMAT, vcpu_id, 0), data)
+
+def write_libxc_pv_vcpu_xsave(vcpu_id, data):
+ write_record(libxc.REC_TYPE_x86_pv_vcpu_xsave,
+ pack(libxc.X86_PV_VCPU_HDR_FORMAT, vcpu_id, 0), data)
+
+def write_page_data(pfns, pages):
+ if fout is None: # Save copying 1M buffers around for no reason
+ return
+
+ new_pfns = [(((x & 0xf0000000) << 32) | (x & 0x0fffffff)) for x in pfns]
+
+ # Optimise the needless buffer copying in write_record()
+ stream_write(pack(libxc.RH_FORMAT,
+ libxc.REC_TYPE_page_data,
+ 8 + (len(new_pfns) * 8) + len(pages)))
+ stream_write(pack(libxc.PAGE_DATA_FORMAT, len(new_pfns), 0))
+ stream_write(pack("Q" * len(new_pfns), *new_pfns))
+ stream_write(pages)
+
+def write_libxc_tsc_info(mode, khz, nsec, incarn):
+ write_record(libxc.REC_TYPE_tsc_info,
+ pack(libxc.TSC_INFO_FORMAT,
+ mode, khz, nsec, incarn, 0))
+
+def write_libxc_hvm_params(params):
+ if pv:
+ raise StreamError("HVM-only param in PV stream")
+ elif len(params) % 2:
+ raise RuntimeError("Expected even length list of hvm parameters")
+
+ write_record(libxc.REC_TYPE_hvm_params,
+ pack(libxc.HVM_PARAMS_FORMAT, len(params) / 2, 0),
+ pack("Q" * len(params), *params))
+
+def write_libxl_end():
+ write_record(libxl.REC_TYPE_end, "")
+
+def write_libxl_libxc_context():
+ write_record(libxl.REC_TYPE_libxc_context, "")
+
+def write_libxl_xenstore_data(data):
+ write_record(libxl.REC_TYPE_xenstore_data, data)
+
+def write_libxl_emulator_context(blob):
+ write_record(libxl.REC_TYPE_emulator_context,
+ pack(libxl.EMULATOR_CONTEXT_FORMAT,
+ libxl.EMULATOR_ID_unknown, 0) + blob)
+
+def rdexact(nr_bytes):
+ """Read exactly nr_bytes from fin"""
+ _ = stream_read(nr_bytes)
+ if len(_) != nr_bytes:
+ raise IOError("Stream truncated")
+ return _
+
+def unpack_exact(fmt):
+ """Unpack a format from fin"""
+ sz = calcsize(fmt)
+ return unpack(fmt, rdexact(sz))
+
+def unpack_ulongs(nr_ulongs):
+ if twidth == 32:
+ return unpack_exact("I" * nr_ulongs)
+ else:
+ return unpack_exact("Q" * nr_ulongs)
+
+def read_pv_extended_info(vm):
+
+ marker, = unpack_ulongs(1)
+
+ if twidth == 32:
+ expected = 0xffffffff
+ else:
+ expected = 0xffffffffffffffff
+
+ if marker != expected:
+ raise StreamError("Unexpected extended info marker 0x%x" % (marker, ))
+
+ total_length, = unpack_exact("I")
+ so_far = 0
+
+ info("Extended Info: length 0x%x" % (total_length, ))
+
+ while so_far < total_length:
+
+ blkid, datasz = unpack_exact("=4sI")
+ so_far += 8
+
+ info(" Record type: %s, size 0x%x" % (blkid, datasz))
+
+ data = rdexact(datasz)
+ so_far += datasz
+
+ # Eww, but this is how it is done :(
+ if blkid == "vcpu":
+
+ vm.basic_len = datasz
+
+ if datasz == 0x1430:
+ vm.width = 8
+ vm.levels = 4
+ info(" 64bit domain, 4 levels")
+ elif datasz == 0xaf0:
+ vm.width = 4
+ vm.levels = 3
+ info(" 32bit domain, 3 levels")
+ else:
+ raise StreamError("Unable to determine guest width/level")
+
+ write_libxc_pv_info(vm)
+
+ elif blkid == "extv":
+ vm.extd = True
+
+ elif blkid == "xcnt":
+ vm.xsave_len, = unpack("I", data[:4])
+ info("xcnt sz 0x%x" % (vm.xsave_len, ))
+
+ else:
+ raise StreamError("Unrecognised extended block")
+
+
+ if so_far != total_length:
+ raise StreamError("Overshot Extended Info size by %d bytes"
+ % (so_far - total_length,))
+
+def read_pv_p2m_frames(vm):
+ fpp = 4096 / vm.width
+ p2m_frame_len = (vm.p2m_size - 1) / fpp + 1
+
+ info("P2M frames: fpp %d, p2m_frame_len %d" % (fpp, p2m_frame_len))
+ write_libxc_pv_p2m_frames(vm, unpack_ulongs(p2m_frame_len))
+
+def read_pv_tail(vm):
+
+ nr_unmapped_pfns, = unpack_exact("I")
+
+ if nr_unmapped_pfns != 0:
+ # "Unmapped" pfns are bogus
+ _ = unpack_ulongs(nr_unmapped_pfns)
+ info("discarding %d bogus 'unmapped pfns'" % (nr_unmapped_pfns, ))
+
+ for vcpu_id in vm.online_vcpu_map:
+
+ basic = rdexact(vm.basic_len)
+ info("Got VCPU basic (size 0x%x)" % (vm.basic_len, ))
+ write_libxc_pv_vcpu_basic(vcpu_id, basic)
+
+ if vm.extd:
+ extd = rdexact(128)
+ info("Got VCPU extd (size 0x%x)" % (128, ))
+ write_libxc_pv_vcpu_extd(vcpu_id, extd)
+
+ if vm.xsave_len:
+ mask, size = unpack_exact("QQ")
+ assert vm.xsave_len - 16 == size
+
+ xsave = rdexact(size)
+ info("Got VCPU xsave (mask 0x%x, size 0x%x)" % (mask, size))
+ write_libxc_pv_vcpu_xsave(vcpu_id, xsave)
+
+ shinfo = rdexact(4096)
+ info("Got shinfo")
+
+ write_record(libxc.REC_TYPE_shared_info, shinfo)
+ write_record(libxc.REC_TYPE_end, "")
+
+
+def read_chunks(vm):
+
+ hvm_params = []
+
+ while True:
+
+ marker, = unpack_exact("=i")
+ if marker <= 0:
+ info("Chunk: %d - %s" %
+ (marker, legacy.chunk_type_to_str.get(marker, "unknown")))
+
+ if marker == legacy.CHUNK_end:
+ info(" End")
+
+ if hvm_params:
+ write_libxc_hvm_params(hvm_params)
+
+ return
+
+ elif marker > 0:
+
+ if marker > legacy.MAX_BATCH:
+ raise StreamError("Page batch (%d) exceeded MAX_BATCH (%d)"
+ % (marker, legacy.MAX_BATCH))
+ pfns = unpack_ulongs(marker)
+
+ # xc_domain_save() leaves many XEN_DOMCTL_PFINFO_XTAB records for
+ # sequences of pfns it cant map. Drop these.
+ pfns = [ x for x in pfns if x != 0xf0000000 ]
+
+ if len(set(pfns)) != len(pfns):
+ raise StreamError("Duplicate pfns in batch")
+
+ nr_pages = len([x for x in pfns if (x & 0xf0000000) < 0xd0000000])
+ pages = rdexact(nr_pages * 4096)
+
+ write_page_data(pfns, pages)
+
+ elif marker == legacy.CHUNK_enable_verify_mode:
+ # For debugging purposes only. Will not be seen in real migration
+ raise RuntimeError("Unable to convert a debug stream")
+
+ elif marker == legacy.CHUNK_vcpu_info:
+ max_id, = unpack_exact("i")
+
+ if max_id > legacy.MAX_VCPU_ID:
+ raise StreamError("Vcpu max_id out of range: %d > %d"
+ % (max_id, legacy.MAX_VCPU_ID))
+
+ vm.max_vcpu_id = max_id
+ bitmap = unpack_exact("Q" * ((max_id/64) + 1))
+
+ for idx, word in enumerate(bitmap):
+ bit_idx = 0
+
+ while word > 0:
+ if word & 1:
+ vm.online_vcpu_map.append((idx * 64) + bit_idx)
+
+ bit_idx += 1
+ word >>= 1
+
+ info(" Vcpu info: max_id %d, online map %s"
+ % (vm.max_vcpu_id, vm.online_vcpu_map))
+
+ elif marker == legacy.CHUNK_hvm_ident_pt:
+ _, ident_pt = unpack_exact("=IQ")
+ info(" EPT Identity Pagetable: 0x%x" % (ident_pt, ))
+ hvm_params.extend([public.HVM_PARAM_IDENT_PT, ident_pt])
+
+ elif marker == legacy.CHUNK_hvm_vm86_tss:
+ _, vm86_tss = unpack_exact("=IQ")
+ info(" VM86 TSS: 0x%x" % (vm86_tss, ))
+ hvm_params.extend([public.HVM_PARAM_VM86_TSS, vm86_tss])
+
+ elif marker == legacy.CHUNK_tmem:
+ raise RuntimeError("todo")
+
+ elif marker == legacy.CHUNK_tmem_extra:
+ raise RuntimeError("todo")
+
+ elif marker == legacy.CHUNK_tsc_info:
+ mode, nsec, khz, incarn = unpack_exact("=IQII")
+ info(" TSC_INFO: mode %s, %d ns, %d khz, %d incarn"
+ % (mode, nsec, khz, incarn))
+ write_libxc_tsc_info(mode, khz, nsec, incarn)
+
+ elif marker == legacy.CHUNK_hvm_console_pfn:
+ _, console_pfn = unpack_exact("=IQ")
+ info(" Console pfn: 0x%x" % (console_pfn, ))
+ hvm_params.extend([public.HVM_PARAM_CONSOLE_PFN, console_pfn])
+
+ elif marker == legacy.CHUNK_last_checkpoint:
+ info(" Last Checkpoint")
+ # Nothing to do
+
+ elif marker == legacy.CHUNK_hvm_acpi_ioports_location:
+ _, loc = unpack_exact("=IQ")
+ info(" ACPI ioport location: 0x%x" % (loc, ))
+ hvm_params.extend([public.HVM_PARAM_ACPI_IOPORTS_LOCATION, loc])
+
+ elif marker == legacy.CHUNK_hvm_viridian:
+ _, loc = unpack_exact("=IQ")
+ info(" Viridian location: 0x%x" % (loc, ))
+ hvm_params.extend([public.HVM_PARAM_VIRIDIAN, loc])
+
+ elif marker == legacy.CHUNK_compressed_data:
+ sz, = unpack_exact("I")
+ data = rdexact(sz)
+ info(" Compressed Data: sz 0x%x" % (sz, ))
+ raise RuntimeError("todo")
+
+ elif marker == legacy.CHUNK_enable_compression:
+ raise RuntimeError("todo")
+
+ elif marker == legacy.CHUNK_hvm_generation_id_addr:
+ _, genid_loc = unpack_exact("=IQ")
+ info(" Generation ID Address: 0x%x" % (genid_loc, ))
+ hvm_params.extend(
+ [public.HVM_PARAM_VM_GENERATION_ID_ADDR, genid_loc])
+
+ elif marker == legacy.CHUNK_hvm_paging_ring_pfn:
+ _, pfn = unpack_exact("=IQ")
+ info(" Paging ring pfn: 0x%x" % (pfn, ))
+ hvm_params.extend([public.HVM_PARAM_PAGING_RING_PFN, pfn])
+
+ elif marker == legacy.CHUNK_hvm_monitor_ring_pfn:
+ _, pfn = unpack_exact("=IQ")
+ info(" Monitor ring pfn: 0x%x" % (pfn, ))
+ hvm_params.extend([public.HVM_PARAM_MONITOR_RING_PFN, pfn])
+
+ elif marker == legacy.CHUNK_hvm_sharing_ring_pfn:
+ _, pfn = unpack_exact("=IQ")
+ info(" Sharing ring pfn: 0x%x" % (pfn, ))
+ hvm_params.extend([public.HVM_PARAM_SHARING_RING_PFN, pfn])
+
+ elif marker == legacy.CHUNK_toolstack:
+ sz, = unpack_exact("I")
+
+ if sz:
+ data = rdexact(sz)
+ info(" Toolstack Data: sz 0x%x" % (sz, ))
+
+ if vm.libxl:
+ vm.xenstore.append(data)
+ else:
+ info(" Discarding")
+
+ elif marker == legacy.CHUNK_hvm_ioreq_server_pfn:
+ _, pfn = unpack_exact("=IQ")
+ info(" IOREQ server pfn: 0x%x" % (pfn, ))
+ hvm_params.extend([public.HVM_PARAM_IOREQ_SERVER_PFN, pfn])
+
+ elif marker == legacy.CHUNK_hvm_nr_ioreq_server_pages:
+ _, nr_pages = unpack_exact("=IQ")
+ info(" IOREQ server pages: %d" % (nr_pages, ))
+ hvm_params.extend(
+ [public.HVM_PARAM_NR_IOREQ_SERVER_PAGES, nr_pages])
+
+ else:
+ raise StreamError("Unrecognised chunk %d" % (marker,))
+
+def read_hvm_tail(vm):
+
+ io, bufio, store = unpack_exact("QQQ")
+ info("Magic pfns: 0x%x 0x%x 0x%x" % (io, bufio, store))
+ write_libxc_hvm_params([public.HVM_PARAM_IOREQ_PFN, io,
+ public.HVM_PARAM_BUFIOREQ_PFN, bufio,
+ public.HVM_PARAM_STORE_PFN, store])
+
+ blobsz, = unpack_exact("I")
+ info("Got HVM Context (0x%x bytes)" % (blobsz, ))
+ blob = rdexact(blobsz)
+
+ write_record(libxc.REC_TYPE_hvm_context, blob)
+ write_record(libxc.REC_TYPE_end, "")
+
+
+
+def read_qemu(vm):
+
+ rawsig = rdexact(21)
+ sig, = unpack("21s", rawsig)
+ info("Qemu signature: %s" % (sig, ))
+
+ if sig == "DeviceModelRecord0002":
+ rawsz = rdexact(4)
+ sz, = unpack("I", rawsz)
+ qdata = rdexact(sz)
+
+ if vm.libxl:
+ write_libxl_emulator_context(qdata)
+ else:
+ stream_write(rawsig)
+ stream_write(rawsz)
+ stream_write(qdata)
+
+ else:
+ raise RuntimeError("Unrecognised Qemu sig '%s'" % (sig, ))
+
+
+def skip_xl_header(fmt):
+ """Skip over an xl header in the stream"""
+
+ hdr = rdexact(len(xl.MAGIC))
+ if hdr != xl.MAGIC:
+ raise StreamError("No xl header")
+
+ byteorder, mflags, oflags, optlen = unpack_exact(xl.HEADER_FORMAT)
+
+ if fmt == "libxl":
+ mflags |= xl.MANDATORY_FLAG_STREAMV2
+
+ opts = pack(xl.HEADER_FORMAT, byteorder, mflags, oflags, optlen)
+
+ optdata = rdexact(optlen)
+
+ info("Processed xl header")
+
+ stream_write(hdr)
+ stream_write(opts)
+ stream_write(optdata)
+
+def read_legacy_stream(vm):
+
+ try:
+ vm.p2m_size, = unpack_ulongs(1)
+ info("P2M Size: 0x%x" % (vm.p2m_size,))
+
+ if vm.libxl:
+ write_libxl_hdr()
+ write_libxl_libxc_context()
+
+ write_libxc_ihdr()
+ write_libxc_dhdr()
+
+ if pv:
+ read_pv_extended_info(vm)
+ read_pv_p2m_frames(vm)
+
+ read_chunks(vm)
+
+ if pv:
+ read_pv_tail(vm)
+ else:
+ read_hvm_tail(vm)
+
+ if vm.libxl:
+ for rec in vm.xenstore:
+ write_libxl_xenstore_data(rec)
+
+ if not pv and (vm.libxl or qemu):
+ read_qemu(vm)
+
+ if vm.libxl:
+ write_libxl_end()
+
+ except (IOError, StreamError):
+ err("Stream Error:")
+ err(traceback.format_exc())
+ return 1
+
+ except RuntimeError:
+ err("Script Error:")
+ err(traceback.format_exc())
+ err("Please fix me")
+ return 2
+ return 0
+
+def open_file_or_fd(val, mode):
+ """
+ If 'val' looks like a decimal integer, open it as an fd. If not, try to
+ open it as a regular file.
+ """
+
+ fd = -1
+ try:
+ # Does it look like an integer?
+ try:
+ fd = int(val, 10)
+ except ValueError:
+ pass
+
+ # Try to open it...
+ if fd != -1:
+ return os.fdopen(fd, mode, 0)
+ else:
+ return open(val, mode, 0)
+
+ except StandardError, e:
+ if fd != -1:
+ err("Unable to open fd %d: %s: %s" %
+ (fd, e.__class__.__name__, e))
+ else:
+ err("Unable to open file '%s': %s: %s" %
+ (val, e.__class__.__name__, e))
+
+ raise SystemExit(1)
+
+
+def main():
+ from optparse import OptionParser
+ global fin, fout, twidth, pv, qemu, verbose
+
+ # Change stdout to be line-buffered.
+ sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 1)
+
+ parser = OptionParser(version = __version__,
+ usage = ("%prog [options] -i INPUT -o OUTPUT"
+ " -w WIDTH -g GUEST"),
+ description =
+ "Convert a legacy stream to a v2 stream")
+
+ # Required options
+ parser.add_option("-i", "--in", dest = "fin", metavar = "<FD or FILE>",
+ help = "Legacy input to convert")
+ parser.add_option("-o", "--out", dest = "fout", metavar = "<FD or FILE>",
+ help = "v2 format output")
+ parser.add_option("-w", "--width", dest = "twidth",
+ metavar = "<32/64>", choices = ["32", "64"],
+ help = "Legacy toolstack bitness")
+ parser.add_option("-g", "--guest-type", dest = "gtype",
+ metavar = "<pv/hvm>", choices = ["pv", "hvm"],
+ help = "Type of guest in stream")
+
+ # Optional options
+ parser.add_option("-f", "--format", dest = "format",
+ metavar = "<libxc|libxl>", default = "libxc",
+ choices = ["libxc", "libxl"],
+ help = "Desired format of the outgoing stream " \
+ "(defaults to libxc)")
+ parser.add_option("-v", "--verbose", action = "store_true", default = False,
+ help = "Summarise stream contents")
+ parser.add_option("-x", "--xl", action = "store_true", default = False,
+ help = ("Is an `xl` header present in the stream?"
+ " (default no)"))
+ parser.add_option("--skip-qemu", action = "store_true", default = False,
+ help = ("Skip processing of the qemu tail?"
+ " (default no)"))
+ parser.add_option("--syslog", action = "store_true", default = False,
+ help = "Log to syslog instead of stdout")
+
+ opts, _ = parser.parse_args()
+
+ if (opts.fin is None or opts.fout is None or
+ opts.twidth is None or opts.gtype is None):
+
+ parser.print_help(sys.stderr)
+ raise SystemExit(1)
+
+ if opts.syslog:
+ global log_to_syslog
+
+ syslog.openlog("convert-legacy-stream", syslog.LOG_PID)
+ log_to_syslog = True
+
+ fin = open_file_or_fd(opts.fin, "rb")
+ fout = open_file_or_fd(opts.fout, "wb")
+ twidth = int(opts.twidth)
+ pv = opts.gtype == "pv"
+ verbose = opts.verbose
+ if opts.skip_qemu:
+ qemu = False
+
+ if opts.xl:
+ skip_xl_header(opts.format)
+
+ rc = read_legacy_stream(VM(opts.format))
+ fout.close()
+
+ return rc
+
+if __name__ == "__main__":
+ try:
+ sys.exit(main())
+ except SystemExit, e:
+ sys.exit(e.code)
+ except KeyboardInterrupt:
+ sys.exit(1)
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 15/27] tools/libxl: Migration v2 stream format
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (13 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 14/27] tools/python: Conversion utility for legacy migration streams Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 9:49 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream Andrew Cooper
` (12 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel
Cc: Ross Lagerwall, Wei Liu, Ian Jackson, Ian Campbell, Andrew Cooper
From: Ross Lagerwall <ross.lagerwall@citrix.com>
C structures describing the Libxl migration v2 stream format
Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v2: Move into libxl__ namespace
---
tools/libxl/libxl_sr_stream_format.h | 57 ++++++++++++++++++++++++++++++++++
1 file changed, 57 insertions(+)
create mode 100644 tools/libxl/libxl_sr_stream_format.h
diff --git a/tools/libxl/libxl_sr_stream_format.h b/tools/libxl/libxl_sr_stream_format.h
new file mode 100644
index 0000000..f4f790b
--- /dev/null
+++ b/tools/libxl/libxl_sr_stream_format.h
@@ -0,0 +1,57 @@
+#ifndef LIBXL__SR_STREAM_FORMAT_H
+#define LIBXL__SR_STREAM_FORMAT_H
+
+/*
+ * C structures for the Migration v2 stream format.
+ * See docs/specs/libxl-migration-stream.pandoc
+ */
+
+#include <stdint.h>
+
+typedef struct libxl__sr_hdr
+{
+ uint64_t ident;
+ uint32_t version;
+ uint32_t options;
+} libxl__sr_hdr;
+
+#define RESTORE_STREAM_IDENT 0x4c6962786c466d74UL
+#define RESTORE_STREAM_VERSION 0x00000002U
+
+#define RESTORE_OPT_BIG_ENDIAN (1u << 0)
+#define RESTORE_OPT_LEGACY (1u << 1)
+
+
+typedef struct libxl__sr_rec_hdr
+{
+ uint32_t type;
+ uint32_t length;
+} libxl__sr_rec_hdr;
+
+/* All records must be aligned up to an 8 octet boundary */
+#define REC_ALIGN_ORDER 3U
+
+#define REC_TYPE_END 0x00000000U
+#define REC_TYPE_LIBXC_CONTEXT 0x00000001U
+#define REC_TYPE_XENSTORE_DATA 0x00000002U
+#define REC_TYPE_EMULATOR_CONTEXT 0x00000003U
+
+typedef struct libxl__sr_emulator_hdr
+{
+ uint32_t id;
+ uint32_t index;
+} libxl__sr_emulator_hdr;
+
+#define EMULATOR_UNKNOWN 0x00000000U
+#define EMULATOR_QEMU_TRADITIONAL 0x00000001U
+#define EMULATOR_QEMU_UPSTREAM 0x00000002U
+
+#endif /* LIBXL__SR_STREAM_FORMAT_H */
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (14 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 15/27] tools/libxl: Migration v2 stream format Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 10:23 ` Ian Campbell
2015-07-10 12:17 ` Ian Jackson
2015-07-09 18:26 ` [PATCH v2 17/27] tools/libxl: Support converting a legacy stream to a " Andrew Cooper
` (11 subsequent siblings)
27 siblings, 2 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel
Cc: Ross Lagerwall, Wei Liu, Ian Jackson, Ian Campbell, Andrew Cooper
From: Ross Lagerwall <ross.lagerwall@citrix.com>
This contains the event machinary and state machines to read an act on a
non-checkpointed migration v2 stream (with the exception of the
xc_domain_restore() handling which is spliced later in a bisectable way).
It also contains some boilerplate to help support checkpointed streams, which
shall be introduced in a later patch.
Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
Large quantities of the logic here are completely overhauled since v1, mostly
as part of fixing the checkpoint buffering bug which was the cause of the
broken Remus failover. The result is actually more simple overall; all
records are buffered in memory (there is no splicing of the emulator records
any more), with normal streams having exactly 0 or 1 records currently in the
buffer, before processing. Remus support later will allow multiple buffered
records.
---
tools/libxl/Makefile | 1 +
tools/libxl/libxl_internal.h | 55 +++++
tools/libxl/libxl_stream_read.c | 504 +++++++++++++++++++++++++++++++++++++++
3 files changed, 560 insertions(+)
create mode 100644 tools/libxl/libxl_stream_read.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index cc9c152..c71c5fe 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -94,6 +94,7 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
libxl_internal.o libxl_utils.o libxl_uuid.o \
libxl_json.o libxl_aoutils.o libxl_numa.o libxl_vnuma.o \
+ libxl_stream_read.o \
libxl_save_callout.o _libxl_save_msgs_callout.o \
libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
LIBXL_OBJS += libxl_genid.o
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 4bd6ea1..0f17e7c 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -19,6 +19,8 @@
#include "libxl_osdeps.h" /* must come before any other headers */
+#include "libxl_sr_stream_format.h"
+
#include <assert.h>
#include <dirent.h>
#include <errno.h>
@@ -3211,6 +3213,58 @@ typedef void libxl__domain_create_cb(libxl__egc *egc,
libxl__domain_create_state*,
int rc, uint32_t domid);
+/* State for manipulating a libxl migration v2 stream */
+typedef struct libxl__stream_read_state libxl__stream_read_state;
+
+typedef struct libxl__sr_record_buf {
+ /* private to stream read helper */
+ LIBXL_STAILQ_ENTRY(struct libxl__sr_record_buf) entry;
+ libxl__sr_rec_hdr hdr;
+ void *body;
+} libxl__sr_record_buf;
+
+struct libxl__stream_read_state {
+ /* filled by the user */
+ libxl__ao *ao;
+ int fd;
+ void (*completion_callback)(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int rc);
+ /* Private */
+ int rc;
+ bool running;
+
+ /* Main stream-reading data */
+ libxl__datacopier_state dc;
+ libxl__sr_hdr hdr;
+ libxl__sr_record_buf *incoming_record;
+ LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue;
+ enum {
+ SRS_PHASE_NORMAL,
+ } phase;
+ bool recursion_guard;
+
+ /* Emulator blob handling */
+ libxl__datacopier_state emu_dc;
+ libxl__carefd *emu_carefd;
+};
+
+_hidden void libxl__stream_read_start(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+
+_hidden void libxl__stream_read_continue(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+
+_hidden void libxl__stream_read_abort(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
+
+static inline bool libxl__stream_read_inuse(
+ const libxl__stream_read_state *stream)
+{
+ return stream->running;
+}
+
+
struct libxl__domain_create_state {
/* filled in by user */
libxl__ao *ao;
@@ -3227,6 +3281,7 @@ struct libxl__domain_create_state {
libxl__stub_dm_spawn_state dmss;
/* If we're not doing stubdom, we use only dmss.dm,
* for the non-stubdom device model. */
+ libxl__stream_read_state srs;
libxl__save_helper_state shs;
/* necessary if the domain creation failed and we have to destroy it */
libxl__domain_destroy_state dds;
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
new file mode 100644
index 0000000..6f5d572
--- /dev/null
+++ b/tools/libxl/libxl_stream_read.c
@@ -0,0 +1,504 @@
+/*
+ * Copyright (C) 2015 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/*
+ * Infrastructure for reading and acting on the contents of a libxl migration
+ * stream. There are a lot of moving parts here.
+ *
+ * Entry points from outside:
+ * - libxl__stream_read_start()
+ * - Set up reading a stream from the start.
+ *
+ * The crux of this logic revolves around the stream_continue() function. It
+ * chooses the next action to perform. Valid actions are:
+ * - Read another record from the stream.
+ * - Process a buffered record.
+ *
+ * There are several chains of event:
+ *
+ * 1) Starting a stream follows:
+ * - libxl__stream_read_start()
+ * - stream_header_done()
+ * - stream_continue()
+ *
+ * 2) Reading a record follows:
+ * - stream_continue()
+ * - record_header_done()
+ * - record_body_done()
+ * - stream_continue()
+ *
+ * 3) Processing a record had several chains to follow, depending on the
+ * record in question.
+ * 3a) "Simple" record:
+ * - process_record()
+ * - stream_continue()
+ * 3b) LIBXC record:
+ * - process_record()
+ * - libxl__xc_domain_restore()
+ * - libxl__xc_domain_restore_done()
+ * - stream_continue()
+ * 3c) EMULATOR record:
+ * - process_record()
+ * - stream_write_emulator()
+ * - stream_write_emulator_done()
+ * - stream_continue()
+ */
+
+/* Stream error/success handling. */
+static void stream_success(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+static void stream_failed(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
+static void stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+
+/* Event callbacks for main reading loop. */
+static void stream_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+
+/* Event chain for reading a record from the stream. */
+static void setup_read_record(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+static void record_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+static void record_body_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+
+static bool process_record(libxl__egc *egc,
+ libxl__stream_read_state *stream);
+
+/* Event chain for writing an emulator blob. */
+static void write_emulator_blob(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ libxl__sr_record_buf *rec);
+static void write_emulator_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+
+/* Helper to set up reading some data from the stream. */
+static int setup_read(libxl__stream_read_state *stream,
+ const char *what, void *ptr, size_t nr_bytes,
+ libxl__datacopier_callback cb)
+{
+ libxl__datacopier_state *dc = &stream->dc;
+
+ dc->readwhat = what;
+ dc->readbuf = ptr;
+ dc->bytes_to_read = nr_bytes;
+ dc->used = 0;
+ dc->callback = cb;
+
+ return libxl__datacopier_start(dc);
+}
+
+void libxl__stream_read_start(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ libxl__datacopier_state *dc = &stream->dc;
+ int ret = 0;
+
+ /* State initialisation. */
+ assert(!stream->running);
+
+ memset(dc, 0, sizeof(*dc));
+ dc->ao = stream->ao;
+ dc->readfd = stream->fd;
+ dc->writefd = -1;
+
+ /* Start reading the stream header. */
+ ret = setup_read(stream, "stream header",
+ &stream->hdr, sizeof(stream->hdr),
+ stream_header_done);
+ if (ret)
+ goto err;
+
+ stream->running = true;
+ stream->phase = SRS_PHASE_NORMAL;
+ LIBXL_STAILQ_INIT(&stream->record_queue);
+ stream->recursion_guard = 0;
+
+ assert(!ret);
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+void libxl__stream_read_abort(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc)
+{
+ stream_failed(egc, stream, rc);
+}
+
+static void stream_success(libxl__egc *egc, libxl__stream_read_state *stream)
+{
+ stream->rc = 0;
+
+ stream_done(egc, stream);
+}
+
+static void stream_failed(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc)
+{
+ assert(rc);
+ stream->rc = rc;
+
+ if (stream->running) {
+ stream_done(egc, stream);
+ }
+}
+
+static void stream_done(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ libxl__sr_record_buf *rec, *trec;
+
+ assert(stream->running);
+ stream->running = false;
+
+ if (stream->emu_carefd)
+ libxl__carefd_close(stream->emu_carefd);
+
+ LIBXL_STAILQ_FOREACH_SAFE(rec, &stream->record_queue, entry, trec) {
+ free(rec->body);
+ free(rec);
+ }
+
+ stream->completion_callback(egc, stream, stream->rc);
+}
+
+static void stream_continue(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ STATE_AO_GC(stream->ao);
+
+ /* Must not mutually recurse with process_record() */
+ assert(stream->recursion_guard == false);
+ stream->recursion_guard = true;
+
+ switch (stream->phase) {
+ case SRS_PHASE_NORMAL:
+ /*
+ * Normal phase of the stream. We arrive here in several senarios.
+ * Alternate between reading another record from the stream, and
+ * processing the record. At no point should there ever be two
+ * records in the queue.
+ */
+ if (LIBXL_STAILQ_EMPTY(&stream->record_queue))
+ setup_read_record(egc, stream);
+ else {
+ if (process_record(egc, stream))
+ setup_read_record(egc, stream);
+ }
+ break;
+
+ default:
+ abort();
+ }
+
+ assert(stream->recursion_guard == true);
+ stream->recursion_guard = false;
+}
+
+static void stream_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
+ libxl__sr_hdr *hdr = &stream->hdr;
+ STATE_AO_GC(dc->ao);
+ int ret = 0;
+
+ if (rc || onwrite || errnoval) {
+ ret = ERROR_FAIL;
+ LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
+ goto err;
+ }
+
+ hdr->ident = be64toh(hdr->ident);
+ hdr->version = be32toh(hdr->version);
+ hdr->options = be32toh(hdr->options);
+
+ if (hdr->ident != RESTORE_STREAM_IDENT) {
+ ret = ERROR_FAIL;
+ LOG(ERROR,
+ "Invalid ident: expected 0x%016"PRIx64", got 0x%016"PRIx64,
+ RESTORE_STREAM_IDENT, hdr->ident);
+ goto err;
+ }
+ if (hdr->version != RESTORE_STREAM_VERSION) {
+ ret = ERROR_FAIL;
+ LOG(ERROR, "Unexpected Version: expected %u, got %u",
+ RESTORE_STREAM_VERSION, hdr->version);
+ goto err;
+ }
+ if (hdr->options & RESTORE_OPT_BIG_ENDIAN) {
+ ret = ERROR_FAIL;
+ LOG(ERROR, "Unable to handle big endian streams");
+ goto err;
+ }
+
+ LOG(DEBUG, "Stream v%u%s", hdr->version,
+ hdr->options & RESTORE_OPT_LEGACY ? " (from legacy)" : "");
+
+ stream_continue(egc, stream);
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+static void setup_read_record(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ STATE_AO_GC(stream->ao);
+ libxl__sr_record_buf *rec = NULL;
+ int ret;
+
+ assert(stream->incoming_record == NULL);
+
+ stream->incoming_record = rec = libxl__zalloc(NOGC, sizeof(*rec));
+ ret = setup_read(stream, "record header",
+ &rec->hdr, sizeof(rec->hdr),
+ record_header_done);
+ if (ret)
+ goto err;
+
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+static void record_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
+ libxl__sr_record_buf *rec = stream->incoming_record;
+ STATE_AO_GC(dc->ao);
+ int ret = 0;
+
+ if (rc || onwrite || errnoval) {
+ ret = ERROR_FAIL;
+ LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
+ goto err;
+ }
+
+ /* No body? All done. */
+ if (rec->hdr.length == 0) {
+ record_body_done(egc, dc, 0, 0, 0);
+ return;
+ }
+
+ size_t bytes_to_read = ROUNDUP(rec->hdr.length, REC_ALIGN_ORDER);
+ rec->body = libxl__malloc(NOGC, bytes_to_read);
+
+ ret = setup_read(stream, "record body",
+ rec->body, bytes_to_read,
+ record_body_done);
+ if (ret)
+ goto err;
+
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+static void record_body_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
+ libxl__sr_record_buf *rec = stream->incoming_record;
+ STATE_AO_GC(dc->ao);
+ int ret = 0;
+
+ if (rc || onwrite || errnoval) {
+ ret = ERROR_FAIL;
+ LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
+ goto err;
+ }
+
+ LIBXL_STAILQ_INSERT_TAIL(&stream->record_queue, rec, entry);
+ stream->incoming_record = NULL;
+
+ stream_continue(egc, stream);
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+/*
+ * Returns a boolean indicating whether a further action should be set up by
+ * the caller. This is needed to prevent mutual recursion with
+ * stream_continue().
+ */
+static bool process_record(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ STATE_AO_GC(stream->ao);
+ libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
+ libxl__sr_record_buf *rec;
+ bool further_action_needed = false;
+ int ret = 0;
+
+ /* Pop a record from the head of the queue. */
+ assert(!LIBXL_STAILQ_EMPTY(&stream->record_queue));
+ rec = LIBXL_STAILQ_FIRST(&stream->record_queue);
+ LIBXL_STAILQ_REMOVE_HEAD(&stream->record_queue, entry);
+
+ LOG(DEBUG, "Record: %u, length %u", rec->hdr.type, rec->hdr.length);
+
+ switch (rec->hdr.type) {
+
+ case REC_TYPE_END:
+ stream_success(egc, stream);
+ break;
+
+ case REC_TYPE_XENSTORE_DATA:
+ ret = libxl__toolstack_restore(dcs->guest_domid, rec->body,
+ rec->hdr.length, &dcs->shs);
+ if (ret)
+ goto err;
+
+ /*
+ * libxl__toolstack_restore() is a synchronous function. Request
+ * that our caller queues another action for us.
+ */
+ further_action_needed = true;
+ break;
+
+ case REC_TYPE_EMULATOR_CONTEXT:
+ write_emulator_blob(egc, stream, rec);
+ break;
+
+ default:
+ LOG(ERROR, "Unrecognised record 0x%08x", rec->hdr.type);
+ ret = ERROR_FAIL;
+ goto err;
+ }
+
+ assert(!ret);
+ free(rec->body);
+ free(rec);
+ return further_action_needed;
+
+ err:
+ assert(ret);
+ free(rec->body);
+ free(rec);
+ stream_failed(egc, stream, ret);
+ return false;
+}
+
+static void write_emulator_blob(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ libxl__sr_record_buf *rec)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
+ libxl__datacopier_state *dc = &stream->emu_dc;
+ libxl__sr_emulator_hdr *emu_hdr;
+ STATE_AO_GC(stream->ao);
+ char path[256];
+ int ret = 0, writefd = -1;
+
+ if ( rec->hdr.length < sizeof(*emu_hdr) ) {
+ ret = ERROR_FAIL;
+ LOG(ERROR, "Emulator record too short to contain header");
+ goto err;
+ }
+ emu_hdr = rec->body;
+
+ sprintf(path, XC_DEVICE_MODEL_RESTORE_FILE".%u", dcs->guest_domid);
+
+ libxl__carefd_begin();
+ writefd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
+ if (writefd == -1) {
+ ret = ERROR_FAIL;
+ LOGE(ERROR, "unable to open %s", path);
+ libxl__carefd_unlock();
+ goto err;
+ }
+ stream->emu_carefd = libxl__carefd_record(CTX, writefd);
+ libxl__carefd_unlock();
+
+
+ memset(dc, 0, sizeof(*dc));
+ dc->ao = stream->ao;
+ dc->writewhat = "qemu save file";
+ dc->writefd = writefd;
+ dc->maxsz = -1;
+ dc->callback = write_emulator_done;
+
+ ret = libxl__datacopier_start(dc);
+ if (ret)
+ goto err;
+
+ libxl__datacopier_prefixdata(egc, dc,
+ rec->body + sizeof(*emu_hdr),
+ rec->hdr.length - sizeof(*emu_hdr));
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+static void write_emulator_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, emu_dc);
+ STATE_AO_GC(dc->ao);
+ int ret = 0;
+
+ libxl__carefd_close(stream->emu_carefd);
+ stream->emu_carefd = NULL;
+
+ if (rc || errnoval) {
+ ret = ERROR_FAIL;
+ LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
+ goto err;
+ }
+
+ stream_continue(egc, stream);
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 17/27] tools/libxl: Support converting a legacy stream to a v2 stream
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (15 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 10:28 ` Ian Campbell
2015-07-10 12:28 ` Ian Jackson
2015-07-09 18:26 ` [PATCH v2 18/27] tools/libxl: Convert a legacy stream if needed Andrew Cooper
` (10 subsequent siblings)
27 siblings, 2 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
When a legacy stream is found, it needs to be converted to a v2 stream for the
reading logic. This is done by exec()ing the python conversion utility.
One complication is that the caller of this interface needs to assume
ownership of the output fd, to prevent it being closed while still in use in a
datacopier.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
No major differences since v1. It has gained a new ao_abrt as a result
of rebasing over the AO ABORT series, and has been made x86-specific.
---
tools/libxl/Makefile | 1 +
tools/libxl/libxl_convert_callout.c | 172 +++++++++++++++++++++++++++++++++++
tools/libxl/libxl_internal.h | 48 ++++++++++
3 files changed, 221 insertions(+)
create mode 100644 tools/libxl/libxl_convert_callout.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index c71c5fe..0ebc35a 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -59,6 +59,7 @@ endif
LIBXL_OBJS-y += libxl_remus_device.o libxl_remus_disk_drbd.o
LIBXL_OBJS-$(CONFIG_X86) += libxl_cpuid.o libxl_x86.o libxl_psr.o
+LIBXL_OBJS-$(CONFIG_X86) += libxl_convert_callout.o
LIBXL_OBJS-$(CONFIG_ARM) += libxl_nocpuid.o libxl_arm.o libxl_libfdt_compat.o
ifeq ($(CONFIG_NetBSD),y)
diff --git a/tools/libxl/libxl_convert_callout.c b/tools/libxl/libxl_convert_callout.c
new file mode 100644
index 0000000..02f3b82
--- /dev/null
+++ b/tools/libxl/libxl_convert_callout.c
@@ -0,0 +1,172 @@
+/*
+ * Copyright (C) 2014 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h"
+
+#include "libxl_internal.h"
+
+/*
+ * Infrastructure for converting a legacy migration stream into a libxl v2
+ * stream.
+ *
+ * This is done by fork()ing the python conversion script, which takes in a
+ * legacy stream, and puts out a suitably-formatted v2 stream.
+ */
+
+static void helper_failed(libxl__egc *egc,
+ libxl__conversion_helper_state *chs, int rc);
+static void helper_stop(libxl__egc *egc, libxl__ao_abortable*, int rc);
+static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
+ pid_t pid, int status);
+static void helper_done(libxl__egc *egc,
+ libxl__conversion_helper_state *chs);
+
+void libxl__conversion_helper_init(libxl__conversion_helper_state *chs)
+{
+ libxl__ao_abortable_init(&chs->abrt);
+ libxl__ev_child_init(&chs->child);
+}
+
+void libxl__convert_legacy_stream(libxl__egc *egc,
+ libxl__conversion_helper_state *chs)
+{
+ STATE_AO_GC(chs->ao);
+ libxl__carefd *child_in = NULL, *child_out = NULL;
+ int ret = 0;
+
+ chs->rc = 0;
+ libxl__conversion_helper_init(chs);
+
+ chs->abrt.ao = chs->ao;
+ chs->abrt.callback = helper_stop;
+ ret = libxl__ao_abortable_register(&chs->abrt);
+ if (ret) goto err;
+
+ libxl__carefd_begin();
+ int fds[2];
+ if (libxl_pipe(CTX, fds)) {
+ ret = ERROR_FAIL;
+ libxl__carefd_unlock();
+ goto err;
+ }
+ child_out = libxl__carefd_record(CTX, fds[0]);
+ child_in = libxl__carefd_record(CTX, fds[1]);
+ libxl__carefd_unlock();
+
+ pid_t pid = libxl__ev_child_fork(gc, &chs->child, helper_exited);
+ if (!pid) {
+ char * const args[] =
+ {
+ getenv("LIBXL_CONVERT_HELPER") ?:
+ LIBEXEC_BIN "/convert-legacy-stream",
+ "--in", GCSPRINTF("%d", chs->legacy_fd),
+ "--out", GCSPRINTF("%d", fds[1]),
+ "--width",
+#ifdef __i386__
+ "32",
+#else
+ "64",
+#endif
+ "--guest", chs->hvm ? "hvm" : "pv",
+ "--format", "libxl",
+ /* "--verbose", */
+ NULL,
+ };
+
+ libxl_fd_set_cloexec(CTX, chs->legacy_fd, 0);
+ libxl_fd_set_cloexec(CTX, libxl__carefd_fd(child_in), 0);
+
+ libxl__exec(gc,
+ -1, -1, -1,
+ args[0], args, NULL);
+ }
+
+ libxl__carefd_close(child_in);
+ chs->v2_carefd = child_out;
+
+ assert(!ret);
+ return;
+
+ err:
+ assert(ret);
+ helper_failed(egc, chs, ret);
+}
+
+void libxl__convert_legacy_stream_abort(libxl__egc *egc,
+ libxl__conversion_helper_state *chs,
+ int rc)
+{
+ helper_stop(egc, &chs->abrt, rc);
+}
+
+static void helper_failed(libxl__egc *egc,
+ libxl__conversion_helper_state *chs, int rc)
+{
+ STATE_AO_GC(chs->ao);
+
+ if (!chs->rc)
+ chs->rc = rc;
+
+ if (!libxl__ev_child_inuse(&chs->child)) {
+ helper_done(egc, chs);
+ return;
+ }
+
+ libxl__kill(gc, chs->child.pid, SIGKILL, "conversion helper");
+}
+
+static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
+{
+ libxl__conversion_helper_state *chs = CONTAINER_OF(abrt, *chs, abrt);
+ STATE_AO_GC(chs->ao);
+
+ if (!chs->rc)
+ chs->rc = rc;
+
+ if (!libxl__ev_child_inuse(&chs->child)) {
+ helper_done(egc, chs);
+ return;
+ }
+
+ libxl__kill(gc, chs->child.pid, SIGTERM, "conversion helper");
+}
+
+static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
+ pid_t pid, int status)
+{
+ libxl__conversion_helper_state *chs = CONTAINER_OF(ch, *chs, child);
+ STATE_AO_GC(chs->ao);
+
+ if (status) {
+ libxl_report_child_exitstatus(CTX, XTL_ERROR, "conversion helper",
+ pid, status);
+ chs->rc = ERROR_FAIL;
+ }
+ else
+ chs->rc = 0;
+
+ helper_done(egc, chs);
+}
+
+static void helper_done(libxl__egc *egc,
+ libxl__conversion_helper_state *chs)
+{
+ STATE_AO_GC(chs->ao);
+
+ assert(!libxl__ev_child_inuse(&chs->child));
+
+ libxl__ao_abortable_deregister(&chs->abrt);
+
+ chs->completion_callback(egc, chs, chs->rc);
+}
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 0f17e7c..68e7f02 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2917,6 +2917,53 @@ _hidden void libxl__remus_devices_commit(libxl__egc *egc,
libxl__remus_devices_state *rds);
_hidden int libxl__netbuffer_enabled(libxl__gc *gc);
+/*----- Legacy conversion helper -----*/
+typedef struct libxl__conversion_helper_state libxl__conversion_helper_state;
+
+struct libxl__conversion_helper_state {
+ /* public */
+ libxl__ao *ao;
+ int legacy_fd;
+ bool hvm; /* pv or hvm domain? */
+ libxl__carefd *v2_carefd; /* Filled by successful call to
+ * libxl__convert_legacy_stream(). Caller
+ * assumes ownership of the fd. */
+ void (*completion_callback)(
+ libxl__egc *egc, libxl__conversion_helper_state *chs, int rc);
+ /* private */
+ int rc;
+ libxl__ao_abortable abrt;
+ libxl__ev_child child;
+};
+
+#if defined(__x86_64__) || defined(__i386__)
+_hidden void libxl__convert_legacy_stream(libxl__egc *egc,
+ libxl__conversion_helper_state *chs);
+
+_hidden void libxl__conversion_helper_init(
+ libxl__conversion_helper_state *chs);
+
+_hidden void libxl__convert_legacy_stream_abort(
+ libxl__egc *egc, libxl__conversion_helper_state *chs, int rc);
+
+static inline bool libxl__convert_legacy_stream_inuse(
+ libxl__conversion_helper_state *chs)
+{
+ return libxl__ev_child_inuse(&chs->child);
+}
+#else
+/* Stubs for non-x86 architecture to reduce code #ifdefary */
+static inline void libxl__conversion_helper_init(
+ libxl__conversion_helper_state *chs) { }
+
+static inline void libxl__convert_legacy_stream_abort(
+ libxl__egc *egc, libxl__conversion_helper_state *chs, int rc) {}
+
+static inline bool libxl__convert_legacy_stream_inuse(
+ libxl__conversion_helper_state *chs) { return 0; }
+#endif
+
+
/*----- Domain suspend (save) state structure -----*/
typedef struct libxl__domain_suspend_state libxl__domain_suspend_state;
@@ -3283,6 +3330,7 @@ struct libxl__domain_create_state {
* for the non-stubdom device model. */
libxl__stream_read_state srs;
libxl__save_helper_state shs;
+ libxl__conversion_helper_state chs;
/* necessary if the domain creation failed and we have to destroy it */
libxl__domain_destroy_state dds;
libxl__multidev multidev;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 18/27] tools/libxl: Convert a legacy stream if needed
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (16 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 17/27] tools/libxl: Support converting a legacy stream to a " Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 10:31 ` Ian Campbell
2015-07-10 12:41 ` Ian Jackson
2015-07-09 18:26 ` [PATCH v2 19/27] tools/libxc+libxl+xl: Restore v2 streams Andrew Cooper
` (9 subsequent siblings)
27 siblings, 2 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
For backwards compatibility, a legacy stream needs converting before it can be
read by the v2 stream logic.
This causes the v2 stream logic to need to juggle two parallel tasks.
check_stream_finished() is introduced for the purpose of joining the tasks in
both success and error cases.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl_internal.h | 7 +++
tools/libxl/libxl_stream_read.c | 98 ++++++++++++++++++++++++++++++++++++++-
2 files changed, 104 insertions(+), 1 deletion(-)
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 68e7f02..1cf1884 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3274,6 +3274,7 @@ struct libxl__stream_read_state {
/* filled by the user */
libxl__ao *ao;
int fd;
+ bool legacy;
void (*completion_callback)(libxl__egc *egc,
libxl__stream_read_state *srs,
int rc);
@@ -3281,6 +3282,12 @@ struct libxl__stream_read_state {
int rc;
bool running;
+ /* Active-stuff handling */
+ int joined_rc;
+
+ /* Conversion helper */
+ libxl__carefd *v2_carefd;
+
/* Main stream-reading data */
libxl__datacopier_state dc;
libxl__sr_hdr hdr;
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index 6f5d572..3011820 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -57,6 +57,10 @@
* - stream_write_emulator()
* - stream_write_emulator_done()
* - stream_continue()
+ *
+ * Depending on the contents of the stream, there are likely to be several
+ * parallel tasks being managed. check_stream_finished() is used to join all
+ * tasks in both success and error cases.
*/
/* Stream error/success handling. */
@@ -67,6 +71,16 @@ static void stream_failed(libxl__egc *egc,
static void stream_done(libxl__egc *egc,
libxl__stream_read_state *stream);
+/* Stream other-active-stuff handling. */
+/* libxl__xc_domain_restore_done() is logically grouped here. */
+#if defined(__x86_64__) || defined(__i386__)
+static void conversion_done(libxl__egc *egc,
+ libxl__conversion_helper_state *chs, int rc);
+#endif
+static void check_stream_finished(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ int rc, const char *what);
+
/* Event callbacks for main reading loop. */
static void stream_header_done(libxl__egc *egc,
libxl__datacopier_state *dc,
@@ -112,12 +126,40 @@ static int setup_read(libxl__stream_read_state *stream,
void libxl__stream_read_start(libxl__egc *egc,
libxl__stream_read_state *stream)
{
+ libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
libxl__datacopier_state *dc = &stream->dc;
+ STATE_AO_GC(stream->ao);
int ret = 0;
/* State initialisation. */
assert(!stream->running);
+ /*
+ * Initialise other moving parts so stream_check_finished() can correctly
+ * work out whether to tear them down.
+ */
+ libxl__conversion_helper_init(&dcs->chs);
+
+#if defined(__x86_64__) || defined(__i386__)
+ if (stream->legacy) {
+ /* Convert a legacy stream, if needed. */
+ dcs->chs.ao = stream->ao;
+ dcs->chs.legacy_fd = stream->fd;
+ dcs->chs.hvm =
+ (dcs->guest_config->b_info.type == LIBXL_DOMAIN_TYPE_HVM);
+ dcs->chs.v2_carefd = NULL;
+ dcs->chs.completion_callback = conversion_done;
+
+ libxl__convert_legacy_stream(egc, &dcs->chs);
+
+ assert(dcs->chs.v2_carefd);
+ stream->v2_carefd = dcs->chs.v2_carefd;
+ dcs->libxc_fd = stream->fd = libxl__carefd_fd(dcs->chs.v2_carefd);
+ }
+#endif
+
+ /* stream->fd is now guarenteed to be a v2 stream. */
+
memset(dc, 0, sizeof(*dc));
dc->ao = stream->ao;
dc->readfd = stream->fd;
@@ -183,7 +225,50 @@ static void stream_done(libxl__egc *egc,
free(rec);
}
- stream->completion_callback(egc, stream, stream->rc);
+ if (stream->v2_carefd)
+ libxl__carefd_close(stream->v2_carefd);
+
+ check_stream_finished(egc, stream, stream->rc, "stream");
+}
+
+static void check_stream_finished(libxl__egc *egc,
+ libxl__stream_read_state *stream,
+ int rc, const char *what)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
+ STATE_AO_GC(stream->ao);
+
+ LOG(DEBUG, "Task '%s' joining (rc %d)", what, rc);
+
+ if (rc && !stream->joined_rc) {
+ bool skip = false;
+ /* First reported failure from joining tasks. Tear everything down */
+ stream->joined_rc = rc;
+
+ if (libxl__stream_read_inuse(&dcs->srs)) {
+ skip = true;
+ libxl__stream_read_abort(egc, &dcs->srs, rc);
+ }
+
+ if (libxl__convert_legacy_stream_inuse(&dcs->chs)) {
+ skip = true;
+ libxl__convert_legacy_stream_abort(egc, &dcs->chs, rc);
+ }
+
+ /* There is at least one more active task to join - wait for its
+ callback */
+ if ( skip )
+ return;
+ }
+
+ if (libxl__stream_read_inuse(&dcs->srs))
+ LOG(DEBUG, "stream still in use");
+ else if (libxl__convert_legacy_stream_inuse(&dcs->chs))
+ LOG(DEBUG, "conversion still in use");
+ else {
+ LOG(DEBUG, "Join complete: result %d", stream->joined_rc);
+ stream->completion_callback(egc, stream, stream->joined_rc);
+ }
}
static void stream_continue(libxl__egc *egc,
@@ -354,6 +439,17 @@ static void record_body_done(libxl__egc *egc,
stream_failed(egc, stream, ret);
}
+#if defined(__x86_64__) || defined(__i386__)
+static void conversion_done(libxl__egc *egc,
+ libxl__conversion_helper_state *chs, int rc)
+{
+ STATE_AO_GC(chs->ao);
+ libxl__domain_create_state *dcs = CONTAINER_OF(chs, *dcs, chs);
+
+ check_stream_finished(egc, &dcs->srs, rc, "conversion");
+}
+#endif
+
/*
* Returns a boolean indicating whether a further action should be set up by
* the caller. This is needed to prevent mutual recursion with
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 19/27] tools/libxc+libxl+xl: Restore v2 streams
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (17 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 18/27] tools/libxl: Convert a legacy stream if needed Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 10:45 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 20/27] tools/libxl: Infrastructure for writing a v2 stream Andrew Cooper
` (8 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
This is a complicated set of changes which must be done together for
bisectability.
* libxl-save-helper is updated to unconditionally use libxc migration v2.
* libxl compatibility workarounds in libxc are disabled for restore operations.
* libxl__stream_read_start() is logically spliced into the event location
where libxl__xc_domain_restore() used to reside.
The parameters 'hvm', 'pae', and 'superpages' were previously superfluous, and
are completely unused in migration v2. callbacks->toolstack_restore is handled
via a migration v2 record now, rather than via a callback from libxc.
NB: this change breaks Remus. Further untangling needs to happen before Remus
will function.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
Since v1:
* Drop "legacy_width" from the IDL
* Gain a LIBXL_HAVE_ to signify support of migration v2 streams
---
tools/libxc/Makefile | 4 ++--
tools/libxl/libxl.h | 17 ++++++++++++++
tools/libxl/libxl_create.c | 48 ++++++++++++---------------------------
tools/libxl/libxl_save_helper.c | 2 +-
tools/libxl/libxl_stream_read.c | 34 +++++++++++++++++++++++++++
tools/libxl/libxl_types.idl | 1 +
tools/libxl/xl_cmdimpl.c | 7 +++++-
7 files changed, 76 insertions(+), 37 deletions(-)
diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
index b659df4..2cd0b1a 100644
--- a/tools/libxc/Makefile
+++ b/tools/libxc/Makefile
@@ -64,8 +64,8 @@ GUEST_SRCS-$(CONFIG_X86) += xc_sr_save_x86_hvm.c
GUEST_SRCS-y += xc_sr_restore.c
GUEST_SRCS-y += xc_sr_save.c
GUEST_SRCS-y += xc_offline_page.c xc_compression.c
-$(patsubst %.c,%.o,$(GUEST_SRCS-y)): CFLAGS += -DXG_LIBXL_HVM_COMPAT
-$(patsubst %.c,%.opic,$(GUEST_SRCS-y)): CFLAGS += -DXG_LIBXL_HVM_COMPAT
+xc_sr_save_x86_hvm.o: CFLAGS += -DXG_LIBXL_HVM_COMPAT
+xc_sr_save_x86_hvm.opic: CFLAGS += -DXG_LIBXL_HVM_COMPAT
else
GUEST_SRCS-y += xc_nomigrate.c
endif
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index e9d63c9..e64a606 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -807,6 +807,23 @@
*/
#define LIBXL_HAVE_SOCKET_BITMAP_ALLOC 1
+/*
+ * LIBXL_HAVE_STREAM_V2
+ *
+ * If this is defined, then the libxl_domain_create_restore() interface takes
+ * a "stream_version" parameter and supports a value of 2.
+ */
+#define LIBXL_HAVE_STREAM_V2 1
+
+/*
+ * LIBXL_HAVE_STREAM_V1
+ *
+ * In the case that LIBXL_HAVE_STREAM_V2 is set, LIBXL_HAVE_STREAM_V1
+ * indicates that libxl_domain_create_restore() can handle a "stream_version"
+ * parameter of 1, and convert the stream format automatically.
+ */
+#define LIBXL_HAVE_STREAM_V1 1
+
typedef char **libxl_string_list;
void libxl_string_list_dispose(libxl_string_list *sl);
int libxl_string_list_length(const libxl_string_list *sl);
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index be13204..2a0063a 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -704,6 +704,10 @@ static void domcreate_attach_dtdev(libxl__egc *egc,
static void domcreate_console_available(libxl__egc *egc,
libxl__domain_create_state *dcs);
+static void domcreate_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int ret);
+
static void domcreate_rebuild_done(libxl__egc *egc,
libxl__domain_create_state *dcs,
int ret);
@@ -933,11 +937,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
/* convenience aliases */
const uint32_t domid = dcs->guest_domid;
libxl_domain_config *const d_config = dcs->guest_config;
- libxl_domain_build_info *const info = &d_config->b_info;
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
- libxl__srm_restore_autogen_callbacks *const callbacks =
- &dcs->shs.callbacks.restore.a;
if (rc) {
domcreate_rebuild_done(egc, dcs, rc);
@@ -970,30 +971,16 @@ static void domcreate_bootloader_done(libxl__egc *egc,
if (rc)
goto out;
- /* read signature */
- int hvm, pae, superpages;
- switch (info->type) {
- case LIBXL_DOMAIN_TYPE_HVM:
- hvm = 1;
- superpages = 1;
- pae = libxl_defbool_val(info->u.hvm.pae);
- callbacks->toolstack_restore = libxl__toolstack_restore;
- break;
- case LIBXL_DOMAIN_TYPE_PV:
- hvm = 0;
- superpages = 0;
- pae = 1;
- break;
- default:
- rc = ERROR_INVAL;
- goto out;
- }
- libxl__xc_domain_restore(egc, dcs,
- hvm, pae, superpages);
+ dcs->srs.ao = ao;
+ dcs->srs.fd = restore_fd;
+ dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
+ dcs->srs.completion_callback = domcreate_stream_done;
+
+ libxl__stream_read_start(egc, &dcs->srs);
return;
out:
- libxl__xc_domain_restore_done(egc, dcs, rc, 0, 0);
+ domcreate_stream_done(egc, &dcs->srs, rc);
}
void libxl__srm_callout_callback_restore_results(unsigned long store_mfn,
@@ -1009,10 +996,11 @@ void libxl__srm_callout_callback_restore_results(unsigned long store_mfn,
shs->need_results = 0;
}
-void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
- int ret, int retval, int errnoval)
+static void domcreate_stream_done(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int ret)
{
- libxl__domain_create_state *dcs = dcs_void;
+ libxl__domain_create_state *dcs = CONTAINER_OF(srs, *dcs, srs);
STATE_AO_GC(dcs->ao);
libxl_ctx *ctx = libxl__gc_owner(gc);
char **vments = NULL, **localents = NULL;
@@ -1029,12 +1017,6 @@ void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
if (ret)
goto out;
- if (retval) {
- LOGEV(ERROR, errnoval, "restoring domain");
- ret = ERROR_FAIL;
- goto out;
- }
-
gettimeofday(&start_time, NULL);
switch (info->type) {
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index 14675ae..efbe2eb 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -313,7 +313,7 @@ int main(int argc, char **argv)
startup("restore");
setup_signals(SIG_DFL);
- r = xc_domain_restore(xch, io_fd, dom, store_evtchn, &store_mfn,
+ r = xc_domain_restore2(xch, io_fd, dom, store_evtchn, &store_mfn,
store_domid, console_evtchn, &console_mfn,
console_domid, hvm, pae, superpages,
checkpointed,
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index 3011820..81095cd 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -139,6 +139,7 @@ void libxl__stream_read_start(libxl__egc *egc,
* work out whether to tear them down.
*/
libxl__conversion_helper_init(&dcs->chs);
+ libxl__save_helper_init(&dcs->shs);
#if defined(__x86_64__) || defined(__i386__)
if (stream->legacy) {
@@ -255,6 +256,11 @@ static void check_stream_finished(libxl__egc *egc,
libxl__convert_legacy_stream_abort(egc, &dcs->chs, rc);
}
+ if (libxl__save_helper_inuse(&dcs->shs)) {
+ skip = true;
+ libxl__save_helper_abort(egc, &dcs->shs);
+ }
+
/* There is at least one more active task to join - wait for its
callback */
if ( skip )
@@ -265,6 +271,8 @@ static void check_stream_finished(libxl__egc *egc,
LOG(DEBUG, "stream still in use");
else if (libxl__convert_legacy_stream_inuse(&dcs->chs))
LOG(DEBUG, "conversion still in use");
+ else if (libxl__save_helper_inuse(&dcs->shs))
+ LOG(DEBUG, "save/restore still in use");
else {
LOG(DEBUG, "Join complete: result %d", stream->joined_rc);
stream->completion_callback(egc, stream, stream->joined_rc);
@@ -439,6 +447,28 @@ static void record_body_done(libxl__egc *egc,
stream_failed(egc, stream, ret);
}
+void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
+ int ret, int retval, int errnoval)
+{
+ libxl__domain_create_state *dcs = dcs_void;
+ STATE_AO_GC(dcs->ao);
+
+ if (ret)
+ goto err;
+
+ if (retval) {
+ LOGEV(ERROR, errnoval, "restoring domain");
+ ret = ERROR_FAIL;
+ goto err;
+ }
+
+ stream_continue(egc, &dcs->srs);
+ return;
+
+ err:
+ check_stream_finished(egc, &dcs->srs, ret, "save/restore helper");
+}
+
#if defined(__x86_64__) || defined(__i386__)
static void conversion_done(libxl__egc *egc,
libxl__conversion_helper_state *chs, int rc)
@@ -477,6 +507,10 @@ static bool process_record(libxl__egc *egc,
stream_success(egc, stream);
break;
+ case REC_TYPE_LIBXC_CONTEXT:
+ libxl__xc_domain_restore(egc, dcs, 0, 0, 0);
+ break;
+
case REC_TYPE_XENSTORE_DATA:
ret = libxl__toolstack_restore(dcs->guest_domid, rec->body,
rec->hdr.length, &dcs->shs);
diff --git a/tools/libxl/libxl_types.idl b/tools/libxl/libxl_types.idl
index 8dacf8d..bc0c4ef 100644
--- a/tools/libxl/libxl_types.idl
+++ b/tools/libxl/libxl_types.idl
@@ -355,6 +355,7 @@ libxl_domain_create_info = Struct("domain_create_info",[
libxl_domain_restore_params = Struct("domain_restore_params", [
("checkpointed_stream", integer),
+ ("stream_version", uint32, {'init_val': '1'}),
])
libxl_domain_sched_params = Struct("domain_sched_params",[
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 26b1e7d..71a22b8 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -110,7 +110,9 @@
#define XL_MANDATORY_FLAG_JSON (1U << 0) /* config data is in JSON format */
#define XL_MANDATORY_FLAG_STREAMv2 (1U << 1) /* stream is v2 */
-#define XL_MANDATORY_FLAG_ALL (XL_MANDATORY_FLAG_JSON)
+#define XL_MANDATORY_FLAG_ALL (XL_MANDATORY_FLAG_JSON | \
+ XL_MANDATORY_FLAG_STREAMv2)
+
struct save_file_header {
char magic[32]; /* savefileheader_magic */
/* All uint32_ts are in domain's byte order. */
@@ -2757,6 +2759,9 @@ static uint32_t create_domain(struct domain_create *dom_info)
libxl_domain_restore_params_init(¶ms);
params.checkpointed_stream = dom_info->checkpointed_stream;
+ params.stream_version =
+ (hdr.mandatory_flags & XL_MANDATORY_FLAG_STREAMv2) ? 2 : 1;
+
ret = libxl_domain_create_restore(ctx, &d_config,
&domid, restore_fd,
¶ms,
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 20/27] tools/libxl: Infrastructure for writing a v2 stream
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (18 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 19/27] tools/libxc+libxl+xl: Restore v2 streams Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 11:10 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 21/27] tools/libxc+libxl+xl: Save v2 streams Andrew Cooper
` (7 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel
Cc: Ross Lagerwall, Wei Liu, Ian Jackson, Ian Campbell, Andrew Cooper
From: Ross Lagerwall <ross.lagerwall@citrix.com>
This contains the event machinary and state machines to write non-checkpointed
migration v2 stream (with the exception of the xc_domain_save() handling which
is spliced later in a bisectable way).
Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
As with the read side of things, this has undergone substantial changes in v2.
---
tools/libxl/Makefile | 2 +-
tools/libxl/libxl_internal.h | 47 ++++
tools/libxl/libxl_stream_write.c | 451 ++++++++++++++++++++++++++++++++++++++
3 files changed, 499 insertions(+), 1 deletion(-)
create mode 100644 tools/libxl/libxl_stream_write.c
diff --git a/tools/libxl/Makefile b/tools/libxl/Makefile
index 0ebc35a..7d44483 100644
--- a/tools/libxl/Makefile
+++ b/tools/libxl/Makefile
@@ -95,7 +95,7 @@ LIBXL_OBJS = flexarray.o libxl.o libxl_create.o libxl_dm.o libxl_pci.o \
libxl_dom.o libxl_exec.o libxl_xshelp.o libxl_device.o \
libxl_internal.o libxl_utils.o libxl_uuid.o \
libxl_json.o libxl_aoutils.o libxl_numa.o libxl_vnuma.o \
- libxl_stream_read.o \
+ libxl_stream_read.o libxl_stream_write.o \
libxl_save_callout.o _libxl_save_msgs_callout.o \
libxl_qmp.o libxl_event.o libxl_fork.o $(LIBXL_OBJS-y)
LIBXL_OBJS += libxl_genid.o
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 1cf1884..2beb534 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2973,6 +2973,52 @@ typedef void libxl__domain_suspend_cb(libxl__egc*,
typedef void libxl__save_device_model_cb(libxl__egc*,
libxl__domain_suspend_state*, int rc);
+/* State for writing a libxl migration v2 stream */
+typedef struct libxl__stream_write_state libxl__stream_write_state;
+
+typedef void (*sws_record_done_cb)(libxl__egc *egc,
+ libxl__stream_write_state *sws);
+
+struct libxl__stream_write_state {
+ /* filled by the user */
+ libxl__ao *ao;
+ int fd;
+ uint32_t domid;
+ void (*completion_callback)(libxl__egc *egc,
+ libxl__stream_write_state *sws,
+ int rc);
+ /* Private */
+ int rc;
+ bool running;
+
+ /* Active-stuff handling */
+ int joined_rc;
+
+ /* Main stream-writing data */
+ size_t padding;
+ libxl__datacopier_state dc;
+ sws_record_done_cb record_done_callback;
+
+ /* Emulator blob handling */
+ libxl__datacopier_state emu_dc;
+ libxl__carefd *emu_carefd;
+ libxl__sr_rec_hdr emu_rec_hdr;
+ void *emu_body;
+};
+
+_hidden void libxl__stream_write_start(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+
+_hidden void libxl__stream_write_abort(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc);
+
+static inline bool libxl__stream_write_inuse(
+ const libxl__stream_write_state *stream)
+{
+ return stream->running;
+}
+
typedef struct libxl__logdirty_switch {
const char *cmd;
const char *cmd_path;
@@ -3013,6 +3059,7 @@ struct libxl__domain_suspend_state {
/* private for libxl__domain_save_device_model */
libxl__save_device_model_cb *save_dm_callback;
libxl__datacopier_state save_dm_datacopier;
+ libxl__stream_write_state sws;
};
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
new file mode 100644
index 0000000..bf568ad
--- /dev/null
+++ b/tools/libxl/libxl_stream_write.c
@@ -0,0 +1,451 @@
+/*
+ * Copyright (C) 2015 Citrix Ltd.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU Lesser General Public License as published
+ * by the Free Software Foundation; version 2.1 only. with the special
+ * exception on linking described in file LICENSE.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+ * GNU Lesser General Public License for more details.
+ */
+
+#include "libxl_osdeps.h" /* must come before any other headers */
+
+#include "libxl_internal.h"
+
+/*
+ * Infrastructure for writing a domain to a libxl migration v2 stream.
+ *
+ * Entry points from outside:
+ * - libxl__stream_write_start()
+ * - Start writing a stream from the start.
+ *
+ * In normal operation, there are two tasks running at once; this stream
+ * processing, and the libxl-save-helper. check_stream_finished() is used to
+ * join all the tasks in both success and error cases.
+ *
+ * Nomenclature for event callbacks:
+ * - $FOO_done(): Completion callback for $FOO
+ * - write_$FOO(): Set up the datacopier to write a $FOO
+ * - $BAR_header(): A $BAR record header only
+ * - $BAR_record(): A complete $BAR record with header and content
+ *
+ * The main loop for a plain VM writes:
+ * - Stream header
+ * - Libxc record
+ * - Toolstack record
+ * - if (hvm), Qemu record
+ * - End record
+ */
+
+static void stream_success(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void stream_failed(libxl__egc *egc,
+ libxl__stream_write_state *stream, int ret);
+static void stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+
+static void check_stream_finished(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc, const char *what);
+
+/* Event callbacks for plain VM. */
+static void stream_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+static void libxc_header_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+/* libxl__xc_domain_save_done() lives here, event-order wise. */
+static void write_toolstack_record(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void toolstack_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void write_emulator_record(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void emulator_read_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval);
+static void emulator_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void write_end_record(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+
+/* Helper to set up reading some data from the stream. */
+static void write_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, dc);
+ STATE_AO_GC(stream->ao);
+ sws_record_done_cb cb = stream->record_done_callback;
+
+ stream->record_done_callback = NULL;
+
+ if (onwrite || errnoval) {
+ LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
+ stream_failed(egc, stream, ERROR_FAIL);
+ }
+ else
+ cb(egc, stream);
+}
+
+static void setup_write(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ const char *what,
+ libxl__sr_rec_hdr *hdr, void *body,
+ sws_record_done_cb cb)
+{
+ static const uint8_t zero_padding[1U << REC_ALIGN_ORDER] = { 0 };
+
+ libxl__datacopier_state *dc = &stream->dc;
+ int ret;
+
+ assert(stream->record_done_callback == NULL);
+
+ dc->writewhat = what;
+ dc->used = 0;
+ dc->callback = write_done;
+
+ ret = libxl__datacopier_start(dc);
+
+ if (ret) {
+ stream_failed(egc, stream, ret);
+ return;
+ }
+
+ size_t padsz = ROUNDUP(hdr->length, REC_ALIGN_ORDER) - hdr->length;
+
+ /* Insert header */
+ libxl__datacopier_prefixdata(egc, dc, hdr, sizeof(*hdr));
+
+ /* Optional body */
+ if (body)
+ libxl__datacopier_prefixdata(egc, dc, body, hdr->length);
+
+ /* Any required padding */
+ if (padsz > 0)
+ libxl__datacopier_prefixdata(egc, dc,
+ zero_padding, padsz);
+ stream->record_done_callback = cb;
+}
+
+void libxl__stream_write_start(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ libxl__datacopier_state *dc = &stream->dc;
+ STATE_AO_GC(stream->ao);
+ struct libxl__sr_hdr hdr = { 0 };
+ int ret = 0;
+
+ assert(!stream->running);
+ stream->running = true;
+
+ memset(dc, 0, sizeof(*dc));
+ dc->ao = ao;
+ dc->readfd = -1;
+
+ dc->writewhat = "save/migration stream";
+ dc->writefd = stream->fd;
+ dc->maxsz = -1;
+ dc->callback = stream_header_done;
+
+ ret = libxl__datacopier_start(dc);
+ if (ret)
+ goto err;
+
+ hdr.ident = htobe64(RESTORE_STREAM_IDENT);
+ hdr.version = htobe32(RESTORE_STREAM_VERSION);
+ hdr.options = htobe32(0);
+
+ libxl__datacopier_prefixdata(egc, dc, &hdr, sizeof(hdr));
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+void libxl__stream_write_abort(libxl__egc *egc,
+ libxl__stream_write_state *stream, int rc)
+{
+ stream_failed(egc, stream, rc);
+}
+
+static void stream_success(libxl__egc *egc, libxl__stream_write_state *stream)
+{
+ stream->rc = 0;
+
+ stream_done(egc, stream);
+}
+
+static void stream_failed(libxl__egc *egc,
+ libxl__stream_write_state *stream, int rc)
+{
+ assert(rc);
+ stream->rc = rc;
+
+ if (stream->running)
+ stream_done(egc, stream);
+}
+
+static void stream_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ assert(stream->running);
+ stream->running = false;
+
+ if (stream->emu_carefd)
+ libxl__carefd_close(stream->emu_carefd);
+ free(stream->emu_body);
+
+ check_stream_finished(egc, stream, stream->rc, "stream");
+}
+
+static void check_stream_finished(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc, const char *what)
+{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
+ STATE_AO_GC(dss->ao);
+
+ LOG(DEBUG, "Task '%s' joining (rc %d)", what, rc);
+
+ if (rc && !stream->joined_rc) {
+ bool skip = false;
+ /* First reported failure from joining tasks. Tear everything down */
+ stream->joined_rc = rc;
+
+ if (libxl__stream_write_inuse(stream)) {
+ skip = true;
+ libxl__stream_write_abort(egc, stream, rc);
+ }
+
+ if (libxl__save_helper_inuse(&dss->shs)) {
+ skip = true;
+ libxl__save_helper_abort(egc, &dss->shs);
+ }
+
+ /* There is at least one more active task to join - wait for its
+ callback */
+ if ( skip )
+ return;
+ }
+
+ if (libxl__stream_write_inuse(stream))
+ LOG(DEBUG, "stream still in use");
+ else if (libxl__save_helper_inuse(&dss->shs))
+ LOG(DEBUG, "save/restore still in use");
+ else {
+ LOG(DEBUG, "Join complete: result %d", stream->joined_rc);
+ stream->completion_callback(egc, stream, stream->joined_rc);
+ }
+}
+
+static void stream_header_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, dc);
+ STATE_AO_GC(stream->ao);
+ struct libxl__sr_rec_hdr rec = { REC_TYPE_LIBXC_CONTEXT };
+ int ret;
+
+ if (onwrite || errnoval) {
+ ret = ERROR_FAIL;
+ LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
+ goto err;
+ }
+
+ setup_write(egc, stream, "libxc header",
+ &rec, NULL, libxc_header_done);
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+static void libxc_header_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
+
+ libxl__xc_domain_save(egc, dss);
+}
+
+static void __attribute__((unused))
+write_toolstack_record(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
+ STATE_AO_GC(stream->ao);
+ struct libxl__sr_rec_hdr rec = { REC_TYPE_XENSTORE_DATA };
+ int ret;
+ uint8_t *toolstack_buf = NULL; /* We must free this. */
+ uint32_t toolstack_len;
+
+ ret = libxl__toolstack_save(dss->domid, &toolstack_buf,
+ &toolstack_len, dss);
+ if (ret)
+ goto err;
+
+ rec.length = toolstack_len;
+
+ setup_write(egc, stream, "toolstack record",
+ &rec, toolstack_buf,
+ toolstack_record_done);
+
+ free(toolstack_buf);
+ return;
+
+ err:
+ assert(ret);
+ free(toolstack_buf);
+ stream_failed(egc, stream, ret);
+}
+
+static void toolstack_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
+
+ if (dss->type == LIBXL_DOMAIN_TYPE_HVM)
+ write_emulator_record(egc, stream);
+ else
+ write_end_record(egc, stream);
+}
+
+static void write_emulator_record(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
+ libxl__datacopier_state *dc = &stream->emu_dc;
+ STATE_AO_GC(stream->ao);
+ struct libxl__sr_rec_hdr *rec = &stream->emu_rec_hdr;
+ struct libxl__sr_emulator_hdr *ehdr = NULL;
+ struct stat st;
+ int ret = 0;
+
+ assert(dss->type == LIBXL_DOMAIN_TYPE_HVM);
+
+ /* Convenience aliases */
+ const char *const filename = dss->dm_savefile;
+ const uint32_t domid = dss->domid;
+
+ libxl__carefd_begin();
+ int readfd = open(filename, O_RDONLY);
+ if (readfd == -1) {
+ ret = ERROR_FAIL;
+ LOGE(ERROR, "unable to open %s", filename);
+ libxl__carefd_unlock();
+ goto err;
+ }
+ stream->emu_carefd = libxl__carefd_record(CTX, readfd);
+ libxl__carefd_unlock();
+
+ if (fstat(readfd, &st)) {
+ ret = ERROR_FAIL;
+ LOGE(ERROR, "unable to fstat %s", filename);
+ goto err;
+ }
+
+ if (!S_ISREG(st.st_mode)) {
+ ret = ERROR_FAIL;
+ LOG(ERROR, "%s is not a plain file!", filename);
+ goto err;
+ }
+
+ rec->type = REC_TYPE_EMULATOR_CONTEXT;
+ rec->length = st.st_size + sizeof(*ehdr);
+ stream->emu_body = ehdr = libxl__malloc(NOGC, rec->length);
+
+ switch(libxl__device_model_version_running(gc, domid)) {
+ case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN_TRADITIONAL:
+ ehdr->id = EMULATOR_QEMU_TRADITIONAL;
+ break;
+
+ case LIBXL_DEVICE_MODEL_VERSION_QEMU_XEN:
+ ehdr->id = EMULATOR_QEMU_UPSTREAM;
+ break;
+
+ default:
+ ret = ERROR_FAIL;
+ goto err;
+ }
+ ehdr->index = 0;
+
+ memset(dc, 0, sizeof(*dc));
+ dc->ao = stream->ao;
+ dc->readwhat = "qemu save file";
+ dc->readfd = readfd;
+ dc->maxsz = -1;
+ dc->readbuf = stream->emu_body + sizeof(*ehdr);
+ dc->bytes_to_read = rec->length - sizeof(*ehdr);
+ dc->callback = emulator_read_done;
+
+ ret = libxl__datacopier_start(dc);
+ if (ret)
+ goto err;
+
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+static void emulator_read_done(libxl__egc *egc,
+ libxl__datacopier_state *dc,
+ int rc, int onwrite, int errnoval)
+{
+ libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, emu_dc);
+ STATE_AO_GC(stream->ao);
+ int ret;
+
+ if (onwrite || onwrite || errnoval) {
+ ret = ERROR_FAIL;
+ LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
+ goto err;
+ }
+
+ libxl__carefd_close(stream->emu_carefd);
+ stream->emu_carefd = NULL;
+
+ setup_write(egc, stream, "emulator record",
+ &stream->emu_rec_hdr, stream->emu_body,
+ emulator_record_done);
+ return;
+
+ err:
+ assert(ret);
+ stream_failed(egc, stream, ret);
+}
+
+static void emulator_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ free(stream->emu_body);
+ stream->emu_body = NULL;
+
+ write_end_record(egc, stream);
+}
+
+static void write_end_record(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ struct libxl__sr_rec_hdr rec = { REC_TYPE_END };
+
+ setup_write(egc, stream, "end record",
+ &rec, NULL, stream_success);
+}
+
+/*
+ * Local variables:
+ * mode: C
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 21/27] tools/libxc+libxl+xl: Save v2 streams
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (19 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 20/27] tools/libxl: Infrastructure for writing a v2 stream Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 10:57 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 22/27] docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams Andrew Cooper
` (6 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
This is a complicated set of changes which must be done together for
bisectability.
* libxl-save-helper is updated to unconditionally use libxc migration v2.
* libxl compatibility workarounds in libxc are disabled for save operations.
* libxl__stream_write_start() is logically spliced into the event location
where libxl__xc_domain_save() used to reside.
* xl is updated to indicate that the stream is now v2
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/Makefile | 2 --
tools/libxl/libxl.h | 2 ++
tools/libxl/libxl_dom.c | 46 +++++++++-----------------------------
tools/libxl/libxl_save_helper.c | 2 +-
tools/libxl/libxl_stream_write.c | 35 +++++++++++++++++++++++++++--
tools/libxl/xl_cmdimpl.c | 1 +
6 files changed, 47 insertions(+), 41 deletions(-)
diff --git a/tools/libxc/Makefile b/tools/libxc/Makefile
index 2cd0b1a..1aec848 100644
--- a/tools/libxc/Makefile
+++ b/tools/libxc/Makefile
@@ -64,8 +64,6 @@ GUEST_SRCS-$(CONFIG_X86) += xc_sr_save_x86_hvm.c
GUEST_SRCS-y += xc_sr_restore.c
GUEST_SRCS-y += xc_sr_save.c
GUEST_SRCS-y += xc_offline_page.c xc_compression.c
-xc_sr_save_x86_hvm.o: CFLAGS += -DXG_LIBXL_HVM_COMPAT
-xc_sr_save_x86_hvm.opic: CFLAGS += -DXG_LIBXL_HVM_COMPAT
else
GUEST_SRCS-y += xc_nomigrate.c
endif
diff --git a/tools/libxl/libxl.h b/tools/libxl/libxl.h
index e64a606..4f24e5f 100644
--- a/tools/libxl/libxl.h
+++ b/tools/libxl/libxl.h
@@ -812,6 +812,8 @@
*
* If this is defined, then the libxl_domain_create_restore() interface takes
* a "stream_version" parameter and supports a value of 2.
+ *
+ * libxl_domain_suspend() will produce a v2 stream.
*/
#define LIBXL_HAVE_STREAM_V2 1
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 8642192..de05124 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1153,6 +1153,8 @@ int libxl__toolstack_restore(uint32_t domid, const uint8_t *ptr,
/*==================== Domain suspend (save) ====================*/
+static void stream_done(libxl__egc *egc,
+ libxl__stream_write_state *sws, int rc);
static void domain_suspend_done(libxl__egc *egc,
libxl__domain_suspend_state *dss, int rc);
static void domain_suspend_callback_common_done(libxl__egc *egc,
@@ -2117,50 +2119,22 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
callbacks->switch_qemu_logdirty = libxl__domain_suspend_common_switch_qemu_logdirty;
dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
- libxl__xc_domain_save(egc, dss);
+ dss->sws.fd = dss->fd;
+ dss->sws.ao = dss->ao;
+ dss->sws.completion_callback = stream_done;
+
+ libxl__stream_write_start(egc, &dss->sws);
return;
out:
domain_suspend_done(egc, dss, rc);
}
-void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void,
- int rc, int retval, int errnoval)
+static void stream_done(libxl__egc *egc,
+ libxl__stream_write_state *sws, int rc)
{
- libxl__domain_suspend_state *dss = dss_void;
- STATE_AO_GC(dss->ao);
-
- /* Convenience aliases */
- const libxl_domain_type type = dss->type;
-
- if (rc)
- goto out;
-
- if (retval) {
- LOGEV(ERROR, errnoval, "saving domain: %s",
- dss->guest_responded ?
- "domain responded to suspend request" :
- "domain did not respond to suspend request");
- if ( !dss->guest_responded )
- rc = ERROR_GUEST_TIMEDOUT;
- else if (dss->rc)
- rc = dss->rc;
- else
- rc = ERROR_FAIL;
- goto out;
- }
-
- if (type == LIBXL_DOMAIN_TYPE_HVM) {
- rc = libxl__domain_suspend_device_model(gc, dss);
- if (rc) goto out;
+ libxl__domain_suspend_state *dss = CONTAINER_OF(sws, *dss, sws);
- libxl__domain_save_device_model(egc, dss, domain_suspend_done);
- return;
- }
-
- rc = 0;
-
-out:
domain_suspend_done(egc, dss, rc);
}
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index efbe2eb..f196786 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -286,7 +286,7 @@ int main(int argc, char **argv)
startup("save");
setup_signals(save_signal_handler);
- r = xc_domain_save(xch, io_fd, dom, max_iters, max_factor, flags,
+ r = xc_domain_save2(xch, io_fd, dom, max_iters, max_factor, flags,
&helper_save_callbacks, hvm);
complete(r);
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
index bf568ad..331173f 100644
--- a/tools/libxl/libxl_stream_write.c
+++ b/tools/libxl/libxl_stream_write.c
@@ -276,8 +276,39 @@ static void libxc_header_done(libxl__egc *egc,
libxl__xc_domain_save(egc, dss);
}
-static void __attribute__((unused))
-write_toolstack_record(libxl__egc *egc,
+void libxl__xc_domain_save_done(libxl__egc *egc, void *dss_void,
+ int rc, int retval, int errnoval)
+{
+ libxl__domain_suspend_state *dss = dss_void;
+ libxl__stream_write_state *stream = &dss->sws;
+ STATE_AO_GC(dss->ao);
+
+ if (rc)
+ goto err;
+
+ if (retval) {
+ LOGEV(ERROR, errnoval, "saving domain: %s",
+ dss->guest_responded ?
+ "domain responded to suspend request" :
+ "domain did not respond to suspend request");
+ if ( !dss->guest_responded )
+ rc = ERROR_GUEST_TIMEDOUT;
+ else if (dss->rc)
+ rc = dss->rc;
+ else
+ rc = ERROR_FAIL;
+ goto err;
+ }
+
+ write_toolstack_record(egc, stream);
+ return;
+
+ err:
+ assert(rc);
+ check_stream_finished(egc, stream, rc, "save/restore helper");
+}
+
+static void write_toolstack_record(libxl__egc *egc,
libxl__stream_write_state *stream)
{
libxl__domain_suspend_state *dss = CONTAINER_OF(stream, *dss, sws);
diff --git a/tools/libxl/xl_cmdimpl.c b/tools/libxl/xl_cmdimpl.c
index 71a22b8..13e154d 100644
--- a/tools/libxl/xl_cmdimpl.c
+++ b/tools/libxl/xl_cmdimpl.c
@@ -3922,6 +3922,7 @@ static void save_domain_core_writeconfig(int fd, const char *source,
memset(&hdr, 0, sizeof(hdr));
memcpy(hdr.magic, savefileheader_magic, sizeof(hdr.magic));
hdr.byteorder = SAVEFILE_BYTEORDER_VALUE;
+ hdr.mandatory_flags = XL_MANDATORY_FLAG_STREAMv2;
optdata_begin= 0;
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 22/27] docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (20 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 21/27] tools/libxc+libxl+xl: Save v2 streams Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 10:59 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 23/27] tools/libxl: Write checkpoint records into the stream Andrew Cooper
` (5 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
In a remus senario, libxc will write a CHECKPOINT record, then hand ownership
of the fd to libxl. Libxl then writes any records required and finishes with
a CHECKPOINT_END record, then hands ownership of the fd back to libxc.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
docs/specs/libxl-migration-stream.pandoc | 14 +++++++++++++-
tools/libxl/libxl_sr_stream_format.h | 1 +
tools/python/xen/migration/libxl.py | 11 +++++++++++
3 files changed, 25 insertions(+), 1 deletion(-)
diff --git a/docs/specs/libxl-migration-stream.pandoc b/docs/specs/libxl-migration-stream.pandoc
index 7235317..674bfdd 100644
--- a/docs/specs/libxl-migration-stream.pandoc
+++ b/docs/specs/libxl-migration-stream.pandoc
@@ -119,7 +119,9 @@ type 0x00000000: END
0x00000003: EMULATOR_CONTEXT
- 0x00000004 - 0x7FFFFFFF: Reserved for future _mandatory_
+ 0x00000004: CHECKPOINT_END
+
+ 0x00000005 - 0x7FFFFFFF: Reserved for future _mandatory_
records.
0x80000000 - 0xFFFFFFFF: Reserved for future _optional_
@@ -203,3 +205,13 @@ index Index of this emulator for the domain, if multiple
emulator_ctx Emulator context blob.
--------------------------------------------------------------------
+
+CHECKPOINT\_END
+---------------
+
+A checkpoint end record marks the end of a checkpoint in the image.
+
+ 0 1 2 3 4 5 6 7 octet
+ +-------------------------------------------------+
+
+The end record contains no fields; its body_length is 0.
diff --git a/tools/libxl/libxl_sr_stream_format.h b/tools/libxl/libxl_sr_stream_format.h
index f4f790b..3f3c497 100644
--- a/tools/libxl/libxl_sr_stream_format.h
+++ b/tools/libxl/libxl_sr_stream_format.h
@@ -35,6 +35,7 @@
#define REC_TYPE_LIBXC_CONTEXT 0x00000001U
#define REC_TYPE_XENSTORE_DATA 0x00000002U
#define REC_TYPE_EMULATOR_CONTEXT 0x00000003U
+#define REC_TYPE_CHECKPOINT_END 0x00000004U
typedef struct libxl__sr_emulator_hdr
{
diff --git a/tools/python/xen/migration/libxl.py b/tools/python/xen/migration/libxl.py
index 4e1f4f8..415502e 100644
--- a/tools/python/xen/migration/libxl.py
+++ b/tools/python/xen/migration/libxl.py
@@ -36,12 +36,14 @@ REC_TYPE_end = 0x00000000
REC_TYPE_libxc_context = 0x00000001
REC_TYPE_xenstore_data = 0x00000002
REC_TYPE_emulator_context = 0x00000003
+REC_TYPE_checkpoint_end = 0x00000004
rec_type_to_str = {
REC_TYPE_end : "End",
REC_TYPE_libxc_context : "Libxc context",
REC_TYPE_xenstore_data : "Xenstore data",
REC_TYPE_emulator_context : "Emulator context",
+ REC_TYPE_checkpoint_end : "Checkpoint end",
}
# emulator_context
@@ -176,6 +178,13 @@ class VerifyLibxl(VerifyBase):
self.info(" Index %d, type %s" % (emu_idx, emulator_id_to_str[emu_id]))
+ def verify_record_checkpoint_end(self, content):
+ """ Checkpoint end record """
+
+ if len(content) != 0:
+ raise RecordError("Checkpoint end record with non-zero length")
+
+
record_verifiers = {
REC_TYPE_end:
VerifyLibxl.verify_record_end,
@@ -185,4 +194,6 @@ record_verifiers = {
VerifyLibxl.verify_record_xenstore_data,
REC_TYPE_emulator_context:
VerifyLibxl.verify_record_emulator_context,
+ REC_TYPE_checkpoint_end:
+ VerifyLibxl.verify_record_checkpoint_end,
}
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 23/27] tools/libxl: Write checkpoint records into the stream
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (21 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 22/27] docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 11:02 ` Ian Campbell
2015-07-10 11:47 ` Wei Liu
2015-07-09 18:26 ` [PATCH v2 24/27] tools/libx{c, l}: Introduce restore_callbacks.checkpoint() Andrew Cooper
` (4 subsequent siblings)
27 siblings, 2 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
when signalled to do so by libxl__remus_domain_checkpoint_callback()
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
This patch has changed substantially in v2 as a result of changes earlier in
the series. No behavioural difference from v1.
---
tools/libxl/libxl_dom.c | 18 ++++-----
tools/libxl/libxl_internal.h | 7 ++++
tools/libxl/libxl_stream_write.c | 80 ++++++++++++++++++++++++++++++++++++--
3 files changed, 91 insertions(+), 14 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index de05124..9f5ddc9 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -1937,8 +1937,8 @@ static void remus_devices_preresume_cb(libxl__egc *egc,
/*----- remus asynchronous checkpoint callback -----*/
-static void remus_checkpoint_dm_saved(libxl__egc *egc,
- libxl__domain_suspend_state *dss, int rc);
+static void remus_checkpoint_stream_written(
+ libxl__egc *egc, libxl__stream_write_state *sws, int rc);
static void remus_devices_commit_cb(libxl__egc *egc,
libxl__remus_devices_state *rds,
int rc);
@@ -1953,17 +1953,14 @@ static void libxl__remus_domain_checkpoint_callback(void *data)
libxl__egc *egc = dss->shs.egc;
STATE_AO_GC(dss->ao);
- /* This would go into tailbuf. */
- if (dss->hvm) {
- libxl__domain_save_device_model(egc, dss, remus_checkpoint_dm_saved);
- } else {
- remus_checkpoint_dm_saved(egc, dss, 0);
- }
+ libxl__stream_write_start_checkpoint(egc, &dss->sws);
}
-static void remus_checkpoint_dm_saved(libxl__egc *egc,
- libxl__domain_suspend_state *dss, int rc)
+static void remus_checkpoint_stream_written(
+ libxl__egc *egc, libxl__stream_write_state *sws, int rc)
{
+ libxl__domain_suspend_state *dss = CONTAINER_OF(sws, *dss, sws);
+
/* Convenience aliases */
libxl__remus_devices_state *const rds = &dss->rds;
@@ -2113,6 +2110,7 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
callbacks->suspend = libxl__remus_domain_suspend_callback;
callbacks->postcopy = libxl__remus_domain_resume_callback;
callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;
+ dss->sws.checkpoint_callback = remus_checkpoint_stream_written;
} else
callbacks->suspend = libxl__domain_suspend_callback;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 2beb534..84e22c2 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2987,9 +2987,13 @@ struct libxl__stream_write_state {
void (*completion_callback)(libxl__egc *egc,
libxl__stream_write_state *sws,
int rc);
+ void (*checkpoint_callback)(libxl__egc *egc,
+ libxl__stream_write_state *sws,
+ int rc);
/* Private */
int rc;
bool running;
+ bool in_checkpoint;
/* Active-stuff handling */
int joined_rc;
@@ -3009,6 +3013,9 @@ struct libxl__stream_write_state {
_hidden void libxl__stream_write_start(libxl__egc *egc,
libxl__stream_write_state *stream);
+_hidden void libxl__stream_write_start_checkpoint(
+ libxl__egc *egc, libxl__stream_write_state *stream);
+
_hidden void libxl__stream_write_abort(libxl__egc *egc,
libxl__stream_write_state *stream,
int rc);
diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
index 331173f..0f5216b 100644
--- a/tools/libxl/libxl_stream_write.c
+++ b/tools/libxl/libxl_stream_write.c
@@ -23,6 +23,9 @@
* - libxl__stream_write_start()
* - Start writing a stream from the start.
*
+ * - libxl__stream_write_start()
+ * - Write the records which form a checkpoint into a stream.
+ *
* In normal operation, there are two tasks running at once; this stream
* processing, and the libxl-save-helper. check_stream_finished() is used to
* join all the tasks in both success and error cases.
@@ -39,6 +42,12 @@
* - Toolstack record
* - if (hvm), Qemu record
* - End record
+ *
+ * For checkpointed stream, there is a second loop which is triggered by a
+ * save-helper checkpoint callback. It writes:
+ * - Toolstack record
+ * - if (hvm), Qemu record
+ * - Checkpoint end record
*/
static void stream_success(libxl__egc *egc,
@@ -73,6 +82,15 @@ static void emulator_record_done(libxl__egc *egc,
static void write_end_record(libxl__egc *egc,
libxl__stream_write_state *stream);
+/* Event callbacks unique to checkpointed streams. */
+static void checkpoint_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc);
+static void write_checkpoint_end_record(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+static void checkpoint_end_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream);
+
/* Helper to set up reading some data from the stream. */
static void write_done(libxl__egc *egc,
libxl__datacopier_state *dc,
@@ -168,6 +186,16 @@ void libxl__stream_write_start(libxl__egc *egc,
stream_failed(egc, stream, ret);
}
+void libxl__stream_write_start_checkpoint(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ assert(stream->running);
+ assert(!stream->in_checkpoint);
+ stream->in_checkpoint = true;
+
+ write_toolstack_record(egc, stream);
+}
+
void libxl__stream_write_abort(libxl__egc *egc,
libxl__stream_write_state *stream, int rc)
{
@@ -176,6 +204,8 @@ void libxl__stream_write_abort(libxl__egc *egc,
static void stream_success(libxl__egc *egc, libxl__stream_write_state *stream)
{
+ assert(!stream->in_checkpoint);
+
stream->rc = 0;
stream_done(egc, stream);
@@ -187,13 +217,24 @@ static void stream_failed(libxl__egc *egc,
assert(rc);
stream->rc = rc;
- if (stream->running)
+ if (stream->running) {
+ /*
+ *If we are in a checkpoint, pass the failure to libxc, which will come
+ * back around to us via libxl__xc_domain_save_done().
+ */
+ if (stream->in_checkpoint) {
+ checkpoint_done(egc, stream, rc);
+ return;
+ }
+
stream_done(egc, stream);
+ }
}
static void stream_done(libxl__egc *egc,
libxl__stream_write_state *stream)
{
+ assert(!stream->in_checkpoint);
assert(stream->running);
stream->running = false;
@@ -345,8 +386,12 @@ static void toolstack_record_done(libxl__egc *egc,
if (dss->type == LIBXL_DOMAIN_TYPE_HVM)
write_emulator_record(egc, stream);
- else
- write_end_record(egc, stream);
+ else {
+ if (stream->in_checkpoint)
+ write_checkpoint_end_record(egc, stream);
+ else
+ write_end_record(egc, stream);
+ }
}
static void write_emulator_record(libxl__egc *egc,
@@ -461,7 +506,10 @@ static void emulator_record_done(libxl__egc *egc,
free(stream->emu_body);
stream->emu_body = NULL;
- write_end_record(egc, stream);
+ if (stream->in_checkpoint)
+ write_checkpoint_end_record(egc, stream);
+ else
+ write_end_record(egc, stream);
}
static void write_end_record(libxl__egc *egc,
@@ -473,6 +521,30 @@ static void write_end_record(libxl__egc *egc,
&rec, NULL, stream_success);
}
+static void checkpoint_done(libxl__egc *egc,
+ libxl__stream_write_state *stream,
+ int rc)
+{
+ assert(stream->in_checkpoint);
+ stream->in_checkpoint = false;
+ stream->checkpoint_callback(egc, stream, rc);
+}
+
+static void write_checkpoint_end_record(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ struct libxl__sr_rec_hdr rec = { REC_TYPE_CHECKPOINT_END };
+
+ setup_write(egc, stream, "checkpoint end record",
+ &rec, NULL, checkpoint_end_record_done);
+}
+
+static void checkpoint_end_record_done(libxl__egc *egc,
+ libxl__stream_write_state *stream)
+{
+ checkpoint_done(egc, stream, 0);
+}
+
/*
* Local variables:
* mode: C
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 24/27] tools/libx{c, l}: Introduce restore_callbacks.checkpoint()
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (22 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 23/27] tools/libxl: Write checkpoint records into the stream Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 11:13 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 25/27] tools/libxl: Handle checkpoint records in a libxl migration v2 stream Andrew Cooper
` (3 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel
Cc: Wei Liu, Ian Campbell, Wen Congyang, Andrew Cooper, Ian Jackson,
Yang Hongyang
And call it when a checkpoint record is found in the libxc stream.
Some parts of this patch have been based on patches from the COLO series.
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
v2: Borrow sufficient fragments from several COLO patches to get
BROKEN_CHANNEL and checkpoint failover to function.
---
tools/libxc/include/xenguest.h | 9 ++++++
tools/libxc/xc_sr_common.h | 7 +++--
tools/libxc/xc_sr_restore.c | 53 ++++++++++++++++++++++++++----------
tools/libxl/libxl_save_msgs_gen.pl | 2 +-
4 files changed, 53 insertions(+), 18 deletions(-)
diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
index 7581263..8799c9f 100644
--- a/tools/libxc/include/xenguest.h
+++ b/tools/libxc/include/xenguest.h
@@ -102,6 +102,15 @@ struct restore_callbacks {
int (*toolstack_restore)(uint32_t domid, const uint8_t *buf,
uint32_t size, void* data);
+ /* A checkpoint record has been found in the stream.
+ *
+ * returns:
+ * 0: (error) terminate processing
+ * 1: (success) continue normally
+ * 2: (failover) failover and resume VM
+ */
+ int (*checkpoint)(void* data);
+
/* to be provided as the last argument to each callback function */
void* data;
};
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 08c66db..1f4d4e4 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -130,10 +130,13 @@ struct xc_sr_restore_ops
* Process an individual record from the stream. The caller shall take
* care of processing common records (e.g. END, PAGE_DATA).
*
- * @return 0 for success, -1 for failure, or the sentinel value
- * RECORD_NOT_PROCESSED.
+ * @return 0 for success, -1 for failure, or the following sentinels:
+ * - RECORD_NOT_PROCESSED
+ * - BROKEN_CHANNEL: under Remus/COLO, this means master may be dead, and
+ * a failover is needed.
*/
#define RECORD_NOT_PROCESSED 1
+#define BROKEN_CHANNEL 2
int (*process_record)(struct xc_sr_context *ctx, struct xc_sr_record *rec);
/**
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index 9e27dba..c9b5213 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -1,5 +1,7 @@
#include <arpa/inet.h>
+#include <assert.h>
+
#include "xc_sr_common.h"
/*
@@ -472,7 +474,7 @@ static int handle_page_data(struct xc_sr_context *ctx, struct xc_sr_record *rec)
static int handle_checkpoint(struct xc_sr_context *ctx)
{
xc_interface *xch = ctx->xch;
- int rc = 0;
+ int rc = 0, ret;
unsigned i;
if ( !ctx->restore.checkpointed )
@@ -482,6 +484,21 @@ static int handle_checkpoint(struct xc_sr_context *ctx)
goto err;
}
+ ret = ctx->restore.callbacks->checkpoint(ctx->restore.callbacks->data);
+ switch ( ret )
+ {
+ case 1: /* Success */
+ break;
+
+ case 2: /* Failover */
+ rc = BROKEN_CHANNEL;
+ goto err;
+
+ default: /* Other fatal error */
+ rc = -1;
+ goto err;
+ }
+
if ( ctx->restore.buffer_all_records )
{
IPRINTF("All records buffered");
@@ -560,19 +577,6 @@ static int process_record(struct xc_sr_context *ctx, struct xc_sr_record *rec)
free(rec->data);
rec->data = NULL;
- if ( rc == RECORD_NOT_PROCESSED )
- {
- if ( rec->type & REC_TYPE_OPTIONAL )
- DPRINTF("Ignoring optional record %#x (%s)",
- rec->type, rec_type_to_str(rec->type));
- else
- {
- ERROR("Mandatory record %#x (%s) not handled",
- rec->type, rec_type_to_str(rec->type));
- rc = -1;
- }
- }
-
return rc;
}
@@ -678,7 +682,22 @@ static int restore(struct xc_sr_context *ctx)
else
{
rc = process_record(ctx, &rec);
- if ( rc )
+ if ( rc == RECORD_NOT_PROCESSED )
+ {
+ if ( rec.type & REC_TYPE_OPTIONAL )
+ DPRINTF("Ignoring optional record %#x (%s)",
+ rec.type, rec_type_to_str(rec.type));
+ else
+ {
+ ERROR("Mandatory record %#x (%s) not handled",
+ rec.type, rec_type_to_str(rec.type));
+ rc = -1;
+ goto err;
+ }
+ }
+ else if ( rc == BROKEN_CHANNEL )
+ goto remus_failover;
+ else if ( rc )
goto err;
}
@@ -735,6 +754,10 @@ int xc_domain_restore2(xc_interface *xch, int io_fd, uint32_t dom,
ctx.restore.checkpointed = checkpointed_stream;
ctx.restore.callbacks = callbacks;
+ /* Sanity checks for callbacks. */
+ if ( checkpointed_stream )
+ assert(callbacks->checkpoint);
+
IPRINTF("In experimental %s", __func__);
DPRINTF("fd %d, dom %u, hvm %u, pae %u, superpages %d"
", checkpointed_stream %d", io_fd, dom, hvm, pae,
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index 6b4b65e..825d5cc 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -25,7 +25,7 @@ our @msgs = (
'unsigned long', 'total'] ],
[ 3, 'scxA', "suspend", [] ],
[ 4, 'scxA', "postcopy", [] ],
- [ 5, 'scxA', "checkpoint", [] ],
+ [ 5, 'srcxA', "checkpoint", [] ],
[ 6, 'scxA', "switch_qemu_logdirty", [qw(int domid
unsigned enable)] ],
# toolstack_save done entirely `by hand'
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 25/27] tools/libxl: Handle checkpoint records in a libxl migration v2 stream
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (23 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 24/27] tools/libx{c, l}: Introduce restore_callbacks.checkpoint() Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 11:18 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 26/27] tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc Andrew Cooper
` (2 subsequent siblings)
27 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Ian Campbell, Wei Liu
This is the final bit of untangling for Remus.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
CC: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
As before, Remus functionality is untested, but the new logic here should
handle failovers correctly. The patch has changed greatly from v1, both in a
functional sence, and because of the knockon effects from earlier changes.
---
tools/libxl/libxl_create.c | 27 +++++++++++
tools/libxl/libxl_internal.h | 8 ++++
tools/libxl/libxl_stream_read.c | 97 +++++++++++++++++++++++++++++++++++++++
3 files changed, 132 insertions(+)
diff --git a/tools/libxl/libxl_create.c b/tools/libxl/libxl_create.c
index 2a0063a..0325bf1 100644
--- a/tools/libxl/libxl_create.c
+++ b/tools/libxl/libxl_create.c
@@ -672,6 +672,29 @@ static int store_libxl_entry(libxl__gc *gc, uint32_t domid,
libxl_device_model_version_to_string(b_info->device_model_version));
}
+/*----- remus asynchronous checkpoint callback -----*/
+
+static void remus_checkpoint_stream_done(
+ libxl__egc *egc, libxl__stream_read_state *srs, int rc);
+
+static void libxl__remus_domain_checkpoint_callback(void *data)
+{
+ libxl__save_helper_state *shs = data;
+ libxl__domain_create_state *dcs = CONTAINER_OF(shs, *dcs, shs);
+ libxl__egc *egc = dcs->shs.egc;
+ STATE_AO_GC(dcs->ao);
+
+ libxl__stream_read_start_checkpoint(egc, &dcs->srs);
+}
+
+static void remus_checkpoint_stream_done(
+ libxl__egc *egc, libxl__stream_read_state *srs, int rc)
+{
+ libxl__domain_create_state *dcs = CONTAINER_OF(srs, *dcs, srs);
+
+ libxl__xc_domain_saverestore_async_callback_done(egc, &dcs->shs, rc);
+}
+
/*----- main domain creation -----*/
/* We have a linear control flow; only one event callback is
@@ -939,6 +962,8 @@ static void domcreate_bootloader_done(libxl__egc *egc,
libxl_domain_config *const d_config = dcs->guest_config;
const int restore_fd = dcs->restore_fd;
libxl__domain_build_state *const state = &dcs->build_state;
+ libxl__srm_restore_autogen_callbacks *const callbacks =
+ &dcs->shs.callbacks.restore.a;
if (rc) {
domcreate_rebuild_done(egc, dcs, rc);
@@ -966,6 +991,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
}
/* Restore */
+ callbacks->checkpoint = libxl__remus_domain_checkpoint_callback;
rc = libxl__build_pre(gc, domid, d_config, state);
if (rc)
@@ -975,6 +1001,7 @@ static void domcreate_bootloader_done(libxl__egc *egc,
dcs->srs.fd = restore_fd;
dcs->srs.legacy = (dcs->restore_params.stream_version == 1);
dcs->srs.completion_callback = domcreate_stream_done;
+ dcs->srs.checkpoint_callback = remus_checkpoint_stream_done;
libxl__stream_read_start(egc, &dcs->srs);
return;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 84e22c2..1b62f25 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -3332,9 +3332,13 @@ struct libxl__stream_read_state {
void (*completion_callback)(libxl__egc *egc,
libxl__stream_read_state *srs,
int rc);
+ void (*checkpoint_callback)(libxl__egc *egc,
+ libxl__stream_read_state *srs,
+ int rc);
/* Private */
int rc;
bool running;
+ bool in_checkpoint;
/* Active-stuff handling */
int joined_rc;
@@ -3349,6 +3353,8 @@ struct libxl__stream_read_state {
LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue;
enum {
SRS_PHASE_NORMAL,
+ SRS_PHASE_BUFFERING,
+ SRS_PHASE_UNBUFFERING,
} phase;
bool recursion_guard;
@@ -3362,6 +3368,8 @@ _hidden void libxl__stream_read_start(libxl__egc *egc,
_hidden void libxl__stream_read_continue(libxl__egc *egc,
libxl__stream_read_state *stream);
+_hidden void libxl__stream_read_start_checkpoint(
+ libxl__egc *egc, libxl__stream_read_state *stream);
_hidden void libxl__stream_read_abort(libxl__egc *egc,
libxl__stream_read_state *stream, int rc);
diff --git a/tools/libxl/libxl_stream_read.c b/tools/libxl/libxl_stream_read.c
index 81095cd..6cfa05c 100644
--- a/tools/libxl/libxl_stream_read.c
+++ b/tools/libxl/libxl_stream_read.c
@@ -123,6 +123,10 @@ static int setup_read(libxl__stream_read_state *stream,
return libxl__datacopier_start(dc);
}
+/* Error handling for checkpoint mini-loop. */
+static void checkpoint_done(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc);
+
void libxl__stream_read_start(libxl__egc *egc,
libxl__stream_read_state *stream)
{
@@ -186,6 +190,18 @@ void libxl__stream_read_start(libxl__egc *egc,
stream_failed(egc, stream, ret);
}
+void libxl__stream_read_start_checkpoint(libxl__egc *egc,
+ libxl__stream_read_state *stream)
+{
+ assert(stream->running);
+ assert(!stream->in_checkpoint);
+
+ stream->in_checkpoint = true;
+ stream->phase = SRS_PHASE_BUFFERING;
+
+ setup_read_record(egc, stream);
+}
+
void libxl__stream_read_abort(libxl__egc *egc,
libxl__stream_read_state *stream, int rc)
{
@@ -206,6 +222,16 @@ static void stream_failed(libxl__egc *egc,
stream->rc = rc;
if (stream->running) {
+
+ /*
+ * If we are in a checkpoint, pass the failure to libxc, which will
+ * come back around to us via libxl__xc_domain_restore_done().
+ */
+ if (stream->in_checkpoint) {
+ checkpoint_done(egc, stream, rc);
+ return;
+ }
+
stream_done(egc, stream);
}
}
@@ -215,6 +241,7 @@ static void stream_done(libxl__egc *egc,
{
libxl__sr_record_buf *rec, *trec;
+ assert(!stream->in_checkpoint);
assert(stream->running);
stream->running = false;
@@ -296,6 +323,8 @@ static void stream_continue(libxl__egc *egc,
* processing the record. At no point should there ever be two
* records in the queue.
*/
+ assert(!stream->in_checkpoint);
+
if (LIBXL_STAILQ_EMPTY(&stream->record_queue))
setup_read_record(egc, stream);
else {
@@ -304,6 +333,45 @@ static void stream_continue(libxl__egc *egc,
}
break;
+ case SRS_PHASE_BUFFERING: {
+ /*
+ * Buffer phase of a checkpoint in the stream. Collect records read
+ * from the stream without processing them. We need to peek at the
+ * tail to spot the CHECKPOINT_END record, and switch to the
+ * unbuffering phase.
+ */
+ libxl__sr_record_buf *rec = LIBXL_STAILQ_LAST(
+ &stream->record_queue, libxl__sr_record_buf, entry);
+
+ assert(stream->in_checkpoint);
+
+ if ( !rec || rec->hdr.type != REC_TYPE_CHECKPOINT_END ) {
+ setup_read_record(egc, stream);
+ break;
+ }
+
+ /*
+ * There are now some number of buffered records, with a
+ * CHECKPOINT_END at the end. Start processing them all.
+ */
+ stream->phase = SRS_PHASE_UNBUFFERING;
+ }
+ /* FALLTHROUGH */
+ case SRS_PHASE_UNBUFFERING:
+ /*
+ * Unbuffering phase of a checkpoint in the stream. Process all
+ * records collected during the buffering phase.
+ */
+ assert(stream->in_checkpoint);
+
+ while (process_record(egc, stream))
+ ; /*
+ * Nothing! process_record() helpfully tells us if no specific
+ * futher actions have been set up, in which case we want to go
+ * ahead and process the next record.
+ */
+ break;
+
default:
abort();
}
@@ -528,6 +596,15 @@ static bool process_record(libxl__egc *egc,
write_emulator_blob(egc, stream, rec);
break;
+ case REC_TYPE_CHECKPOINT_END:
+ if (!stream->in_checkpoint) {
+ LOG(ERROR, "Unexpected CHECKPOINT_END record in stream");
+ ret = ERROR_FAIL;
+ goto err;
+ }
+ checkpoint_done(egc, stream, 0);
+ break;
+
default:
LOG(ERROR, "Unrecognised record 0x%08x", rec->hdr.type);
ret = ERROR_FAIL;
@@ -625,6 +702,26 @@ static void write_emulator_done(libxl__egc *egc,
stream_failed(egc, stream, ret);
}
+static void checkpoint_done(libxl__egc *egc,
+ libxl__stream_read_state *stream, int rc)
+{
+ int ret;
+
+ assert(stream->in_checkpoint);
+
+ if (rc == 0)
+ ret = 0; /* Success */
+ else if (stream->phase == SRS_PHASE_BUFFERING)
+ ret = 2; /* Failover */
+ else
+ ret = 1; /* Error (fatal) */
+
+ stream->checkpoint_callback(egc, stream, ret);
+
+ stream->in_checkpoint = false;
+ stream->phase = SRS_PHASE_NORMAL;
+}
+
/*
* Local variables:
* mode: C
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 26/27] tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (24 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 25/27] tools/libxl: Handle checkpoint records in a libxl migration v2 stream Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 27/27] tools/libxl: Drop all knowledge of toolstack callbacks Andrew Cooper
2015-07-10 3:01 ` [PATCH v2 00/27] Libxl migration v2 Yang Hongyang
27 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Libxl has now been fully adjusted not to need it.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/libxc/xc_sr_common.h | 5 --
tools/libxc/xc_sr_restore.c | 18 -----
tools/libxc/xc_sr_restore_x86_hvm.c | 124 -----------------------------------
tools/libxc/xc_sr_save_x86_hvm.c | 36 ----------
4 files changed, 183 deletions(-)
diff --git a/tools/libxc/xc_sr_common.h b/tools/libxc/xc_sr_common.h
index 1f4d4e4..64f6082 100644
--- a/tools/libxc/xc_sr_common.h
+++ b/tools/libxc/xc_sr_common.h
@@ -309,11 +309,6 @@ struct xc_sr_context
/* HVM context blob. */
void *context;
size_t contextsz;
-
-/* #ifdef XG_LIBXL_HVM_COMPAT */
- uint32_t qlen;
- void *qbuf;
-/* #endif */
} restore;
};
} x86_hvm;
diff --git a/tools/libxc/xc_sr_restore.c b/tools/libxc/xc_sr_restore.c
index c9b5213..2f6a763 100644
--- a/tools/libxc/xc_sr_restore.c
+++ b/tools/libxc/xc_sr_restore.c
@@ -627,9 +627,6 @@ static void cleanup(struct xc_sr_context *ctx)
PERROR("Failed to clean up");
}
-#ifdef XG_LIBXL_HVM_COMPAT
-extern int read_qemu(struct xc_sr_context *ctx);
-#endif
/*
* Restore a domain.
*/
@@ -656,21 +653,6 @@ static int restore(struct xc_sr_context *ctx)
goto err;
}
-#ifdef XG_LIBXL_HVM_COMPAT
- if ( ctx->dominfo.hvm &&
- (rec.type == REC_TYPE_END || rec.type == REC_TYPE_CHECKPOINT) )
- {
- rc = read_qemu(ctx);
- if ( rc )
- {
- if ( ctx->restore.buffer_all_records )
- goto remus_failover;
- else
- goto err;
- }
- }
-#endif
-
if ( ctx->restore.buffer_all_records &&
rec.type != REC_TYPE_END &&
rec.type != REC_TYPE_CHECKPOINT )
diff --git a/tools/libxc/xc_sr_restore_x86_hvm.c b/tools/libxc/xc_sr_restore_x86_hvm.c
index 6f5af0e..49d22c7 100644
--- a/tools/libxc/xc_sr_restore_x86_hvm.c
+++ b/tools/libxc/xc_sr_restore_x86_hvm.c
@@ -3,24 +3,6 @@
#include "xc_sr_common_x86.h"
-#ifdef XG_LIBXL_HVM_COMPAT
-static int handle_toolstack(struct xc_sr_context *ctx, struct xc_sr_record *rec)
-{
- xc_interface *xch = ctx->xch;
- int rc;
-
- if ( !ctx->restore.callbacks || !ctx->restore.callbacks->toolstack_restore )
- return 0;
-
- rc = ctx->restore.callbacks->toolstack_restore(
- ctx->domid, rec->data, rec->length, ctx->restore.callbacks->data);
-
- if ( rc < 0 )
- PERROR("restoring toolstack");
- return rc;
-}
-#endif
-
/*
* Process an HVM_CONTEXT record from the stream.
*/
@@ -93,98 +75,6 @@ static int handle_hvm_params(struct xc_sr_context *ctx,
return 0;
}
-#ifdef XG_LIBXL_HVM_COMPAT
-int read_qemu(struct xc_sr_context *ctx);
-int read_qemu(struct xc_sr_context *ctx)
-{
- xc_interface *xch = ctx->xch;
- char qemusig[21];
- uint32_t qlen;
- void *qbuf = NULL;
- int rc = -1;
-
- if ( read_exact(ctx->fd, qemusig, sizeof(qemusig)) )
- {
- PERROR("Error reading QEMU signature");
- goto out;
- }
-
- if ( !memcmp(qemusig, "DeviceModelRecord0002", sizeof(qemusig)) )
- {
- if ( read_exact(ctx->fd, &qlen, sizeof(qlen)) )
- {
- PERROR("Error reading QEMU record length");
- goto out;
- }
-
- qbuf = malloc(qlen);
- if ( !qbuf )
- {
- PERROR("no memory for device model state");
- goto out;
- }
-
- if ( read_exact(ctx->fd, qbuf, qlen) )
- {
- PERROR("Error reading device model state");
- goto out;
- }
- }
- else
- {
- ERROR("Invalid device model state signature '%*.*s'",
- (int)sizeof(qemusig), (int)sizeof(qemusig), qemusig);
- goto out;
- }
-
- /* With Remus, this could be read many times */
- if ( ctx->x86_hvm.restore.qbuf )
- free(ctx->x86_hvm.restore.qbuf);
- ctx->x86_hvm.restore.qbuf = qbuf;
- ctx->x86_hvm.restore.qlen = qlen;
- rc = 0;
-
-out:
- if (rc)
- free(qbuf);
- return rc;
-}
-
-static int handle_qemu(struct xc_sr_context *ctx)
-{
- xc_interface *xch = ctx->xch;
- char path[256];
- uint32_t qlen = ctx->x86_hvm.restore.qlen;
- void *qbuf = ctx->x86_hvm.restore.qbuf;
- int rc = -1;
- FILE *fp = NULL;
-
- sprintf(path, XC_DEVICE_MODEL_RESTORE_FILE".%u", ctx->domid);
- fp = fopen(path, "wb");
- if ( !fp )
- {
- PERROR("Failed to open '%s' for writing", path);
- goto out;
- }
-
- DPRINTF("Writing %u bytes of QEMU data", qlen);
- if ( fwrite(qbuf, 1, qlen, fp) != qlen )
- {
- PERROR("Failed to write %u bytes of QEMU data", qlen);
- goto out;
- }
-
- rc = 0;
-
- out:
- if ( fp )
- fclose(fp);
- free(qbuf);
-
- return rc;
-}
-#endif
-
/* restore_ops function. */
static bool x86_hvm_pfn_is_valid(const struct xc_sr_context *ctx, xen_pfn_t pfn)
{
@@ -260,11 +150,6 @@ static int x86_hvm_process_record(struct xc_sr_context *ctx,
case REC_TYPE_HVM_PARAMS:
return handle_hvm_params(ctx, rec);
-#ifdef XG_LIBXL_HVM_COMPAT
- case REC_TYPE_TOOLSTACK:
- return handle_toolstack(ctx, rec);
-#endif
-
default:
return RECORD_NOT_PROCESSED;
}
@@ -314,15 +199,6 @@ static int x86_hvm_stream_complete(struct xc_sr_context *ctx)
return rc;
}
-#ifdef XG_LIBXL_HVM_COMPAT
- rc = handle_qemu(ctx);
- if ( rc )
- {
- ERROR("Failed to dump qemu");
- return rc;
- }
-#endif
-
return rc;
}
diff --git a/tools/libxc/xc_sr_save_x86_hvm.c b/tools/libxc/xc_sr_save_x86_hvm.c
index f4604db..cdee774 100644
--- a/tools/libxc/xc_sr_save_x86_hvm.c
+++ b/tools/libxc/xc_sr_save_x86_hvm.c
@@ -118,36 +118,6 @@ static int write_hvm_params(struct xc_sr_context *ctx)
return rc;
}
-#ifdef XG_LIBXL_HVM_COMPAT
-static int write_toolstack(struct xc_sr_context *ctx)
-{
- xc_interface *xch = ctx->xch;
- struct xc_sr_record rec = {
- .type = REC_TYPE_TOOLSTACK,
- .length = 0,
- };
- uint8_t *buf;
- uint32_t len;
- int rc;
-
- if ( !ctx->save.callbacks || !ctx->save.callbacks->toolstack_save )
- return 0;
-
- if ( ctx->save.callbacks->toolstack_save(
- ctx->domid, &buf, &len, ctx->save.callbacks->data) < 0 )
- {
- PERROR("Error calling toolstack_save");
- return -1;
- }
-
- rc = write_split_record(ctx, &rec, buf, len);
- if ( rc < 0 )
- PERROR("Error writing TOOLSTACK record");
- free(buf);
- return rc;
-}
-#endif
-
static xen_pfn_t x86_hvm_pfn_to_gfn(const struct xc_sr_context *ctx,
xen_pfn_t pfn)
{
@@ -199,12 +169,6 @@ static int x86_hvm_end_of_checkpoint(struct xc_sr_context *ctx)
if ( rc )
return rc;
-#ifdef XG_LIBXL_HVM_COMPAT
- rc = write_toolstack(ctx);
- if ( rc )
- return rc;
-#endif
-
/* Write the HVM_CONTEXT record. */
rc = write_hvm_context(ctx);
if ( rc )
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* [PATCH v2 27/27] tools/libxl: Drop all knowledge of toolstack callbacks
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (25 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 26/27] tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc Andrew Cooper
@ 2015-07-09 18:26 ` Andrew Cooper
2015-07-10 3:01 ` [PATCH v2 00/27] Libxl migration v2 Yang Hongyang
27 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-09 18:26 UTC (permalink / raw)
To: Xen-devel; +Cc: Andrew Cooper, Ian Jackson, Wei Liu
Libxl has now been fully adjusted not to need them.
Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
CC: Wei Liu <wei.liu2@citrix.com>
---
tools/libxl/libxl_dom.c | 1 -
tools/libxl/libxl_internal.h | 2 --
tools/libxl/libxl_save_callout.c | 39 +-----------------------------------
tools/libxl/libxl_save_helper.c | 29 ---------------------------
tools/libxl/libxl_save_msgs_gen.pl | 7 ++-----
5 files changed, 3 insertions(+), 75 deletions(-)
diff --git a/tools/libxl/libxl_dom.c b/tools/libxl/libxl_dom.c
index 9f5ddc9..3c765f4 100644
--- a/tools/libxl/libxl_dom.c
+++ b/tools/libxl/libxl_dom.c
@@ -2115,7 +2115,6 @@ void libxl__domain_suspend(libxl__egc *egc, libxl__domain_suspend_state *dss)
callbacks->suspend = libxl__domain_suspend_callback;
callbacks->switch_qemu_logdirty = libxl__domain_suspend_common_switch_qemu_logdirty;
- dss->shs.callbacks.save.toolstack_save = libxl__toolstack_save;
dss->sws.fd = dss->fd;
dss->sws.ao = dss->ao;
diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
index 1b62f25..13e2493 100644
--- a/tools/libxl/libxl_internal.h
+++ b/tools/libxl/libxl_internal.h
@@ -2699,8 +2699,6 @@ _hidden void libxl__datacopier_prefixdata(libxl__egc*, libxl__datacopier_state*,
typedef struct libxl__srm_save_callbacks {
libxl__srm_save_autogen_callbacks a;
- int (*toolstack_save)(uint32_t domid, uint8_t **buf,
- uint32_t *len, void *data);
} libxl__srm_save_callbacks;
typedef struct libxl__srm_restore_callbacks {
diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
index cd18cd2..2a6662f 100644
--- a/tools/libxl/libxl_save_callout.c
+++ b/tools/libxl/libxl_save_callout.c
@@ -78,41 +78,12 @@ void libxl__xc_domain_restore(libxl__egc *egc, libxl__domain_create_state *dcs,
void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss)
{
STATE_AO_GC(dss->ao);
- int r, rc, toolstack_data_fd = -1;
- uint32_t toolstack_data_len = 0;
-
- /* Resources we need to free */
- uint8_t *toolstack_data_buf = 0;
unsigned cbflags = libxl__srm_callout_enumcallbacks_save
(&dss->shs.callbacks.save.a);
- if (dss->shs.callbacks.save.toolstack_save) {
- r = dss->shs.callbacks.save.toolstack_save
- (dss->domid, &toolstack_data_buf, &toolstack_data_len, dss);
- if (r) { rc = ERROR_FAIL; goto out; }
-
- dss->shs.toolstack_data_file = tmpfile();
- if (!dss->shs.toolstack_data_file) {
- LOGE(ERROR, "cannot create toolstack data tmpfile");
- rc = ERROR_FAIL;
- goto out;
- }
- toolstack_data_fd = fileno(dss->shs.toolstack_data_file);
-
- r = libxl_write_exactly(CTX, toolstack_data_fd,
- toolstack_data_buf, toolstack_data_len,
- "toolstack data tmpfile", 0);
- if (r) { rc = ERROR_FAIL; goto out; }
-
- /* file position must be reset before passing to libxl-save-helper. */
- r = lseek(toolstack_data_fd, 0, SEEK_SET);
- if (r) { rc = ERROR_FAIL; goto out; }
- }
-
const unsigned long argnums[] = {
dss->domid, 0, 0, dss->xcflags, dss->hvm,
- toolstack_data_fd, toolstack_data_len,
cbflags,
};
@@ -123,18 +94,10 @@ void libxl__xc_domain_save(libxl__egc *egc, libxl__domain_suspend_state *dss)
dss->shs.caller_state = dss;
dss->shs.need_results = 0;
- free(toolstack_data_buf);
-
run_helper(egc, &dss->shs, "--save-domain", dss->fd,
- &toolstack_data_fd, 1,
+ NULL, 0,
argnums, ARRAY_SIZE(argnums));
return;
-
- out:
- free(toolstack_data_buf);
- if (dss->shs.toolstack_data_file) fclose(dss->shs.toolstack_data_file);
-
- libxl__xc_domain_save_done(egc, dss, rc, 0, 0);
}
diff --git a/tools/libxl/libxl_save_helper.c b/tools/libxl/libxl_save_helper.c
index f196786..1622bb7 100644
--- a/tools/libxl/libxl_save_helper.c
+++ b/tools/libxl/libxl_save_helper.c
@@ -213,32 +213,8 @@ int helper_getreply(void *user)
/*----- other callbacks -----*/
-static int toolstack_save_fd;
-static uint32_t toolstack_save_len;
static struct save_callbacks helper_save_callbacks;
-static int toolstack_save_cb(uint32_t domid, uint8_t **buf,
- uint32_t *len, void *data)
-{
- int r;
-
- assert(toolstack_save_fd > 0);
-
- /* This is a hack for remus */
- if (helper_save_callbacks.checkpoint) {
- r = lseek(toolstack_save_fd, 0, SEEK_SET);
- if (r) fail(errno,"rewind toolstack data tmpfile");
- }
-
- *buf = xmalloc(toolstack_save_len);
- r = read_exactly(toolstack_save_fd, *buf, toolstack_save_len);
- if (r<0) fail(errno,"read toolstack data");
- if (r==0) fail(0,"read toolstack data eof");
-
- *len = toolstack_save_len;
- return 0;
-}
-
static void startup(const char *op) {
xtl_log(&logger,XTL_DEBUG,0,program,"starting %s",op);
@@ -273,14 +249,9 @@ int main(int argc, char **argv)
uint32_t max_factor = strtoul(NEXTARG,0,10);
uint32_t flags = strtoul(NEXTARG,0,10);
int hvm = atoi(NEXTARG);
- toolstack_save_fd = atoi(NEXTARG);
- toolstack_save_len = strtoul(NEXTARG,0,10);
unsigned cbflags = strtoul(NEXTARG,0,10);
assert(!*++argv);
- if (toolstack_save_fd >= 0)
- helper_save_callbacks.toolstack_save = toolstack_save_cb;
-
helper_setcallbacks_save(&helper_save_callbacks, cbflags);
startup("save");
diff --git a/tools/libxl/libxl_save_msgs_gen.pl b/tools/libxl/libxl_save_msgs_gen.pl
index 825d5cc..d6d2967 100755
--- a/tools/libxl/libxl_save_msgs_gen.pl
+++ b/tools/libxl/libxl_save_msgs_gen.pl
@@ -28,12 +28,9 @@ our @msgs = (
[ 5, 'srcxA', "checkpoint", [] ],
[ 6, 'scxA', "switch_qemu_logdirty", [qw(int domid
unsigned enable)] ],
- # toolstack_save done entirely `by hand'
- [ 7, 'rcxW', "toolstack_restore", [qw(uint32_t domid
- BLOCK tsdata)] ],
- [ 8, 'r', "restore_results", ['unsigned long', 'store_mfn',
+ [ 7, 'r', "restore_results", ['unsigned long', 'store_mfn',
'unsigned long', 'console_mfn'] ],
- [ 9, 'srW', "complete", [qw(int retval
+ [ 8, 'srW', "complete", [qw(int retval
int errnoval)] ],
);
--
1.7.10.4
^ permalink raw reply related [flat|nested] 65+ messages in thread
* Re: [PATCH v2 04/27] tools/libxl: Introduce libxl__kill()
2015-07-09 18:26 ` [PATCH v2 04/27] tools/libxl: Introduce libxl__kill() Andrew Cooper
@ 2015-07-10 1:34 ` Yang Hongyang
2015-07-10 8:56 ` Andrew Cooper
2015-07-10 9:08 ` Wei Liu
1 sibling, 1 reply; 65+ messages in thread
From: Yang Hongyang @ 2015-07-10 1:34 UTC (permalink / raw)
To: Andrew Cooper, Xen-devel; +Cc: Wei Liu, Ian Jackson, Ian Campbell
On 07/10/2015 02:26 AM, Andrew Cooper wrote:
> as a wrapper to kill(2), and use it in preference to sendig in
s/sendig/sendsig/
> libxl_save_callout.c.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Ian Campbell <Ian.Campbell@citrix.com>
> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>
>
> ---
> Logically new in v2 - split out from a v1 change which was itself a
> cherrypick-and-modify from the AO Abort series
> ---
> tools/libxl/libxl_aoutils.c | 15 +++++++++++++++
> tools/libxl/libxl_internal.h | 2 ++
> tools/libxl/libxl_save_callout.c | 10 ++--------
> 3 files changed, 19 insertions(+), 8 deletions(-)
>
> diff --git a/tools/libxl/libxl_aoutils.c b/tools/libxl/libxl_aoutils.c
> index 0931eee..274ef39 100644
> --- a/tools/libxl/libxl_aoutils.c
> +++ b/tools/libxl/libxl_aoutils.c
> @@ -621,3 +621,18 @@ bool libxl__async_exec_inuse(const libxl__async_exec_state *aes)
> assert(time_inuse == child_inuse);
> return child_inuse;
> }
> +
> +void libxl__kill(libxl__gc *gc, pid_t pid, int sig, const char *what)
> +{
> + int r = kill(pid, sig);
> + if (r) LOGE(WARN, "failed to kill() %s [%lu] (signal %d)",
> + what, (unsigned long)pid, sig);
> +}
> +
> +/*
> + * Local variables:
> + * mode: C
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 19fc425..9147de1 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -2244,6 +2244,8 @@ struct libxl__async_exec_state {
> int libxl__async_exec_start(libxl__async_exec_state *aes);
> bool libxl__async_exec_inuse(const libxl__async_exec_state *aes);
>
> +void libxl__kill(libxl__gc *gc, pid_t pid, int sig, const char *what);
> +
> /*----- device addition/removal -----*/
>
> typedef struct libxl__ao_device libxl__ao_device;
> diff --git a/tools/libxl/libxl_save_callout.c b/tools/libxl/libxl_save_callout.c
> index 087c2d5..b82a5c1 100644
> --- a/tools/libxl/libxl_save_callout.c
> +++ b/tools/libxl/libxl_save_callout.c
> @@ -244,12 +244,6 @@ static void run_helper(libxl__egc *egc, libxl__save_helper_state *shs,
> libxl__carefd_close(childs_pipes[1]);
> helper_failed(egc, shs, rc);;
> }
> -static void sendsig(libxl__gc *gc, libxl__save_helper_state *shs, int sig)
> -{
> - int r = kill(shs->child.pid, sig);
> - if (r) LOGE(WARN, "failed to kill save/restore helper [%lu] (signal %d)",
> - (unsigned long)shs->child.pid, sig);
> -}
>
> static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs,
> int rc)
> @@ -266,7 +260,7 @@ static void helper_failed(libxl__egc *egc, libxl__save_helper_state *shs,
> return;
> }
>
> - sendsig(gc, shs, SIGKILL);
> + libxl__kill(gc, shs->child.pid, SIGKILL, "save/restore helper");
> }
>
> static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
> @@ -282,7 +276,7 @@ static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
> if (!shs->rc)
> shs->rc = rc;
>
> - sendsig(gc, shs, SIGTERM);
> + libxl__kill(gc, shs->child.pid, SIGTERM, "save/restore helper");
> }
>
> static void helper_stdout_readable(libxl__egc *egc, libxl__ev_fd *ev,
>
--
Thanks,
Yang.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 00/27] Libxl migration v2
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
` (26 preceding siblings ...)
2015-07-09 18:26 ` [PATCH v2 27/27] tools/libxl: Drop all knowledge of toolstack callbacks Andrew Cooper
@ 2015-07-10 3:01 ` Yang Hongyang
27 siblings, 0 replies; 65+ messages in thread
From: Yang Hongyang @ 2015-07-10 3:01 UTC (permalink / raw)
To: Andrew Cooper, Xen-devel; +Cc: Wei Liu, Ian Jackson, Ian Campbell, Wen Congyang
On 07/10/2015 02:26 AM, Andrew Cooper wrote:
> This series adds support for the libxl migration v2 stream, and untangles the
> existing layering violations of the toolstack and qemu records.
>
> It can be found on the branch "libxl-migv2-v2"
> git://xenbits.xen.org/people/andrewcoop/xen.git
> http://xenbits.xen.org/git-http/people/andrewcoop/xen.git
>
> Major changes in v2 are being rebased over the libxl AO-abort series, and a
> redesign of the internal logic to support Remus/COLO buffering and failover.
Great! I've read the series, the redesigned logic seems correct to me, I will
rebase COLO series and test to see if it will work, thanks!
>
> At the end of the series, legacy migration is no longer used.
>
> The Remus code is untested by me. All other combinations of
> suspend/migrate/resume have been tested with PV and HVM guests (qemu-trad and
> qemu-upstream), including 32 -> 64 bit migration (which was the underlying bug
> causing us to write migration v2 in the first place).
>
> Anyway, thoughts/comments welcome. Please test!
>
> ~Andrew
>
> Summary of Acks/Modified/New from v1
>
> N bsd-sys-queue-h-seddery: Massage `offsetof'
> A tools/libxc: Always compile the compat qemu variables into xc_sr_context
> A tools/libxl: Introduce ROUNDUP()
> N tools/libxl: Introduce libxl__kill()
> AM tools/libxl: Stash all restore parameters in domain_create_state
> N tools/libxl: Split libxl__domain_create_state.restore_fd in two
> M tools/libxl: Extra management APIs for the save helper
> AM tools/xl: Mandatory flag indicating the format of the migration stream
> docs: Libxl migration v2 stream specification
> AM tools/python: Libxc migration v2 infrastructure
> AM tools/python: Libxl migration v2 infrastructure
> N tools/python: Other migration infrastructure
> AM tools/python: Verification utility for v2 stream spec compliance
> AM tools/python: Conversion utility for legacy migration streams
> M tools/libxl: Migration v2 stream format
> M tools/libxl: Infrastructure for reading a libxl migration v2 stream
> M tools/libxl: Support converting a legacy stream to a v2 stream
> M tools/libxl: Convert a legacy stream if needed
> M tools/libxc+libxl+xl: Restore v2 streams
> M tools/libxl: Infrastructure for writing a v2 stream
> M tools/libxc+libxl+xl: Save v2 streams
> AM docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams
> M tools/libxl: Write checkpoint records into the stream
> M tools/libx{c,l}: Introduce restore_callbacks.checkpoint()
> M tools/libxl: Handle checkpoint records in a libxl migration v2 stream
> A tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc
> A tools/libxl: Drop all knowledge of toolstack callbacks
>
>
>
> Andrew Cooper (23):
> tools/libxc: Always compile the compat qemu variables into xc_sr_context
> tools/libxl: Introduce ROUNDUP()
> tools/libxl: Introduce libxl__kill()
> tools/libxl: Stash all restore parameters in domain_create_state
> tools/libxl: Split libxl__domain_create_state.restore_fd in two
> tools/libxl: Extra management APIs for the save helper
> tools/xl: Mandatory flag indicating the format of the migration stream
> docs: Libxl migration v2 stream specification
> tools/python: Libxc migration v2 infrastructure
> tools/python: Libxl migration v2 infrastructure
> tools/python: Other migration infrastructure
> tools/python: Verification utility for v2 stream spec compliance
> tools/python: Conversion utility for legacy migration streams
> tools/libxl: Support converting a legacy stream to a v2 stream
> tools/libxl: Convert a legacy stream if needed
> tools/libxc+libxl+xl: Restore v2 streams
> tools/libxc+libxl+xl: Save v2 streams
> docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams
> tools/libxl: Write checkpoint records into the stream
> tools/libx{c,l}: Introduce restore_callbacks.checkpoint()
> tools/libxl: Handle checkpoint records in a libxl migration v2 stream
> tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc
> tools/libxl: Drop all knowledge of toolstack callbacks
>
> Ian Jackson (1):
> bsd-sys-queue-h-seddery: Massage `offsetof'
>
> Ross Lagerwall (3):
> tools/libxl: Migration v2 stream format
> tools/libxl: Infrastructure for reading a libxl migration v2 stream
> tools/libxl: Infrastructure for writing a v2 stream
>
> docs/specs/libxl-migration-stream.pandoc | 217 ++++++
> tools/include/xen-external/bsd-sys-queue-h-seddery | 2 +
> tools/libxc/Makefile | 2 -
> tools/libxc/include/xenguest.h | 9 +
> tools/libxc/xc_sr_common.h | 12 +-
> tools/libxc/xc_sr_restore.c | 71 +-
> tools/libxc/xc_sr_restore_x86_hvm.c | 124 ----
> tools/libxc/xc_sr_save_x86_hvm.c | 36 -
> tools/libxl/Makefile | 2 +
> tools/libxl/libxl.h | 19 +
> tools/libxl/libxl_aoutils.c | 15 +
> tools/libxl/libxl_convert_callout.c | 172 +++++
> tools/libxl/libxl_create.c | 86 ++-
> tools/libxl/libxl_dom.c | 65 +-
> tools/libxl/libxl_internal.h | 192 ++++-
> tools/libxl/libxl_save_callout.c | 70 +-
> tools/libxl/libxl_save_helper.c | 33 +-
> tools/libxl/libxl_save_msgs_gen.pl | 9 +-
> tools/libxl/libxl_sr_stream_format.h | 58 ++
> tools/libxl/libxl_stream_read.c | 731 ++++++++++++++++++++
> tools/libxl/libxl_stream_write.c | 554 +++++++++++++++
> tools/libxl/libxl_types.idl | 1 +
> tools/libxl/xl_cmdimpl.c | 9 +-
> tools/python/Makefile | 4 +
> tools/python/scripts/convert-legacy-stream | 678 ++++++++++++++++++
> tools/python/scripts/verify-stream-v2 | 174 +++++
> tools/python/setup.py | 1 +
> tools/python/xen/migration/legacy.py | 279 ++++++++
> tools/python/xen/migration/libxc.py | 446 ++++++++++++
> tools/python/xen/migration/libxl.py | 199 ++++++
> tools/python/xen/migration/public.py | 21 +
> tools/python/xen/migration/tests.py | 54 ++
> tools/python/xen/migration/verify.py | 37 +
> tools/python/xen/migration/xl.py | 12 +
> 34 files changed, 4014 insertions(+), 380 deletions(-)
> create mode 100644 docs/specs/libxl-migration-stream.pandoc
> create mode 100644 tools/libxl/libxl_convert_callout.c
> create mode 100644 tools/libxl/libxl_sr_stream_format.h
> create mode 100644 tools/libxl/libxl_stream_read.c
> create mode 100644 tools/libxl/libxl_stream_write.c
> create mode 100755 tools/python/scripts/convert-legacy-stream
> create mode 100755 tools/python/scripts/verify-stream-v2
> create mode 100644 tools/python/xen/migration/__init__.py
> create mode 100644 tools/python/xen/migration/legacy.py
> create mode 100644 tools/python/xen/migration/libxc.py
> create mode 100644 tools/python/xen/migration/libxl.py
> create mode 100644 tools/python/xen/migration/public.py
> create mode 100644 tools/python/xen/migration/tests.py
> create mode 100644 tools/python/xen/migration/verify.py
> create mode 100644 tools/python/xen/migration/xl.py
>
--
Thanks,
Yang.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 04/27] tools/libxl: Introduce libxl__kill()
2015-07-10 1:34 ` Yang Hongyang
@ 2015-07-10 8:56 ` Andrew Cooper
0 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-10 8:56 UTC (permalink / raw)
To: Yang Hongyang, Xen-devel; +Cc: Wei Liu, Ian Jackson, Ian Campbell
On 10/07/15 02:34, Yang Hongyang wrote:
> On 07/10/2015 02:26 AM, Andrew Cooper wrote:
>> as a wrapper to kill(2), and use it in preference to sendig in
>
> s/sendig/sendsig/
Oops yes - fixed.
~Andrew
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 04/27] tools/libxl: Introduce libxl__kill()
2015-07-09 18:26 ` [PATCH v2 04/27] tools/libxl: Introduce libxl__kill() Andrew Cooper
2015-07-10 1:34 ` Yang Hongyang
@ 2015-07-10 9:08 ` Wei Liu
2015-07-10 9:25 ` Andrew Cooper
2015-07-10 9:34 ` Ian Campbell
1 sibling, 2 replies; 65+ messages in thread
From: Wei Liu @ 2015-07-10 9:08 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Ian Campbell, Xen-devel
On Thu, Jul 09, 2015 at 07:26:30PM +0100, Andrew Cooper wrote:
> + * Local variables:
> + * mode: C
> + * c-basic-offset: 4
> + * indent-tabs-mode: nil
> + * End:
> + */
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 19fc425..9147de1 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -2244,6 +2244,8 @@ struct libxl__async_exec_state {
> int libxl__async_exec_start(libxl__async_exec_state *aes);
> bool libxl__async_exec_inuse(const libxl__async_exec_state *aes);
>
> +void libxl__kill(libxl__gc *gc, pid_t pid, int sig, const char *what);
_hidden void ...
other than that:
Acked-by: Wei Liu <wei.liu2@citrix.com>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 04/27] tools/libxl: Introduce libxl__kill()
2015-07-10 9:08 ` Wei Liu
@ 2015-07-10 9:25 ` Andrew Cooper
2015-07-10 9:34 ` Ian Campbell
1 sibling, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-10 9:25 UTC (permalink / raw)
To: Wei Liu; +Cc: Ian Jackson, Ian Campbell, Xen-devel
On 10/07/15 10:08, Wei Liu wrote:
> On Thu, Jul 09, 2015 at 07:26:30PM +0100, Andrew Cooper wrote:
>> + * Local variables:
>> + * mode: C
>> + * c-basic-offset: 4
>> + * indent-tabs-mode: nil
>> + * End:
>> + */
>> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
>> index 19fc425..9147de1 100644
>> --- a/tools/libxl/libxl_internal.h
>> +++ b/tools/libxl/libxl_internal.h
>> @@ -2244,6 +2244,8 @@ struct libxl__async_exec_state {
>> int libxl__async_exec_start(libxl__async_exec_state *aes);
>> bool libxl__async_exec_inuse(const libxl__async_exec_state *aes);
>>
>> +void libxl__kill(libxl__gc *gc, pid_t pid, int sig, const char *what);
> _hidden void ...
>
> other than that:
>
> Acked-by: Wei Liu <wei.liu2@citrix.com>
D'oh - fixed as well.
~Andrew
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 01/27] bsd-sys-queue-h-seddery: Massage `offsetof'
2015-07-09 18:26 ` [PATCH v2 01/27] bsd-sys-queue-h-seddery: Massage `offsetof' Andrew Cooper
@ 2015-07-10 9:32 ` Ian Campbell
0 siblings, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 9:32 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> From: Ian Jackson <Ian.Jackson@eu.citrix.com>
>
> For some reason BSD's queue.h uses `__offsetof'. It expects it to
> work just like offsetof. So use offsetof.
>
> Reported-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Signed-off-by: Ian Jackson <Ian.Jackson@eu.citrix.com>
Acked-by: Ian Campbell <ian.campbell@citrix.com>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 04/27] tools/libxl: Introduce libxl__kill()
2015-07-10 9:08 ` Wei Liu
2015-07-10 9:25 ` Andrew Cooper
@ 2015-07-10 9:34 ` Ian Campbell
1 sibling, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 9:34 UTC (permalink / raw)
To: Wei Liu; +Cc: Andrew Cooper, Ian Jackson, Xen-devel
On Fri, 2015-07-10 at 10:08 +0100, Wei Liu wrote:
> On Thu, Jul 09, 2015 at 07:26:30PM +0100, Andrew Cooper wrote:
> > + * Local variables:
> > + * mode: C
> > + * c-basic-offset: 4
> > + * indent-tabs-mode: nil
> > + * End:
> > + */
> > diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> > index 19fc425..9147de1 100644
> > --- a/tools/libxl/libxl_internal.h
> > +++ b/tools/libxl/libxl_internal.h
> > @@ -2244,6 +2244,8 @@ struct libxl__async_exec_state {
> > int libxl__async_exec_start(libxl__async_exec_state *aes);
> > bool libxl__async_exec_inuse(const libxl__async_exec_state *aes);
> >
> > +void libxl__kill(libxl__gc *gc, pid_t pid, int sig, const char *what);
>
> _hidden void ...
>
> other than that:
>
> Acked-by: Wei Liu <wei.liu2@citrix.com>
Likewise.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 06/27] tools/libxl: Split libxl__domain_create_state.restore_fd in two
2015-07-09 18:26 ` [PATCH v2 06/27] tools/libxl: Split libxl__domain_create_state.restore_fd in two Andrew Cooper
@ 2015-07-10 9:37 ` Ian Campbell
0 siblings, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 9:37 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> In a future patch, we shall support automatically converting a legacy stream
> to a v2 stream, in which case libxc needs to read from a different fd.
>
> Simply overwriting restore_fd does not work; the two fd's have different
> circumstances. The restore_fd needs to be returned to its origial state
> before libxl_domain_create_restore() returns, while in the converted case, the
> fd needs allocating and deallocating appropriately.
>
> No functional change.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 07/27] tools/libxl: Extra management APIs for the save helper
2015-07-09 18:26 ` [PATCH v2 07/27] tools/libxl: Extra management APIs for the save helper Andrew Cooper
@ 2015-07-10 9:41 ` Ian Campbell
2015-07-10 9:52 ` Andrew Cooper
0 siblings, 1 reply; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 9:41 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> With migration v2, there are several moving parts needing to be juggled at
> once. This requires the error handling logic to be able to query the state of
> each moving part, possibly before they have been started, and be able to
> cancel them.
Arguably helper_{stop,failed,done} all ought to be patched to use the
new libxl__save_helper_inuse helper. This isn't a blocker IMHO so,
nonetheless:
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 09/27] docs: Libxl migration v2 stream specification
2015-07-09 18:26 ` [PATCH v2 09/27] docs: Libxl migration v2 stream specification Andrew Cooper
@ 2015-07-10 9:46 ` Ian Campbell
0 siblings, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 9:46 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Ian Campbell <Ian.Campbell@citrix.com>
> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>
I think this is unchanged from last time, and my comments then were all
either rebutted or of the "it would be nice but not necessary if.."
type. So:
Acked-by: Ian Campbell <ian.campbell@citrix.com>
I'm still going to nitick your spelling though :-)
> +self-contained, extensible stream with each layer responsibile for its own
"responsible"
> +handed to `xc_domain_restore()`. `libxc` shall be resonsible for reading its
Another one.
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 12/27] tools/python: Other migration infrastructure
2015-07-09 18:26 ` [PATCH v2 12/27] tools/python: Other migration infrastructure Andrew Cooper
@ 2015-07-10 9:48 ` Ian Campbell
0 siblings, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 9:48 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> Contains:
> * Reverse-engineered notes of the legacy format from xg_save_restore.h
> * Python implementation of the legacy format
> * Public HVM Params used in the legacy stream
> * XL header format
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 15/27] tools/libxl: Migration v2 stream format
2015-07-09 18:26 ` [PATCH v2 15/27] tools/libxl: Migration v2 stream format Andrew Cooper
@ 2015-07-10 9:49 ` Ian Campbell
0 siblings, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 9:49 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Ross Lagerwall, Ian Jackson, Wei Liu, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> From: Ross Lagerwall <ross.lagerwall@citrix.com>
>
> C structures describing the Libxl migration v2 stream format
>
> Signed-off-by: Ross Lagerwall <ross.lagerwall@citrix.com>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
(I had an evil thought involving tools/libxl/libxl_types_internal.idl
but I shall spare you that...)
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 07/27] tools/libxl: Extra management APIs for the save helper
2015-07-10 9:41 ` Ian Campbell
@ 2015-07-10 9:52 ` Andrew Cooper
0 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-10 9:52 UTC (permalink / raw)
To: Ian Campbell; +Cc: Wei Liu, Ian Jackson, Xen-devel
On 10/07/15 10:41, Ian Campbell wrote:
> On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
>> With migration v2, there are several moving parts needing to be juggled at
>> once. This requires the error handling logic to be able to query the state of
>> each moving part, possibly before they have been started, and be able to
>> cancel them.
> Arguably helper_{stop,failed,done} all ought to be patched to use the
> new libxl__save_helper_inuse helper.
So they should. Done.
~Andrew
> This isn't a blocker IMHO so,
> nonetheless:
>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
>
>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-09 18:26 ` [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream Andrew Cooper
@ 2015-07-10 10:23 ` Ian Campbell
2015-07-10 10:47 ` Andrew Cooper
2015-07-10 12:17 ` Ian Jackson
1 sibling, 1 reply; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 10:23 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Ross Lagerwall, Ian Jackson, Wei Liu, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> From: Ross Lagerwall <ross.lagerwall@citrix.com>
>
> This contains the event machinary and state machines to read an act on a
"machinery"
[...]
> Large quantities of the logic here are completely overhauled since v1, mostly
> as part of fixing the checkpoint buffering bug which was the cause of the
> broken Remus failover. The result is actually more simple overall;
I agree, it looks much nicer, thanks!
> +struct libxl__stream_read_state {
> + /* filled by the user */
> + libxl__ao *ao;
> + int fd;
> + void (*completion_callback)(libxl__egc *egc,
> + libxl__stream_read_state *srs,
> + int rc);
> + /* Private */
> + int rc;
> + bool running;
[...]
> +void libxl__stream_read_start(libxl__egc *egc,
> + libxl__stream_read_state *stream)
> +{
> + libxl__datacopier_state *dc = &stream->dc;
> + int ret = 0;
> +
> + /* State initialisation. */
> + assert(!stream->running);
Since running is declared private and there is no _init function (I
think _start is effectively filling that role) I'm not sure that the
caller can necessarily be expected to have initialised anything other
than the ao, fd and callback fields.
You might choose to handle this as a request for a doc comment ("must
call LIBXL_FILLZERO on it to init"), or to add a separate init function
containing the memset or to do away with this check. I've not gotten to
the caller yet so I don't know which you will prefer.
> +
> + memset(dc, 0, sizeof(*dc));
> + dc->ao = stream->ao;
> + dc->readfd = stream->fd;
> + dc->writefd = -1;
> +
> + /* Start reading the stream header. */
> + ret = setup_read(stream, "stream header",
> + &stream->hdr, sizeof(stream->hdr),
> + stream_header_done);
> + if (ret)
> + goto err;
> +
> + stream->running = true;
> + stream->phase = SRS_PHASE_NORMAL;
> + LIBXL_STAILQ_INIT(&stream->record_queue);
> + stream->recursion_guard = 0;
> +
> + assert(!ret);
> + return;
> +
> + err:
> + assert(ret);
> + stream_failed(egc, stream, ret);
stream failed looks at stream->running, which due to the above might
also be uninitialised here.
> +static void stream_done(libxl__egc *egc,
> + libxl__stream_read_state *stream)
> +{
> + libxl__sr_record_buf *rec, *trec;
> +
> + assert(stream->running);
> + stream->running = false;
> +
> + if (stream->emu_carefd)
> + libxl__carefd_close(stream->emu_carefd);
> +
> + LIBXL_STAILQ_FOREACH_SAFE(rec, &stream->record_queue, entry, trec) {
> + free(rec->body);
> + free(rec);
> + }
Am I right in thinking that we should only get here with a non-empty
queue on failure? If so then perhaps:
assert(LIBXL_STAILQ_EMPTY(...) || stream->rc);
?
> +
> + stream->completion_callback(egc, stream, stream->rc);
> +}
> +
> +static void stream_continue(libxl__egc *egc,
> + libxl__stream_read_state *stream)
> +{
> + STATE_AO_GC(stream->ao);
> +
> + /* Must not mutually recurse with process_record() */
> + assert(stream->recursion_guard == false);
> + stream->recursion_guard = true;
This smells a bit like it ought to be a SRS_PHASE_PROCESSING or some
such, but lets leave that alone...
> +
> + switch (stream->phase) {
> + case SRS_PHASE_NORMAL:
> + /*
> + * Normal phase of the stream. We arrive here in several senarios.
"scenarios"
> +static void stream_header_done(libxl__egc *egc,
> + libxl__datacopier_state *dc,
> + int rc, int onwrite, int errnoval)
> +{
> + libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
> + libxl__sr_hdr *hdr = &stream->hdr;
> + STATE_AO_GC(dc->ao);
> + int ret = 0;
> +
> + if (rc || onwrite || errnoval) {
> + ret = ERROR_FAIL;
> + LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
Could use LOGEV(ERRRO, errnoval, "rc %d, onweite %d", rc, onwrite);
(for all cases I think).
Actually, doesn't dc guarantee to always have already logged on fail?
Comments in the libxl_internal.h suggest so, apart from the abort case,
so I think maybe you can avoid logging explicitly here.
> + goto err;
> + }
> +
> + hdr->ident = be64toh(hdr->ident);
> + hdr->version = be32toh(hdr->version);
> + hdr->options = be32toh(hdr->options);
> +
> + if (hdr->ident != RESTORE_STREAM_IDENT) {
> + ret = ERROR_FAIL;
Eventually I suspect the xapi people would like to see something more
specific at least for the general "SRS header fail" if not the
individual reasons.
> + LOG(ERROR,
> + "Invalid ident: expected 0x%016"PRIx64", got 0x%016"PRIx64,
> + RESTORE_STREAM_IDENT, hdr->ident);
> + goto err;
> + }
> + if (hdr->version != RESTORE_STREAM_VERSION) {
> + ret = ERROR_FAIL;
> + LOG(ERROR, "Unexpected Version: expected %u, got %u",
hdr->version is a uint32_t, so PRIu32 would be more appropriate.
> + RESTORE_STREAM_VERSION, hdr->version);
> + goto err;
> + }
> + if (hdr->options & RESTORE_OPT_BIG_ENDIAN) {
> + ret = ERROR_FAIL;
> + LOG(ERROR, "Unable to handle big endian streams");
> + goto err;
> + }
> +
> + LOG(DEBUG, "Stream v%u%s", hdr->version,
and again.
Actually looking around since you've used uintXX_t throughout the format
structs, I think you need a lot more PRI[ux]FOO around the place.
_If_ you've compile tested this for both 32- and 64-bit and it works we
could perhaps leave that audit until later.
> +static void setup_read_record(libxl__egc *egc,
> + libxl__stream_read_state *stream)
> +{
> + STATE_AO_GC(stream->ao);
> + libxl__sr_record_buf *rec = NULL;
> + int ret;
> +
> + assert(stream->incoming_record == NULL);
> +
> + stream->incoming_record = rec = libxl__zalloc(NOGC, sizeof(*rec));
I recall Ian J and you discussing NOGC allocations on IRC. Was the
conclusion that it was OK, or that it could be fixed later, or that it
should be fixed now via an nested ao or something similar?
Unless the answer is "fixed now" I think the reason for the NOGC should
be in either the commit log or a comment (in the header, around about
the definition of the allocated data structure).
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 17/27] tools/libxl: Support converting a legacy stream to a v2 stream
2015-07-09 18:26 ` [PATCH v2 17/27] tools/libxl: Support converting a legacy stream to a " Andrew Cooper
@ 2015-07-10 10:28 ` Ian Campbell
2015-07-10 10:39 ` Andrew Cooper
2015-07-10 12:28 ` Ian Jackson
1 sibling, 1 reply; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 10:28 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> When a legacy stream is found, it needs to be converted to a v2 stream for the
> reading logic. This is done by exec()ing the python conversion utility.
>
> One complication is that the caller of this interface needs to assume
> ownership of the output fd, to prevent it being closed while still in use in a
> datacopier.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Ian Campbell <Ian.Campbell@citrix.com>
> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>
I think various internals should use the
libxl__convert_legacy_stream_inuse helper.
With that:
Acked-by: Ian Campbell <ian.campbell@citrix.com>
Perhaps libxl__conversion_helper_state could become empty on !x86? My
Ack stands either way.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 18/27] tools/libxl: Convert a legacy stream if needed
2015-07-09 18:26 ` [PATCH v2 18/27] tools/libxl: Convert a legacy stream if needed Andrew Cooper
@ 2015-07-10 10:31 ` Ian Campbell
2015-07-10 12:41 ` Ian Jackson
1 sibling, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 10:31 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> For backwards compatibility, a legacy stream needs converting before it can be
> read by the v2 stream logic.
>
> This causes the v2 stream logic to need to juggle two parallel tasks.
> check_stream_finished() is introduced for the purpose of joining the tasks in
> both success and error cases.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 17/27] tools/libxl: Support converting a legacy stream to a v2 stream
2015-07-10 10:28 ` Ian Campbell
@ 2015-07-10 10:39 ` Andrew Cooper
0 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-10 10:39 UTC (permalink / raw)
To: Ian Campbell; +Cc: Wei Liu, Ian Jackson, Xen-devel
On 10/07/15 11:28, Ian Campbell wrote:
> On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
>> When a legacy stream is found, it needs to be converted to a v2 stream for the
>> reading logic. This is done by exec()ing the python conversion utility.
>>
>> One complication is that the caller of this interface needs to assume
>> ownership of the output fd, to prevent it being closed while still in use in a
>> datacopier.
>>
>> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
>> CC: Ian Campbell <Ian.Campbell@citrix.com>
>> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
>> CC: Wei Liu <wei.liu2@citrix.com>
> I think various internals should use the
> libxl__convert_legacy_stream_inuse helper.
Already fixed pre-emptively from your patch 7 comments.
>
> With that:
> Acked-by: Ian Campbell <ian.campbell@citrix.com>
>
> Perhaps libxl__conversion_helper_state could become empty on !x86? My
> Ack stands either way.
Will do.
~Andrew
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 19/27] tools/libxc+libxl+xl: Restore v2 streams
2015-07-09 18:26 ` [PATCH v2 19/27] tools/libxc+libxl+xl: Restore v2 streams Andrew Cooper
@ 2015-07-10 10:45 ` Ian Campbell
0 siblings, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 10:45 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> #define LIBXL_HAVE_SOCKET_BITMAP_ALLOC 1
>
> +/*
> + * LIBXL_HAVE_STREAM_V2
These all need another word in them I think. _MIGRATION_, or _SRM_
or ... (I think I prefer SRM since migration is just one aspect)
> @@ -439,6 +447,28 @@ static void record_body_done(libxl__egc *egc,
> stream_failed(egc, stream, ret);
> }
>
> +void libxl__xc_domain_restore_done(libxl__egc *egc, void *dcs_void,
> + int ret, int retval, int errnoval)
> +{
> + libxl__domain_create_state *dcs = dcs_void;
> + STATE_AO_GC(dcs->ao);
> +
> + if (ret)
> + goto err;
> +
> + if (retval) {
> + LOGEV(ERROR, errnoval, "restoring domain");
> + ret = ERROR_FAIL;
> + goto err;
> + }
> +
> + stream_continue(egc, &dcs->srs);
This confused me briefly (done==continue??), a comment indicating that
this callback represents libxc handing back control of the fd to libxl
would maybe be useful?
With those two minor tweaks,
Acked-by: Ian Campbell <ian.campbell@citrix.com>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-10 10:23 ` Ian Campbell
@ 2015-07-10 10:47 ` Andrew Cooper
2015-07-10 11:16 ` Ian Jackson
2015-07-10 11:25 ` Ian Campbell
0 siblings, 2 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-10 10:47 UTC (permalink / raw)
To: Ian Campbell; +Cc: Ross Lagerwall, Ian Jackson, Wei Liu, Xen-devel
On 10/07/15 11:23, Ian Campbell wrote:
> On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
>> From: Ross Lagerwall <ross.lagerwall@citrix.com>
>>
>> This contains the event machinary and state machines to read an act on a
> "machinery"
>
> [...]
>
>
>> Large quantities of the logic here are completely overhauled since v1, mostly
>> as part of fixing the checkpoint buffering bug which was the cause of the
>> broken Remus failover. The result is actually more simple overall;
> I agree, it looks much nicer, thanks!
>
>> +struct libxl__stream_read_state {
>> + /* filled by the user */
>> + libxl__ao *ao;
>> + int fd;
>> + void (*completion_callback)(libxl__egc *egc,
>> + libxl__stream_read_state *srs,
>> + int rc);
>> + /* Private */
>> + int rc;
>> + bool running;
> [...]
>> +void libxl__stream_read_start(libxl__egc *egc,
>> + libxl__stream_read_state *stream)
>> +{
>> + libxl__datacopier_state *dc = &stream->dc;
>> + int ret = 0;
>> +
>> + /* State initialisation. */
>> + assert(!stream->running);
> Since running is declared private and there is no _init function (I
> think _start is effectively filling that role) I'm not sure that the
> caller can necessarily be expected to have initialised anything other
> than the ao, fd and callback fields.
It was a sanity check that _start() doesn't get called twice (guess what
I managed to do while developing). It can probably be dropped.
>
> You might choose to handle this as a request for a doc comment ("must
> call LIBXL_FILLZERO on it to init"), or to add a separate init function
> containing the memset or to do away with this check. I've not gotten to
> the caller yet so I don't know which you will prefer.
It is all zeroed because of the way dcs is constructed. I suppose I can
also drop the zeroing of the dc.
>
>> +
>> + memset(dc, 0, sizeof(*dc));
>> + dc->ao = stream->ao;
>> + dc->readfd = stream->fd;
>> + dc->writefd = -1;
>> +
>> + /* Start reading the stream header. */
>> + ret = setup_read(stream, "stream header",
>> + &stream->hdr, sizeof(stream->hdr),
>> + stream_header_done);
>> + if (ret)
>> + goto err;
>> +
>> + stream->running = true;
>> + stream->phase = SRS_PHASE_NORMAL;
>> + LIBXL_STAILQ_INIT(&stream->record_queue);
>> + stream->recursion_guard = 0;
>> +
>> + assert(!ret);
>> + return;
>> +
>> + err:
>> + assert(ret);
>> + stream_failed(egc, stream, ret);
> stream failed looks at stream->running, which due to the above might
> also be uninitialised here.
Oops yes. I fixed this in the write() side but forgot to propagate the
bugfix back to the read side.
>
>> +static void stream_done(libxl__egc *egc,
>> + libxl__stream_read_state *stream)
>> +{
>> + libxl__sr_record_buf *rec, *trec;
>> +
>> + assert(stream->running);
>> + stream->running = false;
>> +
>> + if (stream->emu_carefd)
>> + libxl__carefd_close(stream->emu_carefd);
>> +
>> + LIBXL_STAILQ_FOREACH_SAFE(rec, &stream->record_queue, entry, trec) {
>> + free(rec->body);
>> + free(rec);
>> + }
> Am I right in thinking that we should only get here with a non-empty
> queue on failure? If so then perhaps:
> assert(LIBXL_STAILQ_EMPTY(...) || stream->rc);
>
> ?
I believe so. There is no way the stream should succeed if there are
outstanding buffered records.
>
>> +
>> + stream->completion_callback(egc, stream, stream->rc);
>> +}
>> +
>> +static void stream_continue(libxl__egc *egc,
>> + libxl__stream_read_state *stream)
>> +{
>> + STATE_AO_GC(stream->ao);
>> +
>> + /* Must not mutually recurse with process_record() */
>> + assert(stream->recursion_guard == false);
>> + stream->recursion_guard = true;
> This smells a bit like it ought to be a SRS_PHASE_PROCESSING or some
> such, but lets leave that alone...
This check is pre-emptively avoid the naive bug which would occur if
process_record() called back into stream_continue() and there were many
TOOLSTACK records back to back in the processing queue.
In that case (and potentially future records as well), the two functions
would mutually recurse based on the contents of the stream.
>
>> +
>> + switch (stream->phase) {
>> + case SRS_PHASE_NORMAL:
>> + /*
>> + * Normal phase of the stream. We arrive here in several senarios.
> "scenarios"
>
>> +static void stream_header_done(libxl__egc *egc,
>> + libxl__datacopier_state *dc,
>> + int rc, int onwrite, int errnoval)
>> +{
>> + libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
>> + libxl__sr_hdr *hdr = &stream->hdr;
>> + STATE_AO_GC(dc->ao);
>> + int ret = 0;
>> +
>> + if (rc || onwrite || errnoval) {
>> + ret = ERROR_FAIL;
>> + LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
> Could use LOGEV(ERRRO, errnoval, "rc %d, onweite %d", rc, onwrite);
> (for all cases I think).
>
> Actually, doesn't dc guarantee to always have already logged on fail?
> Comments in the libxl_internal.h suggest so, apart from the abort case,
> so I think maybe you can avoid logging explicitly here.
So it does. That is handy.
>
>> + goto err;
>> + }
>> +
>> + hdr->ident = be64toh(hdr->ident);
>> + hdr->version = be32toh(hdr->version);
>> + hdr->options = be32toh(hdr->options);
>> +
>> + if (hdr->ident != RESTORE_STREAM_IDENT) {
>> + ret = ERROR_FAIL;
> Eventually I suspect the xapi people would like to see something more
> specific at least for the general "SRS header fail" if not the
> individual reasons.
If you don't object too strongly, I would prefer to leave that
bikeshedding to the error value improvements work.
>
>> + LOG(ERROR,
>> + "Invalid ident: expected 0x%016"PRIx64", got 0x%016"PRIx64,
>> + RESTORE_STREAM_IDENT, hdr->ident);
>> + goto err;
>> + }
>> + if (hdr->version != RESTORE_STREAM_VERSION) {
>> + ret = ERROR_FAIL;
>> + LOG(ERROR, "Unexpected Version: expected %u, got %u",
> hdr->version is a uint32_t, so PRIu32 would be more appropriate.
In both 32 and 64 builds they are equivalent. All parameters are
promoted to unsigned int.
>
>> + RESTORE_STREAM_VERSION, hdr->version);
>> + goto err;
>> + }
>> + if (hdr->options & RESTORE_OPT_BIG_ENDIAN) {
>> + ret = ERROR_FAIL;
>> + LOG(ERROR, "Unable to handle big endian streams");
>> + goto err;
>> + }
>> +
>> + LOG(DEBUG, "Stream v%u%s", hdr->version,
> and again.
>
> Actually looking around since you've used uintXX_t throughout the format
> structs, I think you need a lot more PRI[ux]FOO around the place.
Will do
>
> _If_ you've compile tested this for both 32- and 64-bit and it works we
> could perhaps leave that audit until later.
>
>> +static void setup_read_record(libxl__egc *egc,
>> + libxl__stream_read_state *stream)
>> +{
>> + STATE_AO_GC(stream->ao);
>> + libxl__sr_record_buf *rec = NULL;
>> + int ret;
>> +
>> + assert(stream->incoming_record == NULL);
>> +
>> + stream->incoming_record = rec = libxl__zalloc(NOGC, sizeof(*rec));
> I recall Ian J and you discussing NOGC allocations on IRC. Was the
> conclusion that it was OK, or that it could be fixed later, or that it
> should be fixed now via an nested ao or something similar?
>
> Unless the answer is "fixed now" I think the reason for the NOGC should
> be in either the commit log or a comment (in the header, around about
> the definition of the allocated data structure).
I will add a note about in the commit message. We agreed on IRC that
NOGC was OK. It might be possible to switch to some nested ao later,
but that depends entirely on the COLO work.
~Andrew
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 21/27] tools/libxc+libxl+xl: Save v2 streams
2015-07-09 18:26 ` [PATCH v2 21/27] tools/libxc+libxl+xl: Save v2 streams Andrew Cooper
@ 2015-07-10 10:57 ` Ian Campbell
0 siblings, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 10:57 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> This is a complicated set of changes which must be done together for
> bisectability.
>
> * libxl-save-helper is updated to unconditionally use libxc migration v2.
> * libxl compatibility workarounds in libxc are disabled for save operations.
> * libxl__stream_write_start() is logically spliced into the event location
> where libxl__xc_domain_save() used to reside.
> * xl is updated to indicate that the stream is now v2
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 22/27] docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams
2015-07-09 18:26 ` [PATCH v2 22/27] docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams Andrew Cooper
@ 2015-07-10 10:59 ` Ian Campbell
0 siblings, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 10:59 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> In a remus senario, libxc will write a CHECKPOINT record, then hand ownership
> of the fd to libxl. Libxl then writes any records required and finishes with
> a CHECKPOINT_END record, then hands ownership of the fd back to libxc.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
Acked-by: Ian Campbell <Ian.Campbell@citrix.com>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 23/27] tools/libxl: Write checkpoint records into the stream
2015-07-09 18:26 ` [PATCH v2 23/27] tools/libxl: Write checkpoint records into the stream Andrew Cooper
@ 2015-07-10 11:02 ` Ian Campbell
2015-07-10 11:47 ` Wei Liu
1 sibling, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 11:02 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> @@ -187,13 +217,24 @@ static void stream_failed(libxl__egc *egc,
> assert(rc);
> stream->rc = rc;
>
> - if (stream->running)
> + if (stream->running) {
> + /*
> + *If we are in a checkpoint, pass the failure to libxc, which will come
Missing a space after "*".
Acked-by: Ian Campbell <ian.campbell@citrix.com>
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 20/27] tools/libxl: Infrastructure for writing a v2 stream
2015-07-09 18:26 ` [PATCH v2 20/27] tools/libxl: Infrastructure for writing a v2 stream Andrew Cooper
@ 2015-07-10 11:10 ` Ian Campbell
0 siblings, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 11:10 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Ross Lagerwall, Ian Jackson, Wei Liu, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> From: Ross Lagerwall <ross.lagerwall@citrix.com>
>
> This contains the event machinary and state machines to write non-checkpointed
"machinery"
> /* Helper to set up reading some data from the stream. */
> +static void write_done(libxl__egc *egc,
> + libxl__datacopier_state *dc,
> + int rc, int onwrite, int errnoval)
> +{
> + libxl__stream_write_state *stream = CONTAINER_OF(dc, *stream, dc);
> + STATE_AO_GC(stream->ao);
> + sws_record_done_cb cb = stream->record_done_callback;
> +
> + stream->record_done_callback = NULL;
> +
> + if (onwrite || errnoval) {
> + LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
As with saving, I think the dc has logged.
> +void libxl__stream_write_start(libxl__egc *egc,
> + libxl__stream_write_state *stream)
> +{
> + libxl__datacopier_state *dc = &stream->dc;
> + STATE_AO_GC(stream->ao);
> + struct libxl__sr_hdr hdr = { 0 };
> + int ret = 0;
> +
> + assert(!stream->running);
I think we both know what I'm going to say here ;-)
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 24/27] tools/libx{c, l}: Introduce restore_callbacks.checkpoint()
2015-07-09 18:26 ` [PATCH v2 24/27] tools/libx{c, l}: Introduce restore_callbacks.checkpoint() Andrew Cooper
@ 2015-07-10 11:13 ` Ian Campbell
0 siblings, 0 replies; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 11:13 UTC (permalink / raw)
To: Andrew Cooper
Cc: Wei Liu, Yang Hongyang, Ian Jackson, Wen Congyang, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> And call it when a checkpoint record is found in the libxc stream.
>
> Some parts of this patch have been based on patches from the COLO series.
>
> Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Ian Campbell <Ian.Campbell@citrix.com>
> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>
>
> ---
> v2: Borrow sufficient fragments from several COLO patches to get
> BROKEN_CHANNEL and checkpoint failover to function.
> ---
> tools/libxc/include/xenguest.h | 9 ++++++
> tools/libxc/xc_sr_common.h | 7 +++--
> tools/libxc/xc_sr_restore.c | 53 ++++++++++++++++++++++++++----------
> tools/libxl/libxl_save_msgs_gen.pl | 2 +-
> 4 files changed, 53 insertions(+), 18 deletions(-)
>
> diff --git a/tools/libxc/include/xenguest.h b/tools/libxc/include/xenguest.h
> index 7581263..8799c9f 100644
> --- a/tools/libxc/include/xenguest.h
> +++ b/tools/libxc/include/xenguest.h
> @@ -102,6 +102,15 @@ struct restore_callbacks {
> int (*toolstack_restore)(uint32_t domid, const uint8_t *buf,
> uint32_t size, void* data);
>
> + /* A checkpoint record has been found in the stream.
> + *
> + * returns:
> + * 0: (error) terminate processing
> + * 1: (success) continue normally
Where "continue" means "keep reading from the stream and expect more
checkpoints" (as opposed to continue on to running the guest), I think?
With that clarified: Acked-by: Ian Campbell <ian.campbell@citrix.com>
Ian
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-10 10:47 ` Andrew Cooper
@ 2015-07-10 11:16 ` Ian Jackson
2015-07-10 11:25 ` Ian Campbell
1 sibling, 0 replies; 65+ messages in thread
From: Ian Jackson @ 2015-07-10 11:16 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Ross Lagerwall, Wei Liu, Ian Campbell, Xen-devel
Andrew Cooper writes ("Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream"):
> On 10/07/15 11:23, Ian Campbell wrote:
> >> +void libxl__stream_read_start(libxl__egc *egc,
> >> + libxl__stream_read_state *stream)
> >> +{
> >> + libxl__datacopier_state *dc = &stream->dc;
> >> + int ret = 0;
> >> +
> >> + /* State initialisation. */
> >> + assert(!stream->running);
> >
> > Since running is declared private and there is no _init function (I
> > think _start is effectively filling that role) I'm not sure that the
> > caller can necessarily be expected to have initialised anything other
> > than the ao, fd and callback fields.
>
> It was a sanity check that _start() doesn't get called twice (guess what
> I managed to do while developing). It can probably be dropped.
I don't mind this at all but I think if you do this you should:
* provide an _init method
* document that _init must be called before start
> > You might choose to handle this as a request for a doc comment ("must
> > call LIBXL_FILLZERO on it to init"), or to add a separate init function
> > containing the memset or to do away with this check. I've not gotten to
> > the caller yet so I don't know which you will prefer.
>
> It is all zeroed because of the way dcs is constructed. I suppose I can
> also drop the zeroing of the dc.
Providing an init method would mean that if this thing needs to grow
calls to sub-states' init methods, there is somewhere to put them.
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 25/27] tools/libxl: Handle checkpoint records in a libxl migration v2 stream
2015-07-09 18:26 ` [PATCH v2 25/27] tools/libxl: Handle checkpoint records in a libxl migration v2 stream Andrew Cooper
@ 2015-07-10 11:18 ` Ian Campbell
2015-07-10 14:34 ` Andrew Cooper
0 siblings, 1 reply; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 11:18 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Xen-devel
On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
> @@ -3349,6 +3353,8 @@ struct libxl__stream_read_state {
> LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue;
> enum {
> SRS_PHASE_NORMAL,
> + SRS_PHASE_BUFFERING,
> + SRS_PHASE_UNBUFFERING,
I'd be inclined towards calling the latter DRAINING or PROCESSING or
some such, perhaps tying into my comment about the separate recursion
guard thing earlier on in the series.
But that's a bikeshedding issue on an internal thing so my only real
comment is:
> + if (rc == 0)
> + ret = 0; /* Success */
> + else if (stream->phase == SRS_PHASE_BUFFERING)
> + ret = 2; /* Failover */
> + else
> + ret = 1; /* Error (fatal) */
Maybe we should have had an enum or some #defines for these (this is
really a comment on some previous patch).
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-10 10:47 ` Andrew Cooper
2015-07-10 11:16 ` Ian Jackson
@ 2015-07-10 11:25 ` Ian Campbell
2015-07-10 12:28 ` Andrew Cooper
1 sibling, 1 reply; 65+ messages in thread
From: Ian Campbell @ 2015-07-10 11:25 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Ross Lagerwall, Ian Jackson, Wei Liu, Xen-devel
On Fri, 2015-07-10 at 11:47 +0100, Andrew Cooper wrote:
> >> +void libxl__stream_read_start(libxl__egc *egc,
> >> + libxl__stream_read_state *stream)
> >> +{
> >> + libxl__datacopier_state *dc = &stream->dc;
> >> + int ret = 0;
> >> +
> >> + /* State initialisation. */
> >> + assert(!stream->running);
> > Since running is declared private and there is no _init function (I
> > think _start is effectively filling that role) I'm not sure that the
> > caller can necessarily be expected to have initialised anything other
> > than the ao, fd and callback fields.
>
> It was a sanity check that _start() doesn't get called twice (guess what
> I managed to do while developing). It can probably be dropped.
>
> >
> > You might choose to handle this as a request for a doc comment ("must
> > call LIBXL_FILLZERO on it to init"), or to add a separate init function
> > containing the memset or to do away with this check. I've not gotten to
> > the caller yet so I don't know which you will prefer.
>
> It is all zeroed because of the way dcs is constructed. I suppose I can
> also drop the zeroing of the dc.
Got it. I think a comment that libxl__stream_read_state should be init'd
to zero would still be worthwhile, even if the actual implementation of
that happens implicitly through its containing type.
Then your sanity check could legitimately remain.
> >
> >> +
> >> + stream->completion_callback(egc, stream, stream->rc);
> >> +}
> >> +
> >> +static void stream_continue(libxl__egc *egc,
> >> + libxl__stream_read_state *stream)
> >> +{
> >> + STATE_AO_GC(stream->ao);
> >> +
> >> + /* Must not mutually recurse with process_record() */
> >> + assert(stream->recursion_guard == false);
> >> + stream->recursion_guard = true;
> > This smells a bit like it ought to be a SRS_PHASE_PROCESSING or some
> > such, but lets leave that alone...
>
> This check is pre-emptively avoid the naive bug which would occur if
> process_record() called back into stream_continue() and there were many
> TOOLSTACK records back to back in the processing queue.
>
> In that case (and potentially future records as well), the two functions
> would mutually recurse based on the contents of the stream.
Do you mean they would do so legitimately in that case, or in error?
> >> + goto err;
> >> + }
> >> +
> >> + hdr->ident = be64toh(hdr->ident);
> >> + hdr->version = be32toh(hdr->version);
> >> + hdr->options = be32toh(hdr->options);
> >> +
> >> + if (hdr->ident != RESTORE_STREAM_IDENT) {
> >> + ret = ERROR_FAIL;
> > Eventually I suspect the xapi people would like to see something more
> > specific at least for the general "SRS header fail" if not the
> > individual reasons.
>
> If you don't object too strongly, I would prefer to leave that
> bikeshedding to the error value improvements work.
I don't mind, it's easy enough to change from ERROR_FAIL to something
more specific.
You might incur the wrath of Rob/Euan though ;-)
> >
> > _If_ you've compile tested this for both 32- and 64-bit and it works we
> > could perhaps leave that audit until later.
> >
> >> +static void setup_read_record(libxl__egc *egc,
> >> + libxl__stream_read_state *stream)
> >> +{
> >> + STATE_AO_GC(stream->ao);
> >> + libxl__sr_record_buf *rec = NULL;
> >> + int ret;
> >> +
> >> + assert(stream->incoming_record == NULL);
> >> +
> >> + stream->incoming_record = rec = libxl__zalloc(NOGC, sizeof(*rec));
> > I recall Ian J and you discussing NOGC allocations on IRC. Was the
> > conclusion that it was OK, or that it could be fixed later, or that it
> > should be fixed now via an nested ao or something similar?
> >
> > Unless the answer is "fixed now" I think the reason for the NOGC should
> > be in either the commit log or a comment (in the header, around about
> > the definition of the allocated data structure).
>
> I will add a note about in the commit message. We agreed on IRC that
> NOGC was OK. It might be possible to switch to some nested ao later,
> but that depends entirely on the COLO work.
ACK. Please remember to say why it is ok, not just that it is.
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 23/27] tools/libxl: Write checkpoint records into the stream
2015-07-09 18:26 ` [PATCH v2 23/27] tools/libxl: Write checkpoint records into the stream Andrew Cooper
2015-07-10 11:02 ` Ian Campbell
@ 2015-07-10 11:47 ` Wei Liu
1 sibling, 0 replies; 65+ messages in thread
From: Wei Liu @ 2015-07-10 11:47 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Jackson, Ian Campbell, Xen-devel
On Thu, Jul 09, 2015 at 07:26:49PM +0100, Andrew Cooper wrote:
> when signalled to do so by libxl__remus_domain_checkpoint_callback()
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Ian Campbell <Ian.Campbell@citrix.com>
> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>
>
> ---
[...]
> diff --git a/tools/libxl/libxl_stream_write.c b/tools/libxl/libxl_stream_write.c
> index 331173f..0f5216b 100644
> --- a/tools/libxl/libxl_stream_write.c
> +++ b/tools/libxl/libxl_stream_write.c
> @@ -23,6 +23,9 @@
> * - libxl__stream_write_start()
> * - Start writing a stream from the start.
> *
> + * - libxl__stream_write_start()
libxl__stream_write_start_checkpoint()
> + * - Write the records which form a checkpoint into a stream.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-09 18:26 ` [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream Andrew Cooper
2015-07-10 10:23 ` Ian Campbell
@ 2015-07-10 12:17 ` Ian Jackson
2015-07-10 12:56 ` Andrew Cooper
1 sibling, 1 reply; 65+ messages in thread
From: Ian Jackson @ 2015-07-10 12:17 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Ross Lagerwall, Wei Liu, Ian Campbell, Xen-devel
Andrew Cooper writes ("[PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream"):
> From: Ross Lagerwall <ross.lagerwall@citrix.com>
>
> This contains the event machinary and state machines to read an act on a
> non-checkpointed migration v2 stream (with the exception of the
> xc_domain_restore() handling which is spliced later in a bisectable way).
>
> It also contains some boilerplate to help support checkpointed streams, which
> shall be introduced in a later patch.
...
> +/* State for manipulating a libxl migration v2 stream */
> +typedef struct libxl__stream_read_state libxl__stream_read_state;
...
> +struct libxl__stream_read_state {
> + /* filled by the user */
> + libxl__ao *ao;
> + int fd;
> + void (*completion_callback)(libxl__egc *egc,
> + libxl__stream_read_state *srs,
> + int rc);
> + /* Private */
> + int rc;
> + bool running;
> +
> + /* Main stream-reading data */
> + libxl__datacopier_state dc;
> + libxl__sr_hdr hdr;
> + libxl__sr_record_buf *incoming_record;
> + LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue;
This comes from malloc, not from the gc. That needs a comment. (Is
the same true of incoming_record ? I think it is.)
> + enum {
> + SRS_PHASE_NORMAL,
> + } phase;
> + bool recursion_guard;
> +
> + /* Emulator blob handling */
> + libxl__datacopier_state emu_dc;
> + libxl__carefd *emu_carefd;
> +};
...
The comments in libxl_stream_read.c are good but I'm afraid I'm going
to ask for some commentary about the legal states and/or the
invariants in the data structure.
That is, you've written a state machine but the states are not
explicitly listed.
As far as I can tell, from the outside, this machinery has the usual
Undefined/Idle/Active states. But does it not have more states on the
inside ? PHASE at least needs explaining. See also comments about
the record queue, below.
For an example of the kind of thing I mean, see libxl_spawn.
Particularly, libxl_internal.h:
* Each libxl__spawn_state is in one of these states
* Undefined, Idle, Attached, Detaching
and the much more detailed state and data structure table in
libxl_exec.c
* Full set of possible states of a libxl__spawn_state and its _detachable:
I think you probably don't need to write down the external states for
your srs machinery because the states are Undefined, Idle and Active
like most things. (Assuming you do something about the anomalous
initialisation and checking of `running'.)
What I'm missing is the internal description.
> +/*
> + * Infrastructure for reading and acting on the contents of a libxl migration
> + * stream. There are a lot of moving parts here.
(Can you wrap to 70 or at least 75?)
Thanks.
> +void libxl__stream_read_start(libxl__egc *egc,
> + libxl__stream_read_state *stream)
> +{
> + libxl__datacopier_state *dc = &stream->dc;
> + int ret = 0;
> +
> + /* State initialisation. */
> + assert(!stream->running);
This is being discussed apropos of Ian C's review.
> + memset(dc, 0, sizeof(*dc));
Should be FILLZERO.
> + /* Start reading the stream header. */
> + ret = setup_read(stream, "stream header",
> + &stream->hdr, sizeof(stream->hdr),
> + stream_header_done);
> + if (ret)
> + goto err;
`ret' should be `rc'.
This needs to be changed throughout.
> + stream->running = true;
> + stream->phase = SRS_PHASE_NORMAL;
> + LIBXL_STAILQ_INIT(&stream->record_queue);
> + stream->recursion_guard = 0;
These initialisations-to-empty need to be moved to before any part of
this function which might fail.
As it is, if setup_read fails I think you end up reading from the
uninitialised stream->record_queue, in stream_done. (In your actual
code this does not matter because the caller did a FILLZERO but you
should not be relying on that.)
> +void libxl__stream_read_abort(libxl__egc *egc,
> + libxl__stream_read_state *stream, int rc)
> +{
> + stream_failed(egc, stream, rc);
This has the same signature as stream_failed. Perhaps stream_failed
could be eliminated and replaced with this ? Or maybe...
> +static void stream_success(libxl__egc *egc, libxl__stream_read_state *stream)
> +{
> + stream->rc = 0;
> + stream_done(egc, stream);
This function has one call site. As an alternative to abolishing
stream_failed, maybe stream_failed should be renamed to
`stream_complete' and then the call site for stream_success could
simply say `stream_complete(egc, stream, 0)' ?
> +static void stream_failed(libxl__egc *egc,
> + libxl__stream_read_state *stream, int rc)
> +{
> + assert(rc);
> + stream->rc = rc;
> +
> + if (stream->running) {
> + stream_done(egc, stream);
Is the check for running necessary here ?
Not checking, and instead relying on all the individual parts to be
properly initialised, makes the correctness clearer.
> +static void stream_header_done(libxl__egc *egc,
> + libxl__datacopier_state *dc,
> + int rc, int onwrite, int errnoval)
I think it would be clearer if this function was immediately after
libxl_stream_read_start. Probably moving libxl_stream_read_start is
the right answer.
Also, putting stream_continue after stream_header_done might help.
I appreciate that these functions form a loop so can't be strictly in
execution order, but it is much easier to read if the first iteration
is in the right order. As CODING_STYLE says:
The code for asynchronous operations should be laid out in
chronological order. That is, where there is a chain of callback
functions, each subsequent function should be, textually, the next
function in the file. This will normally involve predeclaring the
callback functions. Synchronous helper functions should be separated
out into a section preceding the main callback chain.
AFAICT there are a few other out-of-order functions.
> + int ret = 0;
> +
> + if (rc || onwrite || errnoval) {
> + ret = ERROR_FAIL;
> + LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
> + goto err;
> + }
> +
> + hdr->ident = be64toh(hdr->ident);
> + hdr->version = be32toh(hdr->version);
> + hdr->options = be32toh(hdr->options);
> +
> + if (hdr->ident != RESTORE_STREAM_IDENT) {
> + ret = ERROR_FAIL;
> + LOG(ERROR,
> + "Invalid ident: expected 0x%016"PRIx64", got 0x%016"PRIx64,
> + RESTORE_STREAM_IDENT, hdr->ident);
> + goto err;
> + }
> + if (hdr->version != RESTORE_STREAM_VERSION) {
> + ret = ERROR_FAIL;
> + LOG(ERROR, "Unexpected Version: expected %u, got %u",
> + RESTORE_STREAM_VERSION, hdr->version);
> + goto err;
> + }
> + if (hdr->options & RESTORE_OPT_BIG_ENDIAN) {
> + ret = ERROR_FAIL;
> + LOG(ERROR, "Unable to handle big endian streams");
> + goto err;
> + }
> +
> + LOG(DEBUG, "Stream v%u%s", hdr->version,
> + hdr->options & RESTORE_OPT_LEGACY ? " (from legacy)" : "");
> +
> + stream_continue(egc, stream);
> + return;
> +
> + err:
> + assert(ret);
> + stream_failed(egc, stream, ret);
> +}
> +
> +static void setup_read_record(libxl__egc *egc,
> + libxl__stream_read_state *stream)
> +{
> + STATE_AO_GC(stream->ao);
> + libxl__sr_record_buf *rec = NULL;
> + int ret;
> +
> + assert(stream->incoming_record == NULL);
> +
> + stream->incoming_record = rec = libxl__zalloc(NOGC, sizeof(*rec));
> + ret = setup_read(stream, "record header",
> + &rec->hdr, sizeof(rec->hdr),
> + record_header_done);
> + if (ret)
> + goto err;
> +
> + return;
> +
> + err:
> + assert(ret);
> + stream_failed(egc, stream, ret);
> +}
> +
> +static void record_header_done(libxl__egc *egc,
> + libxl__datacopier_state *dc,
> + int rc, int onwrite, int errnoval)
> +{
> + libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
> + libxl__sr_record_buf *rec = stream->incoming_record;
> + STATE_AO_GC(dc->ao);
> + int ret = 0;
> +
> + if (rc || onwrite || errnoval) {
> + ret = ERROR_FAIL;
> + LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
> + goto err;
> + }
> +
> + /* No body? All done. */
> + if (rec->hdr.length == 0) {
> + record_body_done(egc, dc, 0, 0, 0);
> + return;
> + }
> +
> + size_t bytes_to_read = ROUNDUP(rec->hdr.length, REC_ALIGN_ORDER);
> + rec->body = libxl__malloc(NOGC, bytes_to_read);
> +
> + ret = setup_read(stream, "record body",
> + rec->body, bytes_to_read,
> + record_body_done);
> + if (ret)
> + goto err;
> +
> + return;
> +
> + err:
> + assert(ret);
> + stream_failed(egc, stream, ret);
> +}
> +
> +static void record_body_done(libxl__egc *egc,
> + libxl__datacopier_state *dc,
> + int rc, int onwrite, int errnoval)
> +{
> + libxl__stream_read_state *stream = CONTAINER_OF(dc, *stream, dc);
> + libxl__sr_record_buf *rec = stream->incoming_record;
> + STATE_AO_GC(dc->ao);
> + int ret = 0;
> +
> + if (rc || onwrite || errnoval) {
> + ret = ERROR_FAIL;
> + LOG(ERROR, "rc %d, onwrite %d, errnoval %d", rc, onwrite, errnoval);
> + goto err;
> + }
> +
> + LIBXL_STAILQ_INSERT_TAIL(&stream->record_queue, rec, entry);
> + stream->incoming_record = NULL;
> +
> + stream_continue(egc, stream);
> + return;
> +
> + err:
> + assert(ret);
> + stream_failed(egc, stream, ret);
> +}
> +
> +/*
> + * Returns a boolean indicating whether a further action should be set up by
> + * the caller. This is needed to prevent mutual recursion with
> + * stream_continue().
> + */
I don't understand why this mutual recursion would be a problem.
AFAICT stream_continue called from process_record ought only to call
process_record if record_queue had more than two items on it.
But if record_queue has more than two items on it, stream_continue
does the wrong thing already.
So err, why is stream_continue not a loop; or, why is record_queue a
queue ?
If the state machine had better doc comments, stating the invariants,
I would perhaps not be asking these foolish questions...
> +static void write_emulator_blob(libxl__egc *egc,
> + libxl__stream_read_state *stream,
> + libxl__sr_record_buf *rec)
> +{
...
> +
> + if ( rec->hdr.length < sizeof(*emu_hdr) ) {
Coding style (whitespace)
> + sprintf(path, XC_DEVICE_MODEL_RESTORE_FILE".%u", dcs->guest_domid);
> +
> + libxl__carefd_begin();
> + writefd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
Does this need to be a carefd ? This is going to be a small amount of
data, and the worst that happens is some other process inherits an fd
onto a file which might otherwise be deleted sooner.
If it does, then please use libxl__carefd_opened. (Best would be to
libxl__carefd_opened to save and restore errno and then you can call
it unconditionally after the open.)
Thanks,
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 17/27] tools/libxl: Support converting a legacy stream to a v2 stream
2015-07-09 18:26 ` [PATCH v2 17/27] tools/libxl: Support converting a legacy stream to a " Andrew Cooper
2015-07-10 10:28 ` Ian Campbell
@ 2015-07-10 12:28 ` Ian Jackson
1 sibling, 0 replies; 65+ messages in thread
From: Ian Jackson @ 2015-07-10 12:28 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Campbell, Xen-devel
Andrew Cooper writes ("[PATCH v2 17/27] tools/libxl: Support converting a legacy stream to a v2 stream"):
> When a legacy stream is found, it needs to be converted to a v2 stream for the
> reading logic. This is done by exec()ing the python conversion utility.
>
> One complication is that the caller of this interface needs to assume
> ownership of the output fd, to prevent it being closed while still in use in a
> datacopier.
Please reformat the commit message to 70 columns at most. (This
applies in general.) Commit messages need to be narrow so that `git
log' output is <80.
AFAICT in this patch there is no call site for the new functionality.
> +static void helper_failed(libxl__egc *egc,
> + libxl__conversion_helper_state *chs, int rc)
> +{
...
> +static void helper_stop(libxl__egc *egc, libxl__ao_abortable *abrt, int rc)
> +{
These two functions are almost identical. Please combine them.
> +static void helper_exited(libxl__egc *egc, libxl__ev_child *ch,
> + pid_t pid, int status)
> +{
> + libxl__conversion_helper_state *chs = CONTAINER_OF(ch, *chs, child);
> + STATE_AO_GC(chs->ao);
> +
> + if (status) {
> + libxl_report_child_exitstatus(CTX, XTL_ERROR, "conversion helper",
> + pid, status);
> + chs->rc = ERROR_FAIL;
Don't overwrite an existing rc. Also, you probably don't want to
report the results of SIGKILL or SIGTERM with XTL_ERROR. So either
report with XTL_DEBUG if rc is already nonzero, or (if you prefer more
work) have the signaller record the signal sent and check with
WIFSIGNALED.
> + }
> + else
> + chs->rc = 0;
Style: `} else {' please. But actually, clearing rc here is wrong I
think.
If rc is already nonzero, but the helper is exiting zero, you should
not be clearing rc.
> +_hidden void libxl__conversion_helper_init(
> + libxl__conversion_helper_state *chs);
Existing style would be
+_hidden void libxl__conversion_helper_init
+ (libxl__conversion_helper_state *chs);
Cf libxl__domain_suspend_common_switch_qemu_logdirty etc.
(There are a few more occurrences of "(\n" in your series; please fix
them too.)
> +#else
> +/* Stubs for non-x86 architecture to reduce code #ifdefary */
> +static inline void libxl__conversion_helper_init(
> + libxl__conversion_helper_state *chs) { }
Please, no. Can we have a separate file instead ? I'm really not a
fan of #ifdefery at all. In header files it is particularly
pernicious.
> typedef struct libxl__domain_suspend_state libxl__domain_suspend_state;
> @@ -3283,6 +3330,7 @@ struct libxl__domain_create_state {
> * for the non-stubdom device model. */
> libxl__stream_read_state srs;
> libxl__save_helper_state shs;
> + libxl__conversion_helper_state chs;
Since there is no call site for this functionality, this struct should
not be introduced here.
In the patch where it /is/ introduced, it should be explicitly
initialised, and probably during the domain create exit path its
idleness should be asserted.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-10 11:25 ` Ian Campbell
@ 2015-07-10 12:28 ` Andrew Cooper
2015-07-10 12:46 ` Ian Jackson
0 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-10 12:28 UTC (permalink / raw)
To: Ian Campbell; +Cc: Ross Lagerwall, Ian Jackson, Wei Liu, Xen-devel
On 10/07/15 12:25, Ian Campbell wrote:
>>>> +
>>>> + stream->completion_callback(egc, stream, stream->rc);
>>>> +}
>>>> +
>>>> +static void stream_continue(libxl__egc *egc,
>>>> + libxl__stream_read_state *stream)
>>>> +{
>>>> + STATE_AO_GC(stream->ao);
>>>> +
>>>> + /* Must not mutually recurse with process_record() */
>>>> + assert(stream->recursion_guard == false);
>>>> + stream->recursion_guard = true;
>>> This smells a bit like it ought to be a SRS_PHASE_PROCESSING or some
>>> such, but lets leave that alone...
>> This check is pre-emptively avoid the naive bug which would occur if
>> process_record() called back into stream_continue() and there were many
>> TOOLSTACK records back to back in the processing queue.
>>
>> In that case (and potentially future records as well), the two functions
>> would mutually recurse based on the contents of the stream.
> Do you mean they would do so legitimately in that case, or in error?
It is wrong in all cases to mutually recurse like this. The data
controlling the degree of mutual recursion is read from a pipe.
The issue with the TOOLSTACK record is that it a synchronous library
call, not an aync one. This is not a problem pe say, but it means that
process_record() must queue something further to do.
In a checkpoint it is possible (although very unlikely) to have $N
thousand TOOLSTACK records back to back.
The guards are in place to prevent introducing a codepath which does end
up in mutual recursion. Such a calltree would function for any
reasonable input, but would fall off the stack given a certain sequence
of records.
~Andrew
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 18/27] tools/libxl: Convert a legacy stream if needed
2015-07-09 18:26 ` [PATCH v2 18/27] tools/libxl: Convert a legacy stream if needed Andrew Cooper
2015-07-10 10:31 ` Ian Campbell
@ 2015-07-10 12:41 ` Ian Jackson
1 sibling, 0 replies; 65+ messages in thread
From: Ian Jackson @ 2015-07-10 12:41 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Wei Liu, Ian Campbell, Xen-devel
Andrew Cooper writes ("[PATCH v2 18/27] tools/libxl: Convert a legacy stream if needed"):
> For backwards compatibility, a legacy stream needs converting before it can be
> read by the v2 stream logic.
>
> This causes the v2 stream logic to need to juggle two parallel tasks.
> check_stream_finished() is introduced for the purpose of joining the tasks in
> both success and error cases.
>
> Signed-off-by: Andrew Cooper <andrew.cooper3@citrix.com>
> CC: Ian Campbell <Ian.Campbell@citrix.com>
> CC: Ian Jackson <Ian.Jackson@eu.citrix.com>
> CC: Wei Liu <wei.liu2@citrix.com>
> ---
> tools/libxl/libxl_internal.h | 7 +++
> tools/libxl/libxl_stream_read.c | 98 ++++++++++++++++++++++++++++++++++++++-
> 2 files changed, 104 insertions(+), 1 deletion(-)
>
> diff --git a/tools/libxl/libxl_internal.h b/tools/libxl/libxl_internal.h
> index 68e7f02..1cf1884 100644
> --- a/tools/libxl/libxl_internal.h
> +++ b/tools/libxl/libxl_internal.h
> @@ -3274,6 +3274,7 @@ struct libxl__stream_read_state {
> /* filled by the user */
> libxl__ao *ao;
> int fd;
> + bool legacy;
> void (*completion_callback)(libxl__egc *egc,
> libxl__stream_read_state *srs,
> int rc);
> @@ -3281,6 +3282,12 @@ struct libxl__stream_read_state {
> int rc;
> bool running;
>
> + /* Active-stuff handling */
> + int joined_rc;
> +
> + /* Conversion helper */
> + libxl__carefd *v2_carefd;
This needs proper documentation of what states this is valid in. See
my observations on 16/ about this. I would expect that this patch
would add appropriate commentary documenting the
invariants/states/whatever for these.
> +static void check_stream_finished(libxl__egc *egc,
> + libxl__stream_read_state *stream,
> + int rc, const char *what)
> +{
> + libxl__domain_create_state *dcs = CONTAINER_OF(stream, *dcs, srs);
> + STATE_AO_GC(stream->ao);
> +
> + LOG(DEBUG, "Task '%s' joining (rc %d)", what, rc);
> +
> + if (rc && !stream->joined_rc) {
> + bool skip = false;
> + /* First reported failure from joining tasks. Tear everything down */
> + stream->joined_rc = rc;
This function has room for improvement, I think.
* You compute
libxl__stream_read_inuse(&dcs->srs) ||
libxl__convert_legacy_stream_inuse(&dcs->chs)
twice, once in the if (rc ...) and once in the straight line.
You could combine these.
* I think libxl__blah_abort are all no-ops on initialised but inactive
objects. (Ie, objects in state Idle.) So you can call them
unconditionally.
* I don't think joined_rc is particularly helpful. Why not simply
combine the incoming rc with the existing rc ? That is, if nothing
has gone wrong already, set the rc to the incoming one; otherwise
keep the existing rc. That assumes that the best rc value to report
is the one from the first detected problem, which is probably
correct. (Consider ERROR_ABORTED.)
> +#if defined(__x86_64__) || defined(__i386__)
> +static void conversion_done(libxl__egc *egc,
> + libxl__conversion_helper_state *chs, int rc)
> +{
> + STATE_AO_GC(chs->ao);
> + libxl__domain_create_state *dcs = CONTAINER_OF(chs, *dcs, chs);
> +
> + check_stream_finished(egc, &dcs->srs, rc, "conversion");
> +}
> +#endif
Again, I would prefer to avoid the ifdeffery if it's not terribly
awkward to do the other way (see libxl_no*.c for some examples). If
you do invent ifdeffery it should definitely have its own #define to
trigger off, rather than directly using __x86_64__ etc.
Thanks,
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-10 12:28 ` Andrew Cooper
@ 2015-07-10 12:46 ` Ian Jackson
2015-07-10 12:50 ` Andrew Cooper
0 siblings, 1 reply; 65+ messages in thread
From: Ian Jackson @ 2015-07-10 12:46 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Ross Lagerwall, Wei Liu, Ian Campbell, Xen-devel
Andrew Cooper writes ("Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream"):
> On 10/07/15 12:25, Ian Campbell wrote:
> > Do you mean they would do so legitimately in that case, or in error?
>
> It is wrong in all cases to mutually recurse like this. The data
> controlling the degree of mutual recursion is read from a pipe.
>
> The issue with the TOOLSTACK record is that it a synchronous library
> call, not an aync one. This is not a problem pe say, but it means that
> process_record() must queue something further to do.
>
> In a checkpoint it is possible (although very unlikely) to have $N
> thousand TOOLSTACK records back to back.
>
> The guards are in place to prevent introducing a codepath which does end
> up in mutual recursion. Such a calltree would function for any
> reasonable input, but would fall off the stack given a certain sequence
> of records.
I just wanted to say that I have read this and it helps my
understanding but I still worry about the correctness of
stream_continue and process_record if the queue has more than one
record.
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-10 12:46 ` Ian Jackson
@ 2015-07-10 12:50 ` Andrew Cooper
0 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-10 12:50 UTC (permalink / raw)
To: Ian Jackson; +Cc: Ross Lagerwall, Wei Liu, Ian Campbell, Xen-devel
On 10/07/15 13:46, Ian Jackson wrote:
> Andrew Cooper writes ("Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream"):
>> On 10/07/15 12:25, Ian Campbell wrote:
>>> Do you mean they would do so legitimately in that case, or in error?
>> It is wrong in all cases to mutually recurse like this. The data
>> controlling the degree of mutual recursion is read from a pipe.
>>
>> The issue with the TOOLSTACK record is that it a synchronous library
>> call, not an aync one. This is not a problem pe say, but it means that
>> process_record() must queue something further to do.
>>
>> In a checkpoint it is possible (although very unlikely) to have $N
>> thousand TOOLSTACK records back to back.
>>
>> The guards are in place to prevent introducing a codepath which does end
>> up in mutual recursion. Such a calltree would function for any
>> reasonable input, but would fall off the stack given a certain sequence
>> of records.
> I just wanted to say that I have read this and it helps my
> understanding but I still worry about the correctness of
> stream_continue and process_record if the queue has more than one
> record.
This might also make more sense with the context from patch 25, which is
where the Remus/COLO buffering is introduced.
~Andrew
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-10 12:17 ` Ian Jackson
@ 2015-07-10 12:56 ` Andrew Cooper
2015-07-10 13:09 ` Ian Jackson
0 siblings, 1 reply; 65+ messages in thread
From: Andrew Cooper @ 2015-07-10 12:56 UTC (permalink / raw)
To: Ian Jackson; +Cc: Ross Lagerwall, Wei Liu, Ian Campbell, Xen-devel
On 10/07/15 13:17, Ian Jackson wrote:
> Andrew Cooper writes ("[PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream"):
>> From: Ross Lagerwall <ross.lagerwall@citrix.com>
>>
>> This contains the event machinary and state machines to read an act on a
>> non-checkpointed migration v2 stream (with the exception of the
>> xc_domain_restore() handling which is spliced later in a bisectable way).
>>
>> It also contains some boilerplate to help support checkpointed streams, which
>> shall be introduced in a later patch.
> ...
>
>
>> +/* State for manipulating a libxl migration v2 stream */
>> +typedef struct libxl__stream_read_state libxl__stream_read_state;
> ...
>> +struct libxl__stream_read_state {
>> + /* filled by the user */
>> + libxl__ao *ao;
>> + int fd;
>> + void (*completion_callback)(libxl__egc *egc,
>> + libxl__stream_read_state *srs,
>> + int rc);
>> + /* Private */
>> + int rc;
>> + bool running;
>> +
>> + /* Main stream-reading data */
>> + libxl__datacopier_state dc;
>> + libxl__sr_hdr hdr;
>> + libxl__sr_record_buf *incoming_record;
>> + LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue;
> This comes from malloc, not from the gc. That needs a comment.
Ok
> (Is the same true of incoming_record ? I think it is.)
Yes. incoming_record is a partially-read record (almost all records
need splitting across two datacopier calls).
Once a record it complete, it is added to the tail of the record queue.
>
>> + enum {
>> + SRS_PHASE_NORMAL,
>> + } phase;
>> + bool recursion_guard;
>> +
>> + /* Emulator blob handling */
>> + libxl__datacopier_state emu_dc;
>> + libxl__carefd *emu_carefd;
>> +};
> ...
>
> The comments in libxl_stream_read.c are good but I'm afraid I'm going
> to ask for some commentary about the legal states and/or the
> invariants in the data structure.
>
> That is, you've written a state machine but the states are not
> explicitly listed.
>
> As far as I can tell, from the outside, this machinery has the usual
> Undefined/Idle/Active states. But does it not have more states on the
> inside ? PHASE at least needs explaining. See also comments about
> the record queue, below.
>
> For an example of the kind of thing I mean, see libxl_spawn.
> Particularly, libxl_internal.h:
>
> * Each libxl__spawn_state is in one of these states
> * Undefined, Idle, Attached, Detaching
>
> and the much more detailed state and data structure table in
> libxl_exec.c
>
> * Full set of possible states of a libxl__spawn_state and its _detachable:
>
> I think you probably don't need to write down the external states for
> your srs machinery because the states are Undefined, Idle and Active
> like most things. (Assuming you do something about the anomalous
> initialisation and checking of `running'.)
>
> What I'm missing is the internal description.
>
I will see what I can do.
>
>> +void libxl__stream_read_abort(libxl__egc *egc,
>> + libxl__stream_read_state *stream, int rc)
>> +{
>> + stream_failed(egc, stream, rc);
> This has the same signature as stream_failed. Perhaps stream_failed
> could be eliminated and replaced with this ? Or maybe...
>
>> +static void stream_success(libxl__egc *egc, libxl__stream_read_state *stream)
>> +{
>> + stream->rc = 0;
>> + stream_done(egc, stream);
> This function has one call site. As an alternative to abolishing
> stream_failed, maybe stream_failed should be renamed to
> `stream_complete' and then the call site for stream_success could
> simply say `stream_complete(egc, stream, 0)' ?
I have to admit that I find that paragdim specifically confusing to
read, which is why I deliberately split the _success() and _failed()
cases.
>
>> +static void stream_failed(libxl__egc *egc,
>> + libxl__stream_read_state *stream, int rc)
>> +{
>> + assert(rc);
>> + stream->rc = rc;
>> +
>> + if (stream->running) {
>> + stream_done(egc, stream);
> Is the check for running necessary here ?
Yes. An _abort() call from outside can happen at any arbitrary time.
We must not deliver the callback twice.
(Although, from another email I see you want different semantics for
_abort(), so will try to follow that.)
>
>> + sprintf(path, XC_DEVICE_MODEL_RESTORE_FILE".%u", dcs->guest_domid);
>> +
>> + libxl__carefd_begin();
>> + writefd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
> Does this need to be a carefd ? This is going to be a small amount of
> data, and the worst that happens is some other process inherits an fd
> onto a file which might otherwise be deleted sooner.
Who says the file will be deleted? That will be down to the exact
semantics of the device model.
(I was asked to switch from a plain open() to a carefd as part of review
from v1).
>
> If it does, then please use libxl__carefd_opened. (Best would be to
> libxl__carefd_opened to save and restore errno and then you can call
> it unconditionally after the open.)
I will go down this route.
~Andrew
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream
2015-07-10 12:56 ` Andrew Cooper
@ 2015-07-10 13:09 ` Ian Jackson
0 siblings, 0 replies; 65+ messages in thread
From: Ian Jackson @ 2015-07-10 13:09 UTC (permalink / raw)
To: Andrew Cooper; +Cc: Ross Lagerwall, Wei Liu, Ian Campbell, Xen-devel
Andrew Cooper writes ("Re: [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream"):
...
> > This function has one call site. As an alternative to abolishing
> > stream_failed, maybe stream_failed should be renamed to
> > `stream_complete' and then the call site for stream_success could
> > simply say `stream_complete(egc, stream, 0)' ?
>
> I have to admit that I find that paragdim specifically confusing to
> read, which is why I deliberately split the _success() and _failed()
> cases.
I'm not sure why it's confusing. It means you have a unified exit
path.
> >> +static void stream_failed(libxl__egc *egc,
> >> + libxl__stream_read_state *stream, int rc)
> >> +{
...
> >> + if (stream->running) {
> >> + stream_done(egc, stream);
> > Is the check for running necessary here ?
>
> Yes. An _abort() call from outside can happen at any arbitrary time.
> We must not deliver the callback twice.
Obviously. But it is surely a serious bug if stream_failed is called
when the stream is not running, because stream_failed is then in
flight in circumstances where the caller might reuse the
libxl__stream_read_state.
> >> + sprintf(path, XC_DEVICE_MODEL_RESTORE_FILE".%u", dcs->guest_domid);
> >> +
> >> + libxl__carefd_begin();
> >> + writefd = open(path, O_WRONLY | O_CREAT | O_TRUNC, 0600);
> > Does this need to be a carefd ? This is going to be a small amount of
> > data, and the worst that happens is some other process inherits an fd
> > onto a file which might otherwise be deleted sooner.
>
> Who says the file will be deleted? That will be down to the exact
> semantics of the device model.
If the file is not going to be deleted then it is even less necessary.
> (I was asked to switch from a plain open() to a carefd as part of review
> from v1).
I tried to find this and all I found was Ian C saying:
Also, should consider whether this fd needs to be subject to the
carefd machinery.
I think it doesn't. But overuse of the carefd machinery is just
pointless and not actually incorrect. So:
> > If it does, then please use libxl__carefd_opened. (Best would be to
> > libxl__carefd_opened to save and restore errno and then you can call
> > it unconditionally after the open.)
>
> I will go down this route.
... if you do this it won't be a blocker.
Ian.
^ permalink raw reply [flat|nested] 65+ messages in thread
* Re: [PATCH v2 25/27] tools/libxl: Handle checkpoint records in a libxl migration v2 stream
2015-07-10 11:18 ` Ian Campbell
@ 2015-07-10 14:34 ` Andrew Cooper
0 siblings, 0 replies; 65+ messages in thread
From: Andrew Cooper @ 2015-07-10 14:34 UTC (permalink / raw)
To: Ian Campbell; +Cc: Wei Liu, Ian Jackson, Xen-devel
On 10/07/15 12:18, Ian Campbell wrote:
> On Thu, 2015-07-09 at 19:26 +0100, Andrew Cooper wrote:
>
>> @@ -3349,6 +3353,8 @@ struct libxl__stream_read_state {
>> LIBXL_STAILQ_HEAD(, libxl__sr_record_buf) record_queue;
>> enum {
>> SRS_PHASE_NORMAL,
>> + SRS_PHASE_BUFFERING,
>> + SRS_PHASE_UNBUFFERING,
> I'd be inclined towards calling the latter DRAINING or PROCESSING or
> some such, perhaps tying into my comment about the separate recursion
> guard thing earlier on in the series.
I considered PROCESSING at first, but it a plausible description of what
is currently NORMAL, so I avoided it.
I have to admit that I don't particularly like UNBUFFERING either, but
it was the least-bad of a number of bad alternatives I came up with.
>
> But that's a bikeshedding issue on an internal thing so my only real
> comment is:
>
>> + if (rc == 0)
>> + ret = 0; /* Success */
>> + else if (stream->phase == SRS_PHASE_BUFFERING)
>> + ret = 2; /* Failover */
>> + else
>> + ret = 1; /* Error (fatal) */
> Maybe we should have had an enum or some #defines for these (this is
> really a comment on some previous patch).
Will do.
~Andrew
^ permalink raw reply [flat|nested] 65+ messages in thread
end of thread, other threads:[~2015-07-10 14:34 UTC | newest]
Thread overview: 65+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-07-09 18:26 [PATCH v2 00/27] Libxl migration v2 Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 01/27] bsd-sys-queue-h-seddery: Massage `offsetof' Andrew Cooper
2015-07-10 9:32 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 02/27] tools/libxc: Always compile the compat qemu variables into xc_sr_context Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 03/27] tools/libxl: Introduce ROUNDUP() Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 04/27] tools/libxl: Introduce libxl__kill() Andrew Cooper
2015-07-10 1:34 ` Yang Hongyang
2015-07-10 8:56 ` Andrew Cooper
2015-07-10 9:08 ` Wei Liu
2015-07-10 9:25 ` Andrew Cooper
2015-07-10 9:34 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 05/27] tools/libxl: Stash all restore parameters in domain_create_state Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 06/27] tools/libxl: Split libxl__domain_create_state.restore_fd in two Andrew Cooper
2015-07-10 9:37 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 07/27] tools/libxl: Extra management APIs for the save helper Andrew Cooper
2015-07-10 9:41 ` Ian Campbell
2015-07-10 9:52 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 08/27] tools/xl: Mandatory flag indicating the format of the migration stream Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 09/27] docs: Libxl migration v2 stream specification Andrew Cooper
2015-07-10 9:46 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 10/27] tools/python: Libxc migration v2 infrastructure Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 11/27] tools/python: Libxl " Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 12/27] tools/python: Other migration infrastructure Andrew Cooper
2015-07-10 9:48 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 13/27] tools/python: Verification utility for v2 stream spec compliance Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 14/27] tools/python: Conversion utility for legacy migration streams Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 15/27] tools/libxl: Migration v2 stream format Andrew Cooper
2015-07-10 9:49 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 16/27] tools/libxl: Infrastructure for reading a libxl migration v2 stream Andrew Cooper
2015-07-10 10:23 ` Ian Campbell
2015-07-10 10:47 ` Andrew Cooper
2015-07-10 11:16 ` Ian Jackson
2015-07-10 11:25 ` Ian Campbell
2015-07-10 12:28 ` Andrew Cooper
2015-07-10 12:46 ` Ian Jackson
2015-07-10 12:50 ` Andrew Cooper
2015-07-10 12:17 ` Ian Jackson
2015-07-10 12:56 ` Andrew Cooper
2015-07-10 13:09 ` Ian Jackson
2015-07-09 18:26 ` [PATCH v2 17/27] tools/libxl: Support converting a legacy stream to a " Andrew Cooper
2015-07-10 10:28 ` Ian Campbell
2015-07-10 10:39 ` Andrew Cooper
2015-07-10 12:28 ` Ian Jackson
2015-07-09 18:26 ` [PATCH v2 18/27] tools/libxl: Convert a legacy stream if needed Andrew Cooper
2015-07-10 10:31 ` Ian Campbell
2015-07-10 12:41 ` Ian Jackson
2015-07-09 18:26 ` [PATCH v2 19/27] tools/libxc+libxl+xl: Restore v2 streams Andrew Cooper
2015-07-10 10:45 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 20/27] tools/libxl: Infrastructure for writing a v2 stream Andrew Cooper
2015-07-10 11:10 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 21/27] tools/libxc+libxl+xl: Save v2 streams Andrew Cooper
2015-07-10 10:57 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 22/27] docs/libxl: Introduce CHECKPOINT_END to support migration v2 remus streams Andrew Cooper
2015-07-10 10:59 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 23/27] tools/libxl: Write checkpoint records into the stream Andrew Cooper
2015-07-10 11:02 ` Ian Campbell
2015-07-10 11:47 ` Wei Liu
2015-07-09 18:26 ` [PATCH v2 24/27] tools/libx{c, l}: Introduce restore_callbacks.checkpoint() Andrew Cooper
2015-07-10 11:13 ` Ian Campbell
2015-07-09 18:26 ` [PATCH v2 25/27] tools/libxl: Handle checkpoint records in a libxl migration v2 stream Andrew Cooper
2015-07-10 11:18 ` Ian Campbell
2015-07-10 14:34 ` Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 26/27] tools/libxc: Drop all XG_LIBXL_HVM_COMPAT code from libxc Andrew Cooper
2015-07-09 18:26 ` [PATCH v2 27/27] tools/libxl: Drop all knowledge of toolstack callbacks Andrew Cooper
2015-07-10 3:01 ` [PATCH v2 00/27] Libxl migration v2 Yang Hongyang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.