* [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT)
@ 2015-09-02 8:22 zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 01/32] configure: Add parameter for configure to enable/disable COLO support zhanghailiang
` (35 more replies)
0 siblings, 36 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
This is the 9th version of COLO.
Please Note that, this version is very different from the previous versions.
since we have decided to realize proxy in qemu, which based on slirp in qemu.
We dropped all the original colo proxy related part.
It will be a long time for proxy to be ready for merging, so here we extract
the basic periodic checkpoint part that not depend on proxy into this series.
Actually, the 'periodic' mode is also what we want to support in COLO, it is
based on Yang Hongyang's netfilter series. and this mode is very like
MicroCheckpointing and Remus.
You can find the discussion about why & how to realize the colo proxy in qemu
from the follow link:
http://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg04069.html
As usual, here is only COLO frame part, you can get the whole codes from github:
https://github.com/coloft/qemu/commits/colo-v2.0-periodic-mode
Compared with previous versions, this version is more easy to test.
Test procedure:
1. Startup qemu
Primary side:
# x86_64-softmmu/qemu-system-x86_64 -enable-kvm -netdev tap,id=bn0 -netfilter buffer,id=f0,netdev=bn0,chain=in -device virtio-net-pci,id=net-pci0,netdev=bn0 -boot c -drive if=virtio,id=disk1,driver=quorum,read-pattern=fifo,cache=none,aio=native,children.0.file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,children.0.driver=raw -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio -S
Secondary side:
# x86_64-softmmu/qemu-system-x86_64 -enable-kvm -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0 -drive if=none,driver=raw,file=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,id=colo1,cache=none,aio=native -drive if=virtio,driver=replication,mode=secondary,throttling.bps-total=70000000,file.file.filename=/mnt/ramfs/active_disk.img,file.driver=qcow2,file.backing.file.filename=/mnt/ramfs/hidden_disk.img,file.backing.driver=qcow2,file.backing.backing.backing_reference=colo1,file.backing.allow-write-backing-file=on -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-table -monitor stdio -incoming tcp:0:8888
2. On Secondary VM's QEMU monitor, issue command
(qemu) nbd_server_start 192.168.2.88:8889
(qemu) nbd_server_add -w colo1
3. On Primary VM's QEMU monitor, issue command:
(qemu) child_add disk1 child.driver=replication,child.mode=primary,child.file.host=192.168.2.88,child.file.port=8889,child.file.export=colo1,child.file.driver=nbd,child.ignore-errors=on
(qemu) migrate_set_capability colo on
(qemu) migrate tcp:192.168.2.88:8888
4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
You can by issue command "migrate_set_parameter checkpoint-delay 2000"
to change the checkpoint period time.
5. Failover test
You can kill PVM and run 'colo_lost_heartbeat' in SVM's
monitor at the same time, then SVM will failover and client will not feel this change.
COLO is a totally new feature which is still in early stage,
your comments and feedback are warmly welcomed.
TODO:
1. checkpoint based on proxy in qemu
2. The capability of continuous FT
v9:
- Drop colo proxy related part (colo-nic.c file)
- Convert COLO protocol name definition to QAPI
- Smash failover related patch (patch 19/20/23)
- Fix colo exit event according Eric's comments.
- Fix some typos from Eric's comments
- Fix bug 'invalid runstate transition: 'colo' -> 'prelaunch' reported
by Dave (patch 27)
- Use migrate_set_parameter intead of ecolo-set-checkpoint-period to set
checkpoint delay time (patch 25)
- Add new patch (patch 29/30) to seperate the process of saving/loading
device and state during checkpoint. which will reduce the data size
for sending and also reduce the qsb size used in checkpoint.
Wen Congyang (1):
COLO: Add block replication into colo process
zhanghailiang (31):
configure: Add parameter for configure to enable/disable COLO support
migration: Introduce capability 'colo' to migration
COLO: migrate colo related info to slave
migration: Add state records for migration incoming
migration: Integrate COLO checkpoint process into migration
migration: Integrate COLO checkpoint process into loadvm
migration: Rename the'file' member of MigrationState and
MigrationIncomingState
COLO/migration: establish a new communication path from destination to
source
COLO: Implement colo checkpoint protocol
COLO: Add a new RunState RUN_STATE_COLO
QEMUSizedBuffer: Introduce two help functions for qsb
COLO: Save PVM state to secondary side when do checkpoint
COLO: Load PVM's dirty pages into SVM's RAM cache temporarily
COLO: Load VMState into qsb before restore it
COLO: Flush PVM's cached RAM into SVM's memory
COLO: synchronize PVM's state to SVM periodically
COLO failover: Introduce a new command to trigger a failover
COLO failover: Introduce state to record failover process
COLO: Implement failover work for Primary VM
COLO: Implement failover work for Secondary VM
COLO: implement default failover treatment
qmp event: Add event notification for COLO error
COLO failover: Shutdown related socket fd when do failover
COLO failover: Don't do failover during loading VM's state
COLO: Control the checkpoint delay time by migrate-set-parameters
command
COLO: Implement shutdown checkpoint
COLO: Update the global runstate after going into colo state
savevm: Split load vm state function qemu_loadvm_state
COLO: Separate the process of saving/loading ram and device state
COLO: Split qemu_savevm_state_begin out of checkpoint process
COLO: Add net packets treatment into COLO
configure | 11 +
docs/qmp/qmp-events.txt | 17 +
hmp-commands.hx | 15 +
hmp.c | 16 +
hmp.h | 1 +
include/exec/cpu-all.h | 1 +
include/migration/colo.h | 44 +++
include/migration/failover.h | 33 ++
include/migration/migration.h | 16 +-
include/migration/qemu-file.h | 3 +-
include/sysemu/sysemu.h | 8 +
migration/Makefile.objs | 2 +
migration/colo-comm.c | 75 ++++
migration/colo-failover.c | 83 +++++
migration/colo.c | 782 ++++++++++++++++++++++++++++++++++++++++++
migration/exec.c | 4 +-
migration/fd.c | 4 +-
migration/migration.c | 184 +++++++---
migration/qemu-file-buf.c | 58 ++++
migration/ram.c | 185 +++++++++-
migration/savevm.c | 309 +++++++++++++----
migration/tcp.c | 4 +-
migration/unix.c | 4 +-
qapi-schema.json | 101 +++++-
qapi/event.json | 17 +
qmp-commands.hx | 20 ++
stubs/Makefile.objs | 1 +
stubs/migration-colo.c | 45 +++
trace-events | 8 +
vl.c | 37 +-
30 files changed, 1930 insertions(+), 158 deletions(-)
create mode 100644 include/migration/colo.h
create mode 100644 include/migration/failover.h
create mode 100644 migration/colo-comm.c
create mode 100644 migration/colo-failover.c
create mode 100644 migration/colo.c
create mode 100644 stubs/migration-colo.c
--
1.8.3.1
^ permalink raw reply [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 01/32] configure: Add parameter for configure to enable/disable COLO support
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-10-02 15:10 ` Dr. David Alan Gilbert
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 02/32] migration: Introduce capability 'colo' to migration zhanghailiang
` (34 subsequent siblings)
35 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
configure --enable-colo/--disable-colo to switch COLO
support on/off.
COLO support is off by default.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
configure | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/configure b/configure
index 9d24d59..e9fff7b 100755
--- a/configure
+++ b/configure
@@ -258,6 +258,7 @@ xfs=""
vhost_net="no"
vhost_scsi="no"
kvm="no"
+colo="no"
rdma=""
gprof="no"
debug_tcg="no"
@@ -929,6 +930,10 @@ for opt do
;;
--enable-kvm) kvm="yes"
;;
+ --disable-colo) colo="no"
+ ;;
+ --enable-colo) colo="yes"
+ ;;
--disable-tcg-interpreter) tcg_interpreter="no"
;;
--enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1339,6 +1344,7 @@ disabled with --disable-FEATURE, default is enabled if available:
fdt fdt device tree
bluez bluez stack connectivity
kvm KVM acceleration support
+ colo COarse-grain LOck-stepping VM for Non-stop Service
rdma RDMA-based migration support
uuid uuid support
vde support for vde network
@@ -4545,6 +4551,7 @@ echo "Linux AIO support $linux_aio"
echo "ATTR/XATTR support $attr"
echo "Install blobs $blobs"
echo "KVM support $kvm"
+echo "COLO support $colo"
echo "RDMA support $rdma"
echo "TCG interpreter $tcg_interpreter"
echo "fdt support $fdt"
@@ -5119,6 +5126,10 @@ if have_backend "ftrace"; then
fi
echo "CONFIG_TRACE_FILE=$trace_file" >> $config_host_mak
+if test "$colo" = "yes"; then
+ echo "CONFIG_COLO=y" >> $config_host_mak
+fi
+
if test "$rdma" = "yes" ; then
echo "CONFIG_RDMA=y" >> $config_host_mak
fi
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 02/32] migration: Introduce capability 'colo' to migration
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 01/32] configure: Add parameter for configure to enable/disable COLO support zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-10-02 16:02 ` Eric Blake
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 03/32] COLO: migrate colo related info to slave zhanghailiang
` (33 subsequent siblings)
35 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, arei.gonglei, stefanha,
amit.shah, yanghy, zhanghailiang
We add helper function colo_supported() to indicate whether
colo is supported or not, with which we use to control whether or not
showing 'colo' string to users, they can use qmp command
'query-migrate-capabilities' or hmp command 'info migrate_capabilities'
to learn if colo is supported.
Cc: Juan Quintela <quintela@redhat.com>
Cc: Amit Shah <amit.shah@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
include/migration/colo.h | 20 ++++++++++++++++++++
include/migration/migration.h | 1 +
migration/Makefile.objs | 1 +
migration/colo.c | 18 ++++++++++++++++++
migration/migration.c | 17 +++++++++++++++++
qapi-schema.json | 6 +++++-
qmp-commands.hx | 1 +
stubs/Makefile.objs | 1 +
stubs/migration-colo.c | 18 ++++++++++++++++++
9 files changed, 82 insertions(+), 1 deletion(-)
create mode 100644 include/migration/colo.h
create mode 100644 migration/colo.c
create mode 100644 stubs/migration-colo.c
diff --git a/include/migration/colo.h b/include/migration/colo.h
new file mode 100644
index 0000000..c60a590
--- /dev/null
+++ b/include/migration/colo.h
@@ -0,0 +1,20 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_COLO_H
+#define QEMU_COLO_H
+
+#include "qemu-common.h"
+
+bool colo_supported(void);
+
+#endif
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 8334621..05de3a1 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -170,6 +170,7 @@ int xbzrle_decode_buffer(uint8_t *src, int slen, uint8_t *dst, int dlen);
int migrate_use_xbzrle(void);
int64_t migrate_xbzrle_cache_size(void);
+bool migrate_enable_colo(void);
int64_t xbzrle_cache_resize(int64_t new_size);
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index d929e96..5a25d39 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -1,4 +1,5 @@
common-obj-y += migration.o tcp.o
+common-obj-$(CONFIG_COLO) += colo.o
common-obj-y += vmstate.o
common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
common-obj-y += xbzrle.o
diff --git a/migration/colo.c b/migration/colo.c
new file mode 100644
index 0000000..2c40d2e
--- /dev/null
+++ b/migration/colo.c
@@ -0,0 +1,18 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#include "migration/colo.h"
+
+bool colo_supported(void)
+{
+ return true;
+}
diff --git a/migration/migration.c b/migration/migration.c
index 662e77e..593cac0 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -29,6 +29,7 @@
#include "trace.h"
#include "qapi/util.h"
#include "qapi-event.h"
+#include "migration/colo.h"
#define MAX_THROTTLE (32 << 20) /* Migration speed throttling */
@@ -345,6 +346,9 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
caps = NULL; /* silence compiler warning */
for (i = 0; i < MIGRATION_CAPABILITY_MAX; i++) {
+ if (i == MIGRATION_CAPABILITY_COLO && !colo_supported()) {
+ continue;
+ }
if (head == NULL) {
head = g_malloc0(sizeof(*caps));
caps = head;
@@ -485,6 +489,13 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
}
for (cap = params; cap; cap = cap->next) {
+ if (cap->value->capability == MIGRATION_CAPABILITY_COLO &&
+ !colo_supported()) {
+ error_setg(errp, "COLO is not currently supported, please"
+ " configure with --enable-colo option in order to"
+ " support COLO feature");
+ continue;
+ }
s->enabled_capabilities[cap->value->capability] = cap->value->state;
}
}
@@ -913,6 +924,12 @@ int64_t migrate_xbzrle_cache_size(void)
return s->xbzrle_cache_size;
}
+bool migrate_enable_colo(void)
+{
+ MigrationState *s = migrate_get_current();
+ return s->enabled_capabilities[MIGRATION_CAPABILITY_COLO];
+}
+
/* migration thread support */
static void *migration_thread(void *opaque)
diff --git a/qapi-schema.json b/qapi-schema.json
index 5f45571..b14d1f4 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -529,11 +529,15 @@
# @auto-converge: If enabled, QEMU will automatically throttle down the guest
# to speed up convergence of RAM migration. (since 1.6)
#
+# @colo: If enabled, migration will never end, and the state of the VM on the
+# primary side will be migrated continuously to the VM on secondary
+# side. (since 2.5)
+#
# Since: 1.2
##
{ 'enum': 'MigrationCapability',
'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
- 'compress', 'events'] }
+ 'compress', 'events', 'colo'] }
##
# @MigrationCapabilityStatus
diff --git a/qmp-commands.hx b/qmp-commands.hx
index e6ed580..b06c09c 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -3489,6 +3489,7 @@ Query current migration capabilities
- "rdma-pin-all" : RDMA Pin Page state (json-bool)
- "auto-converge" : Auto Converge state (json-bool)
- "zero-blocks" : Zero Blocks state (json-bool)
+ - "colo" : COarse-Grain LOck Stepping for Non-stop Service (json-bool)
Arguments:
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index 9937a12..5b124a9 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -38,3 +38,4 @@ stub-obj-$(CONFIG_WIN32) += fd-register.o
stub-obj-y += cpus.o
stub-obj-y += kvm.o
stub-obj-y += qmp_pc_dimm_device_list.o
+stub-obj-y += migration-colo.o
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
new file mode 100644
index 0000000..3d817df
--- /dev/null
+++ b/stubs/migration-colo.c
@@ -0,0 +1,18 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#include "migration/colo.h"
+
+bool colo_supported(void)
+{
+ return false;
+}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 03/32] COLO: migrate colo related info to slave
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 01/32] configure: Add parameter for configure to enable/disable COLO support zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 02/32] migration: Introduce capability 'colo' to migration zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-10-02 18:45 ` Dr. David Alan Gilbert
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 04/32] migration: Add state records for migration incoming zhanghailiang
` (32 subsequent siblings)
35 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
We can know if VM in destination should go into COLO mode by refer to
the info that been migrated from PVM.
We skip this section if colo is not enabled (i.e.
migrate_set_capability colo off), so that, It not break compatibility with migration
however the --enable-colo/disable-colo on the source/destination;
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
include/migration/colo.h | 2 ++
migration/Makefile.objs | 1 +
migration/colo-comm.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++
vl.c | 3 ++-
4 files changed, 59 insertions(+), 1 deletion(-)
create mode 100644 migration/colo-comm.c
diff --git a/include/migration/colo.h b/include/migration/colo.h
index c60a590..9b6662d 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -14,7 +14,9 @@
#define QEMU_COLO_H
#include "qemu-common.h"
+#include "migration/migration.h"
bool colo_supported(void);
+void colo_info_mig_init(void);
#endif
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index 5a25d39..cb7bd30 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -1,5 +1,6 @@
common-obj-y += migration.o tcp.o
common-obj-$(CONFIG_COLO) += colo.o
+common-obj-y += colo-comm.o
common-obj-y += vmstate.o
common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
common-obj-y += xbzrle.o
diff --git a/migration/colo-comm.c b/migration/colo-comm.c
new file mode 100644
index 0000000..4330bd8
--- /dev/null
+++ b/migration/colo-comm.c
@@ -0,0 +1,54 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ *
+ */
+
+#include <migration/colo.h>
+#include "trace.h"
+
+typedef struct {
+ uint32_t colo_requested;
+} COLOInfo;
+
+static COLOInfo colo_info;
+
+static void colo_info_pre_save(void *opaque)
+{
+ COLOInfo *s = opaque;
+
+ if (migrate_enable_colo()) {
+ s->colo_requested = 1;
+ } else {
+ s->colo_requested = 0;
+ }
+}
+
+static bool colo_info_need(void *opaque)
+{
+ return migrate_enable_colo();
+}
+
+static const VMStateDescription colo_state = {
+ .name = "COLOState",
+ .version_id = 1,
+ .minimum_version_id = 1,
+ .pre_save = colo_info_pre_save,
+ .needed = colo_info_need,
+ .fields = (VMStateField[]) {
+ VMSTATE_UINT32(colo_requested, COLOInfo),
+ VMSTATE_END_OF_LIST()
+ },
+};
+
+void colo_info_mig_init(void)
+{
+ vmstate_register(NULL, 0, &colo_state, &colo_info);
+}
diff --git a/vl.c b/vl.c
index aee931a..571347a 100644
--- a/vl.c
+++ b/vl.c
@@ -92,6 +92,7 @@ int main(int argc, char **argv)
#include "sysemu/dma.h"
#include "audio/audio.h"
#include "migration/migration.h"
+#include "migration/colo.h"
#include "sysemu/kvm.h"
#include "qapi/qmp/qjson.h"
#include "qemu/option.h"
@@ -4446,7 +4447,7 @@ int main(int argc, char **argv, char **envp)
blk_mig_init();
ram_mig_init();
-
+ colo_info_mig_init();
/* If the currently selected machine wishes to override the units-per-bus
* property of its default HBA interface type, do so now. */
if (machine_class->units_per_default_bus) {
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 04/32] migration: Add state records for migration incoming
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (2 preceding siblings ...)
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 03/32] COLO: migrate colo related info to slave zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-10-09 16:18 ` Dr. David Alan Gilbert
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 05/32] migration: Integrate COLO checkpoint process into migration zhanghailiang
` (31 subsequent siblings)
35 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
For migration destination, we also need to know its state,
we will use it in COLO.
Here we add a new member 'state' for MigrationIncomingState,
and also use migrate_set_state() to modify its value.
We fix the first parameter of migrate_set_state(), and make it
public.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
---
include/migration/migration.h | 3 +++
migration/migration.c | 38 +++++++++++++++++++++++---------------
2 files changed, 26 insertions(+), 15 deletions(-)
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 05de3a1..a62068f 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -50,6 +50,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
struct MigrationIncomingState {
QEMUFile *file;
+ int state;
/* See savevm.c */
LoadStateEntry_Head loadvm_handlers;
};
@@ -82,6 +83,8 @@ struct MigrationState
int64_t dirty_sync_count;
};
+void migrate_set_state(int *state, int old_state, int new_state);
+
void process_incoming_migration(QEMUFile *f);
void qemu_start_incoming_migration(const char *uri, Error **errp);
diff --git a/migration/migration.c b/migration/migration.c
index 593cac0..98133f1 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -89,6 +89,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
{
mis_current = g_malloc0(sizeof(MigrationIncomingState));
mis_current->file = f;
+ mis_current->state = MIGRATION_STATUS_NONE;
QLIST_INIT(&mis_current->loadvm_handlers);
return mis_current;
@@ -270,11 +271,13 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)
static void process_incoming_migration_co(void *opaque)
{
QEMUFile *f = opaque;
+ MigrationIncomingState *mis;
Error *local_err = NULL;
int ret;
- migration_incoming_state_new(f);
- migrate_generate_event(MIGRATION_STATUS_ACTIVE);
+ mis = migration_incoming_state_new(f);
+ migrate_set_state(&mis->state, MIGRATION_STATUS_NONE,
+ MIGRATION_STATUS_ACTIVE);
ret = qemu_loadvm_state(f);
qemu_fclose(f);
@@ -282,12 +285,14 @@ static void process_incoming_migration_co(void *opaque)
migration_incoming_state_destroy();
if (ret < 0) {
- migrate_generate_event(MIGRATION_STATUS_FAILED);
+ migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+ MIGRATION_STATUS_FAILED);
error_report("load of migration failed: %s", strerror(-ret));
migrate_decompress_threads_join();
exit(EXIT_FAILURE);
}
- migrate_generate_event(MIGRATION_STATUS_COMPLETED);
+ migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+ MIGRATION_STATUS_COMPLETED);
qemu_announce_self();
/* Make sure all file formats flush their mutable metadata */
@@ -543,9 +548,9 @@ void qmp_migrate_set_parameters(bool has_compress_level,
/* shared migration helpers */
-static void migrate_set_state(MigrationState *s, int old_state, int new_state)
+void migrate_set_state(int *state, int old_state, int new_state)
{
- if (atomic_cmpxchg(&s->state, old_state, new_state) == old_state) {
+ if (atomic_cmpxchg(state, old_state, new_state) == old_state) {
trace_migrate_set_state(new_state);
migrate_generate_event(new_state);
}
@@ -574,7 +579,7 @@ static void migrate_fd_cleanup(void *opaque)
if (s->state != MIGRATION_STATUS_COMPLETED) {
qemu_savevm_state_cancel();
if (s->state == MIGRATION_STATUS_CANCELLING) {
- migrate_set_state(s, MIGRATION_STATUS_CANCELLING,
+ migrate_set_state(&s->state, MIGRATION_STATUS_CANCELLING,
MIGRATION_STATUS_CANCELLED);
}
}
@@ -586,7 +591,8 @@ void migrate_fd_error(MigrationState *s)
{
trace_migrate_fd_error();
assert(s->file == NULL);
- migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
+ migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
+ MIGRATION_STATUS_FAILED);
notifier_list_notify(&migration_state_notifiers, s);
}
@@ -602,7 +608,7 @@ static void migrate_fd_cancel(MigrationState *s)
old_state != MIGRATION_STATUS_ACTIVE) {
break;
}
- migrate_set_state(s, old_state, MIGRATION_STATUS_CANCELLING);
+ migrate_set_state(&s->state, old_state, MIGRATION_STATUS_CANCELLING);
} while (s->state != MIGRATION_STATUS_CANCELLING);
/*
@@ -670,7 +676,7 @@ static MigrationState *migrate_init(const MigrationParams *params)
s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
decompress_thread_count;
s->bandwidth_limit = bandwidth_limit;
- migrate_set_state(s, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
+ migrate_set_state(&s->state, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
return s;
@@ -768,7 +774,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
} else {
error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "uri",
"a valid migration protocol");
- migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
+ migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
+ MIGRATION_STATUS_FAILED);
return;
}
@@ -948,7 +955,8 @@ static void *migration_thread(void *opaque)
qemu_savevm_state_begin(s->file, &s->params);
s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
- migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_ACTIVE);
+ migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
+ MIGRATION_STATUS_ACTIVE);
while (s->state == MIGRATION_STATUS_ACTIVE) {
int64_t current_time;
@@ -978,13 +986,13 @@ static void *migration_thread(void *opaque)
qemu_mutex_unlock_iothread();
if (ret < 0) {
- migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
+ migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_FAILED);
break;
}
if (!qemu_file_get_error(s->file)) {
- migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
+ migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_COMPLETED);
break;
}
@@ -992,7 +1000,7 @@ static void *migration_thread(void *opaque)
}
if (qemu_file_get_error(s->file)) {
- migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
+ migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_FAILED);
break;
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 05/32] migration: Integrate COLO checkpoint process into migration
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (3 preceding siblings ...)
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 04/32] migration: Add state records for migration incoming zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-10-09 16:53 ` Dr. David Alan Gilbert
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 06/32] migration: Integrate COLO checkpoint process into loadvm zhanghailiang
` (30 subsequent siblings)
35 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
Add a migrate state: MIGRATION_STATUS_COLO, enter this migration state
after the first live migration successfully finished.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
include/migration/colo.h | 3 +++
migration/colo.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++
migration/migration.c | 23 ++++++++++++++-----
qapi-schema.json | 2 +-
stubs/migration-colo.c | 9 ++++++++
trace-events | 3 +++
6 files changed, 92 insertions(+), 6 deletions(-)
diff --git a/include/migration/colo.h b/include/migration/colo.h
index 9b6662d..dface19 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -19,4 +19,7 @@
bool colo_supported(void);
void colo_info_mig_init(void);
+void colo_init_checkpointer(MigrationState *s);
+bool migration_in_colo_state(void);
+
#endif
diff --git a/migration/colo.c b/migration/colo.c
index 2c40d2e..97e64a3 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -10,9 +10,67 @@
* later. See the COPYING file in the top-level directory.
*/
+#include "sysemu/sysemu.h"
#include "migration/colo.h"
+#include "trace.h"
+
+static QEMUBH *colo_bh;
bool colo_supported(void)
{
return true;
}
+
+bool migration_in_colo_state(void)
+{
+ MigrationState *s = migrate_get_current();
+
+ return (s->state == MIGRATION_STATUS_COLO);
+}
+
+static void *colo_thread(void *opaque)
+{
+ MigrationState *s = opaque;
+
+ qemu_mutex_lock_iothread();
+ vm_start();
+ qemu_mutex_unlock_iothread();
+ trace_colo_vm_state_change("stop", "run");
+
+ /*TODO: COLO checkpoint savevm loop*/
+
+ migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
+ MIGRATION_STATUS_COMPLETED);
+
+ qemu_mutex_lock_iothread();
+ qemu_bh_schedule(s->cleanup_bh);
+ qemu_mutex_unlock_iothread();
+
+ return NULL;
+}
+
+static void colo_start_checkpointer(void *opaque)
+{
+ MigrationState *s = opaque;
+
+ if (colo_bh) {
+ qemu_bh_delete(colo_bh);
+ colo_bh = NULL;
+ }
+
+ qemu_mutex_unlock_iothread();
+ qemu_thread_join(&s->thread);
+ qemu_mutex_lock_iothread();
+
+ migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
+ MIGRATION_STATUS_COLO);
+
+ qemu_thread_create(&s->thread, "colo", colo_thread, s,
+ QEMU_THREAD_JOINABLE);
+}
+
+void colo_init_checkpointer(MigrationState *s)
+{
+ colo_bh = qemu_bh_new(colo_start_checkpointer, s);
+ qemu_bh_schedule(colo_bh);
+}
diff --git a/migration/migration.c b/migration/migration.c
index 98133f1..bee61aa 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -446,6 +446,10 @@ MigrationInfo *qmp_query_migrate(Error **errp)
get_xbzrle_cache_stats(info);
break;
+ case MIGRATION_STATUS_COLO:
+ info->has_status = true;
+ /* TODO: display COLO specific information (checkpoint info etc.) */
+ break;
case MIGRATION_STATUS_COMPLETED:
get_xbzrle_cache_stats(info);
@@ -731,7 +735,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
if (s->state == MIGRATION_STATUS_ACTIVE ||
s->state == MIGRATION_STATUS_SETUP ||
- s->state == MIGRATION_STATUS_CANCELLING) {
+ s->state == MIGRATION_STATUS_CANCELLING ||
+ s->state == MIGRATION_STATUS_COLO) {
error_setg(errp, QERR_MIGRATION_ACTIVE);
return;
}
@@ -948,6 +953,7 @@ static void *migration_thread(void *opaque)
int64_t max_size = 0;
int64_t start_time = initial_time;
bool old_vm_running = false;
+ bool enable_colo = migrate_enable_colo();
rcu_register_thread();
@@ -992,8 +998,10 @@ static void *migration_thread(void *opaque)
}
if (!qemu_file_get_error(s->file)) {
- migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
- MIGRATION_STATUS_COMPLETED);
+ if (!enable_colo) {
+ migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
+ MIGRATION_STATUS_COMPLETED);
+ }
break;
}
}
@@ -1044,11 +1052,16 @@ static void *migration_thread(void *opaque)
}
runstate_set(RUN_STATE_POSTMIGRATE);
} else {
- if (old_vm_running) {
+ if (s->state == MIGRATION_STATUS_ACTIVE && enable_colo) {
+ colo_init_checkpointer(s);
+ } else if (old_vm_running) {
vm_start();
}
}
- qemu_bh_schedule(s->cleanup_bh);
+
+ if (!enable_colo || s->state != MIGRATION_STATUS_ACTIVE) {
+ qemu_bh_schedule(s->cleanup_bh);
+ }
qemu_mutex_unlock_iothread();
rcu_unregister_thread();
diff --git a/qapi-schema.json b/qapi-schema.json
index b14d1f4..6dd5c7c 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -433,7 +433,7 @@
##
{ 'enum': 'MigrationStatus',
'data': [ 'none', 'setup', 'cancelling', 'cancelled',
- 'active', 'completed', 'failed' ] }
+ 'active', 'completed', 'failed', 'colo' ] }
##
# @MigrationInfo
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index 3d817df..51b8f66 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -16,3 +16,12 @@ bool colo_supported(void)
{
return false;
}
+
+bool migration_in_colo_state(void)
+{
+ return false;
+}
+
+void colo_init_checkpointer(MigrationState *s)
+{
+}
diff --git a/trace-events b/trace-events
index 8f9614a..487d1c7 100644
--- a/trace-events
+++ b/trace-events
@@ -1472,6 +1472,9 @@ rdma_start_incoming_migration_after_rdma_listen(void) ""
rdma_start_outgoing_migration_after_rdma_connect(void) ""
rdma_start_outgoing_migration_after_rdma_source_init(void) ""
+# migration/colo.c
+colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
+
# kvm-all.c
kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
kvm_vm_ioctl(int type, void *arg) "type 0x%x, arg %p"
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 06/32] migration: Integrate COLO checkpoint process into loadvm
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (4 preceding siblings ...)
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 05/32] migration: Integrate COLO checkpoint process into migration zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-10-19 9:17 ` Dr. David Alan Gilbert
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 07/32] migration: Rename the'file' member of MigrationState and MigrationIncomingState zhanghailiang
` (29 subsequent siblings)
35 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
Switch from normal migration loadvm process into COLO checkpoint process if
COLO mode is enabled.
We add three new members to struct MigrationIncomingState, 'have_colo_incoming_thread'
and 'colo_incoming_thread' record the colo related threads for secondary VM,
'migration_incoming_co' records the original migration incoming coroutine.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
include/migration/colo.h | 7 +++++++
include/migration/migration.h | 7 +++++++
migration/colo-comm.c | 10 ++++++++++
migration/colo.c | 22 ++++++++++++++++++++++
migration/migration.c | 33 +++++++++++++++++++++++----------
stubs/migration-colo.c | 10 ++++++++++
trace-events | 1 +
7 files changed, 80 insertions(+), 10 deletions(-)
diff --git a/include/migration/colo.h b/include/migration/colo.h
index dface19..58849f7 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -15,6 +15,8 @@
#include "qemu-common.h"
#include "migration/migration.h"
+#include "block/coroutine.h"
+#include "qemu/thread.h"
bool colo_supported(void);
void colo_info_mig_init(void);
@@ -22,4 +24,9 @@ void colo_info_mig_init(void);
void colo_init_checkpointer(MigrationState *s);
bool migration_in_colo_state(void);
+/* loadvm */
+bool migration_incoming_enable_colo(void);
+void migration_incoming_exit_colo(void);
+void *colo_process_incoming_thread(void *opaque);
+bool migration_incoming_in_colo_state(void);
#endif
diff --git a/include/migration/migration.h b/include/migration/migration.h
index a62068f..9cdd6b6 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -22,6 +22,7 @@
#include "migration/vmstate.h"
#include "qapi-types.h"
#include "exec/cpu-common.h"
+#include "block/coroutine.h"
#define QEMU_VM_FILE_MAGIC 0x5145564d
#define QEMU_VM_FILE_VERSION_COMPAT 0x00000002
@@ -51,6 +52,12 @@ struct MigrationIncomingState {
QEMUFile *file;
int state;
+
+ bool have_colo_incoming_thread;
+ QemuThread colo_incoming_thread;
+ /* The coroutine we should enter (back) after failover */
+ Coroutine *migration_incoming_co;
+
/* See savevm.c */
LoadStateEntry_Head loadvm_handlers;
};
diff --git a/migration/colo-comm.c b/migration/colo-comm.c
index 4330bd8..0808d6c 100644
--- a/migration/colo-comm.c
+++ b/migration/colo-comm.c
@@ -52,3 +52,13 @@ void colo_info_mig_init(void)
{
vmstate_register(NULL, 0, &colo_state, &colo_info);
}
+
+bool migration_incoming_enable_colo(void)
+{
+ return colo_info.colo_requested;
+}
+
+void migration_incoming_exit_colo(void)
+{
+ colo_info.colo_requested = 0;
+}
diff --git a/migration/colo.c b/migration/colo.c
index 97e64a3..a341eee 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -13,6 +13,7 @@
#include "sysemu/sysemu.h"
#include "migration/colo.h"
#include "trace.h"
+#include "qemu/error-report.h"
static QEMUBH *colo_bh;
@@ -28,6 +29,13 @@ bool migration_in_colo_state(void)
return (s->state == MIGRATION_STATUS_COLO);
}
+bool migration_incoming_in_colo_state(void)
+{
+ MigrationIncomingState *mis = migration_incoming_get_current();
+
+ return (mis && (mis->state == MIGRATION_STATUS_COLO));
+}
+
static void *colo_thread(void *opaque)
{
MigrationState *s = opaque;
@@ -74,3 +82,17 @@ void colo_init_checkpointer(MigrationState *s)
colo_bh = qemu_bh_new(colo_start_checkpointer, s);
qemu_bh_schedule(colo_bh);
}
+
+void *colo_process_incoming_thread(void *opaque)
+{
+ MigrationIncomingState *mis = opaque;
+
+ migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+ MIGRATION_STATUS_COLO);
+
+ /* TODO: COLO checkpoint restore loop */
+
+ migration_incoming_exit_colo();
+
+ return NULL;
+}
diff --git a/migration/migration.c b/migration/migration.c
index bee61aa..241689f 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -280,7 +280,28 @@ static void process_incoming_migration_co(void *opaque)
MIGRATION_STATUS_ACTIVE);
ret = qemu_loadvm_state(f);
- qemu_fclose(f);
+ if (!ret) {
+ /* Make sure all file formats flush their mutable metadata */
+ bdrv_invalidate_cache_all(&local_err);
+ if (local_err) {
+ error_report_err(local_err);
+ migrate_decompress_threads_join();
+ exit(EXIT_FAILURE);
+ }
+ }
+ /* we get colo info, and know if we are in colo mode */
+ if (!ret && migration_incoming_enable_colo()) {
+ mis->migration_incoming_co = qemu_coroutine_self();
+ qemu_thread_create(&mis->colo_incoming_thread, "colo incoming",
+ colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE);
+ mis->have_colo_incoming_thread = true;
+ qemu_coroutine_yield();
+
+ /* Wait checkpoint incoming thread exit before free resource */
+ qemu_thread_join(&mis->colo_incoming_thread);
+ } else {
+ qemu_fclose(f);
+ }
free_xbzrle_decoded_buf();
migration_incoming_state_destroy();
@@ -295,18 +316,9 @@ static void process_incoming_migration_co(void *opaque)
MIGRATION_STATUS_COMPLETED);
qemu_announce_self();
- /* Make sure all file formats flush their mutable metadata */
- bdrv_invalidate_cache_all(&local_err);
- if (local_err) {
- error_report_err(local_err);
- migrate_decompress_threads_join();
- exit(EXIT_FAILURE);
- }
-
/* If global state section was not received or we are in running
state, we need to obey autostart. Any other state is set with
runstate_set. */
-
if (!global_state_received() ||
global_state_get_runstate() == RUN_STATE_RUNNING) {
if (autostart) {
@@ -740,6 +752,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
error_setg(errp, QERR_MIGRATION_ACTIVE);
return;
}
+
if (runstate_check(RUN_STATE_INMIGRATE)) {
error_setg(errp, "Guest is waiting for an incoming migration");
return;
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index 51b8f66..c49ee1a 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -22,6 +22,16 @@ bool migration_in_colo_state(void)
return false;
}
+bool migration_incoming_in_colo_state(void)
+{
+ return false;
+}
+
void colo_init_checkpointer(MigrationState *s)
{
}
+
+void *colo_process_incoming_thread(void *opaque)
+{
+ return NULL;
+}
diff --git a/trace-events b/trace-events
index 487d1c7..352e9c3 100644
--- a/trace-events
+++ b/trace-events
@@ -1474,6 +1474,7 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) ""
# migration/colo.c
colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
+colo_receive_message(const char *msg) "Receive '%s'"
# kvm-all.c
kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 07/32] migration: Rename the'file' member of MigrationState and MigrationIncomingState
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (5 preceding siblings ...)
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 06/32] migration: Integrate COLO checkpoint process into loadvm zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 08/32] COLO/migration: establish a new communication path from destination to source zhanghailiang
` (28 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
Rename the 'file' member of MigrationState to 'to_dst_file' and
Rename the 'file' member of MigrationIncomingState to 'from_src_file'.
For now, there are only one path direction for migration, it is from
source side to destination side, but for colo and post-copy, we need
both directions communication, so here we rename the file member to indicate
this path.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
include/migration/migration.h | 4 ++--
migration/exec.c | 4 ++--
migration/fd.c | 4 ++--
migration/migration.c | 48 ++++++++++++++++++++++---------------------
migration/tcp.c | 4 ++--
migration/unix.c | 4 ++--
6 files changed, 35 insertions(+), 33 deletions(-)
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 9cdd6b6..6488e03 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -49,7 +49,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
/* State for the incoming migration */
struct MigrationIncomingState {
- QEMUFile *file;
+ QEMUFile *from_src_file;
int state;
@@ -73,7 +73,7 @@ struct MigrationState
size_t xfer_limit;
QemuThread thread;
QEMUBH *cleanup_bh;
- QEMUFile *file;
+ QEMUFile *to_dst_file;
int parameters[MIGRATION_PARAMETER_MAX];
int state;
diff --git a/migration/exec.c b/migration/exec.c
index 8406d2b..9037109 100644
--- a/migration/exec.c
+++ b/migration/exec.c
@@ -36,8 +36,8 @@
void exec_start_outgoing_migration(MigrationState *s, const char *command, Error **errp)
{
- s->file = qemu_popen_cmd(command, "w");
- if (s->file == NULL) {
+ s->to_dst_file = qemu_popen_cmd(command, "w");
+ if (s->to_dst_file == NULL) {
error_setg_errno(errp, errno, "failed to popen the migration target");
return;
}
diff --git a/migration/fd.c b/migration/fd.c
index 3e4bed0..9a9d6c5 100644
--- a/migration/fd.c
+++ b/migration/fd.c
@@ -50,9 +50,9 @@ void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **
}
if (fd_is_socket(fd)) {
- s->file = qemu_fopen_socket(fd, "wb");
+ s->to_dst_file = qemu_fopen_socket(fd, "wb");
} else {
- s->file = qemu_fdopen(fd, "wb");
+ s->to_dst_file = qemu_fdopen(fd, "wb");
}
migrate_fd_connect(s);
diff --git a/migration/migration.c b/migration/migration.c
index 241689f..f7ca2c4 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -88,7 +88,7 @@ MigrationIncomingState *migration_incoming_get_current(void)
MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
{
mis_current = g_malloc0(sizeof(MigrationIncomingState));
- mis_current->file = f;
+ mis_current->from_src_file = f;
mis_current->state = MIGRATION_STATUS_NONE;
QLIST_INIT(&mis_current->loadvm_handlers);
@@ -579,15 +579,15 @@ static void migrate_fd_cleanup(void *opaque)
qemu_bh_delete(s->cleanup_bh);
s->cleanup_bh = NULL;
- if (s->file) {
+ if (s->to_dst_file) {
trace_migrate_fd_cleanup();
qemu_mutex_unlock_iothread();
qemu_thread_join(&s->thread);
qemu_mutex_lock_iothread();
migrate_compress_threads_join();
- qemu_fclose(s->file);
- s->file = NULL;
+ qemu_fclose(s->to_dst_file);
+ s->to_dst_file = NULL;
}
assert(s->state != MIGRATION_STATUS_ACTIVE);
@@ -606,7 +606,7 @@ static void migrate_fd_cleanup(void *opaque)
void migrate_fd_error(MigrationState *s)
{
trace_migrate_fd_error();
- assert(s->file == NULL);
+ assert(s->to_dst_file == NULL);
migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
MIGRATION_STATUS_FAILED);
notifier_list_notify(&migration_state_notifiers, s);
@@ -615,7 +615,7 @@ void migrate_fd_error(MigrationState *s)
static void migrate_fd_cancel(MigrationState *s)
{
int old_state ;
- QEMUFile *f = migrate_get_current()->file;
+ QEMUFile *f = migrate_get_current()->to_dst_file;
trace_migrate_fd_cancel();
do {
@@ -856,8 +856,9 @@ void qmp_migrate_set_speed(int64_t value, Error **errp)
s = migrate_get_current();
s->bandwidth_limit = value;
- if (s->file) {
- qemu_file_set_rate_limit(s->file, s->bandwidth_limit / XFER_LIMIT_RATIO);
+ if (s->to_dst_file) {
+ qemu_file_set_rate_limit(s->to_dst_file,
+ s->bandwidth_limit / XFER_LIMIT_RATIO);
}
}
@@ -970,8 +971,8 @@ static void *migration_thread(void *opaque)
rcu_register_thread();
- qemu_savevm_state_header(s->file);
- qemu_savevm_state_begin(s->file, &s->params);
+ qemu_savevm_state_header(s->to_dst_file);
+ qemu_savevm_state_begin(s->to_dst_file, &s->params);
s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
@@ -981,11 +982,11 @@ static void *migration_thread(void *opaque)
int64_t current_time;
uint64_t pending_size;
- if (!qemu_file_rate_limit(s->file)) {
- pending_size = qemu_savevm_state_pending(s->file, max_size);
+ if (!qemu_file_rate_limit(s->to_dst_file)) {
+ pending_size = qemu_savevm_state_pending(s->to_dst_file, max_size);
trace_migrate_pending(pending_size, max_size);
if (pending_size && pending_size >= max_size) {
- qemu_savevm_state_iterate(s->file);
+ qemu_savevm_state_iterate(s->to_dst_file);
} else {
int ret;
@@ -998,8 +999,8 @@ static void *migration_thread(void *opaque)
if (!ret) {
ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
if (ret >= 0) {
- qemu_file_set_rate_limit(s->file, INT64_MAX);
- qemu_savevm_state_complete(s->file);
+ qemu_file_set_rate_limit(s->to_dst_file, INT64_MAX);
+ qemu_savevm_state_complete(s->to_dst_file);
}
}
qemu_mutex_unlock_iothread();
@@ -1010,7 +1011,7 @@ static void *migration_thread(void *opaque)
break;
}
- if (!qemu_file_get_error(s->file)) {
+ if (!qemu_file_get_error(s->to_dst_file)) {
if (!enable_colo) {
migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_COMPLETED);
@@ -1020,14 +1021,15 @@ static void *migration_thread(void *opaque)
}
}
- if (qemu_file_get_error(s->file)) {
+ if (qemu_file_get_error(s->to_dst_file)) {
migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_FAILED);
break;
}
current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
if (current_time >= initial_time + BUFFER_DELAY) {
- uint64_t transferred_bytes = qemu_ftell(s->file) - initial_bytes;
+ uint64_t transferred_bytes = qemu_ftell(s->to_dst_file) -
+ initial_bytes;
uint64_t time_spent = current_time - initial_time;
double bandwidth = transferred_bytes / time_spent;
max_size = bandwidth * migrate_max_downtime() / 1000000;
@@ -1043,11 +1045,11 @@ static void *migration_thread(void *opaque)
s->expected_downtime = s->dirty_bytes_rate / bandwidth;
}
- qemu_file_reset_rate_limit(s->file);
+ qemu_file_reset_rate_limit(s->to_dst_file);
initial_time = current_time;
- initial_bytes = qemu_ftell(s->file);
+ initial_bytes = qemu_ftell(s->to_dst_file);
}
- if (qemu_file_rate_limit(s->file)) {
+ if (qemu_file_rate_limit(s->to_dst_file)) {
/* usleep expects microseconds */
g_usleep((initial_time + BUFFER_DELAY - current_time)*1000);
}
@@ -1056,7 +1058,7 @@ static void *migration_thread(void *opaque)
qemu_mutex_lock_iothread();
if (s->state == MIGRATION_STATUS_COMPLETED) {
int64_t end_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
- uint64_t transferred_bytes = qemu_ftell(s->file);
+ uint64_t transferred_bytes = qemu_ftell(s->to_dst_file);
s->total_time = end_time - s->total_time;
s->downtime = end_time - start_time;
if (s->total_time) {
@@ -1087,7 +1089,7 @@ void migrate_fd_connect(MigrationState *s)
s->expected_downtime = max_downtime/1000000;
s->cleanup_bh = qemu_bh_new(migrate_fd_cleanup, s);
- qemu_file_set_rate_limit(s->file,
+ qemu_file_set_rate_limit(s->to_dst_file,
s->bandwidth_limit / XFER_LIMIT_RATIO);
/* Notify before starting migration thread */
diff --git a/migration/tcp.c b/migration/tcp.c
index ae89172..e083d68 100644
--- a/migration/tcp.c
+++ b/migration/tcp.c
@@ -39,11 +39,11 @@ static void tcp_wait_for_connect(int fd, Error *err, void *opaque)
if (fd < 0) {
DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
- s->file = NULL;
+ s->to_dst_file = NULL;
migrate_fd_error(s);
} else {
DPRINTF("migrate connect success\n");
- s->file = qemu_fopen_socket(fd, "wb");
+ s->to_dst_file = qemu_fopen_socket(fd, "wb");
migrate_fd_connect(s);
}
}
diff --git a/migration/unix.c b/migration/unix.c
index b591813..5492dd6 100644
--- a/migration/unix.c
+++ b/migration/unix.c
@@ -39,11 +39,11 @@ static void unix_wait_for_connect(int fd, Error *err, void *opaque)
if (fd < 0) {
DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
- s->file = NULL;
+ s->to_dst_file = NULL;
migrate_fd_error(s);
} else {
DPRINTF("migrate connect success\n");
- s->file = qemu_fopen_socket(fd, "wb");
+ s->to_dst_file = qemu_fopen_socket(fd, "wb");
migrate_fd_connect(s);
}
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 08/32] COLO/migration: establish a new communication path from destination to source
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (6 preceding siblings ...)
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 07/32] migration: Rename the'file' member of MigrationState and MigrationIncomingState zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-10-19 9:54 ` Dr. David Alan Gilbert
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 09/32] COLO: Implement colo checkpoint protocol zhanghailiang
` (27 subsequent siblings)
35 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
Add a new member 'to_src_file' to MigrationIncomingState and a
new member 'from_dst_file' to MigrationState.
They will be used for returning messages from destination to source.
It will also be used by post-copy migration.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
include/migration/migration.h | 3 ++-
migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 45 insertions(+), 1 deletion(-)
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 6488e03..0c94103 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -50,7 +50,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
/* State for the incoming migration */
struct MigrationIncomingState {
QEMUFile *from_src_file;
-
+ QEMUFile *to_src_file;
int state;
bool have_colo_incoming_thread;
@@ -74,6 +74,7 @@ struct MigrationState
QemuThread thread;
QEMUBH *cleanup_bh;
QEMUFile *to_dst_file;
+ QEMUFile *from_dst_file;
int parameters[MIGRATION_PARAMETER_MAX];
int state;
diff --git a/migration/colo.c b/migration/colo.c
index a341eee..5f4fb20 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -39,6 +39,20 @@ bool migration_incoming_in_colo_state(void)
static void *colo_thread(void *opaque)
{
MigrationState *s = opaque;
+ int fd, ret = 0;
+
+ /* Dup the fd of to_dst_file */
+ fd = dup(qemu_get_fd(s->to_dst_file));
+ if (fd == -1) {
+ ret = -errno;
+ goto out;
+ }
+ s->from_dst_file = qemu_fopen_socket(fd, "rb");
+ if (!s->from_dst_file) {
+ ret = -EINVAL;
+ error_report("Open QEMUFile failed!");
+ goto out;
+ }
qemu_mutex_lock_iothread();
vm_start();
@@ -47,9 +61,17 @@ static void *colo_thread(void *opaque)
/*TODO: COLO checkpoint savevm loop*/
+out:
+ if (ret < 0) {
+ error_report("Detect some error: %s", strerror(-ret));
+ }
migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
MIGRATION_STATUS_COMPLETED);
+ if (s->from_dst_file) {
+ qemu_fclose(s->from_dst_file);
+ }
+
qemu_mutex_lock_iothread();
qemu_bh_schedule(s->cleanup_bh);
qemu_mutex_unlock_iothread();
@@ -86,12 +108,33 @@ void colo_init_checkpointer(MigrationState *s)
void *colo_process_incoming_thread(void *opaque)
{
MigrationIncomingState *mis = opaque;
+ int fd, ret = 0;
migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_COLO);
+ fd = dup(qemu_get_fd(mis->from_src_file));
+ if (fd < 0) {
+ ret = -errno;
+ goto out;
+ }
+ mis->to_src_file = qemu_fopen_socket(fd, "wb");
+ if (!mis->to_src_file) {
+ ret = -EINVAL;
+ error_report("Can't open incoming channel!");
+ goto out;
+ }
/* TODO: COLO checkpoint restore loop */
+out:
+ if (ret < 0) {
+ error_report("colo incoming thread will exit, detect error: %s",
+ strerror(-ret));
+ }
+
+ if (mis->to_src_file) {
+ qemu_fclose(mis->to_src_file);
+ }
migration_incoming_exit_colo();
return NULL;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 09/32] COLO: Implement colo checkpoint protocol
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (7 preceding siblings ...)
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 08/32] COLO/migration: establish a new communication path from destination to source zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-10-21 12:17 ` Eric Blake
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 10/32] COLO: Add a new RunState RUN_STATE_COLO zhanghailiang
` (26 subsequent siblings)
35 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
We need communications protocol of user-defined to control the checkpoint
process.
The new checkpoint request is started by Primary VM, and the interactive process
like below:
Checkpoint synchronizing points,
Primary Secondary
'checkpoint-request' @ ----------------------------->
Suspend (In hybrid mode)
'checkpoint-reply' <------------------------------ @
Suspend&Save state
'vmstate-send' @ ----------------------------->
Send state Receive state
'vmstate-received' <------------------------------ @
Release packets Load state
'vmstate-load' <------------------------------ @
Resume Resume (In hybrid mode)
Start Comparing (In hybrid mode)
NOTE:
1) '@' who sends the message
2) Every sync-point is synchronized by two sides with only
one handshake(single direction) for low-latency.
If more strict synchronization is required, a opposite direction
sync-point should be added.
3) Since sync-points are single direction, the remote side may
go forward a lot when this side just receives the sync-point.
4) For now, we only support 'periodic' checkpoint, for which
the Secondary VM is not running, later we will support 'hybrid' mode.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
migration/colo.c | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
qapi-schema.json | 26 ++++++++
trace-events | 3 +-
3 files changed, 218 insertions(+), 3 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index 5f4fb20..526ee85 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -10,10 +10,12 @@
* later. See the COPYING file in the top-level directory.
*/
+#include <unistd.h>
#include "sysemu/sysemu.h"
#include "migration/colo.h"
#include "trace.h"
#include "qemu/error-report.h"
+#include "qemu/sockets.h"
static QEMUBH *colo_bh;
@@ -36,6 +38,103 @@ bool migration_incoming_in_colo_state(void)
return (mis && (mis->state == MIGRATION_STATUS_COLO));
}
+/* colo checkpoint control helper */
+static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value)
+{
+ int ret = 0;
+
+ qemu_put_be32(f, cmd);
+ qemu_put_be64(f, value);
+ qemu_fflush(f);
+
+ ret = qemu_file_get_error(f);
+ trace_colo_ctl_put(COLOCmd_lookup[cmd]);
+
+ return ret;
+}
+
+static int colo_ctl_get_cmd(QEMUFile *f, uint32_t *cmd)
+{
+ int ret = 0;
+
+ *cmd = qemu_get_be32(f);
+ ret = qemu_file_get_error(f);
+ if (ret < 0) {
+ return ret;
+ }
+ if (*cmd >= COLO_CMD_MAX) {
+ error_report("Invalid colo command, get cmd:%d", *cmd);
+ return -EINVAL;
+ }
+ trace_colo_ctl_get(COLOCmd_lookup[*cmd]);
+
+ return 0;
+}
+
+static int colo_ctl_get(QEMUFile *f, uint32_t require)
+{
+ int ret;
+ uint32_t cmd;
+ uint64_t value;
+
+ ret = colo_ctl_get_cmd(f, &cmd);
+ if (ret < 0) {
+ return ret;
+ }
+ if (cmd != require) {
+ error_report("Unexpect colo command, expect:%d, but get cmd:%d",
+ require, cmd);
+ return -EINVAL;
+ }
+
+ value = qemu_get_be64(f);
+ ret = qemu_file_get_error(f);
+ if (ret < 0) {
+ return ret;
+ }
+
+ return value;
+}
+
+static int colo_do_checkpoint_transaction(MigrationState *s)
+{
+ int ret;
+
+ ret = colo_ctl_put(s->to_dst_file, COLO_CMD_CHECKPOINT_REQUEST, 0);
+ if (ret < 0) {
+ goto out;
+ }
+
+ ret = colo_ctl_get(s->from_dst_file, COLO_CMD_CHECKPOINT_REPLY);
+ if (ret < 0) {
+ goto out;
+ }
+
+ /* TODO: suspend and save vm state to colo buffer */
+
+ ret = colo_ctl_put(s->to_dst_file, COLO_CMD_VMSTATE_SEND, 0);
+ if (ret < 0) {
+ goto out;
+ }
+
+ /* TODO: send vmstate to Secondary */
+
+ ret = colo_ctl_get(s->from_dst_file, COLO_CMD_VMSTATE_RECEIVED);
+ if (ret < 0) {
+ goto out;
+ }
+
+ ret = colo_ctl_get(s->from_dst_file, COLO_CMD_VMSTATE_LOADED);
+ if (ret < 0) {
+ goto out;
+ }
+
+ /* TODO: resume Primary */
+
+out:
+ return ret;
+}
+
static void *colo_thread(void *opaque)
{
MigrationState *s = opaque;
@@ -54,12 +153,27 @@ static void *colo_thread(void *opaque)
goto out;
}
+ /*
+ * Wait for Secondary finish loading vm states and enter COLO
+ * restore.
+ */
+ ret = colo_ctl_get(s->from_dst_file, COLO_CMD_CHECKPOINT_READY);
+ if (ret < 0) {
+ goto out;
+ }
+
qemu_mutex_lock_iothread();
vm_start();
qemu_mutex_unlock_iothread();
trace_colo_vm_state_change("stop", "run");
- /*TODO: COLO checkpoint savevm loop*/
+ while (s->state == MIGRATION_STATUS_COLO) {
+ /* start a colo checkpoint */
+ ret = colo_do_checkpoint_transaction(s);
+ if (ret < 0) {
+ goto out;
+ }
+ }
out:
if (ret < 0) {
@@ -105,6 +219,39 @@ void colo_init_checkpointer(MigrationState *s)
qemu_bh_schedule(colo_bh);
}
+/*
+ * return:
+ * 0: start a checkpoint
+ * -1: some error happened, exit colo restore
+ */
+static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
+{
+ int ret;
+ uint32_t cmd;
+ uint64_t value;
+
+ ret = colo_ctl_get_cmd(f, &cmd);
+ if (ret < 0) {
+ /* do failover ? */
+ return ret;
+ }
+ /* Fix me: this value should be 0, which is not so good,
+ * should be used for checking ?
+ */
+ value = qemu_get_be64(f);
+ if (value != 0) {
+ return -EINVAL;
+ }
+
+ switch (cmd) {
+ case COLO_CMD_CHECKPOINT_REQUEST:
+ *checkpoint_request = 1;
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
void *colo_process_incoming_thread(void *opaque)
{
MigrationIncomingState *mis = opaque;
@@ -124,7 +271,48 @@ void *colo_process_incoming_thread(void *opaque)
error_report("Can't open incoming channel!");
goto out;
}
- /* TODO: COLO checkpoint restore loop */
+
+ ret = colo_ctl_put(mis->to_src_file, COLO_CMD_CHECKPOINT_READY, 0);
+ if (ret < 0) {
+ goto out;
+ }
+
+ while (mis->state == MIGRATION_STATUS_COLO) {
+ int request = 0;
+ int ret = colo_wait_handle_cmd(mis->from_src_file, &request);
+
+ if (ret < 0) {
+ break;
+ } else {
+ if (!request) {
+ continue;
+ }
+ }
+
+ ret = colo_ctl_put(mis->to_src_file, COLO_CMD_CHECKPOINT_REPLY, 0);
+ if (ret < 0) {
+ goto out;
+ }
+
+ ret = colo_ctl_get(mis->from_src_file, COLO_CMD_VMSTATE_SEND);
+ if (ret < 0) {
+ goto out;
+ }
+
+ /* TODO: read migration data into colo buffer */
+
+ ret = colo_ctl_put(mis->to_src_file, COLO_CMD_VMSTATE_RECEIVED, 0);
+ if (ret < 0) {
+ goto out;
+ }
+
+ /* TODO: load vm state */
+
+ ret = colo_ctl_put(mis->to_src_file, COLO_CMD_VMSTATE_LOADED, 0);
+ if (ret < 0) {
+ goto out;
+ }
+}
out:
if (ret < 0) {
diff --git a/qapi-schema.json b/qapi-schema.json
index 6dd5c7c..0d3d6e2 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -664,6 +664,32 @@
'*tls-port': 'int', '*cert-subject': 'str' } }
##
+# @COLOCmd
+#
+# The colo command
+#
+# @invalid: unknown command
+#
+# @checkpoint-ready: SVM is ready for checkpointing
+#
+# @checkpoint-request: PVM tells SVM to prepare for new checkpointing
+#
+# @checkpoint-reply: SVM gets PVM's checkpoint request
+#
+# @vmstate-send: VM's state will be sent by PVM.
+#
+# @vmstate-received: VM's state has been received by SVM
+#
+# @vmstate-loaded: VM's state has been loaded by SVM
+#
+# Since: 2.5
+##
+{ 'enum': 'COLOCmd',
+ 'data': [ 'invalid', 'checkpoint-ready', 'checkpoint-request',
+ 'checkpoint-reply', 'vmstate-send', 'vmstate-size',
+ 'vmstate-received', 'vmstate-loaded', 'guest-shutdown',
+ 'ram-steal'] }
+
# @MouseInfo:
#
# Information about a mouse device.
diff --git a/trace-events b/trace-events
index 352e9c3..7fdf723 100644
--- a/trace-events
+++ b/trace-events
@@ -1474,7 +1474,8 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) ""
# migration/colo.c
colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
-colo_receive_message(const char *msg) "Receive '%s'"
+colo_ctl_put(const char *msg) "Send '%s'"
+colo_ctl_get(const char *msg) "Receive '%s'"
# kvm-all.c
kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 10/32] COLO: Add a new RunState RUN_STATE_COLO
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (8 preceding siblings ...)
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 09/32] COLO: Implement colo checkpoint protocol zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-10-21 12:18 ` Eric Blake
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 11/32] QEMUSizedBuffer: Introduce two help functions for qsb zhanghailiang
` (25 subsequent siblings)
35 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, arei.gonglei, stefanha,
amit.shah, yanghy, zhanghailiang
Guest will enter this state when paused to save/restore VM state
under colo checkpoint.
Cc: Eric Blake <eblake@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
qapi-schema.json | 7 ++++++-
vl.c | 8 ++++++++
2 files changed, 14 insertions(+), 1 deletion(-)
diff --git a/qapi-schema.json b/qapi-schema.json
index 0d3d6e2..6a3796e 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -148,12 +148,15 @@
# @watchdog: the watchdog action is configured to pause and has been triggered
#
# @guest-panicked: guest has been panicked as a result of guest OS panic
+#
+# @colo: guest is paused to save/restore VM state under colo checkpoint (since
+# 2.5)
##
{ 'enum': 'RunState',
'data': [ 'debug', 'inmigrate', 'internal-error', 'io-error', 'paused',
'postmigrate', 'prelaunch', 'finish-migrate', 'restore-vm',
'running', 'save-vm', 'shutdown', 'suspended', 'watchdog',
- 'guest-panicked' ] }
+ 'guest-panicked', 'colo' ] }
##
# @StatusInfo:
@@ -428,6 +431,8 @@
#
# @failed: some error occurred during migration process.
#
+# @colo: VM is in the process of fault tolerance.
+#
# Since: 2.3
#
##
diff --git a/vl.c b/vl.c
index 571347a..d5e900b 100644
--- a/vl.c
+++ b/vl.c
@@ -584,6 +584,7 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_INMIGRATE, RUN_STATE_SUSPENDED },
{ RUN_STATE_INMIGRATE, RUN_STATE_WATCHDOG },
{ RUN_STATE_INMIGRATE, RUN_STATE_GUEST_PANICKED },
+ { RUN_STATE_INMIGRATE, RUN_STATE_COLO },
{ RUN_STATE_INTERNAL_ERROR, RUN_STATE_PAUSED },
{ RUN_STATE_INTERNAL_ERROR, RUN_STATE_FINISH_MIGRATE },
@@ -593,6 +594,7 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_PAUSED, RUN_STATE_RUNNING },
{ RUN_STATE_PAUSED, RUN_STATE_FINISH_MIGRATE },
+ { RUN_STATE_PAUSED, RUN_STATE_COLO},
{ RUN_STATE_POSTMIGRATE, RUN_STATE_RUNNING },
{ RUN_STATE_POSTMIGRATE, RUN_STATE_FINISH_MIGRATE },
@@ -603,9 +605,12 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_FINISH_MIGRATE, RUN_STATE_RUNNING },
{ RUN_STATE_FINISH_MIGRATE, RUN_STATE_POSTMIGRATE },
+ { RUN_STATE_FINISH_MIGRATE, RUN_STATE_COLO},
{ RUN_STATE_RESTORE_VM, RUN_STATE_RUNNING },
+ { RUN_STATE_COLO, RUN_STATE_RUNNING },
+
{ RUN_STATE_RUNNING, RUN_STATE_DEBUG },
{ RUN_STATE_RUNNING, RUN_STATE_INTERNAL_ERROR },
{ RUN_STATE_RUNNING, RUN_STATE_IO_ERROR },
@@ -616,6 +621,7 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_RUNNING, RUN_STATE_SHUTDOWN },
{ RUN_STATE_RUNNING, RUN_STATE_WATCHDOG },
{ RUN_STATE_RUNNING, RUN_STATE_GUEST_PANICKED },
+ { RUN_STATE_RUNNING, RUN_STATE_COLO},
{ RUN_STATE_SAVE_VM, RUN_STATE_RUNNING },
@@ -626,9 +632,11 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_RUNNING, RUN_STATE_SUSPENDED },
{ RUN_STATE_SUSPENDED, RUN_STATE_RUNNING },
{ RUN_STATE_SUSPENDED, RUN_STATE_FINISH_MIGRATE },
+ { RUN_STATE_SUSPENDED, RUN_STATE_COLO},
{ RUN_STATE_WATCHDOG, RUN_STATE_RUNNING },
{ RUN_STATE_WATCHDOG, RUN_STATE_FINISH_MIGRATE },
+ { RUN_STATE_WATCHDOG, RUN_STATE_COLO},
{ RUN_STATE_GUEST_PANICKED, RUN_STATE_RUNNING },
{ RUN_STATE_GUEST_PANICKED, RUN_STATE_FINISH_MIGRATE },
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 11/32] QEMUSizedBuffer: Introduce two help functions for qsb
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (9 preceding siblings ...)
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 10/32] COLO: Add a new RunState RUN_STATE_COLO zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 12/32] COLO: Save PVM state to secondary side when do checkpoint zhanghailiang
` (24 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
Introduce two new QEMUSizedBuffer APIs which will be used by COLO to buffer
VM state:
One is qsb_put_buffer(), which put the content of a given QEMUSizedBuffer
into QEMUFile, this is used to send buffered VM state to secondary.
Another is qsb_fill_buffer(), read 'size' bytes of data from the file into
qsb, this is used to get VM state from socket into a buffer.
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
include/migration/qemu-file.h | 3 ++-
migration/qemu-file-buf.c | 58 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 60 insertions(+), 1 deletion(-)
diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index ea49f33..62461b5 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -144,7 +144,8 @@ ssize_t qsb_get_buffer(const QEMUSizedBuffer *, off_t start, size_t count,
uint8_t *buf);
ssize_t qsb_write_at(QEMUSizedBuffer *qsb, const uint8_t *buf,
off_t pos, size_t count);
-
+void qsb_put_buffer(QEMUFile *f, QEMUSizedBuffer *qsb, int size);
+int qsb_fill_buffer(QEMUSizedBuffer *qsb, QEMUFile *f, int size);
/*
* For use on files opened with qemu_bufopen
diff --git a/migration/qemu-file-buf.c b/migration/qemu-file-buf.c
index 2de9330..93e67ca 100644
--- a/migration/qemu-file-buf.c
+++ b/migration/qemu-file-buf.c
@@ -366,6 +366,64 @@ ssize_t qsb_write_at(QEMUSizedBuffer *qsb, const uint8_t *source,
return count;
}
+
+/**
+ * Put the content of a given QEMUSizedBuffer into QEMUFile.
+ *
+ * @f: A QEMUFile
+ * @qsb: A QEMUSizedBuffer
+ * @size: size of content to write
+ */
+void qsb_put_buffer(QEMUFile *f, QEMUSizedBuffer *qsb, int size)
+{
+ int i, l;
+
+ for (i = 0; i < qsb->n_iov && size > 0; i++) {
+ l = MIN(qsb->iov[i].iov_len, size);
+ qemu_put_buffer(f, qsb->iov[i].iov_base, l);
+ size -= l;
+ }
+}
+
+/*
+ * Read 'size' bytes of data from the file into qsb.
+ * always fill from pos 0 and used after qsb_create().
+ *
+ * It will return size bytes unless there was an error, in which case it will
+ * return as many as it managed to read (assuming blocking fd's which
+ * all current QEMUFile are)
+ */
+int qsb_fill_buffer(QEMUSizedBuffer *qsb, QEMUFile *f, int size)
+{
+ ssize_t rc = qsb_grow(qsb, size);
+ int pending = size, i;
+ qsb->used = 0;
+ uint8_t *buf = NULL;
+
+ if (rc < 0) {
+ return rc;
+ }
+
+ for (i = 0; i < qsb->n_iov && pending > 0; i++) {
+ int doneone = 0;
+ /* read until iov full */
+ while (doneone < qsb->iov[i].iov_len && pending > 0) {
+ int readone = 0;
+ buf = qsb->iov[i].iov_base;
+ readone = qemu_get_buffer(f, buf,
+ MIN(qsb->iov[i].iov_len - doneone, pending));
+ if (readone == 0) {
+ return qsb->used;
+ }
+ buf += readone;
+ doneone += readone;
+ pending -= readone;
+ qsb->used += readone;
+ }
+ }
+ return qsb->used;
+}
+
typedef struct QEMUBuffer {
QEMUSizedBuffer *qsb;
QEMUFile *file;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 12/32] COLO: Save PVM state to secondary side when do checkpoint
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (10 preceding siblings ...)
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 11/32] QEMUSizedBuffer: Introduce two help functions for qsb zhanghailiang
@ 2015-09-02 8:22 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 13/32] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily zhanghailiang
` (23 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:22 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
The main process of checkpoint is to synchronize SVM with PVM.
VM's state includes ram and device state. So we will migrate PVM's
state to SVM when do checkpoint, just like migration does.
We will cache PVM's state in slave, we use QEMUSizedBuffer
to store the data, we need to know the size of VM state, so in master,
we use qsb to store VM state temporarily, get the data size by call qsb_get_length()
and then migrate the data to the qsb in the secondary side.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 70 +++++++++++++++++++++++++++++++++++++++++++++++++-----
migration/ram.c | 46 +++++++++++++++++++++++++++--------
migration/savevm.c | 2 +-
3 files changed, 101 insertions(+), 17 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index 526ee85..f107032 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -18,6 +18,8 @@
#include "qemu/sockets.h"
static QEMUBH *colo_bh;
+/* colo buffer */
+#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
bool colo_supported(void)
{
@@ -96,9 +98,12 @@ static int colo_ctl_get(QEMUFile *f, uint32_t require)
return value;
}
-static int colo_do_checkpoint_transaction(MigrationState *s)
+static int colo_do_checkpoint_transaction(MigrationState *s,
+ QEMUSizedBuffer *buffer)
{
int ret;
+ size_t size;
+ QEMUFile *trans = NULL;
ret = colo_ctl_put(s->to_dst_file, COLO_CMD_CHECKPOINT_REQUEST, 0);
if (ret < 0) {
@@ -109,15 +114,48 @@ static int colo_do_checkpoint_transaction(MigrationState *s)
if (ret < 0) {
goto out;
}
+ /* Reset colo buffer and open it for write */
+ qsb_set_length(buffer, 0);
+ trans = qemu_bufopen("w", buffer);
+ if (!trans) {
+ error_report("Open colo buffer for write failed");
+ goto out;
+ }
+
+ /* suspend and save vm state to colo buffer */
+ qemu_mutex_lock_iothread();
+ vm_stop_force_state(RUN_STATE_COLO);
+ qemu_mutex_unlock_iothread();
+ trace_colo_vm_state_change("run", "stop");
+
+ /* Disable block migration */
+ s->params.blk = 0;
+ s->params.shared = 0;
+ qemu_savevm_state_header(trans);
+ qemu_savevm_state_begin(trans, &s->params);
+ qemu_mutex_lock_iothread();
+ qemu_savevm_state_complete(trans);
+ qemu_mutex_unlock_iothread();
- /* TODO: suspend and save vm state to colo buffer */
+ qemu_fflush(trans);
ret = colo_ctl_put(s->to_dst_file, COLO_CMD_VMSTATE_SEND, 0);
if (ret < 0) {
goto out;
}
+ /* we send the total size of the vmstate first */
+ size = qsb_get_length(buffer);
+ ret = colo_ctl_put(s->to_dst_file, COLO_CMD_VMSTATE_SIZE, size);
+ if (ret < 0) {
+ goto out;
+ }
- /* TODO: send vmstate to Secondary */
+ qsb_put_buffer(s->to_dst_file, buffer, size);
+ qemu_fflush(s->to_dst_file);
+ ret = qemu_file_get_error(s->to_dst_file);
+ if (ret < 0) {
+ goto out;
+ }
ret = colo_ctl_get(s->from_dst_file, COLO_CMD_VMSTATE_RECEIVED);
if (ret < 0) {
@@ -129,15 +167,25 @@ static int colo_do_checkpoint_transaction(MigrationState *s)
goto out;
}
- /* TODO: resume Primary */
+ ret = 0;
+ /* resume master */
+ qemu_mutex_lock_iothread();
+ vm_start();
+ qemu_mutex_unlock_iothread();
+ trace_colo_vm_state_change("stop", "run");
out:
+ if (trans) {
+ qemu_fclose(trans);
+ }
+
return ret;
}
static void *colo_thread(void *opaque)
{
MigrationState *s = opaque;
+ QEMUSizedBuffer *buffer = NULL;
int fd, ret = 0;
/* Dup the fd of to_dst_file */
@@ -162,6 +210,13 @@ static void *colo_thread(void *opaque)
goto out;
}
+ buffer = qsb_create(NULL, COLO_BUFFER_BASE_SIZE);
+ if (buffer == NULL) {
+ ret = -ENOMEM;
+ error_report("Failed to allocate buffer!");
+ goto out;
+ }
+
qemu_mutex_lock_iothread();
vm_start();
qemu_mutex_unlock_iothread();
@@ -169,7 +224,7 @@ static void *colo_thread(void *opaque)
while (s->state == MIGRATION_STATUS_COLO) {
/* start a colo checkpoint */
- ret = colo_do_checkpoint_transaction(s);
+ ret = colo_do_checkpoint_transaction(s, buffer);
if (ret < 0) {
goto out;
}
@@ -182,7 +237,10 @@ out:
migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
MIGRATION_STATUS_COMPLETED);
- if (s->from_dst_file) {
+ qsb_free(buffer);
+ buffer = NULL;
+
+ if (s->from_dst_file) {
qemu_fclose(s->from_dst_file);
}
diff --git a/migration/ram.c b/migration/ram.c
index 7f007e6..68980be 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -38,6 +38,7 @@
#include "trace.h"
#include "exec/ram_addr.h"
#include "qemu/rcu_queue.h"
+#include "migration/colo.h"
#ifdef DEBUG_MIGRATION_RAM
#define DPRINTF(fmt, ...) \
@@ -1090,15 +1091,8 @@ void migration_bitmap_extend(ram_addr_t old, ram_addr_t new)
}
}
-/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has
- * long-running RCU critical section. When rcu-reclaims in the code
- * start to become numerous it will be necessary to reduce the
- * granularity of these critical sections.
- */
-
-static int ram_save_setup(QEMUFile *f, void *opaque)
+static int ram_save_init_globals(void)
{
- RAMBlock *block;
int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */
mig_throttle_on = false;
@@ -1158,6 +1152,31 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
migration_bitmap_sync();
qemu_mutex_unlock_ramlist();
qemu_mutex_unlock_iothread();
+ rcu_read_unlock();
+
+ return 0;
+}
+
+/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has
+ * long-running RCU critical section. When rcu-reclaims in the code
+ * start to become numerous it will be necessary to reduce the
+ * granularity of these critical sections.
+ */
+
+static int ram_save_setup(QEMUFile *f, void *opaque)
+{
+ RAMBlock *block;
+
+ /*
+ * migration has already setup the bitmap, reuse it.
+ */
+ if (!migration_in_colo_state()) {
+ if (ram_save_init_globals() < 0) {
+ return -1;
+ }
+ }
+
+ rcu_read_lock();
qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
@@ -1257,7 +1276,8 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
while (true) {
int pages;
- pages = ram_find_and_save_block(f, true, &bytes_transferred);
+ pages = ram_find_and_save_block(f, !migration_in_colo_state(),
+ &bytes_transferred);
/* no more blocks to sent */
if (pages == 0) {
break;
@@ -1268,8 +1288,14 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
ram_control_after_iterate(f, RAM_CONTROL_FINISH);
rcu_read_unlock();
+ /*
+ * Since we need to reuse dirty bitmap in colo,
+ * don't cleanup the bitmap.
+ */
+ if (!migrate_enable_colo() || migration_has_failed(migrate_get_current())) {
+ migration_end();
+ }
- migration_end();
qemu_put_be64(f, RAM_SAVE_FLAG_EOS);
return 0;
diff --git a/migration/savevm.c b/migration/savevm.c
index 6071215..c1b40ce 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -48,7 +48,7 @@
#include "qemu/iov.h"
#include "block/snapshot.h"
#include "block/qapi.h"
-
+#include "migration/colo.h"
#ifndef ETH_P_RARP
#define ETH_P_RARP 0x8035
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 13/32] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (11 preceding siblings ...)
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 12/32] COLO: Save PVM state to secondary side when do checkpoint zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 14/32] COLO: Load VMState into qsb before restore it zhanghailiang
` (22 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
We should not load PVM's state directly into SVM, because, there maybe some error
happen when SVM is receving data, which will break SVM.
We need to ensure receving all data before load the state into SVM. We use
an extra memory to cache these data (PVM's ram). The ram cache in secondary side
is initially the same as SVM/PVM's memory. And in the process of checkpoint,
we cache the dirty pages of PVM into this ram cache firstly, so this ram cache
always the same as PVM's memory at every checkpoint, then we flush this cached ram
to SVM after we receive all PVM's state.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
include/exec/cpu-all.h | 1 +
include/migration/colo.h | 3 ++
migration/colo.c | 14 ++++++--
migration/ram.c | 93 ++++++++++++++++++++++++++++++++++++++++++++++--
4 files changed, 107 insertions(+), 4 deletions(-)
diff --git a/include/exec/cpu-all.h b/include/exec/cpu-all.h
index 89db792..d1c1af9 100644
--- a/include/exec/cpu-all.h
+++ b/include/exec/cpu-all.h
@@ -274,6 +274,7 @@ struct RAMBlock {
struct rcu_head rcu;
struct MemoryRegion *mr;
uint8_t *host;
+ uint8_t *host_cache; /* For colo, VM's ram cache */
ram_addr_t offset;
ram_addr_t used_length;
ram_addr_t max_length;
diff --git a/include/migration/colo.h b/include/migration/colo.h
index 58849f7..b8a536e 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -29,4 +29,7 @@ bool migration_incoming_enable_colo(void);
void migration_incoming_exit_colo(void);
void *colo_process_incoming_thread(void *opaque);
bool migration_incoming_in_colo_state(void);
+/* ram cache */
+int colo_init_ram_cache(void);
+void colo_release_ram_cache(void);
#endif
diff --git a/migration/colo.c b/migration/colo.c
index f107032..452a77b 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -330,6 +330,12 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
+ ret = colo_init_ram_cache();
+ if (ret < 0) {
+ error_report("Failed to initialize ram cache");
+ goto out;
+ }
+
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_CHECKPOINT_READY, 0);
if (ret < 0) {
goto out;
@@ -357,14 +363,14 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
- /* TODO: read migration data into colo buffer */
+ /*TODO Load VM state */
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_VMSTATE_RECEIVED, 0);
if (ret < 0) {
goto out;
}
- /* TODO: load vm state */
+ /* TODO: flush vm state */
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_VMSTATE_LOADED, 0);
if (ret < 0) {
@@ -378,6 +384,10 @@ out:
strerror(-ret));
}
+ qemu_mutex_lock_iothread();
+ colo_release_ram_cache();
+ qemu_mutex_unlock_iothread();
+
if (mis->to_src_file) {
qemu_fclose(mis->to_src_file);
}
diff --git a/migration/ram.c b/migration/ram.c
index 68980be..f3b94f2 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -225,6 +225,7 @@ static ram_addr_t last_offset;
static unsigned long *migration_bitmap;
static QemuMutex migration_bitmap_mutex;
static uint64_t migration_dirty_pages;
+static bool ram_cache_enable;
static uint32_t last_version;
static bool ram_bulk_stage;
@@ -1353,6 +1354,8 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
return 0;
}
+static void *memory_region_get_ram_cache_ptr(MemoryRegion *mr, RAMBlock *block);
+
/* Must be called from within a rcu critical section.
* Returns a pointer from within the RCU-protected ram_list.
*/
@@ -1370,7 +1373,20 @@ static inline void *host_from_stream_offset(QEMUFile *f,
return NULL;
}
- return memory_region_get_ram_ptr(block->mr) + offset;
+ if (ram_cache_enable) {
+ /*
+ * During colo checkpoint, we need bitmap of these migrated pages.
+ * It help us to decide which pages in ram cache should be flushed
+ * into VM's RAM later.
+ */
+ long k = (block->mr->ram_addr + offset) >> TARGET_PAGE_BITS;
+ if (!test_and_set_bit(k, migration_bitmap)) {
+ migration_dirty_pages++;
+ }
+ return memory_region_get_ram_cache_ptr(block->mr, block) + offset;
+ } else {
+ return memory_region_get_ram_ptr(block->mr) + offset;
+ }
}
len = qemu_get_byte(f);
@@ -1380,7 +1396,16 @@ static inline void *host_from_stream_offset(QEMUFile *f,
QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
if (!strncmp(id, block->idstr, sizeof(id)) &&
block->max_length > offset) {
- return memory_region_get_ram_ptr(block->mr) + offset;
+ if (ram_cache_enable) {
+ long k = (block->mr->ram_addr + offset) >> TARGET_PAGE_BITS;
+ if (!test_and_set_bit(k, migration_bitmap)) {
+ migration_dirty_pages++;
+ }
+ return memory_region_get_ram_cache_ptr(block->mr, block)
+ + offset;
+ } else {
+ return memory_region_get_ram_ptr(block->mr) + offset;
+ }
}
}
@@ -1631,6 +1656,70 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
return ret;
}
+/*
+ * colo cache: this is for secondary VM, we cache the whole
+ * memory of the secondary VM, it will be called after first migration.
+ */
+int colo_init_ram_cache(void)
+{
+ RAMBlock *block;
+
+ rcu_read_lock();
+ QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
+ block->host_cache = qemu_anon_ram_alloc(block->used_length, NULL);
+ if (!block->host_cache) {
+ goto out_locked;
+ }
+ memcpy(block->host_cache, block->host, block->used_length);
+ }
+ rcu_read_unlock();
+ ram_cache_enable = true;
+ return 0;
+
+out_locked:
+ QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
+ if (block->host_cache) {
+ qemu_anon_ram_free(block->host_cache, block->used_length);
+ block->host_cache = NULL;
+ }
+ }
+
+ rcu_read_unlock();
+ return -errno;
+}
+
+void colo_release_ram_cache(void)
+{
+ RAMBlock *block;
+
+ ram_cache_enable = false;
+
+ rcu_read_lock();
+ QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
+ if (block->host_cache) {
+ qemu_anon_ram_free(block->host_cache, block->used_length);
+ block->host_cache = NULL;
+ }
+ }
+ rcu_read_unlock();
+}
+
+static void *memory_region_get_ram_cache_ptr(MemoryRegion *mr, RAMBlock *block)
+{
+ if (mr->alias) {
+ return memory_region_get_ram_cache_ptr(mr->alias, block) +
+ mr->alias_offset;
+ }
+
+ assert(mr->terminates);
+
+ ram_addr_t addr = mr->ram_addr & TARGET_PAGE_MASK;
+
+ assert(addr - block->offset < block->used_length);
+
+ return block->host_cache + (addr - block->offset);
+}
+
static SaveVMHandlers savevm_ram_handlers = {
.save_live_setup = ram_save_setup,
.save_live_iterate = ram_save_iterate,
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 14/32] COLO: Load VMState into qsb before restore it
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (12 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 13/32] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 15/32] COLO: Flush PVM's cached RAM into SVM's memory zhanghailiang
` (21 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
We should not destroy the state of secondary until we receive the whole
state from the primary, in case the primary fails in the middle of sending
the state, so, here we cache the device state in Secondary before restore it.
Besides, we should call qemu_system_reset() before load VM state,
which can ensure the data is intact.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
migration/colo.c | 49 +++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 47 insertions(+), 2 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index 452a77b..cb6a59d 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -313,6 +313,9 @@ static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
void *colo_process_incoming_thread(void *opaque)
{
MigrationIncomingState *mis = opaque;
+ QEMUFile *fb = NULL;
+ QEMUSizedBuffer *buffer = NULL; /* Cache incoming device state */
+ int total_size;
int fd, ret = 0;
migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
@@ -336,6 +339,12 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
+ buffer = qsb_create(NULL, COLO_BUFFER_BASE_SIZE);
+ if (buffer == NULL) {
+ error_report("Failed to allocate colo buffer!");
+ goto out;
+ }
+
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_CHECKPOINT_READY, 0);
if (ret < 0) {
goto out;
@@ -363,20 +372,50 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
- /*TODO Load VM state */
+ /* read the VM state total size first */
+ total_size = colo_ctl_get(mis->from_src_file, COLO_CMD_VMSTATE_SIZE);
+ if (total_size <= 0) {
+ goto out;
+ }
+
+ /* read vm device state into colo buffer */
+ ret = qsb_fill_buffer(buffer, mis->from_src_file, total_size);
+ if (ret != total_size) {
+ error_report("can't get all migration data");
+ goto out;
+ }
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_VMSTATE_RECEIVED, 0);
if (ret < 0) {
goto out;
}
+ /* open colo buffer for read */
+ fb = qemu_bufopen("r", buffer);
+ if (!fb) {
+ error_report("can't open colo buffer for read");
+ goto out;
+ }
+
+ qemu_mutex_lock_iothread();
+ qemu_system_reset(VMRESET_SILENT);
+ if (qemu_loadvm_state(fb) < 0) {
+ error_report("COLO: loadvm failed");
+ qemu_mutex_unlock_iothread();
+ goto out;
+ }
+ qemu_mutex_unlock_iothread();
+
/* TODO: flush vm state */
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_VMSTATE_LOADED, 0);
if (ret < 0) {
goto out;
}
-}
+
+ qemu_fclose(fb);
+ fb = NULL;
+ }
out:
if (ret < 0) {
@@ -384,6 +423,11 @@ out:
strerror(-ret));
}
+ if (fb) {
+ qemu_fclose(fb);
+ }
+ qsb_free(buffer);
+
qemu_mutex_lock_iothread();
colo_release_ram_cache();
qemu_mutex_unlock_iothread();
@@ -391,6 +435,7 @@ out:
if (mis->to_src_file) {
qemu_fclose(mis->to_src_file);
}
+
migration_incoming_exit_colo();
return NULL;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 15/32] COLO: Flush PVM's cached RAM into SVM's memory
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (13 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 14/32] COLO: Load VMState into qsb before restore it zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 16/32] COLO: synchronize PVM's state to SVM periodically zhanghailiang
` (20 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
During the time of VM's running, PVM may dirty some pages, we will transfer
PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint
time. So, the content of SVM's RAM cache will always be some with PVM's memory
after checkpoint.
Instead of flushing all content of PVM's RAM cache into SVM's MEMORY,
we do this in a more efficient way:
Only flush any page that dirtied by PVM since last checkpoint.
In this way, we can ensure SVM's memory same with PVM's.
Besides, we must ensure flush RAM cache before load device state.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
include/migration/colo.h | 1 +
migration/colo.c | 2 --
migration/ram.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++
3 files changed, 55 insertions(+), 2 deletions(-)
diff --git a/include/migration/colo.h b/include/migration/colo.h
index b8a536e..5c43391 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -32,4 +32,5 @@ bool migration_incoming_in_colo_state(void);
/* ram cache */
int colo_init_ram_cache(void);
void colo_release_ram_cache(void);
+void colo_flush_ram_cache(void);
#endif
diff --git a/migration/colo.c b/migration/colo.c
index cb6a59d..b6a6d66 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -406,8 +406,6 @@ void *colo_process_incoming_thread(void *opaque)
}
qemu_mutex_unlock_iothread();
- /* TODO: flush vm state */
-
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_VMSTATE_LOADED, 0);
if (ret < 0) {
goto out;
diff --git a/migration/ram.c b/migration/ram.c
index f3b94f2..782febd 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -1522,6 +1522,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
int flags = 0, ret = 0;
static uint64_t seq_iter;
int len = 0;
+ bool need_flush = false;
seq_iter++;
@@ -1590,6 +1591,8 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
ret = -EINVAL;
break;
}
+
+ need_flush = true;
ch = qemu_get_byte(f);
ram_handle_compressed(host, ch, TARGET_PAGE_SIZE);
break;
@@ -1600,6 +1603,8 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
ret = -EINVAL;
break;
}
+
+ need_flush = true;
qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
break;
case RAM_SAVE_FLAG_COMPRESS_PAGE:
@@ -1632,6 +1637,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
ret = -EINVAL;
break;
}
+ need_flush = true;
break;
case RAM_SAVE_FLAG_EOS:
/* normal exit */
@@ -1651,6 +1657,11 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
}
rcu_read_unlock();
+
+ if (!ret && ram_cache_enable && need_flush) {
+ DPRINTF("Flush ram_cache\n");
+ colo_flush_ram_cache();
+ }
DPRINTF("Completed load of VM with exit code %d seq iteration "
"%" PRIu64 "\n", ret, seq_iter);
return ret;
@@ -1663,6 +1674,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
int colo_init_ram_cache(void)
{
RAMBlock *block;
+ int64_t ram_cache_pages = last_ram_offset() >> TARGET_PAGE_BITS;
rcu_read_lock();
QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
@@ -1674,6 +1686,14 @@ int colo_init_ram_cache(void)
}
rcu_read_unlock();
ram_cache_enable = true;
+ /*
+ * Record the dirty pages that sent by PVM, we use this dirty bitmap together
+ * with to decide which page in cache should be flushed into SVM's RAM. Here
+ * we use the same name 'migration_bitmap' as for migration.
+ */
+ migration_bitmap = bitmap_new(ram_cache_pages);
+ migration_dirty_pages = 0;
+
return 0;
out_locked:
@@ -1694,6 +1714,11 @@ void colo_release_ram_cache(void)
ram_cache_enable = false;
+ if (migration_bitmap) {
+ g_free(migration_bitmap);
+ migration_bitmap = NULL;
+ }
+
rcu_read_lock();
QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
if (block->host_cache) {
@@ -1720,6 +1745,35 @@ static void *memory_region_get_ram_cache_ptr(MemoryRegion *mr, RAMBlock *block)
return block->host_cache + (addr - block->offset);
}
+/*
+ * Flush content of RAM cache into SVM's memory.
+ * Only flush the pages that be dirtied by PVM or SVM or both.
+ */
+void colo_flush_ram_cache(void)
+{
+ RAMBlock *block = NULL;
+ void *dst_host;
+ void *src_host;
+ ram_addr_t offset = 0;
+
+ rcu_read_lock();
+ block = QLIST_FIRST_RCU(&ram_list.blocks);
+ while (block) {
+ offset = migration_bitmap_find_and_reset_dirty(block->mr, offset);
+ if (offset >= block->used_length) {
+ offset = 0;
+ block = QLIST_NEXT_RCU(block, next);
+ } else {
+ dst_host = memory_region_get_ram_ptr(block->mr) + offset;
+ src_host = memory_region_get_ram_cache_ptr(block->mr, block)
+ + offset;
+ memcpy(dst_host, src_host, TARGET_PAGE_SIZE);
+ }
+ }
+ rcu_read_unlock();
+ assert(migration_dirty_pages == 0);
+}
+
static SaveVMHandlers savevm_ram_handlers = {
.save_live_setup = ram_save_setup,
.save_live_iterate = ram_save_iterate,
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 16/32] COLO: synchronize PVM's state to SVM periodically
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (14 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 15/32] COLO: Flush PVM's cached RAM into SVM's memory zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 17/32] COLO failover: Introduce a new command to trigger a failover zhanghailiang
` (19 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
Do checkpoint periodically, the default interval is 200ms.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration/colo.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/migration/colo.c b/migration/colo.c
index b6a6d66..07e0c1b 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -11,12 +11,19 @@
*/
#include <unistd.h>
+#include "qemu/timer.h"
#include "sysemu/sysemu.h"
#include "migration/colo.h"
#include "trace.h"
#include "qemu/error-report.h"
#include "qemu/sockets.h"
+/*
+ * checkpoint interval: unit ms
+ * Note: Please change this default value to 10000 when we support hybrid mode.
+ */
+#define CHECKPOINT_MAX_PEROID 200
+
static QEMUBH *colo_bh;
/* colo buffer */
#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
@@ -186,6 +193,7 @@ static void *colo_thread(void *opaque)
{
MigrationState *s = opaque;
QEMUSizedBuffer *buffer = NULL;
+ int64_t current_time, checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
int fd, ret = 0;
/* Dup the fd of to_dst_file */
@@ -223,11 +231,17 @@ static void *colo_thread(void *opaque)
trace_colo_vm_state_change("stop", "run");
while (s->state == MIGRATION_STATUS_COLO) {
+ current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ if (current_time - checkpoint_time < CHECKPOINT_MAX_PEROID) {
+ g_usleep(100000);
+ continue;
+ }
/* start a colo checkpoint */
ret = colo_do_checkpoint_transaction(s, buffer);
if (ret < 0) {
goto out;
}
+ checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
}
out:
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 17/32] COLO failover: Introduce a new command to trigger a failover
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (15 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 16/32] COLO: synchronize PVM's state to SVM periodically zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 18/32] COLO failover: Introduce state to record failover process zhanghailiang
` (18 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, Luiz Capitulino,
arei.gonglei, stefanha, amit.shah, yanghy, zhanghailiang
We leave users to choose whatever heartbeat solution they want, if the heartbeat
is lost, or other errors they detect, they can use command
'colo_lost_heartbeat' to tell COLO to do failover, COLO will do operations
accordingly.
For example, if the command is sent to the PVM, the Primary side will
exit COLO mode and take over operation. If sent to the Secondary, the
secondary will run failover work, then take over server operation to
become the new Primary.
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
hmp-commands.hx | 15 +++++++++++++++
hmp.c | 8 ++++++++
hmp.h | 1 +
include/migration/colo.h | 4 ++++
include/migration/failover.h | 20 ++++++++++++++++++++
migration/Makefile.objs | 2 +-
migration/colo-comm.c | 11 +++++++++++
migration/colo-failover.c | 41 +++++++++++++++++++++++++++++++++++++++++
migration/colo.c | 1 +
qapi-schema.json | 26 ++++++++++++++++++++++++++
qmp-commands.hx | 19 +++++++++++++++++++
stubs/migration-colo.c | 8 ++++++++
12 files changed, 155 insertions(+), 1 deletion(-)
create mode 100644 include/migration/failover.h
create mode 100644 migration/colo-failover.c
diff --git a/hmp-commands.hx b/hmp-commands.hx
index e56baec..9065a23 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -1034,6 +1034,21 @@ Set the parameter @var{parameter} for migration.
ETEXI
{
+ .name = "colo_lost_heartbeat",
+ .args_type = "",
+ .params = "",
+ .help = "Tell COLO that heartbeat is lost,\n\t\t\t"
+ "a failover or takeover is needed.",
+ .mhandler.cmd = hmp_colo_lost_heartbeat,
+ },
+
+STEXI
+@item colo_lost_heartbeat
+@findex colo_lost_heartbeat
+Tell COLO that heartbeat is lost, a failover or takeover is needed.
+ETEXI
+
+ {
.name = "client_migrate_info",
.args_type = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
.params = "protocol hostname port tls-port cert-subject",
diff --git a/hmp.c b/hmp.c
index 09e3cda..94b0149 100644
--- a/hmp.c
+++ b/hmp.c
@@ -1273,6 +1273,14 @@ void hmp_client_migrate_info(Monitor *mon, const QDict *qdict)
hmp_handle_error(mon, &err);
}
+void hmp_colo_lost_heartbeat(Monitor *mon, const QDict *qdict)
+{
+ Error *err = NULL;
+
+ qmp_colo_lost_heartbeat(&err);
+ hmp_handle_error(mon, &err);
+}
+
void hmp_set_password(Monitor *mon, const QDict *qdict)
{
const char *protocol = qdict_get_str(qdict, "protocol");
diff --git a/hmp.h b/hmp.h
index a21dbbb..a90ab65 100644
--- a/hmp.h
+++ b/hmp.h
@@ -68,6 +68,7 @@ void hmp_migrate_set_capability(Monitor *mon, const QDict *qdict);
void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict);
void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
void hmp_client_migrate_info(Monitor *mon, const QDict *qdict);
+void hmp_colo_lost_heartbeat(Monitor *mon, const QDict *qdict);
void hmp_set_password(Monitor *mon, const QDict *qdict);
void hmp_expire_password(Monitor *mon, const QDict *qdict);
void hmp_eject(Monitor *mon, const QDict *qdict);
diff --git a/include/migration/colo.h b/include/migration/colo.h
index 5c43391..21f5c5d 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -17,6 +17,7 @@
#include "migration/migration.h"
#include "block/coroutine.h"
#include "qemu/thread.h"
+#include "qemu/main-loop.h"
bool colo_supported(void);
void colo_info_mig_init(void);
@@ -29,6 +30,9 @@ bool migration_incoming_enable_colo(void);
void migration_incoming_exit_colo(void);
void *colo_process_incoming_thread(void *opaque);
bool migration_incoming_in_colo_state(void);
+
+int get_colo_mode(void);
+
/* ram cache */
int colo_init_ram_cache(void);
void colo_release_ram_cache(void);
diff --git a/include/migration/failover.h b/include/migration/failover.h
new file mode 100644
index 0000000..1785b52
--- /dev/null
+++ b/include/migration/failover.h
@@ -0,0 +1,20 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_FAILOVER_H
+#define QEMU_FAILOVER_H
+
+#include "qemu-common.h"
+
+void failover_request_active(Error **errp);
+
+#endif
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index cb7bd30..50d8392 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -1,6 +1,6 @@
common-obj-y += migration.o tcp.o
-common-obj-$(CONFIG_COLO) += colo.o
common-obj-y += colo-comm.o
+common-obj-$(CONFIG_COLO) += colo.o colo-failover.o
common-obj-y += vmstate.o
common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
common-obj-y += xbzrle.o
diff --git a/migration/colo-comm.c b/migration/colo-comm.c
index 0808d6c..1b6f690 100644
--- a/migration/colo-comm.c
+++ b/migration/colo-comm.c
@@ -20,6 +20,17 @@ typedef struct {
static COLOInfo colo_info;
+int get_colo_mode(void)
+{
+ if (migration_in_colo_state()) {
+ return COLO_MODE_PRIMARY;
+ } else if (migration_incoming_in_colo_state()) {
+ return COLO_MODE_SECONDARY;
+ } else {
+ return COLO_MODE_UNKNOWN;
+ }
+}
+
static void colo_info_pre_save(void *opaque)
{
COLOInfo *s = opaque;
diff --git a/migration/colo-failover.c b/migration/colo-failover.c
new file mode 100644
index 0000000..af068d3
--- /dev/null
+++ b/migration/colo-failover.c
@@ -0,0 +1,41 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#include "migration/colo.h"
+#include "migration/failover.h"
+#include "qmp-commands.h"
+#include "qapi/qmp/qerror.h"
+
+static QEMUBH *failover_bh;
+
+static void colo_failover_bh(void *opaque)
+{
+ qemu_bh_delete(failover_bh);
+ failover_bh = NULL;
+ /*TODO: Do failover work */
+}
+
+void failover_request_active(Error **errp)
+{
+ failover_bh = qemu_bh_new(colo_failover_bh, NULL);
+ qemu_bh_schedule(failover_bh);
+}
+
+void qmp_colo_lost_heartbeat(Error **errp)
+{
+ if (get_colo_mode() == COLO_MODE_UNKNOWN) {
+ error_setg(errp, QERR_FEATURE_DISABLED, "colo");
+ return;
+ }
+
+ failover_request_active(errp);
+}
diff --git a/migration/colo.c b/migration/colo.c
index 07e0c1b..574028a 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -17,6 +17,7 @@
#include "trace.h"
#include "qemu/error-report.h"
#include "qemu/sockets.h"
+#include "migration/failover.h"
/*
* checkpoint interval: unit ms
diff --git a/qapi-schema.json b/qapi-schema.json
index 6a3796e..8468552 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -695,6 +695,32 @@
'vmstate-received', 'vmstate-loaded', 'guest-shutdown',
'ram-steal'] }
+##
+# @COLOMode
+#
+# The colo mode
+#
+# @unknown: unknown mode
+#
+# @primary: master side
+#
+# @secondary: slave side
+#
+# Since: 2.5
+##
+{ 'enum': 'COLOMode',
+ 'data': [ 'unknown', 'primary', 'secondary'] }
+
+##
+# @colo-lost-heartbeat
+#
+# Tell qemu that heartbeat is lost, request it to do takeover procedures.
+#
+# Since: 2.5
+##
+{ 'command': 'colo-lost-heartbeat' }
+
+##
# @MouseInfo:
#
# Information about a mouse device.
diff --git a/qmp-commands.hx b/qmp-commands.hx
index b06c09c..fc8ccbb 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -781,6 +781,25 @@ Example:
EQMP
{
+ .name = "colo-lost-heartbeat",
+ .args_type = "",
+ .mhandler.cmd_new = qmp_marshal_input_colo_lost_heartbeat,
+ },
+
+SQMP
+colo-lost-heartbeat
+--------------------
+
+Tell COLO that heartbeat is lost, a failover or takeover is needed.
+
+Example:
+
+-> { "execute": "colo-lost-heartbeat" }
+<- { "return": {} }
+
+EQMP
+
+ {
.name = "client_migrate_info",
.args_type = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
.params = "protocol hostname port tls-port cert-subject",
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index c49ee1a..41dacaf 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -11,6 +11,7 @@
*/
#include "migration/colo.h"
+#include "qmp-commands.h"
bool colo_supported(void)
{
@@ -35,3 +36,10 @@ void *colo_process_incoming_thread(void *opaque)
{
return NULL;
}
+
+void qmp_colo_lost_heartbeat(Error **errp)
+{
+ error_setg(errp, "COLO is not supported, please rerun configure"
+ " with --enable-colo option in order to support"
+ " COLO feature");
+}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 18/32] COLO failover: Introduce state to record failover process
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (16 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 17/32] COLO failover: Introduce a new command to trigger a failover zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 19/32] COLO: Implement failover work for Primary VM zhanghailiang
` (17 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
When handling failover, we do different things according to the different stage
of failover process, here we introduce a global atomic variable to record the
status of failover.
We add four failover status to indicate the different stage of failover process.
You should use the helpers to get and set the value.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
---
include/migration/failover.h | 10 ++++++++++
migration/colo-failover.c | 37 +++++++++++++++++++++++++++++++++++++
migration/colo.c | 4 ++++
trace-events | 1 +
4 files changed, 52 insertions(+)
diff --git a/include/migration/failover.h b/include/migration/failover.h
index 1785b52..882c625 100644
--- a/include/migration/failover.h
+++ b/include/migration/failover.h
@@ -15,6 +15,16 @@
#include "qemu-common.h"
+typedef enum COLOFailoverStatus {
+ FAILOVER_STATUS_NONE = 0,
+ FAILOVER_STATUS_REQUEST = 1, /* Request but not handled */
+ FAILOVER_STATUS_HANDLING = 2, /* In the process of handling failover */
+ FAILOVER_STATUS_COMPLETED = 3, /* Finish the failover process */
+} COLOFailoverStatus;
+
+void failover_init_state(void);
+int failover_set_state(int old_state, int new_state);
+int failover_get_state(void);
void failover_request_active(Error **errp);
#endif
diff --git a/migration/colo-failover.c b/migration/colo-failover.c
index af068d3..e055551 100644
--- a/migration/colo-failover.c
+++ b/migration/colo-failover.c
@@ -14,22 +14,59 @@
#include "migration/failover.h"
#include "qmp-commands.h"
#include "qapi/qmp/qerror.h"
+#include "qemu/error-report.h"
+#include "trace.h"
static QEMUBH *failover_bh;
+static COLOFailoverStatus failover_state;
static void colo_failover_bh(void *opaque)
{
+ int old_state;
+
qemu_bh_delete(failover_bh);
failover_bh = NULL;
+ old_state = failover_set_state(FAILOVER_STATUS_REQUEST,
+ FAILOVER_STATUS_HANDLING);
+ if (old_state != FAILOVER_STATUS_REQUEST) {
+ error_report(" Unkown error for failover, old_state=%d", old_state);
+ return;
+ }
/*TODO: Do failover work */
}
void failover_request_active(Error **errp)
{
+ if (failover_set_state(FAILOVER_STATUS_NONE, FAILOVER_STATUS_REQUEST)
+ != FAILOVER_STATUS_NONE) {
+ error_setg(errp, "COLO failover is already actived");
+ return;
+ }
failover_bh = qemu_bh_new(colo_failover_bh, NULL);
qemu_bh_schedule(failover_bh);
}
+void failover_init_state(void)
+{
+ failover_state = FAILOVER_STATUS_NONE;
+}
+
+int failover_set_state(int old_state, int new_state)
+{
+ int old;
+
+ old = atomic_cmpxchg(&failover_state, old_state, new_state);;
+ if (old == old_state) {
+ trace_colo_failover_set_state(new_state);
+ }
+ return old;
+}
+
+int failover_get_state(void)
+{
+ return atomic_read(&failover_state);
+}
+
void qmp_colo_lost_heartbeat(Error **errp)
{
if (get_colo_mode() == COLO_MODE_UNKNOWN) {
diff --git a/migration/colo.c b/migration/colo.c
index 574028a..6c48dca 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -197,6 +197,8 @@ static void *colo_thread(void *opaque)
int64_t current_time, checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
int fd, ret = 0;
+ failover_init_state();
+
/* Dup the fd of to_dst_file */
fd = dup(qemu_get_fd(s->to_dst_file));
if (fd == -1) {
@@ -336,6 +338,8 @@ void *colo_process_incoming_thread(void *opaque)
migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_COLO);
+ failover_init_state();
+
fd = dup(qemu_get_fd(mis->from_src_file));
if (fd < 0) {
ret = -errno;
diff --git a/trace-events b/trace-events
index 7fdf723..cf378ec 100644
--- a/trace-events
+++ b/trace-events
@@ -1476,6 +1476,7 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) ""
colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
colo_ctl_put(const char *msg) "Send '%s'"
colo_ctl_get(const char *msg) "Receive '%s'"
+colo_failover_set_state(int new_state) "new state %d"
# kvm-all.c
kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 19/32] COLO: Implement failover work for Primary VM
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (17 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 18/32] COLO failover: Introduce state to record failover process zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 20/32] COLO: Implement failover work for Secondary VM zhanghailiang
` (16 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
For PVM, if there is failover request from users.
The colo thread will exit the loop while the failover BH does the
cleanup work and resumes VM.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
include/migration/colo.h | 4 +++
include/migration/failover.h | 1 +
migration/colo-failover.c | 7 ++++-
migration/colo.c | 64 +++++++++++++++++++++++++++++++++++++++-----
4 files changed, 69 insertions(+), 7 deletions(-)
diff --git a/include/migration/colo.h b/include/migration/colo.h
index 21f5c5d..5f6072e 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -37,4 +37,8 @@ int get_colo_mode(void);
int colo_init_ram_cache(void);
void colo_release_ram_cache(void);
void colo_flush_ram_cache(void);
+
+/* failover */
+void colo_do_failover(MigrationState *s);
+
#endif
diff --git a/include/migration/failover.h b/include/migration/failover.h
index 882c625..fba3931 100644
--- a/include/migration/failover.h
+++ b/include/migration/failover.h
@@ -26,5 +26,6 @@ void failover_init_state(void);
int failover_set_state(int old_state, int new_state);
int failover_get_state(void);
void failover_request_active(Error **errp);
+bool failover_request_is_active(void);
#endif
diff --git a/migration/colo-failover.c b/migration/colo-failover.c
index e055551..7299674 100644
--- a/migration/colo-failover.c
+++ b/migration/colo-failover.c
@@ -32,7 +32,7 @@ static void colo_failover_bh(void *opaque)
error_report(" Unkown error for failover, old_state=%d", old_state);
return;
}
- /*TODO: Do failover work */
+ colo_do_failover(NULL);
}
void failover_request_active(Error **errp)
@@ -67,6 +67,11 @@ int failover_get_state(void)
return atomic_read(&failover_state);
}
+bool failover_request_is_active(void)
+{
+ return ((failover_get_state() != FAILOVER_STATUS_NONE));
+}
+
void qmp_colo_lost_heartbeat(Error **errp)
{
if (get_colo_mode() == COLO_MODE_UNKNOWN) {
diff --git a/migration/colo.c b/migration/colo.c
index 6c48dca..6aa6d18 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -48,6 +48,45 @@ bool migration_incoming_in_colo_state(void)
return (mis && (mis->state == MIGRATION_STATUS_COLO));
}
+static bool colo_runstate_is_stopped(void)
+{
+ return runstate_check(RUN_STATE_COLO) || !runstate_is_running();
+}
+
+static void primary_vm_do_failover(void)
+{
+ MigrationState *s = migrate_get_current();
+ int old_state;
+
+ if (s->state != MIGRATION_STATUS_FAILED) {
+ migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
+ MIGRATION_STATUS_COMPLETED);
+ }
+ qemu_bh_schedule(s->cleanup_bh);
+
+ vm_start();
+
+ old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
+ FAILOVER_STATUS_COMPLETED);
+ if (old_state != FAILOVER_STATUS_HANDLING) {
+ error_report("Serious error while do failover for Primary VM,"
+ "old_state: %d", old_state);
+ return;
+ }
+}
+
+void colo_do_failover(MigrationState *s)
+{
+ /* Make sure vm stopped while failover */
+ if (!colo_runstate_is_stopped()) {
+ vm_stop_force_state(RUN_STATE_COLO);
+ }
+
+ if (get_colo_mode() == COLO_MODE_PRIMARY) {
+ primary_vm_do_failover();
+ }
+}
+
/* colo checkpoint control helper */
static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value)
{
@@ -132,9 +171,23 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
/* suspend and save vm state to colo buffer */
qemu_mutex_lock_iothread();
+ if (failover_request_is_active()) {
+ qemu_mutex_unlock_iothread();
+ ret = -1;
+ goto out;
+ }
+ /* suspend and save vm state to colo buffer */
vm_stop_force_state(RUN_STATE_COLO);
qemu_mutex_unlock_iothread();
trace_colo_vm_state_change("run", "stop");
+ /*
+ * failover request bh could be called after
+ * vm_stop_force_state so we check failover_request_is_active() again.
+ */
+ if (failover_request_is_active()) {
+ ret = -1;
+ goto out;
+ }
/* Disable block migration */
s->params.blk = 0;
@@ -234,6 +287,11 @@ static void *colo_thread(void *opaque)
trace_colo_vm_state_change("stop", "run");
while (s->state == MIGRATION_STATUS_COLO) {
+ if (failover_request_is_active()) {
+ error_report("failover request");
+ goto out;
+ }
+
current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
if (current_time - checkpoint_time < CHECKPOINT_MAX_PEROID) {
g_usleep(100000);
@@ -251,8 +309,6 @@ out:
if (ret < 0) {
error_report("Detect some error: %s", strerror(-ret));
}
- migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
- MIGRATION_STATUS_COMPLETED);
qsb_free(buffer);
buffer = NULL;
@@ -261,10 +317,6 @@ out:
qemu_fclose(s->from_dst_file);
}
- qemu_mutex_lock_iothread();
- qemu_bh_schedule(s->cleanup_bh);
- qemu_mutex_unlock_iothread();
-
return NULL;
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 20/32] COLO: Implement failover work for Secondary VM
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (18 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 19/32] COLO: Implement failover work for Primary VM zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 21/32] COLO: implement default failover treatment zhanghailiang
` (15 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
If users require SVM to takeover work, colo incoming thread should
exit from loop while failover BH helps backing to migration incoming
coroutine.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 41 ++++++++++++++++++++++++++++++++++++++---
1 file changed, 38 insertions(+), 3 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index 6aa6d18..141b1c8 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -53,6 +53,33 @@ static bool colo_runstate_is_stopped(void)
return runstate_check(RUN_STATE_COLO) || !runstate_is_running();
}
+static void secondary_vm_do_failover(void)
+{
+ int old_state;
+ MigrationIncomingState *mis = migration_incoming_get_current();
+
+ migrate_set_state(&mis->state, MIGRATION_STATUS_COLO,
+ MIGRATION_STATUS_COMPLETED);
+
+ if (!autostart) {
+ error_report("\"-S\" qemu option will be ignored in secondary side");
+ /* recover runstate to normal migration finish state */
+ autostart = true;
+ }
+
+ old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
+ FAILOVER_STATUS_COMPLETED);
+ if (old_state != FAILOVER_STATUS_HANDLING) {
+ error_report("Serious error while do failover for secondary VM,"
+ "old_state: %d", old_state);
+ return;
+ }
+ /* For Secondary VM, jump to incoming co */
+ if (mis->migration_incoming_co) {
+ qemu_coroutine_enter(mis->migration_incoming_co, NULL);
+ }
+}
+
static void primary_vm_do_failover(void)
{
MigrationState *s = migrate_get_current();
@@ -84,6 +111,8 @@ void colo_do_failover(MigrationState *s)
if (get_colo_mode() == COLO_MODE_PRIMARY) {
primary_vm_do_failover();
+ } else {
+ secondary_vm_do_failover();
}
}
@@ -433,6 +462,11 @@ void *colo_process_incoming_thread(void *opaque)
}
}
+ if (failover_request_is_active()) {
+ error_report("failover request");
+ goto out;
+ }
+
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_CHECKPOINT_REPLY, 0);
if (ret < 0) {
goto out;
@@ -496,10 +530,11 @@ out:
qemu_fclose(fb);
}
qsb_free(buffer);
-
- qemu_mutex_lock_iothread();
+ /* Here, we can ensure BH is hold the global lock, and will join colo
+ * incoming thread, so here it is not necessary to lock here again,
+ * or there will be a deadlock error.
+ */
colo_release_ram_cache();
- qemu_mutex_unlock_iothread();
if (mis->to_src_file) {
qemu_fclose(mis->to_src_file);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 21/32] COLO: implement default failover treatment
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (19 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 20/32] COLO: Implement failover work for Secondary VM zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 22/32] qmp event: Add event notification for COLO error zhanghailiang
` (14 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
If we detect some error in colo, we will wait for some time,
hoping users also detect it. If users don't issue failover command.
We will go into default failover procedure, which the PVM will takeover
work while SVM is exit in default.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)
diff --git a/migration/colo.c b/migration/colo.c
index 141b1c8..9ad285c 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -25,6 +25,14 @@
*/
#define CHECKPOINT_MAX_PEROID 200
+/*
+ * The delay time before qemu begin the procedure of default failover treatment.
+ * Unit: ms
+ * Fix me: This value should be able to change by command
+ * 'migrate-set-parameters'
+ */
+#define DEFAULT_FAILOVER_DELAY 2000
+
static QEMUBH *colo_bh;
/* colo buffer */
#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
@@ -277,6 +285,7 @@ static void *colo_thread(void *opaque)
MigrationState *s = opaque;
QEMUSizedBuffer *buffer = NULL;
int64_t current_time, checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ int64_t error_time;
int fd, ret = 0;
failover_init_state();
@@ -335,8 +344,25 @@ static void *colo_thread(void *opaque)
}
out:
+ current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
if (ret < 0) {
error_report("Detect some error: %s", strerror(-ret));
+ /* Give users time to get involved in this verdict */
+ while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
+ if (failover_request_is_active()) {
+ error_report("Primary VM will take over work");
+ break;
+ }
+ usleep(100 * 1000);
+ current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ }
+
+ qemu_mutex_lock_iothread();
+ if (!failover_request_is_active()) {
+ error_report("Primary VM will take over work in default");
+ failover_request_active(NULL);
+ }
+ qemu_mutex_unlock_iothread();
}
qsb_free(buffer);
@@ -414,6 +440,7 @@ void *colo_process_incoming_thread(void *opaque)
QEMUFile *fb = NULL;
QEMUSizedBuffer *buffer = NULL; /* Cache incoming device state */
int total_size;
+ int64_t error_time, current_time;
int fd, ret = 0;
migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
@@ -521,9 +548,28 @@ void *colo_process_incoming_thread(void *opaque)
}
out:
+ current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
if (ret < 0) {
error_report("colo incoming thread will exit, detect error: %s",
strerror(-ret));
+ /* Give users time to get involved in this verdict */
+ while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
+ if (failover_request_is_active()) {
+ error_report("Secondary VM will take over work");
+ break;
+ }
+ usleep(100 * 1000);
+ current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ }
+ /* check flag again*/
+ if (!failover_request_is_active()) {
+ /*
+ * We assume that Primary VM is still alive according to
+ * heartbeat, just kill Secondary VM
+ */
+ error_report("SVM is going to exit in default!");
+ exit(1);
+ }
}
if (fb) {
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 22/32] qmp event: Add event notification for COLO error
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (20 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 21/32] COLO: implement default failover treatment zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 23/32] COLO failover: Shutdown related socket fd when do failover zhanghailiang
` (13 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, arei.gonglei, stefanha,
amit.shah, yanghy, Michael Roth, zhanghailiang
If some errors happen during VM's COLO FT stage, it's important to notify the users
of this event. Together with 'colo_lost_heartbeat', users can intervene in COLO's
failover work immediately.
If users don't want to get involved in COLO's failover verdict,
it is still necessary to notify users that we exit COLO mode.
Cc: Markus Armbruster <armbru@redhat.com>
Cc: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
docs/qmp/qmp-events.txt | 17 +++++++++++++++++
migration/colo.c | 17 +++++++++++++----
qapi-schema.json | 16 ++++++++++++++++
qapi/event.json | 17 +++++++++++++++++
4 files changed, 63 insertions(+), 4 deletions(-)
diff --git a/docs/qmp/qmp-events.txt b/docs/qmp/qmp-events.txt
index d92cc48..0bd7c41 100644
--- a/docs/qmp/qmp-events.txt
+++ b/docs/qmp/qmp-events.txt
@@ -182,6 +182,23 @@ Example:
Note: The "ready to complete" status is always reset by a BLOCK_JOB_ERROR
event.
+COLO_EXIT
+---------
+
+Emitted when VM finishes COLO mode due to some errors happening or
+the request of users.
+
+Data:
+
+ - "mode": COLO mode, primary or secondary side (json-string)
+ - "reason": the exit reason, internal error or external request. (json-string)
+ - "error": error message (json-string, operation)
+
+Example:
+
+{"timestamp": {"seconds": 2032141960, "microseconds": 417172},
+ "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "request" } }
+
DEVICE_DELETED
--------------
diff --git a/migration/colo.c b/migration/colo.c
index 9ad285c..29b5334 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -18,6 +18,7 @@
#include "qemu/error-report.h"
#include "qemu/sockets.h"
#include "migration/failover.h"
+#include "qapi-event.h"
/*
* checkpoint interval: unit ms
@@ -347,6 +348,9 @@ out:
current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
if (ret < 0) {
error_report("Detect some error: %s", strerror(-ret));
+ qapi_event_send_colo_exit(COLO_MODE_PRIMARY, COLO_EXIT_REASON_ERROR,
+ true, strerror(-ret), NULL);
+
/* Give users time to get involved in this verdict */
while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
if (failover_request_is_active()) {
@@ -363,15 +367,14 @@ out:
failover_request_active(NULL);
}
qemu_mutex_unlock_iothread();
+ } else {
+ qapi_event_send_colo_exit(COLO_MODE_PRIMARY, COLO_EXIT_REASON_REQUEST,
+ false, NULL, NULL);
}
qsb_free(buffer);
buffer = NULL;
- if (s->from_dst_file) {
- qemu_fclose(s->from_dst_file);
- }
-
return NULL;
}
@@ -552,6 +555,9 @@ out:
if (ret < 0) {
error_report("colo incoming thread will exit, detect error: %s",
strerror(-ret));
+ qapi_event_send_colo_exit(COLO_MODE_SECONDARY, COLO_EXIT_REASON_ERROR,
+ true, strerror(-ret), NULL);
+
/* Give users time to get involved in this verdict */
while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
if (failover_request_is_active()) {
@@ -570,6 +576,9 @@ out:
error_report("SVM is going to exit in default!");
exit(1);
}
+ } else {
+ qapi_event_send_colo_exit(COLO_MODE_SECONDARY, COLO_EXIT_REASON_REQUEST,
+ false, NULL, NULL);
}
if (fb) {
diff --git a/qapi-schema.json b/qapi-schema.json
index 8468552..f75e96b 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -712,6 +712,22 @@
'data': [ 'unknown', 'primary', 'secondary'] }
##
+# @COLOExitReason
+#
+# The reason of COLO exit
+#
+# @unknow: unknown reason
+#
+# @request: COLO exit is due to an external request
+#
+# @error: COLO exit is due to an internal error
+#
+# Since: 2.5
+##
+{ 'enum': 'COLOExitReason',
+ 'data': [ 'unknown', 'request', 'error'] }
+
+##
# @colo-lost-heartbeat
#
# Tell qemu that heartbeat is lost, request it to do takeover procedures.
diff --git a/qapi/event.json b/qapi/event.json
index f0cef01..6158ab5 100644
--- a/qapi/event.json
+++ b/qapi/event.json
@@ -255,6 +255,23 @@
'data': {'status': 'MigrationStatus'}}
##
+# @COLO_EXIT
+#
+# Emitted when VM finishes COLO mode due to some errors happening or
+# the request of users.
+#
+# @mode: @COLOMode describing which side of VM is exit.
+#
+# @reason: @COLOExitReason describing the reason of colo exit.
+#
+# @error: #optional, error message. Only present on error happening.
+#
+# Since: 2.5
+##
+{ 'event': 'COLO_EXIT',
+ 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason', '*error': 'str' } }
+
+##
# @ACPI_DEVICE_OST
#
# Emitted when guest executes ACPI _OST method.
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 23/32] COLO failover: Shutdown related socket fd when do failover
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (21 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 22/32] qmp event: Add event notification for COLO error zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 24/32] COLO failover: Don't do failover during loading VM's state zhanghailiang
` (12 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
If the net connection between COLO's two sides is broken while colo/colo incoming
thread is blocked in 'read'/'write' socket fd. It will not detect this error until
connect timeout. It will be a long time.
Here we shutdown all the related socket file descriptors to wake up the blocking
operation in failover BH. Besides, we should close the corresponding file descriptors
after failvoer BH shutdown them, or there will be an error.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 35 +++++++++++++++++++++++++++++++----
1 file changed, 31 insertions(+), 4 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index 29b5334..e443dfa 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -75,6 +75,13 @@ static void secondary_vm_do_failover(void)
/* recover runstate to normal migration finish state */
autostart = true;
}
+ /* Make sure colo incoming thread not block in recv */
+ if (mis->from_src_file) {
+ qemu_file_shutdown(mis->from_src_file);
+ }
+ if (mis->to_src_file) {
+ qemu_file_shutdown(mis->to_src_file);
+ }
old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
FAILOVER_STATUS_COMPLETED);
@@ -98,6 +105,14 @@ static void primary_vm_do_failover(void)
migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
MIGRATION_STATUS_COMPLETED);
}
+
+ if (s->from_dst_file) { /* Make sure colo thread no block in recv */
+ qemu_file_shutdown(s->from_dst_file);
+ }
+ if (s->to_dst_file) {
+ qemu_file_shutdown(s->to_dst_file);
+ }
+
qemu_bh_schedule(s->cleanup_bh);
vm_start();
@@ -207,7 +222,6 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
goto out;
}
- /* suspend and save vm state to colo buffer */
qemu_mutex_lock_iothread();
if (failover_request_is_active()) {
qemu_mutex_unlock_iothread();
@@ -346,7 +360,7 @@ static void *colo_thread(void *opaque)
out:
current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
- if (ret < 0) {
+ if (ret < 0 || (!ret && !failover_request_is_active())) {
error_report("Detect some error: %s", strerror(-ret));
qapi_event_send_colo_exit(COLO_MODE_PRIMARY, COLO_EXIT_REASON_ERROR,
true, strerror(-ret), NULL);
@@ -375,6 +389,15 @@ out:
qsb_free(buffer);
buffer = NULL;
+ /* Hope this not to be too long to loop here */
+ while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
+ ;
+ }
+ /* Must be called after failover BH is completed */
+ if (s->from_dst_file) {
+ qemu_fclose(s->from_dst_file);
+ }
+
return NULL;
}
@@ -552,7 +575,7 @@ void *colo_process_incoming_thread(void *opaque)
out:
current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
- if (ret < 0) {
+ if (ret < 0 || (!ret && !failover_request_is_active())) {
error_report("colo incoming thread will exit, detect error: %s",
strerror(-ret));
qapi_event_send_colo_exit(COLO_MODE_SECONDARY, COLO_EXIT_REASON_ERROR,
@@ -591,11 +614,15 @@ out:
*/
colo_release_ram_cache();
+ /* Hope this not to be too long to loop here */
+ while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
+ ;
+ }
+ /* Must be called after failover BH is completed */
if (mis->to_src_file) {
qemu_fclose(mis->to_src_file);
}
migration_incoming_exit_colo();
-
return NULL;
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 24/32] COLO failover: Don't do failover during loading VM's state
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (22 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 23/32] COLO failover: Shutdown related socket fd when do failover zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 25/32] COLO: Control the checkpoint delay time by migrate-set-parameters command zhanghailiang
` (11 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
We should not do failover work while the main thread is loading
VM's state, otherwise it will destroy the consistent of VM's memory and
device state.
Here we add a new failover status 'RELAUNCH' which means we should
relaunch the process of failover.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
include/migration/failover.h | 2 ++
migration/colo.c | 24 ++++++++++++++++++++++++
2 files changed, 26 insertions(+)
diff --git a/include/migration/failover.h b/include/migration/failover.h
index fba3931..e115d25 100644
--- a/include/migration/failover.h
+++ b/include/migration/failover.h
@@ -20,6 +20,8 @@ typedef enum COLOFailoverStatus {
FAILOVER_STATUS_REQUEST = 1, /* Request but not handled */
FAILOVER_STATUS_HANDLING = 2, /* In the process of handling failover */
FAILOVER_STATUS_COMPLETED = 3, /* Finish the failover process */
+ /* Optional, Relaunch the failover process, again 'NONE' -> 'COMPLETED' */
+ FAILOVER_STATUS_RELAUNCH = 4,
} COLOFailoverStatus;
void failover_init_state(void);
diff --git a/migration/colo.c b/migration/colo.c
index e443dfa..6bfebd1 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -35,6 +35,7 @@
#define DEFAULT_FAILOVER_DELAY 2000
static QEMUBH *colo_bh;
+static bool vmstate_loading;
/* colo buffer */
#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
@@ -67,6 +68,19 @@ static void secondary_vm_do_failover(void)
int old_state;
MigrationIncomingState *mis = migration_incoming_get_current();
+ /* Can not do failover during the process of VM's loading VMstate, Or
+ * it will break the secondary VM.
+ */
+ if (vmstate_loading) {
+ old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
+ FAILOVER_STATUS_RELAUNCH);
+ if (old_state != FAILOVER_STATUS_HANDLING) {
+ error_report("Unknow error while do failover for secondary VM,"
+ "old_state: %d", old_state);
+ }
+ return;
+ }
+
migrate_set_state(&mis->state, MIGRATION_STATUS_COLO,
MIGRATION_STATUS_COMPLETED);
@@ -557,13 +571,23 @@ void *colo_process_incoming_thread(void *opaque)
qemu_mutex_lock_iothread();
qemu_system_reset(VMRESET_SILENT);
+ vmstate_loading = true;
if (qemu_loadvm_state(fb) < 0) {
error_report("COLO: loadvm failed");
+ vmstate_loading = false;
qemu_mutex_unlock_iothread();
goto out;
}
+
+ vmstate_loading = false;
qemu_mutex_unlock_iothread();
+ if (failover_get_state() == FAILOVER_STATUS_RELAUNCH) {
+ failover_set_state(FAILOVER_STATUS_RELAUNCH, FAILOVER_STATUS_NONE);
+ failover_request_active(NULL);
+ goto out;
+ }
+
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_VMSTATE_LOADED, 0);
if (ret < 0) {
goto out;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 25/32] COLO: Control the checkpoint delay time by migrate-set-parameters command
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (23 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 24/32] COLO failover: Don't do failover during loading VM's state zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 26/32] COLO: Implement shutdown checkpoint zhanghailiang
` (10 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, Luiz Capitulino,
arei.gonglei, stefanha, amit.shah, yanghy, zhanghailiang
Add checkpoint-delay parameter for migrate-set-parameters, so that
we can control the checkpoint frequency when COLO is in periodic mode.
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
hmp.c | 8 ++++++++
migration/colo.c | 12 +++++-------
migration/migration.c | 27 ++++++++++++++++++++++++++-
qapi-schema.json | 17 ++++++++++++++---
4 files changed, 53 insertions(+), 11 deletions(-)
diff --git a/hmp.c b/hmp.c
index 94b0149..2f78666 100644
--- a/hmp.c
+++ b/hmp.c
@@ -273,6 +273,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
monitor_printf(mon, " %s: %" PRId64,
MigrationParameter_lookup[MIGRATION_PARAMETER_DECOMPRESS_THREADS],
params->decompress_threads);
+ monitor_printf(mon, " %s: %" PRId64,
+ MigrationParameter_lookup[MIGRATION_PARAMETER_CHECKPOINT_DELAY],
+ params->checkpoint_delay);
monitor_printf(mon, "\n");
}
@@ -1222,6 +1225,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
bool has_compress_level = false;
bool has_compress_threads = false;
bool has_decompress_threads = false;
+ bool has_checkpoint_delay = false;
int i;
for (i = 0; i < MIGRATION_PARAMETER_MAX; i++) {
@@ -1236,10 +1240,14 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
case MIGRATION_PARAMETER_DECOMPRESS_THREADS:
has_decompress_threads = true;
break;
+ case MIGRATION_PARAMETER_CHECKPOINT_DELAY:
+ has_checkpoint_delay = true;
+ break;
}
qmp_migrate_set_parameters(has_compress_level, value,
has_compress_threads, value,
has_decompress_threads, value,
+ has_checkpoint_delay, value,
&err);
break;
}
diff --git a/migration/colo.c b/migration/colo.c
index 6bfebd1..0113e1b 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -19,12 +19,8 @@
#include "qemu/sockets.h"
#include "migration/failover.h"
#include "qapi-event.h"
-
-/*
- * checkpoint interval: unit ms
- * Note: Please change this default value to 10000 when we support hybrid mode.
- */
-#define CHECKPOINT_MAX_PEROID 200
+#include "qmp-commands.h"
+#include "qapi-types.h"
/*
* The delay time before qemu begin the procedure of default failover treatment.
@@ -36,6 +32,7 @@
static QEMUBH *colo_bh;
static bool vmstate_loading;
+
/* colo buffer */
#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
@@ -360,7 +357,8 @@ static void *colo_thread(void *opaque)
}
current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
- if (current_time - checkpoint_time < CHECKPOINT_MAX_PEROID) {
+ if (current_time - checkpoint_time <
+ s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) {
g_usleep(100000);
continue;
}
diff --git a/migration/migration.c b/migration/migration.c
index f7ca2c4..b140678 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -49,6 +49,11 @@
/* Migration XBZRLE default cache size */
#define DEFAULT_MIGRATE_CACHE_SIZE (64 * 1024 * 1024)
+/* The delay time (in ms) between two COLO checkpoints
+ * Note: Please change this default value to 10000 when we support hybrid mode.
+ */
+#define DEFAULT_MIGRATE_CHECKPOINT_DELAY 200
+
static NotifierList migration_state_notifiers =
NOTIFIER_LIST_INITIALIZER(migration_state_notifiers);
@@ -72,6 +77,8 @@ MigrationState *migrate_get_current(void)
DEFAULT_MIGRATE_COMPRESS_THREAD_COUNT,
.parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
DEFAULT_MIGRATE_DECOMPRESS_THREAD_COUNT,
+ .parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] =
+ DEFAULT_MIGRATE_CHECKPOINT_DELAY,
};
return ¤t_migration;
@@ -393,6 +400,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
s->parameters[MIGRATION_PARAMETER_COMPRESS_THREADS];
params->decompress_threads =
s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS];
+ params->checkpoint_delay =
+ s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY];
return params;
}
@@ -526,7 +535,10 @@ void qmp_migrate_set_parameters(bool has_compress_level,
bool has_compress_threads,
int64_t compress_threads,
bool has_decompress_threads,
- int64_t decompress_threads, Error **errp)
+ int64_t decompress_threads,
+ bool has_checkpoint_delay,
+ int64_t checkpoint_delay,
+ Error **errp)
{
MigrationState *s = migrate_get_current();
@@ -549,6 +561,11 @@ void qmp_migrate_set_parameters(bool has_compress_level,
"is invalid, it should be in the range of 1 to 255");
return;
}
+ if (has_checkpoint_delay && (checkpoint_delay < 0)) {
+ error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
+ "checkpoint_delay",
+ "is invalid, it should be positive");
+ }
if (has_compress_level) {
s->parameters[MIGRATION_PARAMETER_COMPRESS_LEVEL] = compress_level;
@@ -560,6 +577,9 @@ void qmp_migrate_set_parameters(bool has_compress_level,
s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
decompress_threads;
}
+ if (has_checkpoint_delay) {
+ s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] = checkpoint_delay;
+ }
}
/* shared migration helpers */
@@ -676,6 +696,8 @@ static MigrationState *migrate_init(const MigrationParams *params)
s->parameters[MIGRATION_PARAMETER_COMPRESS_THREADS];
int decompress_thread_count =
s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS];
+ int checkpoint_delay =
+ s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY];
memcpy(enabled_capabilities, s->enabled_capabilities,
sizeof(enabled_capabilities));
@@ -691,6 +713,9 @@ static MigrationState *migrate_init(const MigrationParams *params)
compress_thread_count;
s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
decompress_thread_count;
+ s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] =
+ checkpoint_delay;
+
s->bandwidth_limit = bandwidth_limit;
migrate_set_state(&s->state, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
diff --git a/qapi-schema.json b/qapi-schema.json
index f75e96b..75e1770 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -599,10 +599,14 @@
# compression, so set the decompress-threads to the number about 1/4
# of compress-threads is adequate.
#
+# @checkpoint-delay: The delay time (in ms) between two COLO checkpoints in
+# periodic mode.
+#
# Since: 2.4
##
{ 'enum': 'MigrationParameter',
- 'data': ['compress-level', 'compress-threads', 'decompress-threads'] }
+ 'data': ['compress-level', 'compress-threads', 'decompress-threads',
+ 'checkpoint-delay' ] }
#
# @migrate-set-parameters
@@ -615,12 +619,15 @@
#
# @decompress-threads: decompression thread count
#
+# @checkpoint-delay: the delay time between two checkpoints
+#
# Since: 2.4
##
{ 'command': 'migrate-set-parameters',
'data': { '*compress-level': 'int',
'*compress-threads': 'int',
- '*decompress-threads': 'int'} }
+ '*decompress-threads': 'int',
+ '*checkpoint-delay': 'int' } }
#
# @MigrationParameters
@@ -631,12 +638,16 @@
#
# @decompress-threads: decompression thread count
#
+# @checkpoint-delay: the delay time between two COLO checkpoints
+#
# Since: 2.4
##
{ 'struct': 'MigrationParameters',
'data': { 'compress-level': 'int',
'compress-threads': 'int',
- 'decompress-threads': 'int'} }
+ 'decompress-threads': 'int',
+ 'checkpoint-delay': 'int'} }
+
##
# @query-migrate-parameters
#
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 26/32] COLO: Implement shutdown checkpoint
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (24 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 25/32] COLO: Control the checkpoint delay time by migrate-set-parameters command zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 27/32] COLO: Update the global runstate after going into colo state zhanghailiang
` (9 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, Paolo Bonzini,
yanghy, zhanghailiang
For Secondary VM, we forbid it shutdown directly when in COLO mode,
FOR Primary VM's shutdown, we should do some work to ensure the consistent action
between PVM and SVM.
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
include/sysemu/sysemu.h | 3 +++
migration/colo.c | 27 +++++++++++++++++++++++----
qapi-schema.json | 5 +++--
vl.c | 26 ++++++++++++++++++++++++--
4 files changed, 53 insertions(+), 8 deletions(-)
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 15d6d00..e576f65 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -52,6 +52,8 @@ typedef enum WakeupReason {
QEMU_WAKEUP_REASON_OTHER,
} WakeupReason;
+extern int colo_shutdown_requested;
+
void qemu_system_reset_request(void);
void qemu_system_suspend_request(void);
void qemu_register_suspend_notifier(Notifier *notifier);
@@ -59,6 +61,7 @@ void qemu_system_wakeup_request(WakeupReason reason);
void qemu_system_wakeup_enable(WakeupReason reason, bool enabled);
void qemu_register_wakeup_notifier(Notifier *notifier);
void qemu_system_shutdown_request(void);
+void qemu_system_shutdown_request_core(void);
void qemu_system_powerdown_request(void);
void qemu_register_powerdown_notifier(Notifier *notifier);
void qemu_system_debug_request(void);
diff --git a/migration/colo.c b/migration/colo.c
index 0113e1b..ca303c1 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -212,7 +212,7 @@ static int colo_ctl_get(QEMUFile *f, uint32_t require)
static int colo_do_checkpoint_transaction(MigrationState *s,
QEMUSizedBuffer *buffer)
{
- int ret;
+ int colo_shutdown, ret;
size_t size;
QEMUFile *trans = NULL;
@@ -239,7 +239,7 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
ret = -1;
goto out;
}
- /* suspend and save vm state to colo buffer */
+ colo_shutdown = colo_shutdown_requested;
vm_stop_force_state(RUN_STATE_COLO);
qemu_mutex_unlock_iothread();
trace_colo_vm_state_change("run", "stop");
@@ -291,6 +291,15 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
goto out;
}
+ if (colo_shutdown) {
+ colo_ctl_put(s->to_dst_file, COLO_CMD_GUEST_SHUTDOWN, 0);
+ qemu_fflush(s->to_dst_file);
+ colo_shutdown_requested = 0;
+ qemu_system_shutdown_request_core();
+ /* Fix me: Just let the colo thread exit ? */
+ qemu_thread_exit(0);
+ }
+
ret = 0;
/* resume master */
qemu_mutex_lock_iothread();
@@ -357,8 +366,9 @@ static void *colo_thread(void *opaque)
}
current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
- if (current_time - checkpoint_time <
- s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) {
+ if ((current_time - checkpoint_time <
+ s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) &&
+ !colo_shutdown_requested) {
g_usleep(100000);
continue;
}
@@ -467,6 +477,15 @@ static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
case COLO_CMD_CHECKPOINT_REQUEST:
*checkpoint_request = 1;
return 0;
+ case COLO_CMD_GUEST_SHUTDOWN:
+ qemu_mutex_lock_iothread();
+ vm_stop_force_state(RUN_STATE_COLO);
+ qemu_system_shutdown_request_core();
+ qemu_mutex_unlock_iothread();
+ /* the main thread will exit and termiante the whole
+ * process, do we need some cleanup?
+ */
+ qemu_thread_exit(0);
default:
return -EINVAL;
}
diff --git a/qapi-schema.json b/qapi-schema.json
index 75e1770..c19cf97 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -698,13 +698,14 @@
#
# @vmstate-loaded: VM's state has been loaded by SVM
#
+# @guest-shutdown: shutdown require from PVM to SVM
+#
# Since: 2.5
##
{ 'enum': 'COLOCmd',
'data': [ 'invalid', 'checkpoint-ready', 'checkpoint-request',
'checkpoint-reply', 'vmstate-send', 'vmstate-size',
- 'vmstate-received', 'vmstate-loaded', 'guest-shutdown',
- 'ram-steal'] }
+ 'vmstate-received', 'vmstate-loaded', 'guest-shutdown' ] }
##
# @COLOMode
diff --git a/vl.c b/vl.c
index d5e900b..bee3095 100644
--- a/vl.c
+++ b/vl.c
@@ -1637,6 +1637,8 @@ static NotifierList wakeup_notifiers =
NOTIFIER_LIST_INITIALIZER(wakeup_notifiers);
static uint32_t wakeup_reason_mask = ~(1 << QEMU_WAKEUP_REASON_NONE);
+int colo_shutdown_requested;
+
int qemu_shutdown_requested_get(void)
{
return shutdown_requested;
@@ -1753,6 +1755,10 @@ void qemu_system_reset(bool report)
void qemu_system_reset_request(void)
{
if (no_reboot) {
+ qemu_system_shutdown_request();
+ if (!shutdown_requested) {/* colo handle it ? */
+ return;
+ }
shutdown_requested = 1;
} else {
reset_requested = 1;
@@ -1821,13 +1827,29 @@ void qemu_system_killed(int signal, pid_t pid)
qemu_system_shutdown_request();
}
-void qemu_system_shutdown_request(void)
+void qemu_system_shutdown_request_core(void)
{
- trace_qemu_system_shutdown_request();
shutdown_requested = 1;
qemu_notify_event();
}
+void qemu_system_shutdown_request(void)
+{
+ trace_qemu_system_shutdown_request();
+ /*
+ * if in colo mode, we need do some significant work before respond to the
+ * shutdown request.
+ */
+ if (migration_incoming_in_colo_state()) {
+ return ; /* primary's responsibility */
+ }
+ if (migration_in_colo_state()) {
+ colo_shutdown_requested = 1;
+ return;
+ }
+ qemu_system_shutdown_request_core();
+}
+
static void qemu_system_powerdown(void)
{
qapi_event_send_powerdown(&error_abort);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 27/32] COLO: Update the global runstate after going into colo state
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (25 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 26/32] COLO: Implement shutdown checkpoint zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 28/32] savevm: Split load vm state function qemu_loadvm_state zhanghailiang
` (8 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
If we start qemu with -S, the runstate will change from 'prelaunch' to 'running'
after going into colo state.
So it is necessary to update the global runstate after going into colo state.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/migration/colo.c b/migration/colo.c
index ca303c1..e7d733e 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -359,6 +359,11 @@ static void *colo_thread(void *opaque)
qemu_mutex_unlock_iothread();
trace_colo_vm_state_change("stop", "run");
+ ret = global_state_store();
+ if (ret < 0) {
+ goto out;
+ }
+
while (s->state == MIGRATION_STATUS_COLO) {
if (failover_request_is_active()) {
error_report("failover request");
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 28/32] savevm: Split load vm state function qemu_loadvm_state
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (26 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 27/32] COLO: Update the global runstate after going into colo state zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 29/32] COLO: Separate the process of saving/loading ram and device state zhanghailiang
` (7 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
qemu_loadvm_state is too long, and we can simplify it by splitting up
with three helper functions.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
---
migration/savevm.c | 165 +++++++++++++++++++++++++++++++----------------------
1 file changed, 96 insertions(+), 69 deletions(-)
diff --git a/migration/savevm.c b/migration/savevm.c
index c1b40ce..069645b 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1056,6 +1056,100 @@ void loadvm_free_handlers(MigrationIncomingState *mis)
}
}
+static int
+qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis)
+{
+ uint32_t instance_id, version_id, section_id;
+ SaveStateEntry *se;
+ LoadStateEntry *le;
+ char idstr[256];
+ int ret;
+
+ /* Read section start */
+ section_id = qemu_get_be32(f);
+ if (!qemu_get_counted_string(f, idstr)) {
+ error_report("Unable to read ID string for section %u",
+ section_id);
+ return -EINVAL;
+ }
+ instance_id = qemu_get_be32(f);
+ version_id = qemu_get_be32(f);
+
+ trace_qemu_loadvm_state_section_startfull(section_id, idstr,
+ instance_id, version_id);
+ /* Find savevm section */
+ se = find_se(idstr, instance_id);
+ if (se == NULL) {
+ error_report("Unknown savevm section or instance '%s' %d",
+ idstr, instance_id);
+ ret = -EINVAL;
+ return ret;
+ }
+
+ /* Validate version */
+ if (version_id > se->version_id) {
+ error_report("savevm: unsupported version %d for '%s' v%d",
+ version_id, idstr, se->version_id);
+ ret = -EINVAL;
+ return ret;
+ }
+
+ /* Add entry */
+ le = g_malloc0(sizeof(*le));
+
+ le->se = se;
+ le->section_id = section_id;
+ le->version_id = version_id;
+ QLIST_INSERT_HEAD(&mis->loadvm_handlers, le, entry);
+
+ ret = vmstate_load(f, le->se, le->version_id);
+ if (ret < 0) {
+ error_report("error while loading state for instance 0x%x of"
+ " device '%s'", instance_id, idstr);
+ return ret;
+ }
+ if (!check_section_footer(f, le)) {
+ ret = -EINVAL;
+ return ret;
+ }
+
+ return 0;
+}
+
+static int
+qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState *mis)
+{
+ uint32_t section_id;
+ LoadStateEntry *le;
+ int ret;
+
+ section_id = qemu_get_be32(f);
+
+ trace_qemu_loadvm_state_section_partend(section_id);
+ QLIST_FOREACH(le, &mis->loadvm_handlers, entry) {
+ if (le->section_id == section_id) {
+ break;
+ }
+ }
+ if (le == NULL) {
+ error_report("Unknown savevm section %d", section_id);
+ ret = -EINVAL;
+ return ret;
+ }
+
+ ret = vmstate_load(f, le->se, le->version_id);
+ if (ret < 0) {
+ error_report("error while loading state section id %d(%s)",
+ section_id, le->se->idstr);
+ return ret;
+ }
+ if (!check_section_footer(f, le)) {
+ ret = -EINVAL;
+ return ret;
+ }
+
+ return 0;
+}
int qemu_loadvm_state(QEMUFile *f)
{
MigrationIncomingState *mis = migration_incoming_get_current();
@@ -1099,87 +1193,20 @@ int qemu_loadvm_state(QEMUFile *f)
}
while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
- uint32_t instance_id, version_id, section_id;
- SaveStateEntry *se;
- LoadStateEntry *le;
- char idstr[256];
trace_qemu_loadvm_state_section(section_type);
switch (section_type) {
case QEMU_VM_SECTION_START:
case QEMU_VM_SECTION_FULL:
- /* Read section start */
- section_id = qemu_get_be32(f);
- if (!qemu_get_counted_string(f, idstr)) {
- error_report("Unable to read ID string for section %u",
- section_id);
- return -EINVAL;
- }
- instance_id = qemu_get_be32(f);
- version_id = qemu_get_be32(f);
-
- trace_qemu_loadvm_state_section_startfull(section_id, idstr,
- instance_id, version_id);
- /* Find savevm section */
- se = find_se(idstr, instance_id);
- if (se == NULL) {
- error_report("Unknown savevm section or instance '%s' %d",
- idstr, instance_id);
- ret = -EINVAL;
- goto out;
- }
-
- /* Validate version */
- if (version_id > se->version_id) {
- error_report("savevm: unsupported version %d for '%s' v%d",
- version_id, idstr, se->version_id);
- ret = -EINVAL;
- goto out;
- }
-
- /* Add entry */
- le = g_malloc0(sizeof(*le));
-
- le->se = se;
- le->section_id = section_id;
- le->version_id = version_id;
- QLIST_INSERT_HEAD(&mis->loadvm_handlers, le, entry);
-
- ret = vmstate_load(f, le->se, le->version_id);
+ ret = qemu_loadvm_section_start_full(f, mis);
if (ret < 0) {
- error_report("error while loading state for instance 0x%x of"
- " device '%s'", instance_id, idstr);
- goto out;
- }
- if (!check_section_footer(f, le)) {
- ret = -EINVAL;
goto out;
}
break;
case QEMU_VM_SECTION_PART:
case QEMU_VM_SECTION_END:
- section_id = qemu_get_be32(f);
-
- trace_qemu_loadvm_state_section_partend(section_id);
- QLIST_FOREACH(le, &mis->loadvm_handlers, entry) {
- if (le->section_id == section_id) {
- break;
- }
- }
- if (le == NULL) {
- error_report("Unknown savevm section %d", section_id);
- ret = -EINVAL;
- goto out;
- }
-
- ret = vmstate_load(f, le->se, le->version_id);
+ ret = qemu_loadvm_section_part_end(f, mis);
if (ret < 0) {
- error_report("error while loading state section id %d(%s)",
- section_id, le->se->idstr);
- goto out;
- }
- if (!check_section_footer(f, le)) {
- ret = -EINVAL;
goto out;
}
break;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 29/32] COLO: Separate the process of saving/loading ram and device state
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (27 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 28/32] savevm: Split load vm state function qemu_loadvm_state zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 30/32] COLO: Split qemu_savevm_state_begin out of checkpoint process zhanghailiang
` (6 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
We separate the process of saving/loading ram and device state when do checkpoint,
we add new helpers for save/load ram/device. With this change, we can directly
transfer ram from master to slave without using QEMUSizeBuffer as assistant,
which also reduce the size of extra memory been used during checkpoint.
Besides, we move the colo_flush_ram_cache to the proper position after the
above change.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
include/sysemu/sysemu.h | 5 ++
migration/colo.c | 43 +++++++++++----
migration/ram.c | 8 ---
migration/savevm.c | 142 +++++++++++++++++++++++++++++++++++++++++++++++-
4 files changed, 177 insertions(+), 21 deletions(-)
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index e576f65..637959f 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -93,7 +93,12 @@ int qemu_savevm_state_iterate(QEMUFile *f);
void qemu_savevm_state_complete(QEMUFile *f);
void qemu_savevm_state_cancel(void);
uint64_t qemu_savevm_state_pending(QEMUFile *f, uint64_t max_size);
+int qemu_save_ram_state(QEMUFile *f);
+int qemu_save_device_state(QEMUFile *f);
int qemu_loadvm_state(QEMUFile *f);
+int qemu_loadvm_state_begin(QEMUFile *f);
+int qemu_load_ram_state(QEMUFile *f);
+int qemu_load_device_state(QEMUFile *f);
typedef enum DisplayType
{
diff --git a/migration/colo.c b/migration/colo.c
index e7d733e..22ca7e5 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -252,21 +252,32 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
goto out;
}
+ ret = colo_ctl_put(s->to_dst_file, COLO_CMD_VMSTATE_SEND, 0);
+ if (ret < 0) {
+ goto out;
+ }
/* Disable block migration */
s->params.blk = 0;
s->params.shared = 0;
- qemu_savevm_state_header(trans);
- qemu_savevm_state_begin(trans, &s->params);
- qemu_mutex_lock_iothread();
- qemu_savevm_state_complete(trans);
- qemu_mutex_unlock_iothread();
-
- qemu_fflush(trans);
+ qemu_savevm_state_begin(s->to_dst_file, &s->params);
+ ret = qemu_file_get_error(s->to_dst_file);
+ if (ret < 0) {
+ error_report("save vm state begin error\n");
+ goto out;
+ }
- ret = colo_ctl_put(s->to_dst_file, COLO_CMD_VMSTATE_SEND, 0);
+ qemu_mutex_lock_iothread();
+ /* Note: device state is saved into buffer */
+ ret = qemu_save_device_state(trans);
if (ret < 0) {
+ error_report("save device state error\n");
+ qemu_mutex_unlock_iothread();
goto out;
}
+ qemu_fflush(trans);
+ qemu_save_ram_state(s->to_dst_file);
+ qemu_mutex_unlock_iothread();
+
/* we send the total size of the vmstate first */
size = qsb_get_length(buffer);
ret = colo_ctl_put(s->to_dst_file, COLO_CMD_VMSTATE_SIZE, size);
@@ -566,6 +577,16 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
+ ret = qemu_loadvm_state_begin(mis->from_src_file);
+ if (ret < 0) {
+ error_report("load vm state begin error, ret=%d", ret);
+ goto out;
+ }
+ ret = qemu_load_ram_state(mis->from_src_file);
+ if (ret < 0) {
+ error_report("load ram state error");
+ goto out;
+ }
/* read the VM state total size first */
total_size = colo_ctl_get(mis->from_src_file, COLO_CMD_VMSTATE_SIZE);
if (total_size <= 0) {
@@ -594,8 +615,10 @@ void *colo_process_incoming_thread(void *opaque)
qemu_mutex_lock_iothread();
qemu_system_reset(VMRESET_SILENT);
vmstate_loading = true;
- if (qemu_loadvm_state(fb) < 0) {
- error_report("COLO: loadvm failed");
+ colo_flush_ram_cache();
+ ret = qemu_load_device_state(fb);
+ if (ret < 0) {
+ error_report("COLO: load device state failed\n");
vmstate_loading = false;
qemu_mutex_unlock_iothread();
goto out;
diff --git a/migration/ram.c b/migration/ram.c
index 782febd..8476783 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -1522,7 +1522,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
int flags = 0, ret = 0;
static uint64_t seq_iter;
int len = 0;
- bool need_flush = false;
seq_iter++;
@@ -1592,7 +1591,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
break;
}
- need_flush = true;
ch = qemu_get_byte(f);
ram_handle_compressed(host, ch, TARGET_PAGE_SIZE);
break;
@@ -1604,7 +1602,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
break;
}
- need_flush = true;
qemu_get_buffer(f, host, TARGET_PAGE_SIZE);
break;
case RAM_SAVE_FLAG_COMPRESS_PAGE:
@@ -1637,7 +1634,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
ret = -EINVAL;
break;
}
- need_flush = true;
break;
case RAM_SAVE_FLAG_EOS:
/* normal exit */
@@ -1658,10 +1654,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
rcu_read_unlock();
- if (!ret && ram_cache_enable && need_flush) {
- DPRINTF("Flush ram_cache\n");
- colo_flush_ram_cache();
- }
DPRINTF("Completed load of VM with exit code %d seq iteration "
"%" PRIu64 "\n", ret, seq_iter);
return ret;
diff --git a/migration/savevm.c b/migration/savevm.c
index 069645b..64e4bae 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -755,6 +755,10 @@ void qemu_savevm_state_begin(QEMUFile *f,
break;
}
}
+ if (migration_in_colo_state()) {
+ qemu_put_byte(f, QEMU_VM_EOF);
+ qemu_fflush(f);
+ }
}
/*
@@ -952,13 +956,44 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
return ret;
}
-static int qemu_save_device_state(QEMUFile *f)
+int qemu_save_ram_state(QEMUFile *f)
{
SaveStateEntry *se;
+ int ret = 0;
- qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
- qemu_put_be32(f, QEMU_VM_FILE_VERSION);
+ QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
+ if (!se->ops || !se->ops->save_live_complete) {
+ continue;
+ }
+ if (se->ops && se->ops->is_active) {
+ if (!se->ops->is_active(se->opaque)) {
+ continue;
+ }
+ }
+ trace_savevm_section_start(se->idstr, se->section_id);
+
+ save_section_header(f, se, QEMU_VM_SECTION_END);
+
+ ret = se->ops->save_live_complete(f, se->opaque);
+ trace_savevm_section_end(se->idstr, se->section_id, ret);
+ save_section_footer(f, se);
+ if (ret < 0) {
+ qemu_file_set_error(f, ret);
+ return ret;
+ }
+ }
+ qemu_put_byte(f, QEMU_VM_EOF);
+ return 0;
+}
+
+int qemu_save_device_state(QEMUFile *f)
+{
+ SaveStateEntry *se;
+
+ if (!migration_in_colo_state()) {
+ qemu_savevm_state_header(f);
+ }
cpu_synchronize_all_states();
QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
@@ -1267,6 +1302,107 @@ out:
return ret;
}
+int qemu_loadvm_state_begin(QEMUFile *f)
+{
+ uint8_t section_type;
+ int ret = -1;
+ MigrationIncomingState *mis = migration_incoming_get_current();
+
+ if (!mis) {
+ error_report("qemu_loadvm_state_begin");
+ return -EINVAL;
+ }
+ /* CleanUp */
+ loadvm_free_handlers(mis);
+
+ if (qemu_savevm_state_blocked(NULL)) {
+ return -EINVAL;
+ }
+
+ if (!savevm_state.skip_configuration) {
+ if (qemu_get_byte(f) != QEMU_VM_CONFIGURATION) {
+ error_report("Configuration section missing");
+ return -EINVAL;
+ }
+ ret = vmstate_load_state(f, &vmstate_configuration, &savevm_state, 0);
+
+ if (ret) {
+ return ret;
+ }
+ }
+
+ while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
+ if (section_type != QEMU_VM_SECTION_START) {
+ error_report("QEMU_VM_SECTION_START");
+ ret = -EINVAL;
+ goto out;
+ }
+ ret = qemu_loadvm_section_start_full(f, mis);
+ if (ret < 0) {
+ goto out;
+ }
+ }
+ ret = qemu_file_get_error(f);
+ if (ret == 0) {
+ return 0;
+ }
+out:
+ return ret;
+}
+
+int qemu_load_ram_state(QEMUFile *f)
+{
+ uint8_t section_type;
+ MigrationIncomingState *mis = migration_incoming_get_current();
+ int ret = -1;
+
+ while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
+ if (section_type != QEMU_VM_SECTION_PART &&
+ section_type != QEMU_VM_SECTION_END) {
+ error_report("load ram state, not get "
+ "QEMU_VM_SECTION_FULL or QEMU_VM_SECTION_END");
+ return -EINVAL;
+ }
+ ret = qemu_loadvm_section_part_end(f, mis);
+ if (ret < 0) {
+ goto out;
+ }
+ }
+ ret = qemu_file_get_error(f);
+ if (ret == 0) {
+ return 0;
+ }
+out:
+ return ret;
+}
+
+int qemu_load_device_state(QEMUFile *f)
+{
+ uint8_t section_type;
+ MigrationIncomingState *mis = migration_incoming_get_current();
+ int ret = -1;
+
+ while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
+ if (section_type != QEMU_VM_SECTION_FULL) {
+ error_report("load device state error: "
+ "Not get QEMU_VM_SECTION_FULL");
+ return -EINVAL;
+ }
+ ret = qemu_loadvm_section_start_full(f, mis);
+ if (ret < 0) {
+ goto out;
+ }
+ }
+
+ ret = qemu_file_get_error(f);
+
+ cpu_synchronize_all_post_init();
+ if (ret == 0) {
+ return 0;
+ }
+out:
+ return ret;
+}
static BlockDriverState *find_vmstate_bs(void)
{
BlockDriverState *bs = NULL;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 30/32] COLO: Split qemu_savevm_state_begin out of checkpoint process
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (28 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 29/32] COLO: Separate the process of saving/loading ram and device state zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 31/32] COLO: Add block replication into colo process zhanghailiang
` (5 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
It is unnecessary to call qemu_savevm_state_begin() in every checkponit process.
It mainly sets up devices and does the first device state pass. These data will
not change during the later checkpoint process. So, we split it out of
colo_do_checkpoint_transaction(), in this way, we can reduce these data
transferring in the later checkpoint.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 51 +++++++++++++++++++++++++++++++++++++--------------
1 file changed, 37 insertions(+), 14 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index 22ca7e5..fdbda79 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -256,15 +256,6 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
if (ret < 0) {
goto out;
}
- /* Disable block migration */
- s->params.blk = 0;
- s->params.shared = 0;
- qemu_savevm_state_begin(s->to_dst_file, &s->params);
- ret = qemu_file_get_error(s->to_dst_file);
- if (ret < 0) {
- error_report("save vm state begin error\n");
- goto out;
- }
qemu_mutex_lock_iothread();
/* Note: device state is saved into buffer */
@@ -326,6 +317,21 @@ out:
return ret;
}
+static int colo_prepare_before_save(MigrationState *s)
+{
+ int ret;
+ /* Disable block migration */
+ s->params.blk = 0;
+ s->params.shared = 0;
+ qemu_savevm_state_begin(s->to_dst_file, &s->params);
+ ret = qemu_file_get_error(s->to_dst_file);
+ if (ret < 0) {
+ error_report("save vm state begin error\n");
+ return ret;
+ }
+ return 0;
+}
+
static void *colo_thread(void *opaque)
{
MigrationState *s = opaque;
@@ -349,6 +355,11 @@ static void *colo_thread(void *opaque)
goto out;
}
+ ret = colo_prepare_before_save(s);
+ if (ret < 0) {
+ goto out;
+ }
+
/*
* Wait for Secondary finish loading vm states and enter COLO
* restore.
@@ -507,6 +518,18 @@ static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
}
}
+static int colo_prepare_before_load(QEMUFile *f)
+{
+ int ret;
+
+ ret = qemu_loadvm_state_begin(f);
+ if (ret < 0) {
+ error_report("load vm state begin error, ret=%d", ret);
+ return ret;
+ }
+ return 0;
+}
+
void *colo_process_incoming_thread(void *opaque)
{
MigrationIncomingState *mis = opaque;
@@ -545,6 +568,11 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
+ ret = colo_prepare_before_load(mis->from_src_file);
+ if (ret < 0) {
+ goto out;
+ }
+
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_CHECKPOINT_READY, 0);
if (ret < 0) {
goto out;
@@ -577,11 +605,6 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
- ret = qemu_loadvm_state_begin(mis->from_src_file);
- if (ret < 0) {
- error_report("load vm state begin error, ret=%d", ret);
- goto out;
- }
ret = qemu_load_ram_state(mis->from_src_file);
if (ret < 0) {
error_report("load ram state error");
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 31/32] COLO: Add block replication into colo process
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (29 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 30/32] COLO: Split qemu_savevm_state_begin out of checkpoint process zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 32/32] COLO: Add net packets treatment into COLO zhanghailiang
` (4 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
From: Wen Congyang <wency@cn.fujitsu.com>
Make sure master start block replication after slave's block replication started.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 61 +++++++++++++++++++++++++++++++++++++++++++++++++++++++-
trace-events | 2 ++
2 files changed, 62 insertions(+), 1 deletion(-)
diff --git a/migration/colo.c b/migration/colo.c
index fdbda79..d7a14ba 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -21,6 +21,7 @@
#include "qapi-event.h"
#include "qmp-commands.h"
#include "qapi-types.h"
+#include "block/block_int.h"
/*
* The delay time before qemu begin the procedure of default failover treatment.
@@ -64,6 +65,7 @@ static void secondary_vm_do_failover(void)
{
int old_state;
MigrationIncomingState *mis = migration_incoming_get_current();
+ Error *local_err = NULL;
/* Can not do failover during the process of VM's loading VMstate, Or
* it will break the secondary VM.
@@ -81,6 +83,12 @@ static void secondary_vm_do_failover(void)
migrate_set_state(&mis->state, MIGRATION_STATUS_COLO,
MIGRATION_STATUS_COMPLETED);
+ bdrv_stop_replication_all(true, &local_err);
+ if (local_err) {
+ error_report_err(local_err);
+ }
+ trace_colo_stop_block_replication("failover");
+
if (!autostart) {
error_report("\"-S\" qemu option will be ignored in secondary side");
/* recover runstate to normal migration finish state */
@@ -111,6 +119,7 @@ static void primary_vm_do_failover(void)
{
MigrationState *s = migrate_get_current();
int old_state;
+ Error *local_err = NULL;
if (s->state != MIGRATION_STATUS_FAILED) {
migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
@@ -126,6 +135,12 @@ static void primary_vm_do_failover(void)
qemu_bh_schedule(s->cleanup_bh);
+ bdrv_stop_replication_all(true, &local_err);
+ if (local_err) {
+ error_report_err(local_err);
+ }
+ trace_colo_stop_block_replication("failover");
+
vm_start();
old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
@@ -215,6 +230,7 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
int colo_shutdown, ret;
size_t size;
QEMUFile *trans = NULL;
+ Error *local_err = NULL;
ret = colo_ctl_put(s->to_dst_file, COLO_CMD_CHECKPOINT_REQUEST, 0);
if (ret < 0) {
@@ -252,6 +268,16 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
goto out;
}
+ /* we call this api although this may do nothing on primary side */
+ qemu_mutex_lock_iothread();
+ bdrv_do_checkpoint_all(&local_err);
+ qemu_mutex_unlock_iothread();
+ if (local_err) {
+ error_report_err(local_err);
+ ret = -1;
+ goto out;
+ }
+
ret = colo_ctl_put(s->to_dst_file, COLO_CMD_VMSTATE_SEND, 0);
if (ret < 0) {
goto out;
@@ -294,6 +320,10 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
}
if (colo_shutdown) {
+ qemu_mutex_lock_iothread();
+ bdrv_stop_replication_all(false, NULL);
+ trace_colo_stop_block_replication("shutdown");
+ qemu_mutex_unlock_iothread();
colo_ctl_put(s->to_dst_file, COLO_CMD_GUEST_SHUTDOWN, 0);
qemu_fflush(s->to_dst_file);
colo_shutdown_requested = 0;
@@ -339,6 +369,7 @@ static void *colo_thread(void *opaque)
int64_t current_time, checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
int64_t error_time;
int fd, ret = 0;
+ Error *local_err = NULL;
failover_init_state();
@@ -377,6 +408,14 @@ static void *colo_thread(void *opaque)
}
qemu_mutex_lock_iothread();
+ /* start block replication */
+ bdrv_start_replication_all(REPLICATION_MODE_PRIMARY, &local_err);
+ if (local_err) {
+ qemu_mutex_unlock_iothread();
+ error_report_err(local_err);
+ goto out;
+ }
+ trace_colo_start_block_replication();
vm_start();
qemu_mutex_unlock_iothread();
trace_colo_vm_state_change("stop", "run");
@@ -507,6 +546,8 @@ static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
case COLO_CMD_GUEST_SHUTDOWN:
qemu_mutex_lock_iothread();
vm_stop_force_state(RUN_STATE_COLO);
+ bdrv_stop_replication_all(false, NULL);
+ trace_colo_stop_block_replication("shutdown");
qemu_system_shutdown_request_core();
qemu_mutex_unlock_iothread();
/* the main thread will exit and termiante the whole
@@ -538,6 +579,7 @@ void *colo_process_incoming_thread(void *opaque)
int total_size;
int64_t error_time, current_time;
int fd, ret = 0;
+ Error *local_err = NULL;
migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_COLO);
@@ -573,6 +615,16 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
+ qemu_mutex_lock_iothread();
+ /* start block replication */
+ bdrv_start_replication_all(REPLICATION_MODE_SECONDARY, &local_err);
+ qemu_mutex_unlock_iothread();
+ if (local_err) {
+ error_report_err(local_err);
+ goto out;
+ }
+ trace_colo_start_block_replication();
+
ret = colo_ctl_put(mis->to_src_file, COLO_CMD_CHECKPOINT_READY, 0);
if (ret < 0) {
goto out;
@@ -647,8 +699,15 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
- vmstate_loading = false;
+ /* discard colo disk buffer */
+ bdrv_do_checkpoint_all(&local_err);
qemu_mutex_unlock_iothread();
+ if (local_err) {
+ vmstate_loading = false;
+ goto out;
+ }
+
+ vmstate_loading = false;
if (failover_get_state() == FAILOVER_STATUS_RELAUNCH) {
failover_set_state(FAILOVER_STATUS_RELAUNCH, FAILOVER_STATUS_NONE);
diff --git a/trace-events b/trace-events
index cf378ec..d3ba02d 100644
--- a/trace-events
+++ b/trace-events
@@ -1477,6 +1477,8 @@ colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
colo_ctl_put(const char *msg) "Send '%s'"
colo_ctl_get(const char *msg) "Receive '%s'"
colo_failover_set_state(int new_state) "new state %d"
+colo_start_block_replication(void) "Block replication is started"
+colo_stop_block_replication(const char *reason) "Block replication is stopped(reason: '%s')"
# kvm-all.c
kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v9 32/32] COLO: Add net packets treatment into COLO
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (30 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 31/32] COLO: Add block replication into colo process zhanghailiang
@ 2015-09-02 8:23 ` zhanghailiang
2015-09-02 9:03 ` [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) Yang Hongyang
` (3 subsequent siblings)
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 8:23 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy,
zhanghailiang
We will buffer all the net packets that sended by PVM, and release
them during the checkpoint process.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
---
migration/colo.c | 3 +++
1 file changed, 3 insertions(+)
diff --git a/migration/colo.c b/migration/colo.c
index d7a14ba..f59f6e0 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -22,6 +22,7 @@
#include "qmp-commands.h"
#include "qapi-types.h"
#include "block/block_int.h"
+#include "net/filter.h"
/*
* The delay time before qemu begin the procedure of default failover treatment.
@@ -319,6 +320,8 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
goto out;
}
+ filter_buffer_release_all();
+
if (colo_shutdown) {
qemu_mutex_lock_iothread();
bdrv_stop_replication_all(false, NULL);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT)
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (31 preceding siblings ...)
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 32/32] COLO: Add net packets treatment into COLO zhanghailiang
@ 2015-09-02 9:03 ` Yang Hongyang
2015-09-02 9:17 ` zhanghailiang
2015-09-09 3:36 ` zhanghailiang
` (2 subsequent siblings)
35 siblings, 1 reply; 59+ messages in thread
From: Yang Hongyang @ 2015-09-02 9:03 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah
Hi Stefan,
As we discussed at KVM Forum, a stripped down version of COLO is realized,
which only contains periodic checkpoint mode. By the advantage of QEMU space
netbuffer(we can drop all those complex network settings), this version is more
easy to setup and test, all you need is a pair of machine, both checkout
https://github.com/coloft/qemu/commits/colo-v2.0-periodic-mode
and compile. Issue the test procedure mentioned below. It's done.
Thanks!
On 09/02/2015 04:22 PM, zhanghailiang wrote:
> This is the 9th version of COLO.
>
> Please Note that, this version is very different from the previous versions.
> since we have decided to realize proxy in qemu, which based on slirp in qemu.
> We dropped all the original colo proxy related part.
>
> It will be a long time for proxy to be ready for merging, so here we extract
> the basic periodic checkpoint part that not depend on proxy into this series.
> Actually, the 'periodic' mode is also what we want to support in COLO, it is
> based on Yang Hongyang's netfilter series. and this mode is very like
> MicroCheckpointing and Remus.
>
> You can find the discussion about why & how to realize the colo proxy in qemu
> from the follow link:
> http://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg04069.html
>
> As usual, here is only COLO frame part, you can get the whole codes from github:
> https://github.com/coloft/qemu/commits/colo-v2.0-periodic-mode
>
> Compared with previous versions, this version is more easy to test.
>
> Test procedure:
> 1. Startup qemu
> Primary side:
> # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -netdev tap,id=bn0 -netfilter buffer,id=f0,netdev=bn0,chain=in -device virtio-net-pci,id=net-pci0,netdev=bn0 -boot c -drive if=virtio,id=disk1,driver=quorum,read-pattern=fifo,cache=none,aio=native,children.0.file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,children.0.driver=raw -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio -S
>
> Secondary side:
> # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0 -drive if=none,driver=raw,file=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,id=colo1,cache=none,aio=native -drive if=virtio,driver=replication,mode=secondary,throttling.bps-total=70000000,file.file.filename=/mnt/ramfs/active_disk.img,file.driver=qcow2,file.backing.file.filename=/mnt/ramfs/hidden_disk.img,file.backing.driver=qcow2,file.backing.backing.backing_reference=colo1,file.backing.allow-write-backing-file=on -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-table -monitor stdio -incoming tcp:0:8888
>
> 2. On Secondary VM's QEMU monitor, issue command
> (qemu) nbd_server_start 192.168.2.88:8889
> (qemu) nbd_server_add -w colo1
>
> 3. On Primary VM's QEMU monitor, issue command:
> (qemu) child_add disk1 child.driver=replication,child.mode=primary,child.file.host=192.168.2.88,child.file.port=8889,child.file.export=colo1,child.file.driver=nbd,child.ignore-errors=on
> (qemu) migrate_set_capability colo on
> (qemu) migrate tcp:192.168.2.88:8888
>
> 4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
> You can by issue command "migrate_set_parameter checkpoint-delay 2000"
> to change the checkpoint period time.
>
> 5. Failover test
> You can kill PVM and run 'colo_lost_heartbeat' in SVM's
> monitor at the same time, then SVM will failover and client will not feel this change.
>
> COLO is a totally new feature which is still in early stage,
> your comments and feedback are warmly welcomed.
>
> TODO:
> 1. checkpoint based on proxy in qemu
> 2. The capability of continuous FT
>
> v9:
> - Drop colo proxy related part (colo-nic.c file)
> - Convert COLO protocol name definition to QAPI
> - Smash failover related patch (patch 19/20/23)
> - Fix colo exit event according Eric's comments.
> - Fix some typos from Eric's comments
> - Fix bug 'invalid runstate transition: 'colo' -> 'prelaunch' reported
> by Dave (patch 27)
> - Use migrate_set_parameter intead of ecolo-set-checkpoint-period to set
> checkpoint delay time (patch 25)
> - Add new patch (patch 29/30) to seperate the process of saving/loading
> device and state during checkpoint. which will reduce the data size
> for sending and also reduce the qsb size used in checkpoint.
>
> Wen Congyang (1):
> COLO: Add block replication into colo process
>
> zhanghailiang (31):
> configure: Add parameter for configure to enable/disable COLO support
> migration: Introduce capability 'colo' to migration
> COLO: migrate colo related info to slave
> migration: Add state records for migration incoming
> migration: Integrate COLO checkpoint process into migration
> migration: Integrate COLO checkpoint process into loadvm
> migration: Rename the'file' member of MigrationState and
> MigrationIncomingState
> COLO/migration: establish a new communication path from destination to
> source
> COLO: Implement colo checkpoint protocol
> COLO: Add a new RunState RUN_STATE_COLO
> QEMUSizedBuffer: Introduce two help functions for qsb
> COLO: Save PVM state to secondary side when do checkpoint
> COLO: Load PVM's dirty pages into SVM's RAM cache temporarily
> COLO: Load VMState into qsb before restore it
> COLO: Flush PVM's cached RAM into SVM's memory
> COLO: synchronize PVM's state to SVM periodically
> COLO failover: Introduce a new command to trigger a failover
> COLO failover: Introduce state to record failover process
> COLO: Implement failover work for Primary VM
> COLO: Implement failover work for Secondary VM
> COLO: implement default failover treatment
> qmp event: Add event notification for COLO error
> COLO failover: Shutdown related socket fd when do failover
> COLO failover: Don't do failover during loading VM's state
> COLO: Control the checkpoint delay time by migrate-set-parameters
> command
> COLO: Implement shutdown checkpoint
> COLO: Update the global runstate after going into colo state
> savevm: Split load vm state function qemu_loadvm_state
> COLO: Separate the process of saving/loading ram and device state
> COLO: Split qemu_savevm_state_begin out of checkpoint process
> COLO: Add net packets treatment into COLO
>
> configure | 11 +
> docs/qmp/qmp-events.txt | 17 +
> hmp-commands.hx | 15 +
> hmp.c | 16 +
> hmp.h | 1 +
> include/exec/cpu-all.h | 1 +
> include/migration/colo.h | 44 +++
> include/migration/failover.h | 33 ++
> include/migration/migration.h | 16 +-
> include/migration/qemu-file.h | 3 +-
> include/sysemu/sysemu.h | 8 +
> migration/Makefile.objs | 2 +
> migration/colo-comm.c | 75 ++++
> migration/colo-failover.c | 83 +++++
> migration/colo.c | 782 ++++++++++++++++++++++++++++++++++++++++++
> migration/exec.c | 4 +-
> migration/fd.c | 4 +-
> migration/migration.c | 184 +++++++---
> migration/qemu-file-buf.c | 58 ++++
> migration/ram.c | 185 +++++++++-
> migration/savevm.c | 309 +++++++++++++----
> migration/tcp.c | 4 +-
> migration/unix.c | 4 +-
> qapi-schema.json | 101 +++++-
> qapi/event.json | 17 +
> qmp-commands.hx | 20 ++
> stubs/Makefile.objs | 1 +
> stubs/migration-colo.c | 45 +++
> trace-events | 8 +
> vl.c | 37 +-
> 30 files changed, 1930 insertions(+), 158 deletions(-)
> create mode 100644 include/migration/colo.h
> create mode 100644 include/migration/failover.h
> create mode 100644 migration/colo-comm.c
> create mode 100644 migration/colo-failover.c
> create mode 100644 migration/colo.c
> create mode 100644 stubs/migration-colo.c
>
--
Thanks,
Yang.
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT)
2015-09-02 9:03 ` [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) Yang Hongyang
@ 2015-09-02 9:17 ` zhanghailiang
0 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-02 9:17 UTC (permalink / raw)
To: Yang Hongyang, qemu-devel, stefanha
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, amit.shah
On 2015/9/2 17:03, Yang Hongyang wrote:
> Hi Stefan,
>
> As we discussed at KVM Forum, a stripped down version of COLO is realized,
> which only contains periodic checkpoint mode. By the advantage of QEMU space
> netbuffer(we can drop all those complex network settings), this version is more
> easy to setup and test, all you need is a pair of machine, both checkout
> https://github.com/coloft/qemu/commits/colo-v2.0-periodic-mode
> and compile. Issue the test procedure mentioned below. It's done.
>
Er, sorry, i have lost one step for the test.
In Slave side, before startup qemu, we should create the ramdisk for the disk.
# qemu-img create -f qcow2 /mnt/ramfs/active_disk.img 10G
# qemu-img create -f qcow2 /mnt/ramfs/hidden_disk.img 10G
Please note, the size should be same with VM's disk.
Thanks.
zhanghailiang
> Thanks!
>
> On 09/02/2015 04:22 PM, zhanghailiang wrote:
>> This is the 9th version of COLO.
>>
>> Please Note that, this version is very different from the previous versions.
>> since we have decided to realize proxy in qemu, which based on slirp in qemu.
>> We dropped all the original colo proxy related part.
>>
>> It will be a long time for proxy to be ready for merging, so here we extract
>> the basic periodic checkpoint part that not depend on proxy into this series.
>> Actually, the 'periodic' mode is also what we want to support in COLO, it is
>> based on Yang Hongyang's netfilter series. and this mode is very like
>> MicroCheckpointing and Remus.
>>
>> You can find the discussion about why & how to realize the colo proxy in qemu
>> from the follow link:
>> http://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg04069.html
>>
>> As usual, here is only COLO frame part, you can get the whole codes from github:
>> https://github.com/coloft/qemu/commits/colo-v2.0-periodic-mode
>>
>> Compared with previous versions, this version is more easy to test.
>>
>> Test procedure:
>> 1. Startup qemu
>> Primary side:
>> # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -netdev tap,id=bn0 -netfilter buffer,id=f0,netdev=bn0,chain=in -device virtio-net-pci,id=net-pci0,netdev=bn0 -boot c -drive if=virtio,id=disk1,driver=quorum,read-pattern=fifo,cache=none,aio=native,children.0.file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,children.0.driver=raw -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio -S
>>
>> Secondary side:
>> # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0 -drive if=none,driver=raw,file=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,id=colo1,cache=none,aio=native -drive if=virtio,driver=replication,mode=secondary,throttling.bps-total=70000000,file.file.filename=/mnt/ramfs/active_disk.img,file.driver=qcow2,file.backing.file.filename=/mnt/ramfs/hidden_disk.img,file.backing.driver=qcow2,file.backing.backing.backing_reference=colo1,file.backing.allow-write-backing-file=on -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-table -monitor stdio -incoming tcp:0:8888
>>
>> 2. On Secondary VM's QEMU monitor, issue command
>> (qemu) nbd_server_start 192.168.2.88:8889
>> (qemu) nbd_server_add -w colo1
>>
>> 3. On Primary VM's QEMU monitor, issue command:
>> (qemu) child_add disk1 child.driver=replication,child.mode=primary,child.file.host=192.168.2.88,child.file.port=8889,child.file.export=colo1,child.file.driver=nbd,child.ignore-errors=on
>> (qemu) migrate_set_capability colo on
>> (qemu) migrate tcp:192.168.2.88:8888
>>
>> 4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
>> You can by issue command "migrate_set_parameter checkpoint-delay 2000"
>> to change the checkpoint period time.
>>
>> 5. Failover test
>> You can kill PVM and run 'colo_lost_heartbeat' in SVM's
>> monitor at the same time, then SVM will failover and client will not feel this change.
>>
>> COLO is a totally new feature which is still in early stage,
>> your comments and feedback are warmly welcomed.
>>
>> TODO:
>> 1. checkpoint based on proxy in qemu
>> 2. The capability of continuous FT
>>
>> v9:
>> - Drop colo proxy related part (colo-nic.c file)
>> - Convert COLO protocol name definition to QAPI
>> - Smash failover related patch (patch 19/20/23)
>> - Fix colo exit event according Eric's comments.
>> - Fix some typos from Eric's comments
>> - Fix bug 'invalid runstate transition: 'colo' -> 'prelaunch' reported
>> by Dave (patch 27)
>> - Use migrate_set_parameter intead of ecolo-set-checkpoint-period to set
>> checkpoint delay time (patch 25)
>> - Add new patch (patch 29/30) to seperate the process of saving/loading
>> device and state during checkpoint. which will reduce the data size
>> for sending and also reduce the qsb size used in checkpoint.
>>
>> Wen Congyang (1):
>> COLO: Add block replication into colo process
>>
>> zhanghailiang (31):
>> configure: Add parameter for configure to enable/disable COLO support
>> migration: Introduce capability 'colo' to migration
>> COLO: migrate colo related info to slave
>> migration: Add state records for migration incoming
>> migration: Integrate COLO checkpoint process into migration
>> migration: Integrate COLO checkpoint process into loadvm
>> migration: Rename the'file' member of MigrationState and
>> MigrationIncomingState
>> COLO/migration: establish a new communication path from destination to
>> source
>> COLO: Implement colo checkpoint protocol
>> COLO: Add a new RunState RUN_STATE_COLO
>> QEMUSizedBuffer: Introduce two help functions for qsb
>> COLO: Save PVM state to secondary side when do checkpoint
>> COLO: Load PVM's dirty pages into SVM's RAM cache temporarily
>> COLO: Load VMState into qsb before restore it
>> COLO: Flush PVM's cached RAM into SVM's memory
>> COLO: synchronize PVM's state to SVM periodically
>> COLO failover: Introduce a new command to trigger a failover
>> COLO failover: Introduce state to record failover process
>> COLO: Implement failover work for Primary VM
>> COLO: Implement failover work for Secondary VM
>> COLO: implement default failover treatment
>> qmp event: Add event notification for COLO error
>> COLO failover: Shutdown related socket fd when do failover
>> COLO failover: Don't do failover during loading VM's state
>> COLO: Control the checkpoint delay time by migrate-set-parameters
>> command
>> COLO: Implement shutdown checkpoint
>> COLO: Update the global runstate after going into colo state
>> savevm: Split load vm state function qemu_loadvm_state
>> COLO: Separate the process of saving/loading ram and device state
>> COLO: Split qemu_savevm_state_begin out of checkpoint process
>> COLO: Add net packets treatment into COLO
>>
>> configure | 11 +
>> docs/qmp/qmp-events.txt | 17 +
>> hmp-commands.hx | 15 +
>> hmp.c | 16 +
>> hmp.h | 1 +
>> include/exec/cpu-all.h | 1 +
>> include/migration/colo.h | 44 +++
>> include/migration/failover.h | 33 ++
>> include/migration/migration.h | 16 +-
>> include/migration/qemu-file.h | 3 +-
>> include/sysemu/sysemu.h | 8 +
>> migration/Makefile.objs | 2 +
>> migration/colo-comm.c | 75 ++++
>> migration/colo-failover.c | 83 +++++
>> migration/colo.c | 782 ++++++++++++++++++++++++++++++++++++++++++
>> migration/exec.c | 4 +-
>> migration/fd.c | 4 +-
>> migration/migration.c | 184 +++++++---
>> migration/qemu-file-buf.c | 58 ++++
>> migration/ram.c | 185 +++++++++-
>> migration/savevm.c | 309 +++++++++++++----
>> migration/tcp.c | 4 +-
>> migration/unix.c | 4 +-
>> qapi-schema.json | 101 +++++-
>> qapi/event.json | 17 +
>> qmp-commands.hx | 20 ++
>> stubs/Makefile.objs | 1 +
>> stubs/migration-colo.c | 45 +++
>> trace-events | 8 +
>> vl.c | 37 +-
>> 30 files changed, 1930 insertions(+), 158 deletions(-)
>> create mode 100644 include/migration/colo.h
>> create mode 100644 include/migration/failover.h
>> create mode 100644 migration/colo-comm.c
>> create mode 100644 migration/colo-failover.c
>> create mode 100644 migration/colo.c
>> create mode 100644 stubs/migration-colo.c
>>
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT)
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (32 preceding siblings ...)
2015-09-02 9:03 ` [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) Yang Hongyang
@ 2015-09-09 3:36 ` zhanghailiang
2015-09-15 10:40 ` zhanghailiang
2015-10-21 14:10 ` Dr. David Alan Gilbert
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-09 3:36 UTC (permalink / raw)
To: quintela, amit.shah; +Cc: yanghy, qemu-devel, peter.huangpeng
Ping...
Hi Juan & Amit,
Could you please help review this series ?
Since it is already comes v9, i really hope to get your feedback on this series :)
Thanks,
zhanghailiang
On 2015/9/2 16:22, zhanghailiang wrote:
> This is the 9th version of COLO.
>
> Please Note that, this version is very different from the previous versions.
> since we have decided to realize proxy in qemu, which based on slirp in qemu.
> We dropped all the original colo proxy related part.
>
> It will be a long time for proxy to be ready for merging, so here we extract
> the basic periodic checkpoint part that not depend on proxy into this series.
> Actually, the 'periodic' mode is also what we want to support in COLO, it is
> based on Yang Hongyang's netfilter series. and this mode is very like
> MicroCheckpointing and Remus.
>
> You can find the discussion about why & how to realize the colo proxy in qemu
> from the follow link:
> http://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg04069.html
>
> As usual, here is only COLO frame part, you can get the whole codes from github:
> https://github.com/coloft/qemu/commits/colo-v2.0-periodic-mode
>
> Compared with previous versions, this version is more easy to test.
>
> Test procedure:
> 1. Startup qemu
> Primary side:
> # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -netdev tap,id=bn0 -netfilter buffer,id=f0,netdev=bn0,chain=in -device virtio-net-pci,id=net-pci0,netdev=bn0 -boot c -drive if=virtio,id=disk1,driver=quorum,read-pattern=fifo,cache=none,aio=native,children.0.file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,children.0.driver=raw -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio -S
>
> Secondary side:
> # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0 -drive if=none,driver=raw,file=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,id=colo1,cache=none,aio=native -drive if=virtio,driver=replication,mode=secondary,throttling.bps-total=70000000,file.file.filename=/mnt/ramfs/active_disk.img,file.driver=qcow2,file.backing.file.filename=/mnt/ramfs/hidden_disk.img,file.backing.driver=qcow2,file.backing.backing.backing_reference=colo1,file.backing.allow-write-backing-file=on -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-table -monitor stdio -incoming tcp:0:8888
>
> 2. On Secondary VM's QEMU monitor, issue command
> (qemu) nbd_server_start 192.168.2.88:8889
> (qemu) nbd_server_add -w colo1
>
> 3. On Primary VM's QEMU monitor, issue command:
> (qemu) child_add disk1 child.driver=replication,child.mode=primary,child.file.host=192.168.2.88,child.file.port=8889,child.file.export=colo1,child.file.driver=nbd,child.ignore-errors=on
> (qemu) migrate_set_capability colo on
> (qemu) migrate tcp:192.168.2.88:8888
>
> 4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
> You can by issue command "migrate_set_parameter checkpoint-delay 2000"
> to change the checkpoint period time.
>
> 5. Failover test
> You can kill PVM and run 'colo_lost_heartbeat' in SVM's
> monitor at the same time, then SVM will failover and client will not feel this change.
>
> COLO is a totally new feature which is still in early stage,
> your comments and feedback are warmly welcomed.
>
> TODO:
> 1. checkpoint based on proxy in qemu
> 2. The capability of continuous FT
>
> v9:
> - Drop colo proxy related part (colo-nic.c file)
> - Convert COLO protocol name definition to QAPI
> - Smash failover related patch (patch 19/20/23)
> - Fix colo exit event according Eric's comments.
> - Fix some typos from Eric's comments
> - Fix bug 'invalid runstate transition: 'colo' -> 'prelaunch' reported
> by Dave (patch 27)
> - Use migrate_set_parameter intead of ecolo-set-checkpoint-period to set
> checkpoint delay time (patch 25)
> - Add new patch (patch 29/30) to seperate the process of saving/loading
> device and state during checkpoint. which will reduce the data size
> for sending and also reduce the qsb size used in checkpoint.
>
> Wen Congyang (1):
> COLO: Add block replication into colo process
>
> zhanghailiang (31):
> configure: Add parameter for configure to enable/disable COLO support
> migration: Introduce capability 'colo' to migration
> COLO: migrate colo related info to slave
> migration: Add state records for migration incoming
> migration: Integrate COLO checkpoint process into migration
> migration: Integrate COLO checkpoint process into loadvm
> migration: Rename the'file' member of MigrationState and
> MigrationIncomingState
> COLO/migration: establish a new communication path from destination to
> source
> COLO: Implement colo checkpoint protocol
> COLO: Add a new RunState RUN_STATE_COLO
> QEMUSizedBuffer: Introduce two help functions for qsb
> COLO: Save PVM state to secondary side when do checkpoint
> COLO: Load PVM's dirty pages into SVM's RAM cache temporarily
> COLO: Load VMState into qsb before restore it
> COLO: Flush PVM's cached RAM into SVM's memory
> COLO: synchronize PVM's state to SVM periodically
> COLO failover: Introduce a new command to trigger a failover
> COLO failover: Introduce state to record failover process
> COLO: Implement failover work for Primary VM
> COLO: Implement failover work for Secondary VM
> COLO: implement default failover treatment
> qmp event: Add event notification for COLO error
> COLO failover: Shutdown related socket fd when do failover
> COLO failover: Don't do failover during loading VM's state
> COLO: Control the checkpoint delay time by migrate-set-parameters
> command
> COLO: Implement shutdown checkpoint
> COLO: Update the global runstate after going into colo state
> savevm: Split load vm state function qemu_loadvm_state
> COLO: Separate the process of saving/loading ram and device state
> COLO: Split qemu_savevm_state_begin out of checkpoint process
> COLO: Add net packets treatment into COLO
>
> configure | 11 +
> docs/qmp/qmp-events.txt | 17 +
> hmp-commands.hx | 15 +
> hmp.c | 16 +
> hmp.h | 1 +
> include/exec/cpu-all.h | 1 +
> include/migration/colo.h | 44 +++
> include/migration/failover.h | 33 ++
> include/migration/migration.h | 16 +-
> include/migration/qemu-file.h | 3 +-
> include/sysemu/sysemu.h | 8 +
> migration/Makefile.objs | 2 +
> migration/colo-comm.c | 75 ++++
> migration/colo-failover.c | 83 +++++
> migration/colo.c | 782 ++++++++++++++++++++++++++++++++++++++++++
> migration/exec.c | 4 +-
> migration/fd.c | 4 +-
> migration/migration.c | 184 +++++++---
> migration/qemu-file-buf.c | 58 ++++
> migration/ram.c | 185 +++++++++-
> migration/savevm.c | 309 +++++++++++++----
> migration/tcp.c | 4 +-
> migration/unix.c | 4 +-
> qapi-schema.json | 101 +++++-
> qapi/event.json | 17 +
> qmp-commands.hx | 20 ++
> stubs/Makefile.objs | 1 +
> stubs/migration-colo.c | 45 +++
> trace-events | 8 +
> vl.c | 37 +-
> 30 files changed, 1930 insertions(+), 158 deletions(-)
> create mode 100644 include/migration/colo.h
> create mode 100644 include/migration/failover.h
> create mode 100644 migration/colo-comm.c
> create mode 100644 migration/colo-failover.c
> create mode 100644 migration/colo.c
> create mode 100644 stubs/migration-colo.c
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT)
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (33 preceding siblings ...)
2015-09-09 3:36 ` zhanghailiang
@ 2015-09-15 10:40 ` zhanghailiang
2015-10-21 14:10 ` Dr. David Alan Gilbert
35 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-09-15 10:40 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy
Ping again ...
On 2015/9/2 16:22, zhanghailiang wrote:
> This is the 9th version of COLO.
>
> Please Note that, this version is very different from the previous versions.
> since we have decided to realize proxy in qemu, which based on slirp in qemu.
> We dropped all the original colo proxy related part.
>
> It will be a long time for proxy to be ready for merging, so here we extract
> the basic periodic checkpoint part that not depend on proxy into this series.
> Actually, the 'periodic' mode is also what we want to support in COLO, it is
> based on Yang Hongyang's netfilter series. and this mode is very like
> MicroCheckpointing and Remus.
>
> You can find the discussion about why & how to realize the colo proxy in qemu
> from the follow link:
> http://lists.nongnu.org/archive/html/qemu-devel/2015-07/msg04069.html
>
> As usual, here is only COLO frame part, you can get the whole codes from github:
> https://github.com/coloft/qemu/commits/colo-v2.0-periodic-mode
>
> Compared with previous versions, this version is more easy to test.
>
> Test procedure:
> 1. Startup qemu
> Primary side:
> # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -netdev tap,id=bn0 -netfilter buffer,id=f0,netdev=bn0,chain=in -device virtio-net-pci,id=net-pci0,netdev=bn0 -boot c -drive if=virtio,id=disk1,driver=quorum,read-pattern=fifo,cache=none,aio=native,children.0.file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,children.0.driver=raw -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-tablet -monitor stdio -S
>
> Secondary side:
> # x86_64-softmmu/qemu-system-x86_64 -enable-kvm -netdev tap,id=bn0 -device virtio-net-pci,id=net-pci0,netdev=bn0 -drive if=none,driver=raw,file=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,id=colo1,cache=none,aio=native -drive if=virtio,driver=replication,mode=secondary,throttling.bps-total=70000000,file.file.filename=/mnt/ramfs/active_disk.img,file.driver=qcow2,file.backing.file.filename=/mnt/ramfs/hidden_disk.img,file.backing.driver=qcow2,file.backing.backing.backing_reference=colo1,file.backing.allow-write-backing-file=on -vnc :7 -m 2048 -smp 2 -device piix3-usb-uhci -device usb-table -monitor stdio -incoming tcp:0:8888
>
> 2. On Secondary VM's QEMU monitor, issue command
> (qemu) nbd_server_start 192.168.2.88:8889
> (qemu) nbd_server_add -w colo1
>
> 3. On Primary VM's QEMU monitor, issue command:
> (qemu) child_add disk1 child.driver=replication,child.mode=primary,child.file.host=192.168.2.88,child.file.port=8889,child.file.export=colo1,child.file.driver=nbd,child.ignore-errors=on
> (qemu) migrate_set_capability colo on
> (qemu) migrate tcp:192.168.2.88:8888
>
> 4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
> You can by issue command "migrate_set_parameter checkpoint-delay 2000"
> to change the checkpoint period time.
>
> 5. Failover test
> You can kill PVM and run 'colo_lost_heartbeat' in SVM's
> monitor at the same time, then SVM will failover and client will not feel this change.
>
> COLO is a totally new feature which is still in early stage,
> your comments and feedback are warmly welcomed.
>
> TODO:
> 1. checkpoint based on proxy in qemu
> 2. The capability of continuous FT
>
> v9:
> - Drop colo proxy related part (colo-nic.c file)
> - Convert COLO protocol name definition to QAPI
> - Smash failover related patch (patch 19/20/23)
> - Fix colo exit event according Eric's comments.
> - Fix some typos from Eric's comments
> - Fix bug 'invalid runstate transition: 'colo' -> 'prelaunch' reported
> by Dave (patch 27)
> - Use migrate_set_parameter intead of ecolo-set-checkpoint-period to set
> checkpoint delay time (patch 25)
> - Add new patch (patch 29/30) to seperate the process of saving/loading
> device and state during checkpoint. which will reduce the data size
> for sending and also reduce the qsb size used in checkpoint.
>
> Wen Congyang (1):
> COLO: Add block replication into colo process
>
> zhanghailiang (31):
> configure: Add parameter for configure to enable/disable COLO support
> migration: Introduce capability 'colo' to migration
> COLO: migrate colo related info to slave
> migration: Add state records for migration incoming
> migration: Integrate COLO checkpoint process into migration
> migration: Integrate COLO checkpoint process into loadvm
> migration: Rename the'file' member of MigrationState and
> MigrationIncomingState
> COLO/migration: establish a new communication path from destination to
> source
> COLO: Implement colo checkpoint protocol
> COLO: Add a new RunState RUN_STATE_COLO
> QEMUSizedBuffer: Introduce two help functions for qsb
> COLO: Save PVM state to secondary side when do checkpoint
> COLO: Load PVM's dirty pages into SVM's RAM cache temporarily
> COLO: Load VMState into qsb before restore it
> COLO: Flush PVM's cached RAM into SVM's memory
> COLO: synchronize PVM's state to SVM periodically
> COLO failover: Introduce a new command to trigger a failover
> COLO failover: Introduce state to record failover process
> COLO: Implement failover work for Primary VM
> COLO: Implement failover work for Secondary VM
> COLO: implement default failover treatment
> qmp event: Add event notification for COLO error
> COLO failover: Shutdown related socket fd when do failover
> COLO failover: Don't do failover during loading VM's state
> COLO: Control the checkpoint delay time by migrate-set-parameters
> command
> COLO: Implement shutdown checkpoint
> COLO: Update the global runstate after going into colo state
> savevm: Split load vm state function qemu_loadvm_state
> COLO: Separate the process of saving/loading ram and device state
> COLO: Split qemu_savevm_state_begin out of checkpoint process
> COLO: Add net packets treatment into COLO
>
> configure | 11 +
> docs/qmp/qmp-events.txt | 17 +
> hmp-commands.hx | 15 +
> hmp.c | 16 +
> hmp.h | 1 +
> include/exec/cpu-all.h | 1 +
> include/migration/colo.h | 44 +++
> include/migration/failover.h | 33 ++
> include/migration/migration.h | 16 +-
> include/migration/qemu-file.h | 3 +-
> include/sysemu/sysemu.h | 8 +
> migration/Makefile.objs | 2 +
> migration/colo-comm.c | 75 ++++
> migration/colo-failover.c | 83 +++++
> migration/colo.c | 782 ++++++++++++++++++++++++++++++++++++++++++
> migration/exec.c | 4 +-
> migration/fd.c | 4 +-
> migration/migration.c | 184 +++++++---
> migration/qemu-file-buf.c | 58 ++++
> migration/ram.c | 185 +++++++++-
> migration/savevm.c | 309 +++++++++++++----
> migration/tcp.c | 4 +-
> migration/unix.c | 4 +-
> qapi-schema.json | 101 +++++-
> qapi/event.json | 17 +
> qmp-commands.hx | 20 ++
> stubs/Makefile.objs | 1 +
> stubs/migration-colo.c | 45 +++
> trace-events | 8 +
> vl.c | 37 +-
> 30 files changed, 1930 insertions(+), 158 deletions(-)
> create mode 100644 include/migration/colo.h
> create mode 100644 include/migration/failover.h
> create mode 100644 migration/colo-comm.c
> create mode 100644 migration/colo-failover.c
> create mode 100644 migration/colo.c
> create mode 100644 stubs/migration-colo.c
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 01/32] configure: Add parameter for configure to enable/disable COLO support
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 01/32] configure: Add parameter for configure to enable/disable COLO support zhanghailiang
@ 2015-10-02 15:10 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 59+ messages in thread
From: Dr. David Alan Gilbert @ 2015-10-02 15:10 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> configure --enable-colo/--disable-colo to switch COLO
> support on/off.
> COLO support is off by default.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> configure | 11 +++++++++++
> 1 file changed, 11 insertions(+)
>
> diff --git a/configure b/configure
> index 9d24d59..e9fff7b 100755
> --- a/configure
> +++ b/configure
> @@ -258,6 +258,7 @@ xfs=""
> vhost_net="no"
> vhost_scsi="no"
> kvm="no"
> +colo="no"
> rdma=""
> gprof="no"
> debug_tcg="no"
> @@ -929,6 +930,10 @@ for opt do
> ;;
> --enable-kvm) kvm="yes"
> ;;
> + --disable-colo) colo="no"
> + ;;
> + --enable-colo) colo="yes"
> + ;;
> --disable-tcg-interpreter) tcg_interpreter="no"
> ;;
> --enable-tcg-interpreter) tcg_interpreter="yes"
> @@ -1339,6 +1344,7 @@ disabled with --disable-FEATURE, default is enabled if available:
> fdt fdt device tree
> bluez bluez stack connectivity
> kvm KVM acceleration support
> + colo COarse-grain LOck-stepping VM for Non-stop Service
> rdma RDMA-based migration support
> uuid uuid support
> vde support for vde network
> @@ -4545,6 +4551,7 @@ echo "Linux AIO support $linux_aio"
> echo "ATTR/XATTR support $attr"
> echo "Install blobs $blobs"
> echo "KVM support $kvm"
> +echo "COLO support $colo"
> echo "RDMA support $rdma"
> echo "TCG interpreter $tcg_interpreter"
> echo "fdt support $fdt"
> @@ -5119,6 +5126,10 @@ if have_backend "ftrace"; then
> fi
> echo "CONFIG_TRACE_FILE=$trace_file" >> $config_host_mak
>
> +if test "$colo" = "yes"; then
> + echo "CONFIG_COLO=y" >> $config_host_mak
> +fi
> +
> if test "$rdma" = "yes" ; then
> echo "CONFIG_RDMA=y" >> $config_host_mak
> fi
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 02/32] migration: Introduce capability 'colo' to migration
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 02/32] migration: Introduce capability 'colo' to migration zhanghailiang
@ 2015-10-02 16:02 ` Eric Blake
2015-10-08 6:34 ` zhanghailiang
0 siblings, 1 reply; 59+ messages in thread
From: Eric Blake @ 2015-10-02 16:02 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, arei.gonglei, stefanha,
amit.shah, yanghy
[-- Attachment #1: Type: text/plain, Size: 5542 bytes --]
On 09/02/2015 02:22 AM, zhanghailiang wrote:
> We add helper function colo_supported() to indicate whether
> colo is supported or not, with which we use to control whether or not
> showing 'colo' string to users, they can use qmp command
> 'query-migrate-capabilities' or hmp command 'info migrate_capabilities'
> to learn if colo is supported.
>
> Cc: Juan Quintela <quintela@redhat.com>
> Cc: Amit Shah <amit.shah@redhat.com>
> Cc: Eric Blake <eblake@redhat.com>
> Cc: Markus Armbruster <armbru@redhat.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
> +++ b/migration/colo.c
> @@ -0,0 +1,18 @@
> +/*
> + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
> + * (a.k.a. Fault Tolerance or Continuous Replication)
> + *
> + * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
> + * Copyright (c) 2015 FUJITSU LIMITED
> + * Copyright (c) 2015 Intel Corporation
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or
> + * later. See the COPYING file in the top-level directory.
> + */
> +
> +#include "migration/colo.h"
> +
> +bool colo_supported(void)
> +{
> + return true;
> +}
So this exists because you have a configure option that says whether to
include or exclude this .o file (the backup stub version returns false
if this file is not included)...
> diff --git a/migration/migration.c b/migration/migration.c
> index 662e77e..593cac0 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -29,6 +29,7 @@
> #include "trace.h"
> #include "qapi/util.h"
> #include "qapi-event.h"
> +#include "migration/colo.h"
>
> #define MAX_THROTTLE (32 << 20) /* Migration speed throttling */
>
> @@ -345,6 +346,9 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
>
> caps = NULL; /* silence compiler warning */
> for (i = 0; i < MIGRATION_CAPABILITY_MAX; i++) {
> + if (i == MIGRATION_CAPABILITY_COLO && !colo_supported()) {
> + continue;
> + }
...and here, you use the result to explicitly remove the colo migration
property from the runtime result of query-migrate-capabilities if
support was not compiled in. That's a good thing, because it means that
a client can call query-migrate-capabilities, and immediately know if
qemu supports colo by whether the migration property is present.
Especially since the new query-qmp-schema WILL show colo as a member of
the enum of possible values. Knowing that the enum supports a value
doesn't tell you whether runtime supports use of the value, so your
approach definitely helps in a place where static introspection doesn't
solve everything.
> if (head == NULL) {
> head = g_malloc0(sizeof(*caps));
> caps = head;
> @@ -485,6 +489,13 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
> }
>
> for (cap = params; cap; cap = cap->next) {
> + if (cap->value->capability == MIGRATION_CAPABILITY_COLO &&
> + !colo_supported()) {
> + error_setg(errp, "COLO is not currently supported, please"
> + " configure with --enable-colo option in order to"
> + " support COLO feature");
> + continue;
Likewise, you explicitly reject attempts to change the property, where
introspection says the enum supports the property. It might be
acceptable to silently permit attempts to set the capability to false
(the value it already has) and only reject attempts to set it to true,
but I'm not sure if the difference is worth it.
> + }
> s->enabled_capabilities[cap->value->capability] = cap->value->state;
> }
> }
> @@ -913,6 +924,12 @@ int64_t migrate_xbzrle_cache_size(void)
> return s->xbzrle_cache_size;
> }
>
> +bool migrate_enable_colo(void)
> +{
> + MigrationState *s = migrate_get_current();
> + return s->enabled_capabilities[MIGRATION_CAPABILITY_COLO];
Function name is wrong - it sounds like an imperative command (call this
to enable colo), when it is really a query command (call this to learn
if colo is enabled). Maybe migrate_colo_enabled()?
> +}
> +
> /* migration thread support */
>
> static void *migration_thread(void *opaque)
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 5f45571..b14d1f4 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -529,11 +529,15 @@
> # @auto-converge: If enabled, QEMU will automatically throttle down the guest
> # to speed up convergence of RAM migration. (since 1.6)
> #
> +# @colo: If enabled, migration will never end, and the state of the VM on the
> +# primary side will be migrated continuously to the VM on secondary
> +# side. (since 2.5)
You may want to name this property 'x-colo' to mark it experimental;
especially since there are still a lot of other things waiting to go in
and we are getting closer to 2.5 freeze. Making the property
experimental will relieve the pressure of having to get everything right
on the first try, although it also means that libvirt won't use the
property until we graduate it from experimental 'x-colo' to
fully-supported 'colo'.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 03/32] COLO: migrate colo related info to slave
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 03/32] COLO: migrate colo related info to slave zhanghailiang
@ 2015-10-02 18:45 ` Dr. David Alan Gilbert
2015-10-08 6:48 ` zhanghailiang
0 siblings, 1 reply; 59+ messages in thread
From: Dr. David Alan Gilbert @ 2015-10-02 18:45 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> We can know if VM in destination should go into COLO mode by refer to
> the info that been migrated from PVM.
>
> We skip this section if colo is not enabled (i.e.
> migrate_set_capability colo off), so that, It not break compatibility with migration
> however the --enable-colo/disable-colo on the source/destination;
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
> include/migration/colo.h | 2 ++
> migration/Makefile.objs | 1 +
> migration/colo-comm.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++
> vl.c | 3 ++-
> 4 files changed, 59 insertions(+), 1 deletion(-)
> create mode 100644 migration/colo-comm.c
>
> diff --git a/include/migration/colo.h b/include/migration/colo.h
> index c60a590..9b6662d 100644
> --- a/include/migration/colo.h
> +++ b/include/migration/colo.h
> @@ -14,7 +14,9 @@
> #define QEMU_COLO_H
>
> #include "qemu-common.h"
> +#include "migration/migration.h"
>
> bool colo_supported(void);
> +void colo_info_mig_init(void);
>
> #endif
> diff --git a/migration/Makefile.objs b/migration/Makefile.objs
> index 5a25d39..cb7bd30 100644
> --- a/migration/Makefile.objs
> +++ b/migration/Makefile.objs
> @@ -1,5 +1,6 @@
> common-obj-y += migration.o tcp.o
> common-obj-$(CONFIG_COLO) += colo.o
> +common-obj-y += colo-comm.o
> common-obj-y += vmstate.o
> common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
> common-obj-y += xbzrle.o
> diff --git a/migration/colo-comm.c b/migration/colo-comm.c
> new file mode 100644
> index 0000000..4330bd8
> --- /dev/null
> +++ b/migration/colo-comm.c
> @@ -0,0 +1,54 @@
> +/*
> + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
> + * (a.k.a. Fault Tolerance or Continuous Replication)
> + *
> + * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
> + * Copyright (c) 2015 FUJITSU LIMITED
> + * Copyright (c) 2015 Intel Corporation
> + *
> + * This work is licensed under the terms of the GNU GPL, version 2 or
> + * later. See the COPYING file in the top-level directory.
> + *
> + */
> +
> +#include <migration/colo.h>
> +#include "trace.h"
> +
> +typedef struct {
> + uint32_t colo_requested;
> +} COLOInfo;
> +
> +static COLOInfo colo_info;
> +
> +static void colo_info_pre_save(void *opaque)
> +{
> + COLOInfo *s = opaque;
> +
> + if (migrate_enable_colo()) {
> + s->colo_requested = 1;
> + } else {
> + s->colo_requested = 0;
> + }
> +}
> +
> +static bool colo_info_need(void *opaque)
> +{
> + return migrate_enable_colo();
> +}
> +
> +static const VMStateDescription colo_state = {
> + .name = "COLOState",
> + .version_id = 1,
> + .minimum_version_id = 1,
> + .pre_save = colo_info_pre_save,
> + .needed = colo_info_need,
> + .fields = (VMStateField[]) {
> + VMSTATE_UINT32(colo_requested, COLOInfo),
Any reason not to do it as a bool, and use VMSTATE_BOOL?
(That would also simplify the pre_save).
(Oddly you don't actually need the state at all - just
the presence of the section is enough, but I don't worry
about that).
Dave
> + VMSTATE_END_OF_LIST()
> + },
> +};
> +
> +void colo_info_mig_init(void)
> +{
> + vmstate_register(NULL, 0, &colo_state, &colo_info);
> +}
> diff --git a/vl.c b/vl.c
> index aee931a..571347a 100644
> --- a/vl.c
> +++ b/vl.c
> @@ -92,6 +92,7 @@ int main(int argc, char **argv)
> #include "sysemu/dma.h"
> #include "audio/audio.h"
> #include "migration/migration.h"
> +#include "migration/colo.h"
> #include "sysemu/kvm.h"
> #include "qapi/qmp/qjson.h"
> #include "qemu/option.h"
> @@ -4446,7 +4447,7 @@ int main(int argc, char **argv, char **envp)
>
> blk_mig_init();
> ram_mig_init();
> -
> + colo_info_mig_init();
> /* If the currently selected machine wishes to override the units-per-bus
> * property of its default HBA interface type, do so now. */
> if (machine_class->units_per_default_bus) {
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 02/32] migration: Introduce capability 'colo' to migration
2015-10-02 16:02 ` Eric Blake
@ 2015-10-08 6:34 ` zhanghailiang
0 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-10-08 6:34 UTC (permalink / raw)
To: Eric Blake, qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, arei.gonglei, stefanha,
amit.shah, yanghy
On 2015/10/3 0:02, Eric Blake wrote:
> On 09/02/2015 02:22 AM, zhanghailiang wrote:
>> We add helper function colo_supported() to indicate whether
>> colo is supported or not, with which we use to control whether or not
>> showing 'colo' string to users, they can use qmp command
>> 'query-migrate-capabilities' or hmp command 'info migrate_capabilities'
>> to learn if colo is supported.
>>
>> Cc: Juan Quintela <quintela@redhat.com>
>> Cc: Amit Shah <amit.shah@redhat.com>
>> Cc: Eric Blake <eblake@redhat.com>
>> Cc: Markus Armbruster <armbru@redhat.com>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>
>> +++ b/migration/colo.c
>> @@ -0,0 +1,18 @@
>> +/*
>> + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
>> + * (a.k.a. Fault Tolerance or Continuous Replication)
>> + *
>> + * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
>> + * Copyright (c) 2015 FUJITSU LIMITED
>> + * Copyright (c) 2015 Intel Corporation
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or
>> + * later. See the COPYING file in the top-level directory.
>> + */
>> +
>> +#include "migration/colo.h"
>> +
>> +bool colo_supported(void)
>> +{
>> + return true;
>> +}
>
> So this exists because you have a configure option that says whether to
> include or exclude this .o file (the backup stub version returns false
> if this file is not included)...
>
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 662e77e..593cac0 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -29,6 +29,7 @@
>> #include "trace.h"
>> #include "qapi/util.h"
>> #include "qapi-event.h"
>> +#include "migration/colo.h"
>>
>> #define MAX_THROTTLE (32 << 20) /* Migration speed throttling */
>>
>> @@ -345,6 +346,9 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
>>
>> caps = NULL; /* silence compiler warning */
>> for (i = 0; i < MIGRATION_CAPABILITY_MAX; i++) {
>> + if (i == MIGRATION_CAPABILITY_COLO && !colo_supported()) {
>> + continue;
>> + }
>
> ...and here, you use the result to explicitly remove the colo migration
> property from the runtime result of query-migrate-capabilities if
> support was not compiled in. That's a good thing, because it means that
> a client can call query-migrate-capabilities, and immediately know if
> qemu supports colo by whether the migration property is present.
>
> Especially since the new query-qmp-schema WILL show colo as a member of
> the enum of possible values. Knowing that the enum supports a value
> doesn't tell you whether runtime supports use of the value, so your
> approach definitely helps in a place where static introspection doesn't
> solve everything.
>
Yes, that is the reason we add it.
>> if (head == NULL) {
>> head = g_malloc0(sizeof(*caps));
>> caps = head;
>> @@ -485,6 +489,13 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
>> }
>>
>> for (cap = params; cap; cap = cap->next) {
>> + if (cap->value->capability == MIGRATION_CAPABILITY_COLO &&
>> + !colo_supported()) {
>> + error_setg(errp, "COLO is not currently supported, please"
>> + " configure with --enable-colo option in order to"
>> + " support COLO feature");
>> + continue;
>
> Likewise, you explicitly reject attempts to change the property, where
> introspection says the enum supports the property. It might be
> acceptable to silently permit attempts to set the capability to false
> (the value it already has) and only reject attempts to set it to true,
> but I'm not sure if the difference is worth it.
>
Ha, maybe you have forgotten, in last version, you pointed out that maybe it was
better to give this error message regardless of true/false being requested,
and i think it is reasonable, so i will keep it like this ;)
>> + }
>> s->enabled_capabilities[cap->value->capability] = cap->value->state;
>> }
>> }
>> @@ -913,6 +924,12 @@ int64_t migrate_xbzrle_cache_size(void)
>> return s->xbzrle_cache_size;
>> }
>>
>> +bool migrate_enable_colo(void)
>> +{
>> + MigrationState *s = migrate_get_current();
>> + return s->enabled_capabilities[MIGRATION_CAPABILITY_COLO];
>
> Function name is wrong - it sounds like an imperative command (call this
> to enable colo), when it is really a query command (call this to learn
> if colo is enabled). Maybe migrate_colo_enabled()?
>
Sounds reasonable, will fix in next version.
>> +}
>> +
>> /* migration thread support */
>>
>> static void *migration_thread(void *opaque)
>> diff --git a/qapi-schema.json b/qapi-schema.json
>> index 5f45571..b14d1f4 100644
>> --- a/qapi-schema.json
>> +++ b/qapi-schema.json
>> @@ -529,11 +529,15 @@
>> # @auto-converge: If enabled, QEMU will automatically throttle down the guest
>> # to speed up convergence of RAM migration. (since 1.6)
>> #
>> +# @colo: If enabled, migration will never end, and the state of the VM on the
>> +# primary side will be migrated continuously to the VM on secondary
>> +# side. (since 2.5)
>
> You may want to name this property 'x-colo' to mark it experimental;
> especially since there are still a lot of other things waiting to go in
> and we are getting closer to 2.5 freeze. Making the property
> experimental will relieve the pressure of having to get everything right
> on the first try, although it also means that libvirt won't use the
> property until we graduate it from experimental 'x-colo' to
> fully-supported 'colo'.
>
Got it, will fix it in next version, thanks.
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 03/32] COLO: migrate colo related info to slave
2015-10-02 18:45 ` Dr. David Alan Gilbert
@ 2015-10-08 6:48 ` zhanghailiang
0 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-10-08 6:48 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
On 2015/10/3 2:45, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> We can know if VM in destination should go into COLO mode by refer to
>> the info that been migrated from PVM.
>>
>> We skip this section if colo is not enabled (i.e.
>> migrate_set_capability colo off), so that, It not break compatibility with migration
>> however the --enable-colo/disable-colo on the source/destination;
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>> include/migration/colo.h | 2 ++
>> migration/Makefile.objs | 1 +
>> migration/colo-comm.c | 54 ++++++++++++++++++++++++++++++++++++++++++++++++
>> vl.c | 3 ++-
>> 4 files changed, 59 insertions(+), 1 deletion(-)
>> create mode 100644 migration/colo-comm.c
>>
>> diff --git a/include/migration/colo.h b/include/migration/colo.h
>> index c60a590..9b6662d 100644
>> --- a/include/migration/colo.h
>> +++ b/include/migration/colo.h
>> @@ -14,7 +14,9 @@
>> #define QEMU_COLO_H
>>
>> #include "qemu-common.h"
>> +#include "migration/migration.h"
>>
>> bool colo_supported(void);
>> +void colo_info_mig_init(void);
>>
>> #endif
>> diff --git a/migration/Makefile.objs b/migration/Makefile.objs
>> index 5a25d39..cb7bd30 100644
>> --- a/migration/Makefile.objs
>> +++ b/migration/Makefile.objs
>> @@ -1,5 +1,6 @@
>> common-obj-y += migration.o tcp.o
>> common-obj-$(CONFIG_COLO) += colo.o
>> +common-obj-y += colo-comm.o
>> common-obj-y += vmstate.o
>> common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
>> common-obj-y += xbzrle.o
>> diff --git a/migration/colo-comm.c b/migration/colo-comm.c
>> new file mode 100644
>> index 0000000..4330bd8
>> --- /dev/null
>> +++ b/migration/colo-comm.c
>> @@ -0,0 +1,54 @@
>> +/*
>> + * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
>> + * (a.k.a. Fault Tolerance or Continuous Replication)
>> + *
>> + * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
>> + * Copyright (c) 2015 FUJITSU LIMITED
>> + * Copyright (c) 2015 Intel Corporation
>> + *
>> + * This work is licensed under the terms of the GNU GPL, version 2 or
>> + * later. See the COPYING file in the top-level directory.
>> + *
>> + */
>> +
>> +#include <migration/colo.h>
>> +#include "trace.h"
>> +
>> +typedef struct {
>> + uint32_t colo_requested;
>> +} COLOInfo;
>> +
>> +static COLOInfo colo_info;
>> +
>> +static void colo_info_pre_save(void *opaque)
>> +{
>> + COLOInfo *s = opaque;
>> +
>> + if (migrate_enable_colo()) {
>> + s->colo_requested = 1;
>> + } else {
>> + s->colo_requested = 0;
>> + }
>> +}
>> +
>> +static bool colo_info_need(void *opaque)
>> +{
>> + return migrate_enable_colo();
>> +}
>> +
>> +static const VMStateDescription colo_state = {
>> + .name = "COLOState",
>> + .version_id = 1,
>> + .minimum_version_id = 1,
>> + .pre_save = colo_info_pre_save,
>> + .needed = colo_info_need,
>> + .fields = (VMStateField[]) {
>> + VMSTATE_UINT32(colo_requested, COLOInfo),
>
> Any reason not to do it as a bool, and use VMSTATE_BOOL?
> (That would also simplify the pre_save).
>
No special reason, use VMSTATE_BOOL is a good idea, will fix
in next version, thanks.
zhanghailiang
> (Oddly you don't actually need the state at all - just
> the presence of the section is enough, but I don't worry
> about that).
>
> Dave
>
>> + VMSTATE_END_OF_LIST()
>> + },
>> +};
>> +
>> +void colo_info_mig_init(void)
>> +{
>> + vmstate_register(NULL, 0, &colo_state, &colo_info);
>> +}
>> diff --git a/vl.c b/vl.c
>> index aee931a..571347a 100644
>> --- a/vl.c
>> +++ b/vl.c
>> @@ -92,6 +92,7 @@ int main(int argc, char **argv)
>> #include "sysemu/dma.h"
>> #include "audio/audio.h"
>> #include "migration/migration.h"
>> +#include "migration/colo.h"
>> #include "sysemu/kvm.h"
>> #include "qapi/qmp/qjson.h"
>> #include "qemu/option.h"
>> @@ -4446,7 +4447,7 @@ int main(int argc, char **argv, char **envp)
>>
>> blk_mig_init();
>> ram_mig_init();
>> -
>> + colo_info_mig_init();
>> /* If the currently selected machine wishes to override the units-per-bus
>> * property of its default HBA interface type, do so now. */
>> if (machine_class->units_per_default_bus) {
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 04/32] migration: Add state records for migration incoming
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 04/32] migration: Add state records for migration incoming zhanghailiang
@ 2015-10-09 16:18 ` Dr. David Alan Gilbert
2015-10-10 7:07 ` zhanghailiang
0 siblings, 1 reply; 59+ messages in thread
From: Dr. David Alan Gilbert @ 2015-10-09 16:18 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, jdenemar, yanghy
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> For migration destination, we also need to know its state,
> we will use it in COLO.
>
> Here we add a new member 'state' for MigrationIncomingState,
> and also use migrate_set_state() to modify its value.
> We fix the first parameter of migrate_set_state(), and make it
> public.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
You should split this patch out and submit it by itself; it can go in
without waiting for any other COLO bits.
Dave
> ---
> include/migration/migration.h | 3 +++
> migration/migration.c | 38 +++++++++++++++++++++++---------------
> 2 files changed, 26 insertions(+), 15 deletions(-)
>
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index 05de3a1..a62068f 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -50,6 +50,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
> struct MigrationIncomingState {
> QEMUFile *file;
>
> + int state;
> /* See savevm.c */
> LoadStateEntry_Head loadvm_handlers;
> };
> @@ -82,6 +83,8 @@ struct MigrationState
> int64_t dirty_sync_count;
> };
>
> +void migrate_set_state(int *state, int old_state, int new_state);
> +
> void process_incoming_migration(QEMUFile *f);
>
> void qemu_start_incoming_migration(const char *uri, Error **errp);
> diff --git a/migration/migration.c b/migration/migration.c
> index 593cac0..98133f1 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -89,6 +89,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
> {
> mis_current = g_malloc0(sizeof(MigrationIncomingState));
> mis_current->file = f;
> + mis_current->state = MIGRATION_STATUS_NONE;
> QLIST_INIT(&mis_current->loadvm_handlers);
>
> return mis_current;
> @@ -270,11 +271,13 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)
> static void process_incoming_migration_co(void *opaque)
> {
> QEMUFile *f = opaque;
> + MigrationIncomingState *mis;
> Error *local_err = NULL;
> int ret;
>
> - migration_incoming_state_new(f);
> - migrate_generate_event(MIGRATION_STATUS_ACTIVE);
> + mis = migration_incoming_state_new(f);
> + migrate_set_state(&mis->state, MIGRATION_STATUS_NONE,
> + MIGRATION_STATUS_ACTIVE);
> ret = qemu_loadvm_state(f);
>
> qemu_fclose(f);
> @@ -282,12 +285,14 @@ static void process_incoming_migration_co(void *opaque)
> migration_incoming_state_destroy();
>
> if (ret < 0) {
> - migrate_generate_event(MIGRATION_STATUS_FAILED);
> + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> + MIGRATION_STATUS_FAILED);
> error_report("load of migration failed: %s", strerror(-ret));
> migrate_decompress_threads_join();
> exit(EXIT_FAILURE);
> }
> - migrate_generate_event(MIGRATION_STATUS_COMPLETED);
> + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> + MIGRATION_STATUS_COMPLETED);
> qemu_announce_self();
>
> /* Make sure all file formats flush their mutable metadata */
> @@ -543,9 +548,9 @@ void qmp_migrate_set_parameters(bool has_compress_level,
>
> /* shared migration helpers */
>
> -static void migrate_set_state(MigrationState *s, int old_state, int new_state)
> +void migrate_set_state(int *state, int old_state, int new_state)
> {
> - if (atomic_cmpxchg(&s->state, old_state, new_state) == old_state) {
> + if (atomic_cmpxchg(state, old_state, new_state) == old_state) {
> trace_migrate_set_state(new_state);
> migrate_generate_event(new_state);
> }
> @@ -574,7 +579,7 @@ static void migrate_fd_cleanup(void *opaque)
> if (s->state != MIGRATION_STATUS_COMPLETED) {
> qemu_savevm_state_cancel();
> if (s->state == MIGRATION_STATUS_CANCELLING) {
> - migrate_set_state(s, MIGRATION_STATUS_CANCELLING,
> + migrate_set_state(&s->state, MIGRATION_STATUS_CANCELLING,
> MIGRATION_STATUS_CANCELLED);
> }
> }
> @@ -586,7 +591,8 @@ void migrate_fd_error(MigrationState *s)
> {
> trace_migrate_fd_error();
> assert(s->file == NULL);
> - migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
> + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> + MIGRATION_STATUS_FAILED);
> notifier_list_notify(&migration_state_notifiers, s);
> }
>
> @@ -602,7 +608,7 @@ static void migrate_fd_cancel(MigrationState *s)
> old_state != MIGRATION_STATUS_ACTIVE) {
> break;
> }
> - migrate_set_state(s, old_state, MIGRATION_STATUS_CANCELLING);
> + migrate_set_state(&s->state, old_state, MIGRATION_STATUS_CANCELLING);
> } while (s->state != MIGRATION_STATUS_CANCELLING);
>
> /*
> @@ -670,7 +676,7 @@ static MigrationState *migrate_init(const MigrationParams *params)
> s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
> decompress_thread_count;
> s->bandwidth_limit = bandwidth_limit;
> - migrate_set_state(s, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
> + migrate_set_state(&s->state, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
>
> s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
> return s;
> @@ -768,7 +774,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
> } else {
> error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "uri",
> "a valid migration protocol");
> - migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
> + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> + MIGRATION_STATUS_FAILED);
> return;
> }
>
> @@ -948,7 +955,8 @@ static void *migration_thread(void *opaque)
> qemu_savevm_state_begin(s->file, &s->params);
>
> s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
> - migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_ACTIVE);
> + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> + MIGRATION_STATUS_ACTIVE);
>
> while (s->state == MIGRATION_STATUS_ACTIVE) {
> int64_t current_time;
> @@ -978,13 +986,13 @@ static void *migration_thread(void *opaque)
> qemu_mutex_unlock_iothread();
>
> if (ret < 0) {
> - migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
> MIGRATION_STATUS_FAILED);
> break;
> }
>
> if (!qemu_file_get_error(s->file)) {
> - migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
> MIGRATION_STATUS_COMPLETED);
> break;
> }
> @@ -992,7 +1000,7 @@ static void *migration_thread(void *opaque)
> }
>
> if (qemu_file_get_error(s->file)) {
> - migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
> MIGRATION_STATUS_FAILED);
> break;
> }
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 05/32] migration: Integrate COLO checkpoint process into migration
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 05/32] migration: Integrate COLO checkpoint process into migration zhanghailiang
@ 2015-10-09 16:53 ` Dr. David Alan Gilbert
2015-10-10 6:25 ` zhanghailiang
0 siblings, 1 reply; 59+ messages in thread
From: Dr. David Alan Gilbert @ 2015-10-09 16:53 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> Add a migrate state: MIGRATION_STATUS_COLO, enter this migration state
> after the first live migration successfully finished.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
If I understand this correctly, I think what you're doing is:
migration_thread
does initial migrate
init_checkpointer
creates a bh
main thread
bh
start_checkpointer
creates colo_thread
colo thread
outbound side loop
Why not just keep the migration thread going and reuse that?
Couldn't you just call the colo_thread() function at the place you
currently call colo_init_checkpointer, and add the init code
to the start of colo_thread?
It seems simpler.
Other than that it's OK.
Dave
> ---
> include/migration/colo.h | 3 +++
> migration/colo.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++
> migration/migration.c | 23 ++++++++++++++-----
> qapi-schema.json | 2 +-
> stubs/migration-colo.c | 9 ++++++++
> trace-events | 3 +++
> 6 files changed, 92 insertions(+), 6 deletions(-)
>
> diff --git a/include/migration/colo.h b/include/migration/colo.h
> index 9b6662d..dface19 100644
> --- a/include/migration/colo.h
> +++ b/include/migration/colo.h
> @@ -19,4 +19,7 @@
> bool colo_supported(void);
> void colo_info_mig_init(void);
>
> +void colo_init_checkpointer(MigrationState *s);
> +bool migration_in_colo_state(void);
> +
> #endif
> diff --git a/migration/colo.c b/migration/colo.c
> index 2c40d2e..97e64a3 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -10,9 +10,67 @@
> * later. See the COPYING file in the top-level directory.
> */
>
> +#include "sysemu/sysemu.h"
> #include "migration/colo.h"
> +#include "trace.h"
> +
> +static QEMUBH *colo_bh;
>
> bool colo_supported(void)
> {
> return true;
> }
> +
> +bool migration_in_colo_state(void)
> +{
> + MigrationState *s = migrate_get_current();
> +
> + return (s->state == MIGRATION_STATUS_COLO);
> +}
> +
> +static void *colo_thread(void *opaque)
> +{
> + MigrationState *s = opaque;
> +
> + qemu_mutex_lock_iothread();
> + vm_start();
> + qemu_mutex_unlock_iothread();
> + trace_colo_vm_state_change("stop", "run");
> +
> + /*TODO: COLO checkpoint savevm loop*/
> +
> + migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
> + MIGRATION_STATUS_COMPLETED);
> +
> + qemu_mutex_lock_iothread();
> + qemu_bh_schedule(s->cleanup_bh);
> + qemu_mutex_unlock_iothread();
> +
> + return NULL;
> +}
> +
> +static void colo_start_checkpointer(void *opaque)
> +{
> + MigrationState *s = opaque;
> +
> + if (colo_bh) {
> + qemu_bh_delete(colo_bh);
> + colo_bh = NULL;
> + }
> +
> + qemu_mutex_unlock_iothread();
> + qemu_thread_join(&s->thread);
> + qemu_mutex_lock_iothread();
> +
> + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
> + MIGRATION_STATUS_COLO);
> +
> + qemu_thread_create(&s->thread, "colo", colo_thread, s,
> + QEMU_THREAD_JOINABLE);
> +}
> +
> +void colo_init_checkpointer(MigrationState *s)
> +{
> + colo_bh = qemu_bh_new(colo_start_checkpointer, s);
> + qemu_bh_schedule(colo_bh);
> +}
> diff --git a/migration/migration.c b/migration/migration.c
> index 98133f1..bee61aa 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -446,6 +446,10 @@ MigrationInfo *qmp_query_migrate(Error **errp)
>
> get_xbzrle_cache_stats(info);
> break;
> + case MIGRATION_STATUS_COLO:
> + info->has_status = true;
> + /* TODO: display COLO specific information (checkpoint info etc.) */
> + break;
> case MIGRATION_STATUS_COMPLETED:
> get_xbzrle_cache_stats(info);
>
> @@ -731,7 +735,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>
> if (s->state == MIGRATION_STATUS_ACTIVE ||
> s->state == MIGRATION_STATUS_SETUP ||
> - s->state == MIGRATION_STATUS_CANCELLING) {
> + s->state == MIGRATION_STATUS_CANCELLING ||
> + s->state == MIGRATION_STATUS_COLO) {
> error_setg(errp, QERR_MIGRATION_ACTIVE);
> return;
> }
> @@ -948,6 +953,7 @@ static void *migration_thread(void *opaque)
> int64_t max_size = 0;
> int64_t start_time = initial_time;
> bool old_vm_running = false;
> + bool enable_colo = migrate_enable_colo();
>
> rcu_register_thread();
>
> @@ -992,8 +998,10 @@ static void *migration_thread(void *opaque)
> }
>
> if (!qemu_file_get_error(s->file)) {
> - migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
> - MIGRATION_STATUS_COMPLETED);
> + if (!enable_colo) {
> + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
> + MIGRATION_STATUS_COMPLETED);
> + }
> break;
> }
> }
> @@ -1044,11 +1052,16 @@ static void *migration_thread(void *opaque)
> }
> runstate_set(RUN_STATE_POSTMIGRATE);
> } else {
> - if (old_vm_running) {
> + if (s->state == MIGRATION_STATUS_ACTIVE && enable_colo) {
> + colo_init_checkpointer(s);
> + } else if (old_vm_running) {
> vm_start();
> }
> }
> - qemu_bh_schedule(s->cleanup_bh);
> +
> + if (!enable_colo || s->state != MIGRATION_STATUS_ACTIVE) {
> + qemu_bh_schedule(s->cleanup_bh);
> + }
> qemu_mutex_unlock_iothread();
>
> rcu_unregister_thread();
> diff --git a/qapi-schema.json b/qapi-schema.json
> index b14d1f4..6dd5c7c 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -433,7 +433,7 @@
> ##
> { 'enum': 'MigrationStatus',
> 'data': [ 'none', 'setup', 'cancelling', 'cancelled',
> - 'active', 'completed', 'failed' ] }
> + 'active', 'completed', 'failed', 'colo' ] }
>
> ##
> # @MigrationInfo
> diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
> index 3d817df..51b8f66 100644
> --- a/stubs/migration-colo.c
> +++ b/stubs/migration-colo.c
> @@ -16,3 +16,12 @@ bool colo_supported(void)
> {
> return false;
> }
> +
> +bool migration_in_colo_state(void)
> +{
> + return false;
> +}
> +
> +void colo_init_checkpointer(MigrationState *s)
> +{
> +}
> diff --git a/trace-events b/trace-events
> index 8f9614a..487d1c7 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -1472,6 +1472,9 @@ rdma_start_incoming_migration_after_rdma_listen(void) ""
> rdma_start_outgoing_migration_after_rdma_connect(void) ""
> rdma_start_outgoing_migration_after_rdma_source_init(void) ""
>
> +# migration/colo.c
> +colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
> +
> # kvm-all.c
> kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
> kvm_vm_ioctl(int type, void *arg) "type 0x%x, arg %p"
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 05/32] migration: Integrate COLO checkpoint process into migration
2015-10-09 16:53 ` Dr. David Alan Gilbert
@ 2015-10-10 6:25 ` zhanghailiang
0 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-10-10 6:25 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
On 2015/10/10 0:53, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> Add a migrate state: MIGRATION_STATUS_COLO, enter this migration state
>> after the first live migration successfully finished.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>
> If I understand this correctly, I think what you're doing is:
> migration_thread
> does initial migrate
> init_checkpointer
> creates a bh
>
> main thread
> bh
> start_checkpointer
> creates colo_thread
>
> colo thread
> outbound side loop
>
> Why not just keep the migration thread going and reuse that?
> Couldn't you just call the colo_thread() function at the place you
> currently call colo_init_checkpointer, and add the init code
> to the start of colo_thread?
> It seems simpler.
>
Of course, it is really a good idea to reuse migration thread,
i will fix it like that. and the helper function colo_thread will be renamed :)
Thanks,
zhanghailiang
> Other than that it's OK.
>
> Dave
>
>
>> ---
>> include/migration/colo.h | 3 +++
>> migration/colo.c | 58 ++++++++++++++++++++++++++++++++++++++++++++++++
>> migration/migration.c | 23 ++++++++++++++-----
>> qapi-schema.json | 2 +-
>> stubs/migration-colo.c | 9 ++++++++
>> trace-events | 3 +++
>> 6 files changed, 92 insertions(+), 6 deletions(-)
>>
>> diff --git a/include/migration/colo.h b/include/migration/colo.h
>> index 9b6662d..dface19 100644
>> --- a/include/migration/colo.h
>> +++ b/include/migration/colo.h
>> @@ -19,4 +19,7 @@
>> bool colo_supported(void);
>> void colo_info_mig_init(void);
>>
>> +void colo_init_checkpointer(MigrationState *s);
>> +bool migration_in_colo_state(void);
>> +
>> #endif
>> diff --git a/migration/colo.c b/migration/colo.c
>> index 2c40d2e..97e64a3 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -10,9 +10,67 @@
>> * later. See the COPYING file in the top-level directory.
>> */
>>
>> +#include "sysemu/sysemu.h"
>> #include "migration/colo.h"
>> +#include "trace.h"
>> +
>> +static QEMUBH *colo_bh;
>>
>> bool colo_supported(void)
>> {
>> return true;
>> }
>> +
>> +bool migration_in_colo_state(void)
>> +{
>> + MigrationState *s = migrate_get_current();
>> +
>> + return (s->state == MIGRATION_STATUS_COLO);
>> +}
>> +
>> +static void *colo_thread(void *opaque)
>> +{
>> + MigrationState *s = opaque;
>> +
>> + qemu_mutex_lock_iothread();
>> + vm_start();
>> + qemu_mutex_unlock_iothread();
>> + trace_colo_vm_state_change("stop", "run");
>> +
>> + /*TODO: COLO checkpoint savevm loop*/
>> +
>> + migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
>> + MIGRATION_STATUS_COMPLETED);
>> +
>> + qemu_mutex_lock_iothread();
>> + qemu_bh_schedule(s->cleanup_bh);
>> + qemu_mutex_unlock_iothread();
>> +
>> + return NULL;
>> +}
>> +
>> +static void colo_start_checkpointer(void *opaque)
>> +{
>> + MigrationState *s = opaque;
>> +
>> + if (colo_bh) {
>> + qemu_bh_delete(colo_bh);
>> + colo_bh = NULL;
>> + }
>> +
>> + qemu_mutex_unlock_iothread();
>> + qemu_thread_join(&s->thread);
>> + qemu_mutex_lock_iothread();
>> +
>> + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
>> + MIGRATION_STATUS_COLO);
>> +
>> + qemu_thread_create(&s->thread, "colo", colo_thread, s,
>> + QEMU_THREAD_JOINABLE);
>> +}
>> +
>> +void colo_init_checkpointer(MigrationState *s)
>> +{
>> + colo_bh = qemu_bh_new(colo_start_checkpointer, s);
>> + qemu_bh_schedule(colo_bh);
>> +}
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 98133f1..bee61aa 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -446,6 +446,10 @@ MigrationInfo *qmp_query_migrate(Error **errp)
>>
>> get_xbzrle_cache_stats(info);
>> break;
>> + case MIGRATION_STATUS_COLO:
>> + info->has_status = true;
>> + /* TODO: display COLO specific information (checkpoint info etc.) */
>> + break;
>> case MIGRATION_STATUS_COMPLETED:
>> get_xbzrle_cache_stats(info);
>>
>> @@ -731,7 +735,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>>
>> if (s->state == MIGRATION_STATUS_ACTIVE ||
>> s->state == MIGRATION_STATUS_SETUP ||
>> - s->state == MIGRATION_STATUS_CANCELLING) {
>> + s->state == MIGRATION_STATUS_CANCELLING ||
>> + s->state == MIGRATION_STATUS_COLO) {
>> error_setg(errp, QERR_MIGRATION_ACTIVE);
>> return;
>> }
>> @@ -948,6 +953,7 @@ static void *migration_thread(void *opaque)
>> int64_t max_size = 0;
>> int64_t start_time = initial_time;
>> bool old_vm_running = false;
>> + bool enable_colo = migrate_enable_colo();
>>
>> rcu_register_thread();
>>
>> @@ -992,8 +998,10 @@ static void *migration_thread(void *opaque)
>> }
>>
>> if (!qemu_file_get_error(s->file)) {
>> - migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
>> - MIGRATION_STATUS_COMPLETED);
>> + if (!enable_colo) {
>> + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
>> + MIGRATION_STATUS_COMPLETED);
>> + }
>> break;
>> }
>> }
>> @@ -1044,11 +1052,16 @@ static void *migration_thread(void *opaque)
>> }
>> runstate_set(RUN_STATE_POSTMIGRATE);
>> } else {
>> - if (old_vm_running) {
>> + if (s->state == MIGRATION_STATUS_ACTIVE && enable_colo) {
>> + colo_init_checkpointer(s);
>> + } else if (old_vm_running) {
>> vm_start();
>> }
>> }
>> - qemu_bh_schedule(s->cleanup_bh);
>> +
>> + if (!enable_colo || s->state != MIGRATION_STATUS_ACTIVE) {
>> + qemu_bh_schedule(s->cleanup_bh);
>> + }
>> qemu_mutex_unlock_iothread();
>>
>> rcu_unregister_thread();
>> diff --git a/qapi-schema.json b/qapi-schema.json
>> index b14d1f4..6dd5c7c 100644
>> --- a/qapi-schema.json
>> +++ b/qapi-schema.json
>> @@ -433,7 +433,7 @@
>> ##
>> { 'enum': 'MigrationStatus',
>> 'data': [ 'none', 'setup', 'cancelling', 'cancelled',
>> - 'active', 'completed', 'failed' ] }
>> + 'active', 'completed', 'failed', 'colo' ] }
>>
>> ##
>> # @MigrationInfo
>> diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
>> index 3d817df..51b8f66 100644
>> --- a/stubs/migration-colo.c
>> +++ b/stubs/migration-colo.c
>> @@ -16,3 +16,12 @@ bool colo_supported(void)
>> {
>> return false;
>> }
>> +
>> +bool migration_in_colo_state(void)
>> +{
>> + return false;
>> +}
>> +
>> +void colo_init_checkpointer(MigrationState *s)
>> +{
>> +}
>> diff --git a/trace-events b/trace-events
>> index 8f9614a..487d1c7 100644
>> --- a/trace-events
>> +++ b/trace-events
>> @@ -1472,6 +1472,9 @@ rdma_start_incoming_migration_after_rdma_listen(void) ""
>> rdma_start_outgoing_migration_after_rdma_connect(void) ""
>> rdma_start_outgoing_migration_after_rdma_source_init(void) ""
>>
>> +# migration/colo.c
>> +colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
>> +
>> # kvm-all.c
>> kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
>> kvm_vm_ioctl(int type, void *arg) "type 0x%x, arg %p"
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 04/32] migration: Add state records for migration incoming
2015-10-09 16:18 ` Dr. David Alan Gilbert
@ 2015-10-10 7:07 ` zhanghailiang
2015-10-16 11:14 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-10-10 7:07 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, jdenemar, yanghy
On 2015/10/10 0:18, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> For migration destination, we also need to know its state,
>> we will use it in COLO.
>>
>> Here we add a new member 'state' for MigrationIncomingState,
>> and also use migrate_set_state() to modify its value.
>> We fix the first parameter of migrate_set_state(), and make it
>> public.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
> You should split this patch out and submit it by itself; it can go in
> without waiting for any other COLO bits.
>
Ok, will do that.
Besides, there is still an independent patch in this series.
"[PATCH COLO-Frame v9 07/32] migration: Rename the'file' member of MigrationState and
MigrationIncomingState", i know you have done the same thing in your postcopy series.
(I noticed that you didn't change the 'file' member in MigartionState ...)
Could you send that modification as an independent series ? So that i can drop that patch in
next version.
BTW, is there any plan to merge your postcopy series ?
Thanks,
zhanghailiang
> Dave
>
>> ---
>> include/migration/migration.h | 3 +++
>> migration/migration.c | 38 +++++++++++++++++++++++---------------
>> 2 files changed, 26 insertions(+), 15 deletions(-)
>>
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index 05de3a1..a62068f 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -50,6 +50,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
>> struct MigrationIncomingState {
>> QEMUFile *file;
>>
>> + int state;
>> /* See savevm.c */
>> LoadStateEntry_Head loadvm_handlers;
>> };
>> @@ -82,6 +83,8 @@ struct MigrationState
>> int64_t dirty_sync_count;
>> };
>>
>> +void migrate_set_state(int *state, int old_state, int new_state);
>> +
>> void process_incoming_migration(QEMUFile *f);
>>
>> void qemu_start_incoming_migration(const char *uri, Error **errp);
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 593cac0..98133f1 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -89,6 +89,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
>> {
>> mis_current = g_malloc0(sizeof(MigrationIncomingState));
>> mis_current->file = f;
>> + mis_current->state = MIGRATION_STATUS_NONE;
>> QLIST_INIT(&mis_current->loadvm_handlers);
>>
>> return mis_current;
>> @@ -270,11 +271,13 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)
>> static void process_incoming_migration_co(void *opaque)
>> {
>> QEMUFile *f = opaque;
>> + MigrationIncomingState *mis;
>> Error *local_err = NULL;
>> int ret;
>>
>> - migration_incoming_state_new(f);
>> - migrate_generate_event(MIGRATION_STATUS_ACTIVE);
>> + mis = migration_incoming_state_new(f);
>> + migrate_set_state(&mis->state, MIGRATION_STATUS_NONE,
>> + MIGRATION_STATUS_ACTIVE);
>> ret = qemu_loadvm_state(f);
>>
>> qemu_fclose(f);
>> @@ -282,12 +285,14 @@ static void process_incoming_migration_co(void *opaque)
>> migration_incoming_state_destroy();
>>
>> if (ret < 0) {
>> - migrate_generate_event(MIGRATION_STATUS_FAILED);
>> + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>> + MIGRATION_STATUS_FAILED);
>> error_report("load of migration failed: %s", strerror(-ret));
>> migrate_decompress_threads_join();
>> exit(EXIT_FAILURE);
>> }
>> - migrate_generate_event(MIGRATION_STATUS_COMPLETED);
>> + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>> + MIGRATION_STATUS_COMPLETED);
>> qemu_announce_self();
>>
>> /* Make sure all file formats flush their mutable metadata */
>> @@ -543,9 +548,9 @@ void qmp_migrate_set_parameters(bool has_compress_level,
>>
>> /* shared migration helpers */
>>
>> -static void migrate_set_state(MigrationState *s, int old_state, int new_state)
>> +void migrate_set_state(int *state, int old_state, int new_state)
>> {
>> - if (atomic_cmpxchg(&s->state, old_state, new_state) == old_state) {
>> + if (atomic_cmpxchg(state, old_state, new_state) == old_state) {
>> trace_migrate_set_state(new_state);
>> migrate_generate_event(new_state);
>> }
>> @@ -574,7 +579,7 @@ static void migrate_fd_cleanup(void *opaque)
>> if (s->state != MIGRATION_STATUS_COMPLETED) {
>> qemu_savevm_state_cancel();
>> if (s->state == MIGRATION_STATUS_CANCELLING) {
>> - migrate_set_state(s, MIGRATION_STATUS_CANCELLING,
>> + migrate_set_state(&s->state, MIGRATION_STATUS_CANCELLING,
>> MIGRATION_STATUS_CANCELLED);
>> }
>> }
>> @@ -586,7 +591,8 @@ void migrate_fd_error(MigrationState *s)
>> {
>> trace_migrate_fd_error();
>> assert(s->file == NULL);
>> - migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
>> + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
>> + MIGRATION_STATUS_FAILED);
>> notifier_list_notify(&migration_state_notifiers, s);
>> }
>>
>> @@ -602,7 +608,7 @@ static void migrate_fd_cancel(MigrationState *s)
>> old_state != MIGRATION_STATUS_ACTIVE) {
>> break;
>> }
>> - migrate_set_state(s, old_state, MIGRATION_STATUS_CANCELLING);
>> + migrate_set_state(&s->state, old_state, MIGRATION_STATUS_CANCELLING);
>> } while (s->state != MIGRATION_STATUS_CANCELLING);
>>
>> /*
>> @@ -670,7 +676,7 @@ static MigrationState *migrate_init(const MigrationParams *params)
>> s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
>> decompress_thread_count;
>> s->bandwidth_limit = bandwidth_limit;
>> - migrate_set_state(s, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
>> + migrate_set_state(&s->state, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
>>
>> s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
>> return s;
>> @@ -768,7 +774,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>> } else {
>> error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "uri",
>> "a valid migration protocol");
>> - migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
>> + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
>> + MIGRATION_STATUS_FAILED);
>> return;
>> }
>>
>> @@ -948,7 +955,8 @@ static void *migration_thread(void *opaque)
>> qemu_savevm_state_begin(s->file, &s->params);
>>
>> s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
>> - migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_ACTIVE);
>> + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
>> + MIGRATION_STATUS_ACTIVE);
>>
>> while (s->state == MIGRATION_STATUS_ACTIVE) {
>> int64_t current_time;
>> @@ -978,13 +986,13 @@ static void *migration_thread(void *opaque)
>> qemu_mutex_unlock_iothread();
>>
>> if (ret < 0) {
>> - migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
>> + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
>> MIGRATION_STATUS_FAILED);
>> break;
>> }
>>
>> if (!qemu_file_get_error(s->file)) {
>> - migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
>> + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
>> MIGRATION_STATUS_COMPLETED);
>> break;
>> }
>> @@ -992,7 +1000,7 @@ static void *migration_thread(void *opaque)
>> }
>>
>> if (qemu_file_get_error(s->file)) {
>> - migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
>> + migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
>> MIGRATION_STATUS_FAILED);
>> break;
>> }
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 04/32] migration: Add state records for migration incoming
2015-10-10 7:07 ` zhanghailiang
@ 2015-10-16 11:14 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 59+ messages in thread
From: Dr. David Alan Gilbert @ 2015-10-16 11:14 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, jdenemar, yanghy
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> On 2015/10/10 0:18, Dr. David Alan Gilbert wrote:
> >* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> >>For migration destination, we also need to know its state,
> >>we will use it in COLO.
> >>
> >>Here we add a new member 'state' for MigrationIncomingState,
> >>and also use migrate_set_state() to modify its value.
> >>We fix the first parameter of migrate_set_state(), and make it
> >>public.
> >>
> >>Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> >
> >Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> >
> >You should split this patch out and submit it by itself; it can go in
> >without waiting for any other COLO bits.
> >
>
> Ok, will do that.
> Besides, there is still an independent patch in this series.
> "[PATCH COLO-Frame v9 07/32] migration: Rename the'file' member of MigrationState and
> MigrationIncomingState", i know you have done the same thing in your postcopy series.
> (I noticed that you didn't change the 'file' member in MigartionState ...)
> Could you send that modification as an independent series ? So that i can drop that patch in
> next version.
> BTW, is there any plan to merge your postcopy series ?
Hopefully, it's waiting for review, but you know that can take some time.
Dave
>
> Thanks,
> zhanghailiang
>
> >Dave
> >
> >>---
> >> include/migration/migration.h | 3 +++
> >> migration/migration.c | 38 +++++++++++++++++++++++---------------
> >> 2 files changed, 26 insertions(+), 15 deletions(-)
> >>
> >>diff --git a/include/migration/migration.h b/include/migration/migration.h
> >>index 05de3a1..a62068f 100644
> >>--- a/include/migration/migration.h
> >>+++ b/include/migration/migration.h
> >>@@ -50,6 +50,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
> >> struct MigrationIncomingState {
> >> QEMUFile *file;
> >>
> >>+ int state;
> >> /* See savevm.c */
> >> LoadStateEntry_Head loadvm_handlers;
> >> };
> >>@@ -82,6 +83,8 @@ struct MigrationState
> >> int64_t dirty_sync_count;
> >> };
> >>
> >>+void migrate_set_state(int *state, int old_state, int new_state);
> >>+
> >> void process_incoming_migration(QEMUFile *f);
> >>
> >> void qemu_start_incoming_migration(const char *uri, Error **errp);
> >>diff --git a/migration/migration.c b/migration/migration.c
> >>index 593cac0..98133f1 100644
> >>--- a/migration/migration.c
> >>+++ b/migration/migration.c
> >>@@ -89,6 +89,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
> >> {
> >> mis_current = g_malloc0(sizeof(MigrationIncomingState));
> >> mis_current->file = f;
> >>+ mis_current->state = MIGRATION_STATUS_NONE;
> >> QLIST_INIT(&mis_current->loadvm_handlers);
> >>
> >> return mis_current;
> >>@@ -270,11 +271,13 @@ void qemu_start_incoming_migration(const char *uri, Error **errp)
> >> static void process_incoming_migration_co(void *opaque)
> >> {
> >> QEMUFile *f = opaque;
> >>+ MigrationIncomingState *mis;
> >> Error *local_err = NULL;
> >> int ret;
> >>
> >>- migration_incoming_state_new(f);
> >>- migrate_generate_event(MIGRATION_STATUS_ACTIVE);
> >>+ mis = migration_incoming_state_new(f);
> >>+ migrate_set_state(&mis->state, MIGRATION_STATUS_NONE,
> >>+ MIGRATION_STATUS_ACTIVE);
> >> ret = qemu_loadvm_state(f);
> >>
> >> qemu_fclose(f);
> >>@@ -282,12 +285,14 @@ static void process_incoming_migration_co(void *opaque)
> >> migration_incoming_state_destroy();
> >>
> >> if (ret < 0) {
> >>- migrate_generate_event(MIGRATION_STATUS_FAILED);
> >>+ migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> >>+ MIGRATION_STATUS_FAILED);
> >> error_report("load of migration failed: %s", strerror(-ret));
> >> migrate_decompress_threads_join();
> >> exit(EXIT_FAILURE);
> >> }
> >>- migrate_generate_event(MIGRATION_STATUS_COMPLETED);
> >>+ migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> >>+ MIGRATION_STATUS_COMPLETED);
> >> qemu_announce_self();
> >>
> >> /* Make sure all file formats flush their mutable metadata */
> >>@@ -543,9 +548,9 @@ void qmp_migrate_set_parameters(bool has_compress_level,
> >>
> >> /* shared migration helpers */
> >>
> >>-static void migrate_set_state(MigrationState *s, int old_state, int new_state)
> >>+void migrate_set_state(int *state, int old_state, int new_state)
> >> {
> >>- if (atomic_cmpxchg(&s->state, old_state, new_state) == old_state) {
> >>+ if (atomic_cmpxchg(state, old_state, new_state) == old_state) {
> >> trace_migrate_set_state(new_state);
> >> migrate_generate_event(new_state);
> >> }
> >>@@ -574,7 +579,7 @@ static void migrate_fd_cleanup(void *opaque)
> >> if (s->state != MIGRATION_STATUS_COMPLETED) {
> >> qemu_savevm_state_cancel();
> >> if (s->state == MIGRATION_STATUS_CANCELLING) {
> >>- migrate_set_state(s, MIGRATION_STATUS_CANCELLING,
> >>+ migrate_set_state(&s->state, MIGRATION_STATUS_CANCELLING,
> >> MIGRATION_STATUS_CANCELLED);
> >> }
> >> }
> >>@@ -586,7 +591,8 @@ void migrate_fd_error(MigrationState *s)
> >> {
> >> trace_migrate_fd_error();
> >> assert(s->file == NULL);
> >>- migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
> >>+ migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> >>+ MIGRATION_STATUS_FAILED);
> >> notifier_list_notify(&migration_state_notifiers, s);
> >> }
> >>
> >>@@ -602,7 +608,7 @@ static void migrate_fd_cancel(MigrationState *s)
> >> old_state != MIGRATION_STATUS_ACTIVE) {
> >> break;
> >> }
> >>- migrate_set_state(s, old_state, MIGRATION_STATUS_CANCELLING);
> >>+ migrate_set_state(&s->state, old_state, MIGRATION_STATUS_CANCELLING);
> >> } while (s->state != MIGRATION_STATUS_CANCELLING);
> >>
> >> /*
> >>@@ -670,7 +676,7 @@ static MigrationState *migrate_init(const MigrationParams *params)
> >> s->parameters[MIGRATION_PARAMETER_DECOMPRESS_THREADS] =
> >> decompress_thread_count;
> >> s->bandwidth_limit = bandwidth_limit;
> >>- migrate_set_state(s, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
> >>+ migrate_set_state(&s->state, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
> >>
> >> s->total_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
> >> return s;
> >>@@ -768,7 +774,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
> >> } else {
> >> error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "uri",
> >> "a valid migration protocol");
> >>- migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
> >>+ migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> >>+ MIGRATION_STATUS_FAILED);
> >> return;
> >> }
> >>
> >>@@ -948,7 +955,8 @@ static void *migration_thread(void *opaque)
> >> qemu_savevm_state_begin(s->file, &s->params);
> >>
> >> s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
> >>- migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_ACTIVE);
> >>+ migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> >>+ MIGRATION_STATUS_ACTIVE);
> >>
> >> while (s->state == MIGRATION_STATUS_ACTIVE) {
> >> int64_t current_time;
> >>@@ -978,13 +986,13 @@ static void *migration_thread(void *opaque)
> >> qemu_mutex_unlock_iothread();
> >>
> >> if (ret < 0) {
> >>- migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> >>+ migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
> >> MIGRATION_STATUS_FAILED);
> >> break;
> >> }
> >>
> >> if (!qemu_file_get_error(s->file)) {
> >>- migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> >>+ migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
> >> MIGRATION_STATUS_COMPLETED);
> >> break;
> >> }
> >>@@ -992,7 +1000,7 @@ static void *migration_thread(void *opaque)
> >> }
> >>
> >> if (qemu_file_get_error(s->file)) {
> >>- migrate_set_state(s, MIGRATION_STATUS_ACTIVE,
> >>+ migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
> >> MIGRATION_STATUS_FAILED);
> >> break;
> >> }
> >>--
> >>1.8.3.1
> >>
> >>
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> >.
> >
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 06/32] migration: Integrate COLO checkpoint process into loadvm
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 06/32] migration: Integrate COLO checkpoint process into loadvm zhanghailiang
@ 2015-10-19 9:17 ` Dr. David Alan Gilbert
2015-10-20 8:04 ` zhanghailiang
0 siblings, 1 reply; 59+ messages in thread
From: Dr. David Alan Gilbert @ 2015-10-19 9:17 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> Switch from normal migration loadvm process into COLO checkpoint process if
> COLO mode is enabled.
> We add three new members to struct MigrationIncomingState, 'have_colo_incoming_thread'
> and 'colo_incoming_thread' record the colo related threads for secondary VM,
> 'migration_incoming_co' records the original migration incoming coroutine.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
Mostly OK, some mostly minor comments, and one question below:
> ---
> include/migration/colo.h | 7 +++++++
> include/migration/migration.h | 7 +++++++
> migration/colo-comm.c | 10 ++++++++++
> migration/colo.c | 22 ++++++++++++++++++++++
> migration/migration.c | 33 +++++++++++++++++++++++----------
> stubs/migration-colo.c | 10 ++++++++++
> trace-events | 1 +
> 7 files changed, 80 insertions(+), 10 deletions(-)
>
> diff --git a/include/migration/colo.h b/include/migration/colo.h
> index dface19..58849f7 100644
> --- a/include/migration/colo.h
> +++ b/include/migration/colo.h
> @@ -15,6 +15,8 @@
>
> #include "qemu-common.h"
> #include "migration/migration.h"
> +#include "block/coroutine.h"
> +#include "qemu/thread.h"
>
> bool colo_supported(void);
> void colo_info_mig_init(void);
> @@ -22,4 +24,9 @@ void colo_info_mig_init(void);
> void colo_init_checkpointer(MigrationState *s);
> bool migration_in_colo_state(void);
>
> +/* loadvm */
> +bool migration_incoming_enable_colo(void);
> +void migration_incoming_exit_colo(void);
> +void *colo_process_incoming_thread(void *opaque);
> +bool migration_incoming_in_colo_state(void);
> #endif
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index a62068f..9cdd6b6 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -22,6 +22,7 @@
> #include "migration/vmstate.h"
> #include "qapi-types.h"
> #include "exec/cpu-common.h"
> +#include "block/coroutine.h"
>
> #define QEMU_VM_FILE_MAGIC 0x5145564d
> #define QEMU_VM_FILE_VERSION_COMPAT 0x00000002
> @@ -51,6 +52,12 @@ struct MigrationIncomingState {
> QEMUFile *file;
>
> int state;
> +
> + bool have_colo_incoming_thread;
> + QemuThread colo_incoming_thread;
> + /* The coroutine we should enter (back) after failover */
> + Coroutine *migration_incoming_co;
> +
> /* See savevm.c */
> LoadStateEntry_Head loadvm_handlers;
> };
> diff --git a/migration/colo-comm.c b/migration/colo-comm.c
> index 4330bd8..0808d6c 100644
> --- a/migration/colo-comm.c
> +++ b/migration/colo-comm.c
> @@ -52,3 +52,13 @@ void colo_info_mig_init(void)
> {
> vmstate_register(NULL, 0, &colo_state, &colo_info);
> }
> +
> +bool migration_incoming_enable_colo(void)
> +{
> + return colo_info.colo_requested;
> +}
> +
> +void migration_incoming_exit_colo(void)
> +{
> + colo_info.colo_requested = 0;
> +}
> diff --git a/migration/colo.c b/migration/colo.c
> index 97e64a3..a341eee 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -13,6 +13,7 @@
> #include "sysemu/sysemu.h"
> #include "migration/colo.h"
> #include "trace.h"
> +#include "qemu/error-report.h"
>
> static QEMUBH *colo_bh;
>
> @@ -28,6 +29,13 @@ bool migration_in_colo_state(void)
> return (s->state == MIGRATION_STATUS_COLO);
> }
>
> +bool migration_incoming_in_colo_state(void)
> +{
> + MigrationIncomingState *mis = migration_incoming_get_current();
> +
> + return (mis && (mis->state == MIGRATION_STATUS_COLO));
Can remove outer brackets.
> +}
> +
> static void *colo_thread(void *opaque)
> {
> MigrationState *s = opaque;
> @@ -74,3 +82,17 @@ void colo_init_checkpointer(MigrationState *s)
> colo_bh = qemu_bh_new(colo_start_checkpointer, s);
> qemu_bh_schedule(colo_bh);
> }
> +
> +void *colo_process_incoming_thread(void *opaque)
> +{
> + MigrationIncomingState *mis = opaque;
> +
> + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> + MIGRATION_STATUS_COLO);
> +
> + /* TODO: COLO checkpoint restore loop */
> +
> + migration_incoming_exit_colo();
> +
> + return NULL;
> +}
> diff --git a/migration/migration.c b/migration/migration.c
> index bee61aa..241689f 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -280,7 +280,28 @@ static void process_incoming_migration_co(void *opaque)
> MIGRATION_STATUS_ACTIVE);
> ret = qemu_loadvm_state(f);
>
> - qemu_fclose(f);
> + if (!ret) {
> + /* Make sure all file formats flush their mutable metadata */
> + bdrv_invalidate_cache_all(&local_err);
> + if (local_err) {
> + error_report_err(local_err);
> + migrate_decompress_threads_join();
> + exit(EXIT_FAILURE);
> + }
> + }
> + /* we get colo info, and know if we are in colo mode */
> + if (!ret && migration_incoming_enable_colo()) {
> + mis->migration_incoming_co = qemu_coroutine_self();
> + qemu_thread_create(&mis->colo_incoming_thread, "colo incoming",
> + colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE);
> + mis->have_colo_incoming_thread = true;
> + qemu_coroutine_yield();
> +
> + /* Wait checkpoint incoming thread exit before free resource */
> + qemu_thread_join(&mis->colo_incoming_thread);
> + } else {
> + qemu_fclose(f);
> + }
Why is that qemu_fclose(f) in the else ? If you do colo, and do the colo
thread, when the incoming colo thread exits, don't we return here - and if
we return here then don't we want to close f? (I guess this only happens
on a failover to secondary?)
> free_xbzrle_decoded_buf();
> migration_incoming_state_destroy();
>
> @@ -295,18 +316,9 @@ static void process_incoming_migration_co(void *opaque)
> MIGRATION_STATUS_COMPLETED);
> qemu_announce_self();
>
> - /* Make sure all file formats flush their mutable metadata */
> - bdrv_invalidate_cache_all(&local_err);
> - if (local_err) {
> - error_report_err(local_err);
> - migrate_decompress_threads_join();
> - exit(EXIT_FAILURE);
> - }
> -
> /* If global state section was not received or we are in running
> state, we need to obey autostart. Any other state is set with
> runstate_set. */
> -
> if (!global_state_received() ||
> global_state_get_runstate() == RUN_STATE_RUNNING) {
> if (autostart) {
> @@ -740,6 +752,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
> error_setg(errp, QERR_MIGRATION_ACTIVE);
> return;
> }
> +
> if (runstate_check(RUN_STATE_INMIGRATE)) {
> error_setg(errp, "Guest is waiting for an incoming migration");
> return;
> diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
> index 51b8f66..c49ee1a 100644
> --- a/stubs/migration-colo.c
> +++ b/stubs/migration-colo.c
> @@ -22,6 +22,16 @@ bool migration_in_colo_state(void)
> return false;
> }
>
> +bool migration_incoming_in_colo_state(void)
> +{
> + return false;
> +}
> +
> void colo_init_checkpointer(MigrationState *s)
> {
> }
> +
> +void *colo_process_incoming_thread(void *opaque)
> +{
> + return NULL;
> +}
> diff --git a/trace-events b/trace-events
> index 487d1c7..352e9c3 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -1474,6 +1474,7 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) ""
>
> # migration/colo.c
> colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
> +colo_receive_message(const char *msg) "Receive '%s'"
Should be in a different patch?
Dave
> # kvm-all.c
> kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 08/32] COLO/migration: establish a new communication path from destination to source
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 08/32] COLO/migration: establish a new communication path from destination to source zhanghailiang
@ 2015-10-19 9:54 ` Dr. David Alan Gilbert
2015-10-20 8:30 ` zhanghailiang
0 siblings, 1 reply; 59+ messages in thread
From: Dr. David Alan Gilbert @ 2015-10-19 9:54 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> Add a new member 'to_src_file' to MigrationIncomingState and a
> new member 'from_dst_file' to MigrationState.
> They will be used for returning messages from destination to source.
> It will also be used by post-copy migration.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> include/migration/migration.h | 3 ++-
> migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++
> 2 files changed, 45 insertions(+), 1 deletion(-)
>
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index 6488e03..0c94103 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -50,7 +50,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
> /* State for the incoming migration */
> struct MigrationIncomingState {
> QEMUFile *from_src_file;
> -
> + QEMUFile *to_src_file;
> int state;
>
> bool have_colo_incoming_thread;
> @@ -74,6 +74,7 @@ struct MigrationState
> QemuThread thread;
> QEMUBH *cleanup_bh;
> QEMUFile *to_dst_file;
> + QEMUFile *from_dst_file;
> int parameters[MIGRATION_PARAMETER_MAX];
>
> int state;
> diff --git a/migration/colo.c b/migration/colo.c
> index a341eee..5f4fb20 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -39,6 +39,20 @@ bool migration_incoming_in_colo_state(void)
> static void *colo_thread(void *opaque)
> {
> MigrationState *s = opaque;
> + int fd, ret = 0;
> +
> + /* Dup the fd of to_dst_file */
> + fd = dup(qemu_get_fd(s->to_dst_file));
> + if (fd == -1) {
> + ret = -errno;
> + goto out;
> + }
> + s->from_dst_file = qemu_fopen_socket(fd, "rb");
In my postcopy code I add the return-path opening as a new
method on QEMUFile, that way if we get a return path working
on another transport (RDMA which I hope to do) then it
works; have a look at 'Return path: Open a return path on QEMUFile for sockets'
> + if (!s->from_dst_file) {
> + ret = -EINVAL;
> + error_report("Open QEMUFile failed!");
In errors, try to give detail of where a problem was;
e.g. 'colo_thread: Open QEMUFile from_dst failed'.
> + goto out;
> + }
>
> qemu_mutex_lock_iothread();
> vm_start();
> @@ -47,9 +61,17 @@ static void *colo_thread(void *opaque)
>
> /*TODO: COLO checkpoint savevm loop*/
>
> +out:
> + if (ret < 0) {
> + error_report("Detect some error: %s", strerror(-ret));
> + }
Again, best to say where the error happened.
> migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
> MIGRATION_STATUS_COMPLETED);
>
> + if (s->from_dst_file) {
> + qemu_fclose(s->from_dst_file);
> + }
> +
> qemu_mutex_lock_iothread();
> qemu_bh_schedule(s->cleanup_bh);
> qemu_mutex_unlock_iothread();
> @@ -86,12 +108,33 @@ void colo_init_checkpointer(MigrationState *s)
> void *colo_process_incoming_thread(void *opaque)
> {
> MigrationIncomingState *mis = opaque;
> + int fd, ret = 0;
>
> migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> MIGRATION_STATUS_COLO);
>
> + fd = dup(qemu_get_fd(mis->from_src_file));
> + if (fd < 0) {
> + ret = -errno;
> + goto out;
> + }
> + mis->to_src_file = qemu_fopen_socket(fd, "wb");
> + if (!mis->to_src_file) {
> + ret = -EINVAL;
> + error_report("Can't open incoming channel!");
> + goto out;
> + }
Same as above.
Dave
> /* TODO: COLO checkpoint restore loop */
>
> +out:
> + if (ret < 0) {
> + error_report("colo incoming thread will exit, detect error: %s",
> + strerror(-ret));
> + }
> +
> + if (mis->to_src_file) {
> + qemu_fclose(mis->to_src_file);
> + }
> migration_incoming_exit_colo();
>
> return NULL;
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 06/32] migration: Integrate COLO checkpoint process into loadvm
2015-10-19 9:17 ` Dr. David Alan Gilbert
@ 2015-10-20 8:04 ` zhanghailiang
0 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-10-20 8:04 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
On 2015/10/19 17:17, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> Switch from normal migration loadvm process into COLO checkpoint process if
>> COLO mode is enabled.
>> We add three new members to struct MigrationIncomingState, 'have_colo_incoming_thread'
>> and 'colo_incoming_thread' record the colo related threads for secondary VM,
>> 'migration_incoming_co' records the original migration incoming coroutine.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
>
> Mostly OK, some mostly minor comments, and one question below:
>
>> ---
>> include/migration/colo.h | 7 +++++++
>> include/migration/migration.h | 7 +++++++
>> migration/colo-comm.c | 10 ++++++++++
>> migration/colo.c | 22 ++++++++++++++++++++++
>> migration/migration.c | 33 +++++++++++++++++++++++----------
>> stubs/migration-colo.c | 10 ++++++++++
>> trace-events | 1 +
>> 7 files changed, 80 insertions(+), 10 deletions(-)
>>
>> diff --git a/include/migration/colo.h b/include/migration/colo.h
>> index dface19..58849f7 100644
>> --- a/include/migration/colo.h
>> +++ b/include/migration/colo.h
>> @@ -15,6 +15,8 @@
>>
>> #include "qemu-common.h"
>> #include "migration/migration.h"
>> +#include "block/coroutine.h"
>> +#include "qemu/thread.h"
>>
>> bool colo_supported(void);
>> void colo_info_mig_init(void);
>> @@ -22,4 +24,9 @@ void colo_info_mig_init(void);
>> void colo_init_checkpointer(MigrationState *s);
>> bool migration_in_colo_state(void);
>>
>> +/* loadvm */
>> +bool migration_incoming_enable_colo(void);
>> +void migration_incoming_exit_colo(void);
>> +void *colo_process_incoming_thread(void *opaque);
>> +bool migration_incoming_in_colo_state(void);
>> #endif
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index a62068f..9cdd6b6 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -22,6 +22,7 @@
>> #include "migration/vmstate.h"
>> #include "qapi-types.h"
>> #include "exec/cpu-common.h"
>> +#include "block/coroutine.h"
>>
>> #define QEMU_VM_FILE_MAGIC 0x5145564d
>> #define QEMU_VM_FILE_VERSION_COMPAT 0x00000002
>> @@ -51,6 +52,12 @@ struct MigrationIncomingState {
>> QEMUFile *file;
>>
>> int state;
>> +
>> + bool have_colo_incoming_thread;
>> + QemuThread colo_incoming_thread;
>> + /* The coroutine we should enter (back) after failover */
>> + Coroutine *migration_incoming_co;
>> +
>> /* See savevm.c */
>> LoadStateEntry_Head loadvm_handlers;
>> };
>> diff --git a/migration/colo-comm.c b/migration/colo-comm.c
>> index 4330bd8..0808d6c 100644
>> --- a/migration/colo-comm.c
>> +++ b/migration/colo-comm.c
>> @@ -52,3 +52,13 @@ void colo_info_mig_init(void)
>> {
>> vmstate_register(NULL, 0, &colo_state, &colo_info);
>> }
>> +
>> +bool migration_incoming_enable_colo(void)
>> +{
>> + return colo_info.colo_requested;
>> +}
>> +
>> +void migration_incoming_exit_colo(void)
>> +{
>> + colo_info.colo_requested = 0;
>> +}
>> diff --git a/migration/colo.c b/migration/colo.c
>> index 97e64a3..a341eee 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -13,6 +13,7 @@
>> #include "sysemu/sysemu.h"
>> #include "migration/colo.h"
>> #include "trace.h"
>> +#include "qemu/error-report.h"
>>
>> static QEMUBH *colo_bh;
>>
>> @@ -28,6 +29,13 @@ bool migration_in_colo_state(void)
>> return (s->state == MIGRATION_STATUS_COLO);
>> }
>>
>> +bool migration_incoming_in_colo_state(void)
>> +{
>> + MigrationIncomingState *mis = migration_incoming_get_current();
>> +
>> + return (mis && (mis->state == MIGRATION_STATUS_COLO));
>
> Can remove outer brackets.
>
OK.
>> +}
>> +
>> static void *colo_thread(void *opaque)
>> {
>> MigrationState *s = opaque;
>> @@ -74,3 +82,17 @@ void colo_init_checkpointer(MigrationState *s)
>> colo_bh = qemu_bh_new(colo_start_checkpointer, s);
>> qemu_bh_schedule(colo_bh);
>> }
>> +
>> +void *colo_process_incoming_thread(void *opaque)
>> +{
>> + MigrationIncomingState *mis = opaque;
>> +
>> + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>> + MIGRATION_STATUS_COLO);
>> +
>> + /* TODO: COLO checkpoint restore loop */
>> +
>> + migration_incoming_exit_colo();
>> +
>> + return NULL;
>> +}
>> diff --git a/migration/migration.c b/migration/migration.c
>> index bee61aa..241689f 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -280,7 +280,28 @@ static void process_incoming_migration_co(void *opaque)
>> MIGRATION_STATUS_ACTIVE);
>> ret = qemu_loadvm_state(f);
>>
>> - qemu_fclose(f);
>> + if (!ret) {
>> + /* Make sure all file formats flush their mutable metadata */
>> + bdrv_invalidate_cache_all(&local_err);
>> + if (local_err) {
>> + error_report_err(local_err);
>> + migrate_decompress_threads_join();
>> + exit(EXIT_FAILURE);
>> + }
>> + }
>> + /* we get colo info, and know if we are in colo mode */
>> + if (!ret && migration_incoming_enable_colo()) {
>> + mis->migration_incoming_co = qemu_coroutine_self();
>> + qemu_thread_create(&mis->colo_incoming_thread, "colo incoming",
>> + colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE);
>> + mis->have_colo_incoming_thread = true;
>> + qemu_coroutine_yield();
>> +
>> + /* Wait checkpoint incoming thread exit before free resource */
>> + qemu_thread_join(&mis->colo_incoming_thread);
>> + } else {
>> + qemu_fclose(f);
>> + }
>
> Why is that qemu_fclose(f) in the else ? If you do colo, and do the colo
> thread, when the incoming colo thread exits, don't we return here - and if
> we return here then don't we want to close f? (I guess this only happens
> on a failover to secondary?)
>
Er,it is a bug, we closed it in colo incoming thread before in old version, but after
several version, we lost it.
I will fix it in next version.
>> free_xbzrle_decoded_buf();
>> migration_incoming_state_destroy();
>>
>> @@ -295,18 +316,9 @@ static void process_incoming_migration_co(void *opaque)
>> MIGRATION_STATUS_COMPLETED);
>> qemu_announce_self();
>>
>> - /* Make sure all file formats flush their mutable metadata */
>> - bdrv_invalidate_cache_all(&local_err);
>> - if (local_err) {
>> - error_report_err(local_err);
>> - migrate_decompress_threads_join();
>> - exit(EXIT_FAILURE);
>> - }
>> -
>> /* If global state section was not received or we are in running
>> state, we need to obey autostart. Any other state is set with
>> runstate_set. */
>> -
>> if (!global_state_received() ||
>> global_state_get_runstate() == RUN_STATE_RUNNING) {
>> if (autostart) {
>> @@ -740,6 +752,7 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
>> error_setg(errp, QERR_MIGRATION_ACTIVE);
>> return;
>> }
>> +
>> if (runstate_check(RUN_STATE_INMIGRATE)) {
>> error_setg(errp, "Guest is waiting for an incoming migration");
>> return;
>> diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
>> index 51b8f66..c49ee1a 100644
>> --- a/stubs/migration-colo.c
>> +++ b/stubs/migration-colo.c
>> @@ -22,6 +22,16 @@ bool migration_in_colo_state(void)
>> return false;
>> }
>>
>> +bool migration_incoming_in_colo_state(void)
>> +{
>> + return false;
>> +}
>> +
>> void colo_init_checkpointer(MigrationState *s)
>> {
>> }
>> +
>> +void *colo_process_incoming_thread(void *opaque)
>> +{
>> + return NULL;
>> +}
>> diff --git a/trace-events b/trace-events
>> index 487d1c7..352e9c3 100644
>> --- a/trace-events
>> +++ b/trace-events
>> @@ -1474,6 +1474,7 @@ rdma_start_outgoing_migration_after_rdma_source_init(void) ""
>>
>> # migration/colo.c
>> colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
>> +colo_receive_message(const char *msg) "Receive '%s'"
>
> Should be in a different patch?
>
This is a residual, i have removed it.
Thanks,
zhanghailiang
> Dave
>
>> # kvm-all.c
>> kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 08/32] COLO/migration: establish a new communication path from destination to source
2015-10-19 9:54 ` Dr. David Alan Gilbert
@ 2015-10-20 8:30 ` zhanghailiang
2015-10-20 19:32 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 59+ messages in thread
From: zhanghailiang @ 2015-10-20 8:30 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
On 2015/10/19 17:54, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> Add a new member 'to_src_file' to MigrationIncomingState and a
>> new member 'from_dst_file' to MigrationState.
>> They will be used for returning messages from destination to source.
>> It will also be used by post-copy migration.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> ---
>> include/migration/migration.h | 3 ++-
>> migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++
>> 2 files changed, 45 insertions(+), 1 deletion(-)
>>
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index 6488e03..0c94103 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -50,7 +50,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
>> /* State for the incoming migration */
>> struct MigrationIncomingState {
>> QEMUFile *from_src_file;
>> -
>> + QEMUFile *to_src_file;
>> int state;
>>
>> bool have_colo_incoming_thread;
>> @@ -74,6 +74,7 @@ struct MigrationState
>> QemuThread thread;
>> QEMUBH *cleanup_bh;
>> QEMUFile *to_dst_file;
>> + QEMUFile *from_dst_file;
>> int parameters[MIGRATION_PARAMETER_MAX];
>>
>> int state;
>> diff --git a/migration/colo.c b/migration/colo.c
>> index a341eee..5f4fb20 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -39,6 +39,20 @@ bool migration_incoming_in_colo_state(void)
>> static void *colo_thread(void *opaque)
>> {
>> MigrationState *s = opaque;
>> + int fd, ret = 0;
>> +
>> + /* Dup the fd of to_dst_file */
>> + fd = dup(qemu_get_fd(s->to_dst_file));
>> + if (fd == -1) {
>> + ret = -errno;
>> + goto out;
>> + }
>> + s->from_dst_file = qemu_fopen_socket(fd, "rb");
>
> In my postcopy code I add the return-path opening as a new
> method on QEMUFile, that way if we get a return path working
> on another transport (RDMA which I hope to do) then it
> works; have a look at 'Return path: Open a return path on QEMUFile for sockets'
>
I have looked at it. That's a good solution, we use the same fd for return path, and
i don't have to call qemu_file_shutdown two times in failover process.
>> + if (!s->from_dst_file) {
>> + ret = -EINVAL;
>> + error_report("Open QEMUFile failed!");
>
> In errors, try to give detail of where a problem was;
> e.g. 'colo_thread: Open QEMUFile from_dst failed'.
>
OK. I will fix it in next version.
>> + goto out;
>> + }
>>
>> qemu_mutex_lock_iothread();
>> vm_start();
>> @@ -47,9 +61,17 @@ static void *colo_thread(void *opaque)
>>
>> /*TODO: COLO checkpoint savevm loop*/
>>
>> +out:
>> + if (ret < 0) {
>> + error_report("Detect some error: %s", strerror(-ret));
>> + }
>
> Again, best to say where the error happened.
>
Hmm, it is a little difficult to say where exactly this error happened here,
what i can do is to error out in the place where the error happened.
Here is only a summary for the error.
>
>> migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
>> MIGRATION_STATUS_COMPLETED);
>>
>> + if (s->from_dst_file) {
>> + qemu_fclose(s->from_dst_file);
>> + }
>> +
>> qemu_mutex_lock_iothread();
>> qemu_bh_schedule(s->cleanup_bh);
>> qemu_mutex_unlock_iothread();
>> @@ -86,12 +108,33 @@ void colo_init_checkpointer(MigrationState *s)
>> void *colo_process_incoming_thread(void *opaque)
>> {
>> MigrationIncomingState *mis = opaque;
>> + int fd, ret = 0;
>>
>> migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>> MIGRATION_STATUS_COLO);
>>
>> + fd = dup(qemu_get_fd(mis->from_src_file));
>> + if (fd < 0) {
>> + ret = -errno;
>> + goto out;
>> + }
>> + mis->to_src_file = qemu_fopen_socket(fd, "wb");
>> + if (!mis->to_src_file) {
>> + ret = -EINVAL;
>> + error_report("Can't open incoming channel!");
>> + goto out;
>> + }
>
> Same as above.
OK. Will fix it.
Thanks,
zhanghailiang
>
>> /* TODO: COLO checkpoint restore loop */
>>
>> +out:
>> + if (ret < 0) {
>
>> + error_report("colo incoming thread will exit, detect error: %s",
>> + strerror(-ret));
>> + }
>> +
>> + if (mis->to_src_file) {
>> + qemu_fclose(mis->to_src_file);
>> + }
>> migration_incoming_exit_colo();
>>
>> return NULL;
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 08/32] COLO/migration: establish a new communication path from destination to source
2015-10-20 8:30 ` zhanghailiang
@ 2015-10-20 19:32 ` Dr. David Alan Gilbert
2015-10-21 8:33 ` zhanghailiang
0 siblings, 1 reply; 59+ messages in thread
From: Dr. David Alan Gilbert @ 2015-10-20 19:32 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> On 2015/10/19 17:54, Dr. David Alan Gilbert wrote:
> >* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> >>Add a new member 'to_src_file' to MigrationIncomingState and a
> >>new member 'from_dst_file' to MigrationState.
> >>They will be used for returning messages from destination to source.
> >>It will also be used by post-copy migration.
> >>
> >>Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> >>Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> >>Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
> >>---
> >> include/migration/migration.h | 3 ++-
> >> migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++
> >> 2 files changed, 45 insertions(+), 1 deletion(-)
> >>
> >>diff --git a/include/migration/migration.h b/include/migration/migration.h
> >>index 6488e03..0c94103 100644
> >>--- a/include/migration/migration.h
> >>+++ b/include/migration/migration.h
> >>@@ -50,7 +50,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
> >> /* State for the incoming migration */
> >> struct MigrationIncomingState {
> >> QEMUFile *from_src_file;
> >>-
> >>+ QEMUFile *to_src_file;
> >> int state;
> >>
> >> bool have_colo_incoming_thread;
> >>@@ -74,6 +74,7 @@ struct MigrationState
> >> QemuThread thread;
> >> QEMUBH *cleanup_bh;
> >> QEMUFile *to_dst_file;
> >>+ QEMUFile *from_dst_file;
> >> int parameters[MIGRATION_PARAMETER_MAX];
> >>
> >> int state;
> >>diff --git a/migration/colo.c b/migration/colo.c
> >>index a341eee..5f4fb20 100644
> >>--- a/migration/colo.c
> >>+++ b/migration/colo.c
> >>@@ -39,6 +39,20 @@ bool migration_incoming_in_colo_state(void)
> >> static void *colo_thread(void *opaque)
> >> {
> >> MigrationState *s = opaque;
> >>+ int fd, ret = 0;
> >>+
> >>+ /* Dup the fd of to_dst_file */
> >>+ fd = dup(qemu_get_fd(s->to_dst_file));
> >>+ if (fd == -1) {
> >>+ ret = -errno;
> >>+ goto out;
> >>+ }
> >>+ s->from_dst_file = qemu_fopen_socket(fd, "rb");
> >
> >In my postcopy code I add the return-path opening as a new
> >method on QEMUFile, that way if we get a return path working
> >on another transport (RDMA which I hope to do) then it
> >works; have a look at 'Return path: Open a return path on QEMUFile for sockets'
> >
>
> I have looked at it. That's a good solution, we use the same fd for return path, and
> i don't have to call qemu_file_shutdown two times in failover process.
>
> >>+ if (!s->from_dst_file) {
> >>+ ret = -EINVAL;
> >>+ error_report("Open QEMUFile failed!");
> >
> >In errors, try to give detail of where a problem was;
> >e.g. 'colo_thread: Open QEMUFile from_dst failed'.
> >
>
> OK. I will fix it in next version.
>
> >>+ goto out;
> >>+ }
> >>
> >> qemu_mutex_lock_iothread();
> >> vm_start();
> >>@@ -47,9 +61,17 @@ static void *colo_thread(void *opaque)
> >>
> >> /*TODO: COLO checkpoint savevm loop*/
> >>
> >>+out:
> >>+ if (ret < 0) {
> >>+ error_report("Detect some error: %s", strerror(-ret));
> >>+ }
> >
> >Again, best to say where the error happened.
> >
>
> Hmm, it is a little difficult to say where exactly this error happened here,
> what i can do is to error out in the place where the error happened.
> Here is only a summary for the error.
OK, just add the function name then, e.g.
error_report("%s: %s", __func__, strerror(-ret));
it just helps when you're looking at a log file, to understand where it came
from; especially if the person who sent you the log file didn't tell you much :-)
Dave
>
> >
> >> migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
> >> MIGRATION_STATUS_COMPLETED);
> >>
> >>+ if (s->from_dst_file) {
> >>+ qemu_fclose(s->from_dst_file);
> >>+ }
> >>+
> >> qemu_mutex_lock_iothread();
> >> qemu_bh_schedule(s->cleanup_bh);
> >> qemu_mutex_unlock_iothread();
> >>@@ -86,12 +108,33 @@ void colo_init_checkpointer(MigrationState *s)
> >> void *colo_process_incoming_thread(void *opaque)
> >> {
> >> MigrationIncomingState *mis = opaque;
> >>+ int fd, ret = 0;
> >>
> >> migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> >> MIGRATION_STATUS_COLO);
> >>
> >>+ fd = dup(qemu_get_fd(mis->from_src_file));
> >>+ if (fd < 0) {
> >>+ ret = -errno;
> >>+ goto out;
> >>+ }
> >>+ mis->to_src_file = qemu_fopen_socket(fd, "wb");
> >>+ if (!mis->to_src_file) {
> >>+ ret = -EINVAL;
> >>+ error_report("Can't open incoming channel!");
> >>+ goto out;
> >>+ }
> >
> >Same as above.
>
> OK. Will fix it.
>
> Thanks,
> zhanghailiang
> >
> >> /* TODO: COLO checkpoint restore loop */
> >>
> >>+out:
> >>+ if (ret < 0) {
> >
> >>+ error_report("colo incoming thread will exit, detect error: %s",
> >>+ strerror(-ret));
> >>+ }
> >>+
> >>+ if (mis->to_src_file) {
> >>+ qemu_fclose(mis->to_src_file);
> >>+ }
> >> migration_incoming_exit_colo();
> >>
> >> return NULL;
> >>--
> >>1.8.3.1
> >>
> >>
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> >.
> >
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 08/32] COLO/migration: establish a new communication path from destination to source
2015-10-20 19:32 ` Dr. David Alan Gilbert
@ 2015-10-21 8:33 ` zhanghailiang
0 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-10-21 8:33 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
On 2015/10/21 3:32, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> On 2015/10/19 17:54, Dr. David Alan Gilbert wrote:
>>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>>>> Add a new member 'to_src_file' to MigrationIncomingState and a
>>>> new member 'from_dst_file' to MigrationState.
>>>> They will be used for returning messages from destination to source.
>>>> It will also be used by post-copy migration.
>>>>
>>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>>>> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
>>>> ---
>>>> include/migration/migration.h | 3 ++-
>>>> migration/colo.c | 43 +++++++++++++++++++++++++++++++++++++++++++
>>>> 2 files changed, 45 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>>>> index 6488e03..0c94103 100644
>>>> --- a/include/migration/migration.h
>>>> +++ b/include/migration/migration.h
>>>> @@ -50,7 +50,7 @@ typedef QLIST_HEAD(, LoadStateEntry) LoadStateEntry_Head;
>>>> /* State for the incoming migration */
>>>> struct MigrationIncomingState {
>>>> QEMUFile *from_src_file;
>>>> -
>>>> + QEMUFile *to_src_file;
>>>> int state;
>>>>
>>>> bool have_colo_incoming_thread;
>>>> @@ -74,6 +74,7 @@ struct MigrationState
>>>> QemuThread thread;
>>>> QEMUBH *cleanup_bh;
>>>> QEMUFile *to_dst_file;
>>>> + QEMUFile *from_dst_file;
>>>> int parameters[MIGRATION_PARAMETER_MAX];
>>>>
>>>> int state;
>>>> diff --git a/migration/colo.c b/migration/colo.c
>>>> index a341eee..5f4fb20 100644
>>>> --- a/migration/colo.c
>>>> +++ b/migration/colo.c
>>>> @@ -39,6 +39,20 @@ bool migration_incoming_in_colo_state(void)
>>>> static void *colo_thread(void *opaque)
>>>> {
>>>> MigrationState *s = opaque;
>>>> + int fd, ret = 0;
>>>> +
>>>> + /* Dup the fd of to_dst_file */
>>>> + fd = dup(qemu_get_fd(s->to_dst_file));
>>>> + if (fd == -1) {
>>>> + ret = -errno;
>>>> + goto out;
>>>> + }
>>>> + s->from_dst_file = qemu_fopen_socket(fd, "rb");
>>>
>>> In my postcopy code I add the return-path opening as a new
>>> method on QEMUFile, that way if we get a return path working
>>> on another transport (RDMA which I hope to do) then it
>>> works; have a look at 'Return path: Open a return path on QEMUFile for sockets'
>>>
>>
>> I have looked at it. That's a good solution, we use the same fd for return path, and
>> i don't have to call qemu_file_shutdown two times in failover process.
>>
>>>> + if (!s->from_dst_file) {
>>>> + ret = -EINVAL;
>>>> + error_report("Open QEMUFile failed!");
>>>
>>> In errors, try to give detail of where a problem was;
>>> e.g. 'colo_thread: Open QEMUFile from_dst failed'.
>>>
>>
>> OK. I will fix it in next version.
>>
>>>> + goto out;
>>>> + }
>>>>
>>>> qemu_mutex_lock_iothread();
>>>> vm_start();
>>>> @@ -47,9 +61,17 @@ static void *colo_thread(void *opaque)
>>>>
>>>> /*TODO: COLO checkpoint savevm loop*/
>>>>
>>>> +out:
>>>> + if (ret < 0) {
>>>> + error_report("Detect some error: %s", strerror(-ret));
>>>> + }
>>>
>>> Again, best to say where the error happened.
>>>
>>
>> Hmm, it is a little difficult to say where exactly this error happened here,
>> what i can do is to error out in the place where the error happened.
>> Here is only a summary for the error.
>
> OK, just add the function name then, e.g.
> error_report("%s: %s", __func__, strerror(-ret));
>
> it just helps when you're looking at a log file, to understand where it came
> from; especially if the person who sent you the log file didn't tell you much :-)
>
OK, that's a good idea, will fix in next version. Thanks.
>>
>>>
>>>> migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
>>>> MIGRATION_STATUS_COMPLETED);
>>>>
>>>> + if (s->from_dst_file) {
>>>> + qemu_fclose(s->from_dst_file);
>>>> + }
>>>> +
>>>> qemu_mutex_lock_iothread();
>>>> qemu_bh_schedule(s->cleanup_bh);
>>>> qemu_mutex_unlock_iothread();
>>>> @@ -86,12 +108,33 @@ void colo_init_checkpointer(MigrationState *s)
>>>> void *colo_process_incoming_thread(void *opaque)
>>>> {
>>>> MigrationIncomingState *mis = opaque;
>>>> + int fd, ret = 0;
>>>>
>>>> migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>>>> MIGRATION_STATUS_COLO);
>>>>
>>>> + fd = dup(qemu_get_fd(mis->from_src_file));
>>>> + if (fd < 0) {
>>>> + ret = -errno;
>>>> + goto out;
>>>> + }
>>>> + mis->to_src_file = qemu_fopen_socket(fd, "wb");
>>>> + if (!mis->to_src_file) {
>>>> + ret = -EINVAL;
>>>> + error_report("Can't open incoming channel!");
>>>> + goto out;
>>>> + }
>>>
>>> Same as above.
>>
>> OK. Will fix it.
>>
>> Thanks,
>> zhanghailiang
>>>
>>>> /* TODO: COLO checkpoint restore loop */
>>>>
>>>> +out:
>>>> + if (ret < 0) {
>>>
>>>> + error_report("colo incoming thread will exit, detect error: %s",
>>>> + strerror(-ret));
>>>> + }
>>>> +
>>>> + if (mis->to_src_file) {
>>>> + qemu_fclose(mis->to_src_file);
>>>> + }
>>>> migration_incoming_exit_colo();
>>>>
>>>> return NULL;
>>>> --
>>>> 1.8.3.1
>>>>
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>>> .
>>>
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 09/32] COLO: Implement colo checkpoint protocol
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 09/32] COLO: Implement colo checkpoint protocol zhanghailiang
@ 2015-10-21 12:17 ` Eric Blake
2015-10-22 7:13 ` zhanghailiang
0 siblings, 1 reply; 59+ messages in thread
From: Eric Blake @ 2015-10-21 12:17 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy
[-- Attachment #1: Type: text/plain, Size: 3330 bytes --]
On 09/02/2015 02:22 AM, zhanghailiang wrote:
> We need communications protocol of user-defined to control the checkpoint
> process.
>
> The new checkpoint request is started by Primary VM, and the interactive process
> like below:
> Checkpoint synchronizing points,
>
> Primary Secondary
> 'checkpoint-request' @ ----------------------------->
> Suspend (In hybrid mode)
> 'checkpoint-reply' <------------------------------ @
> Suspend&Save state
> 'vmstate-send' @ ----------------------------->
> Send state Receive state
> 'vmstate-received' <------------------------------ @
> Release packets Load state
> 'vmstate-load' <------------------------------ @
> Resume Resume (In hybrid mode)
>
> Start Comparing (In hybrid mode)
> NOTE:
> 1) '@' who sends the message
> 2) Every sync-point is synchronized by two sides with only
> one handshake(single direction) for low-latency.
> If more strict synchronization is required, a opposite direction
> sync-point should be added.
> 3) Since sync-points are single direction, the remote side may
> go forward a lot when this side just receives the sync-point.
> 4) For now, we only support 'periodic' checkpoint, for which
> the Secondary VM is not running, later we will support 'hybrid' mode.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
> migration/colo.c | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> qapi-schema.json | 26 ++++++++
> trace-events | 3 +-
> 3 files changed, 218 insertions(+), 3 deletions(-)
Just a qapi review:
> +++ b/qapi-schema.json
> @@ -664,6 +664,32 @@
> '*tls-port': 'int', '*cert-subject': 'str' } }
>
> ##
> +# @COLOCmd
Any reason this can't be COLOCommand? We tend to avoid abbreviations in
the public interface, although arguably type names are not ABI.
> +#
> +# The colo command
> +#
> +# @invalid: unknown command
> +#
> +# @checkpoint-ready: SVM is ready for checkpointing
> +#
> +# @checkpoint-request: PVM tells SVM to prepare for new checkpointing
> +#
> +# @checkpoint-reply: SVM gets PVM's checkpoint request
> +#
> +# @vmstate-send: VM's state will be sent by PVM.
> +#
> +# @vmstate-received: VM's state has been received by SVM
> +#
> +# @vmstate-loaded: VM's state has been loaded by SVM
7 documentation strings...
> +#
> +# Since: 2.5
> +##
> +{ 'enum': 'COLOCmd',
> + 'data': [ 'invalid', 'checkpoint-ready', 'checkpoint-request',
> + 'checkpoint-reply', 'vmstate-send', 'vmstate-size',
> + 'vmstate-received', 'vmstate-loaded', 'guest-shutdown',
> + 'ram-steal'] }
...10 enum values. Missing vmstate-size, guest-shutdown, ram-steal.
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 10/32] COLO: Add a new RunState RUN_STATE_COLO
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 10/32] COLO: Add a new RunState RUN_STATE_COLO zhanghailiang
@ 2015-10-21 12:18 ` Eric Blake
2015-10-22 6:58 ` zhanghailiang
0 siblings, 1 reply; 59+ messages in thread
From: Eric Blake @ 2015-10-21 12:18 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, arei.gonglei, stefanha,
amit.shah, yanghy
[-- Attachment #1: Type: text/plain, Size: 1769 bytes --]
On 09/02/2015 02:22 AM, zhanghailiang wrote:
> Guest will enter this state when paused to save/restore VM state
> under colo checkpoint.
>
> Cc: Eric Blake <eblake@redhat.com>
> Cc: Markus Armbruster <armbru@redhat.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> qapi-schema.json | 7 ++++++-
> vl.c | 8 ++++++++
> 2 files changed, 14 insertions(+), 1 deletion(-)
>
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 0d3d6e2..6a3796e 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -148,12 +148,15 @@
> # @watchdog: the watchdog action is configured to pause and has been triggered
> #
> # @guest-panicked: guest has been panicked as a result of guest OS panic
> +#
> +# @colo: guest is paused to save/restore VM state under colo checkpoint (since
> +# 2.5)
> ##
> { 'enum': 'RunState',
> 'data': [ 'debug', 'inmigrate', 'internal-error', 'io-error', 'paused',
> 'postmigrate', 'prelaunch', 'finish-migrate', 'restore-vm',
> 'running', 'save-vm', 'shutdown', 'suspended', 'watchdog',
> - 'guest-panicked' ] }
> + 'guest-panicked', 'colo' ] }
>
Looks okay.
> ##
> # @StatusInfo:
> @@ -428,6 +431,8 @@
> #
> # @failed: some error occurred during migration process.
> #
> +# @colo: VM is in the process of fault tolerance.
> +#
Missing '(since 2.5)'
With that fixed,
Reviewed-by: Eric Blake <eblake@redhat.com>
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT)
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (34 preceding siblings ...)
2015-09-15 10:40 ` zhanghailiang
@ 2015-10-21 14:10 ` Dr. David Alan Gilbert
2015-10-22 9:01 ` zhanghailiang
35 siblings, 1 reply; 59+ messages in thread
From: Dr. David Alan Gilbert @ 2015-10-21 14:10 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
Hi,
Not something I found on this set; but on your older set, where you have
the experimental 'live' ram transfer; we found a test that was slowed
down (5x) by the calls to 'colo_need_live_migrate_ram'; even if it decides
not to send any RAM in the live phase, the call to 'qemu_savevm_state_pending'
can be really expensive.
Just thought it worth mentioning.
Dave
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 10/32] COLO: Add a new RunState RUN_STATE_COLO
2015-10-21 12:18 ` Eric Blake
@ 2015-10-22 6:58 ` zhanghailiang
0 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-10-22 6:58 UTC (permalink / raw)
To: Eric Blake, qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, arei.gonglei, stefanha,
amit.shah, yanghy
On 2015/10/21 20:18, Eric Blake wrote:
> On 09/02/2015 02:22 AM, zhanghailiang wrote:
>> Guest will enter this state when paused to save/restore VM state
>> under colo checkpoint.
>>
>> Cc: Eric Blake <eblake@redhat.com>
>> Cc: Markus Armbruster <armbru@redhat.com>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> ---
>> qapi-schema.json | 7 ++++++-
>> vl.c | 8 ++++++++
>> 2 files changed, 14 insertions(+), 1 deletion(-)
>>
>> diff --git a/qapi-schema.json b/qapi-schema.json
>> index 0d3d6e2..6a3796e 100644
>> --- a/qapi-schema.json
>> +++ b/qapi-schema.json
>> @@ -148,12 +148,15 @@
>> # @watchdog: the watchdog action is configured to pause and has been triggered
>> #
>> # @guest-panicked: guest has been panicked as a result of guest OS panic
>> +#
>> +# @colo: guest is paused to save/restore VM state under colo checkpoint (since
>> +# 2.5)
>> ##
>> { 'enum': 'RunState',
>> 'data': [ 'debug', 'inmigrate', 'internal-error', 'io-error', 'paused',
>> 'postmigrate', 'prelaunch', 'finish-migrate', 'restore-vm',
>> 'running', 'save-vm', 'shutdown', 'suspended', 'watchdog',
>> - 'guest-panicked' ] }
>> + 'guest-panicked', 'colo' ] }
>>
>
> Looks okay.
>
>> ##
>> # @StatusInfo:
>> @@ -428,6 +431,8 @@
>> #
>> # @failed: some error occurred during migration process.
>> #
>> +# @colo: VM is in the process of fault tolerance.
>> +#
>
> Missing '(since 2.5)'
>
> With that fixed,
> Reviewed-by: Eric Blake <eblake@redhat.com>
>
Fixed, Thanks.
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 09/32] COLO: Implement colo checkpoint protocol
2015-10-21 12:17 ` Eric Blake
@ 2015-10-22 7:13 ` zhanghailiang
0 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-10-22 7:13 UTC (permalink / raw)
To: Eric Blake, qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, yanghy
On 2015/10/21 20:17, Eric Blake wrote:
> On 09/02/2015 02:22 AM, zhanghailiang wrote:
>> We need communications protocol of user-defined to control the checkpoint
>> process.
>>
>> The new checkpoint request is started by Primary VM, and the interactive process
>> like below:
>> Checkpoint synchronizing points,
>>
>> Primary Secondary
>> 'checkpoint-request' @ ----------------------------->
>> Suspend (In hybrid mode)
>> 'checkpoint-reply' <------------------------------ @
>> Suspend&Save state
>> 'vmstate-send' @ ----------------------------->
>> Send state Receive state
>> 'vmstate-received' <------------------------------ @
>> Release packets Load state
>> 'vmstate-load' <------------------------------ @
>> Resume Resume (In hybrid mode)
>>
>> Start Comparing (In hybrid mode)
>> NOTE:
>> 1) '@' who sends the message
>> 2) Every sync-point is synchronized by two sides with only
>> one handshake(single direction) for low-latency.
>> If more strict synchronization is required, a opposite direction
>> sync-point should be added.
>> 3) Since sync-points are single direction, the remote side may
>> go forward a lot when this side just receives the sync-point.
>> 4) For now, we only support 'periodic' checkpoint, for which
>> the Secondary VM is not running, later we will support 'hybrid' mode.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Yang Hongyang <yanghy@cn.fujitsu.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>> migration/colo.c | 192 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>> qapi-schema.json | 26 ++++++++
>> trace-events | 3 +-
>> 3 files changed, 218 insertions(+), 3 deletions(-)
>
> Just a qapi review:
>
>
>> +++ b/qapi-schema.json
>> @@ -664,6 +664,32 @@
>> '*tls-port': 'int', '*cert-subject': 'str' } }
>>
>> ##
>> +# @COLOCmd
>
> Any reason this can't be COLOCommand? We tend to avoid abbreviations in
> the public interface, although arguably type names are not ABI.
>
No special reason, will rename it in next version. :)
>> +#
>> +# The colo command
>> +#
>> +# @invalid: unknown command
>> +#
>> +# @checkpoint-ready: SVM is ready for checkpointing
>> +#
>> +# @checkpoint-request: PVM tells SVM to prepare for new checkpointing
>> +#
>> +# @checkpoint-reply: SVM gets PVM's checkpoint request
>> +#
>> +# @vmstate-send: VM's state will be sent by PVM.
>> +#
>> +# @vmstate-received: VM's state has been received by SVM
>> +#
>> +# @vmstate-loaded: VM's state has been loaded by SVM
>
> 7 documentation strings...
>
>> +#
>> +# Since: 2.5
>> +##
>> +{ 'enum': 'COLOCmd',
>> + 'data': [ 'invalid', 'checkpoint-ready', 'checkpoint-request',
>> + 'checkpoint-reply', 'vmstate-send', 'vmstate-size',
>> + 'vmstate-received', 'vmstate-loaded', 'guest-shutdown',
>> + 'ram-steal'] }
>
> ...10 enum values. Missing vmstate-size, guest-shutdown, ram-steal.
>
Yes, this is a mistake, these three values shouldn't be added in this patch, we
didn't refer to them in this patch, they should appear in the later corresponding
patch. I will fix it in next version.
Thanks,
zhanghailiang
^ permalink raw reply [flat|nested] 59+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT)
2015-10-21 14:10 ` Dr. David Alan Gilbert
@ 2015-10-22 9:01 ` zhanghailiang
0 siblings, 0 replies; 59+ messages in thread
From: zhanghailiang @ 2015-10-22 9:01 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, yanghy
Hi Dave,
Thanks for feedback, i have recorded this problem, maybe we can optimize this
calling in the future.
Thanks,
zhanghailiang
On 2015/10/21 22:10, Dr. David Alan Gilbert wrote:
> Hi,
> Not something I found on this set; but on your older set, where you have
> the experimental 'live' ram transfer; we found a test that was slowed
> down (5x) by the calls to 'colo_need_live_migrate_ram'; even if it decides
> not to send any RAM in the live phase, the call to 'qemu_savevm_state_pending'
> can be really expensive.
>
> Just thought it worth mentioning.
>
> Dave
>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 59+ messages in thread
end of thread, other threads:[~2015-10-22 9:05 UTC | newest]
Thread overview: 59+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-09-02 8:22 [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 01/32] configure: Add parameter for configure to enable/disable COLO support zhanghailiang
2015-10-02 15:10 ` Dr. David Alan Gilbert
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 02/32] migration: Introduce capability 'colo' to migration zhanghailiang
2015-10-02 16:02 ` Eric Blake
2015-10-08 6:34 ` zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 03/32] COLO: migrate colo related info to slave zhanghailiang
2015-10-02 18:45 ` Dr. David Alan Gilbert
2015-10-08 6:48 ` zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 04/32] migration: Add state records for migration incoming zhanghailiang
2015-10-09 16:18 ` Dr. David Alan Gilbert
2015-10-10 7:07 ` zhanghailiang
2015-10-16 11:14 ` Dr. David Alan Gilbert
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 05/32] migration: Integrate COLO checkpoint process into migration zhanghailiang
2015-10-09 16:53 ` Dr. David Alan Gilbert
2015-10-10 6:25 ` zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 06/32] migration: Integrate COLO checkpoint process into loadvm zhanghailiang
2015-10-19 9:17 ` Dr. David Alan Gilbert
2015-10-20 8:04 ` zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 07/32] migration: Rename the'file' member of MigrationState and MigrationIncomingState zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 08/32] COLO/migration: establish a new communication path from destination to source zhanghailiang
2015-10-19 9:54 ` Dr. David Alan Gilbert
2015-10-20 8:30 ` zhanghailiang
2015-10-20 19:32 ` Dr. David Alan Gilbert
2015-10-21 8:33 ` zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 09/32] COLO: Implement colo checkpoint protocol zhanghailiang
2015-10-21 12:17 ` Eric Blake
2015-10-22 7:13 ` zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 10/32] COLO: Add a new RunState RUN_STATE_COLO zhanghailiang
2015-10-21 12:18 ` Eric Blake
2015-10-22 6:58 ` zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 11/32] QEMUSizedBuffer: Introduce two help functions for qsb zhanghailiang
2015-09-02 8:22 ` [Qemu-devel] [PATCH COLO-Frame v9 12/32] COLO: Save PVM state to secondary side when do checkpoint zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 13/32] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 14/32] COLO: Load VMState into qsb before restore it zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 15/32] COLO: Flush PVM's cached RAM into SVM's memory zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 16/32] COLO: synchronize PVM's state to SVM periodically zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 17/32] COLO failover: Introduce a new command to trigger a failover zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 18/32] COLO failover: Introduce state to record failover process zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 19/32] COLO: Implement failover work for Primary VM zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 20/32] COLO: Implement failover work for Secondary VM zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 21/32] COLO: implement default failover treatment zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 22/32] qmp event: Add event notification for COLO error zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 23/32] COLO failover: Shutdown related socket fd when do failover zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 24/32] COLO failover: Don't do failover during loading VM's state zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 25/32] COLO: Control the checkpoint delay time by migrate-set-parameters command zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 26/32] COLO: Implement shutdown checkpoint zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 27/32] COLO: Update the global runstate after going into colo state zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 28/32] savevm: Split load vm state function qemu_loadvm_state zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 29/32] COLO: Separate the process of saving/loading ram and device state zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 30/32] COLO: Split qemu_savevm_state_begin out of checkpoint process zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 31/32] COLO: Add block replication into colo process zhanghailiang
2015-09-02 8:23 ` [Qemu-devel] [PATCH COLO-Frame v9 32/32] COLO: Add net packets treatment into COLO zhanghailiang
2015-09-02 9:03 ` [Qemu-devel] [PATCH COLO-Frame v9 00/32] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) Yang Hongyang
2015-09-02 9:17 ` zhanghailiang
2015-09-09 3:36 ` zhanghailiang
2015-09-15 10:40 ` zhanghailiang
2015-10-21 14:10 ` Dr. David Alan Gilbert
2015-10-22 9:01 ` zhanghailiang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.