* [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT)
@ 2015-11-24 9:25 zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 01/39] configure: Add parameter for configure to enable/disable COLO support zhanghailiang
` (38 more replies)
0 siblings, 39 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
This is the 11th version of COLO.
As usual, this version of COLO is only support periodic checkpoint,
just like MicroCheckpointing and Remus does.
The 'peroidic' mode is based on netfilter which has been merged.
It uses the 'filter-buffer' to buffer and release packets.
We add each netdev a default buffer filter and the name is 'nop'.
The 'nop' buffer filter will not buffer any packets in default.
So it has no side effect on netdev.
As usual, here is only COLO frame part, you can get the whole codes from github:
https://github.com/coloft/qemu/commits/colo-v2.2-periodic-mode
Test procedure:
1. Startup qemu
Primary side:
#x86_64-softmmu/qemu-system-x86_64 -enable-kvm -boot c -m 2048 -smp 8 -qmp stdio -vnc :7 -name primary -cpu qemu64,+kvmclock -device piix3-usb-uhci -device usb-tablet -netdev tap,id=hn0 -device virtio-net-pci,id=net-pci0,netdev=hn0 -drive if=virtio,id=colo-disk0,driver=quorum,read-pattern=fifo,vote-threshold=1,children.0.file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,children.0.driver=raw
Secondary side:
#x86_64-softmmu/qemu-system-x86_64 -enable-kvm -boot c -m 2048 -smp 8 -qmp stdio -vnc :7 -name secondary -cpu qemu64,+kvmclock -device piix3-usb-uhci -device usb-tablet -netdev tap,id=hn0 -device virtio-net-pci,id=net-pci0,netdev=hn0 -drive if=none,id=colo-disk0,file.filename=/dev/null,driver=raw -drive if=virtio,id=active-disk0,throttling.bps-total=70000000,driver=replication,mode=secondary,file.driver=qcow2,file.file.filename=/mnt/ramfs/active_disk.img,file.backing.allow-write-backing-file=on,file.backing.driver=qcow2,file.backing.file.filename=/mnt/ramfs/hidden_disk.img,file.backing.backing.allow-write-backing-file=on,file.backing.backing.file.filename=/mnt/sdd/pure_IMG/linux/redhat/rhel_6.5_64_2U_ide,file.backing.backing.driver=raw,file.backing.backing.node-name=node0 -incoming tcp:0:8888
2. On Secondary VM's QEMU monitor, issue command
(Note: Hmp command is unstable, so here we recommand using qmp command)
{'execute':'qmp_capabilities'}
{'execute': 'blockdev-remove-medium', 'arguments': {'device': 'colo-disk0'} }
{'execute': 'blockdev-insert-medium', 'arguments': {'device': 'colo-disk0', 'node-name': 'node0'} }
{'execute': 'nbd-server-start', 'arguments': {'addr': {'type': 'inet', 'data': {'host': '192.168.2.88', 'port': '8889'} } } }
{'execute': 'nbd-server-add', 'arguments': {'device': 'colo-disk0', 'writable': true } }
{'execute': 'trace-event-set-state', 'arguments': {'name': 'colo*', 'enable': true} }
3. On Primary VM's QEMU monitor, issue command:
{'execute':'qmp_capabilities'}
{'execute': 'human-monitor-command', 'arguments': {'command-line': 'drive_add buddy driver=replication,mode=primary,file.driver=nbd,file.host=9.61.1.7,file.port=8889,file.export=colo-disk0,node-name=node0,if=none'}}
{'execute':'x-blockdev-change', 'arguments':{'parent': 'colo-disk0', 'node': 'node0' } }
{'execute': 'migrate-set-capabilities', 'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': true } ] } }
{'execute': 'migrate', 'arguments': {'uri': 'tcp:192.168.2.88:8888' } }
4. After the above steps, you will see, whenever you make changes to PVM, SVM will be synced.
You can by issue command '{ "execute": "migrate-set-parameters" , "arguments":{ "checkpoint-delay": 2000 } }'
to change the checkpoint period time.
5. Failover test
You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM's
monitor at the same time, then SVM will failover and client will not feel this
change.
The qmp command is '{ "execute": "x-colo-lost-heartbeat" }'.
COLO is a totally new feature which is still in early stage,
your comments and feedback are warmly welcomed.
TODO:
1. implement packets compare module (proxy) in qemu
2. checkpoint based on proxy in qemu
3. The capability of continuous FT
v11:
- Re-implement buffer/release packets based on filter-buffer according
to Jason Wang's suggestion. (patch 34, patch 36 ~ patch 38)
- Rebase master to re-use some stuff introduced by post-copy.
- Address several comments from Eric and Dave, the fixing record can
be found in each patch.
v10:
- Rename 'colo_lost_heartbeat' command to experimental 'x_colo_lost_heartbeat'
- Rename migration capability 'colo' to 'x-colo' (Eric's suggestion)
- Simplify the process of primary side by dropping colo thread and reusing
migration thread. (Dave's suggestion)
- Add several netfilter related APIs to support buffer/release packets
for COLO (patch 32 ~ patch 36)
zhanghailiang (39):
configure: Add parameter for configure to enable/disable COLO support
migration: Introduce capability 'x-colo' to migration
COLO: migrate colo related info to secondary node
migration: Export migrate_set_state()
migration: Add state records for migration incoming
migration: Integrate COLO checkpoint process into migration
migration: Integrate COLO checkpoint process into loadvm
migration: Rename the'file' member of MigrationState
COLO/migration: Create a new communication path from destination to
source
COLO: Implement colo checkpoint protocol
COLO: Add a new RunState RUN_STATE_COLO
QEMUSizedBuffer: Introduce two help functions for qsb
COLO: Save PVM state to secondary side when do checkpoint
ram: Split host_from_stream_offset() into two helper functions
COLO: Load PVM's dirty pages into SVM's RAM cache temporarily
ram/COLO: Record the dirty pages that SVM received
COLO: Load VMState into qsb before restore it
COLO: Flush PVM's cached RAM into SVM's memory
COLO: Add checkpoint-delay parameter for migrate-set-parameters
COLO: synchronize PVM's state to SVM periodically
COLO failover: Introduce a new command to trigger a failover
COLO failover: Introduce state to record failover process
COLO: Implement failover work for Primary VM
COLO: Implement failover work for Secondary VM
COLO: implement default failover treatment
qmp event: Add event notification for COLO error
COLO failover: Shutdown related socket fd when do failover
COLO failover: Don't do failover during loading VM's state
COLO: Process shutdown command for VM in COLO state
COLO: Update the global runstate after going into colo state
savevm: Split load vm state function qemu_loadvm_state
COLO: Separate the process of saving/loading ram and device state
COLO: Split qemu_savevm_state_begin out of checkpoint process
net/filter-buffer: Add default filter-buffer for each netdev
filter-buffer: Accept zero interval
filter-buffer: Introduce a helper function to enable/disable default
filter
filter-buffer: Introduce a helper function to release packets
colo: Use default buffer-filter to buffer and release packets
COLO: Add block replication into colo process
configure | 11 +
docs/qmp-events.txt | 17 +
hmp-commands.hx | 15 +
hmp.c | 15 +
hmp.h | 1 +
include/exec/ram_addr.h | 1 +
include/migration/colo.h | 38 +++
include/migration/failover.h | 33 ++
include/migration/migration.h | 18 +-
include/migration/qemu-file.h | 3 +-
include/net/filter.h | 5 +
include/net/net.h | 4 +
include/sysemu/sysemu.h | 9 +
migration/Makefile.objs | 2 +
migration/colo-comm.c | 71 ++++
migration/colo-failover.c | 83 +++++
migration/colo.c | 773 ++++++++++++++++++++++++++++++++++++++++++
migration/exec.c | 4 +-
migration/fd.c | 4 +-
migration/migration.c | 215 ++++++++----
migration/postcopy-ram.c | 6 +-
migration/qemu-file-buf.c | 61 ++++
migration/ram.c | 195 ++++++++++-
migration/savevm.c | 295 ++++++++++++----
migration/tcp.c | 4 +-
migration/unix.c | 4 +-
net/filter-buffer.c | 123 ++++++-
net/net.c | 37 ++
qapi-schema.json | 110 +++++-
qapi/event.json | 17 +
qmp-commands.hx | 22 +-
stubs/Makefile.objs | 1 +
stubs/migration-colo.c | 45 +++
trace-events | 9 +
vl.c | 37 +-
35 files changed, 2105 insertions(+), 183 deletions(-)
create mode 100644 include/migration/colo.h
create mode 100644 include/migration/failover.h
create mode 100644 migration/colo-comm.c
create mode 100644 migration/colo-failover.c
create mode 100644 migration/colo.c
create mode 100644 stubs/migration-colo.c
--
1.8.3.1
^ permalink raw reply [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 01/39] configure: Add parameter for configure to enable/disable COLO support
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 02/39] migration: Introduce capability 'x-colo' to migration zhanghailiang
` (37 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
configure --enable-colo/--disable-colo to switch COLO
support on/off.
COLO support is On by default.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
---
v11:
- Turn COLO on by default (Eric's suggestion)
---
configure | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/configure b/configure
index 71d6cbc..7e0d947 100755
--- a/configure
+++ b/configure
@@ -260,6 +260,7 @@ xfs=""
vhost_net="no"
vhost_scsi="no"
kvm="no"
+colo="yes"
rdma=""
gprof="no"
debug_tcg="no"
@@ -937,6 +938,10 @@ for opt do
;;
--enable-kvm) kvm="yes"
;;
+ --disable-colo) colo="no"
+ ;;
+ --enable-colo) colo="yes"
+ ;;
--disable-tcg-interpreter) tcg_interpreter="no"
;;
--enable-tcg-interpreter) tcg_interpreter="yes"
@@ -1360,6 +1365,7 @@ disabled with --disable-FEATURE, default is enabled if available:
fdt fdt device tree
bluez bluez stack connectivity
kvm KVM acceleration support
+ colo COarse-grain LOck-stepping VM for Non-stop Service
rdma RDMA-based migration support
uuid uuid support
vde support for vde network
@@ -4760,6 +4766,7 @@ echo "Linux AIO support $linux_aio"
echo "ATTR/XATTR support $attr"
echo "Install blobs $blobs"
echo "KVM support $kvm"
+echo "COLO support $colo"
echo "RDMA support $rdma"
echo "TCG interpreter $tcg_interpreter"
echo "fdt support $fdt"
@@ -5349,6 +5356,10 @@ if have_backend "ftrace"; then
fi
echo "CONFIG_TRACE_FILE=$trace_file" >> $config_host_mak
+if test "$colo" = "yes"; then
+ echo "CONFIG_COLO=y" >> $config_host_mak
+fi
+
if test "$rdma" = "yes" ; then
echo "CONFIG_RDMA=y" >> $config_host_mak
fi
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 02/39] migration: Introduce capability 'x-colo' to migration
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 01/39] configure: Add parameter for configure to enable/disable COLO support zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 03/39] COLO: migrate colo related info to secondary node zhanghailiang
` (36 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, zhanghailiang,
arei.gonglei, stefanha, amit.shah, hongyang.yang
We add helper function colo_supported() to indicate whether
colo is supported or not, with which we use to control whether or not
showing 'x-colo' string to users, they can use qmp command
'query-migrate-capabilities' or hmp command 'info migrate_capabilities'
to learn if colo is supported.
Cc: Juan Quintela <quintela@redhat.com>
Cc: Amit Shah <amit.shah@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
v10:
- Rename capability 'colo' to experimental 'x-colo' (Eric's suggestion).
- Rename migrate_enable_colo() to migrate_colo_enabled() (Eric's suggestion).
---
include/migration/colo.h | 20 ++++++++++++++++++++
include/migration/migration.h | 1 +
migration/Makefile.objs | 1 +
migration/colo.c | 18 ++++++++++++++++++
migration/migration.c | 17 +++++++++++++++++
qapi-schema.json | 6 +++++-
qmp-commands.hx | 1 +
stubs/Makefile.objs | 1 +
stubs/migration-colo.c | 18 ++++++++++++++++++
9 files changed, 82 insertions(+), 1 deletion(-)
create mode 100644 include/migration/colo.h
create mode 100644 migration/colo.c
create mode 100644 stubs/migration-colo.c
diff --git a/include/migration/colo.h b/include/migration/colo.h
new file mode 100644
index 0000000..c60a590
--- /dev/null
+++ b/include/migration/colo.h
@@ -0,0 +1,20 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_COLO_H
+#define QEMU_COLO_H
+
+#include "qemu-common.h"
+
+bool colo_supported(void);
+
+#endif
diff --git a/include/migration/migration.h b/include/migration/migration.h
index fd018b7..1f004e4 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -268,6 +268,7 @@ int xbzrle_decode_buffer(uint8_t *src, int slen, uint8_t *dst, int dlen);
int migrate_use_xbzrle(void);
int64_t migrate_xbzrle_cache_size(void);
+bool migrate_colo_enabled(void);
int64_t xbzrle_cache_resize(int64_t new_size);
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index 0cac6d7..65ecc35 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -1,4 +1,5 @@
common-obj-y += migration.o tcp.o
+common-obj-$(CONFIG_COLO) += colo.o
common-obj-y += vmstate.o
common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
common-obj-y += xbzrle.o postcopy-ram.o
diff --git a/migration/colo.c b/migration/colo.c
new file mode 100644
index 0000000..2c40d2e
--- /dev/null
+++ b/migration/colo.c
@@ -0,0 +1,18 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#include "migration/colo.h"
+
+bool colo_supported(void)
+{
+ return true;
+}
diff --git a/migration/migration.c b/migration/migration.c
index 1a42aee..efacf52 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -33,6 +33,7 @@
#include "qom/cpu.h"
#include "exec/memory.h"
#include "exec/address-spaces.h"
+#include "migration/colo.h"
#define MAX_THROTTLE (32 << 20) /* Migration transfer speed throttling */
@@ -480,6 +481,9 @@ MigrationCapabilityStatusList *qmp_query_migrate_capabilities(Error **errp)
caps = NULL; /* silence compiler warning */
for (i = 0; i < MIGRATION_CAPABILITY_MAX; i++) {
+ if (i == MIGRATION_CAPABILITY_X_COLO && !colo_supported()) {
+ continue;
+ }
if (head == NULL) {
head = g_malloc0(sizeof(*caps));
caps = head;
@@ -679,6 +683,13 @@ void qmp_migrate_set_capabilities(MigrationCapabilityStatusList *params,
}
for (cap = params; cap; cap = cap->next) {
+ if (cap->value->capability == MIGRATION_CAPABILITY_X_COLO &&
+ !colo_supported()) {
+ error_setg(errp, "COLO is not currently supported, please"
+ " configure with --enable-colo option in order to"
+ " support COLO feature");
+ continue;
+ }
s->enabled_capabilities[cap->value->capability] = cap->value->state;
}
@@ -1581,6 +1592,12 @@ fail:
migrate_set_state(s, current_active_state, MIGRATION_STATUS_FAILED);
}
+bool migrate_colo_enabled(void)
+{
+ MigrationState *s = migrate_get_current();
+ return s->enabled_capabilities[MIGRATION_CAPABILITY_X_COLO];
+}
+
/*
* Master migration thread on the source VM.
* It drives the migration and pumps the data down the outgoing channel.
diff --git a/qapi-schema.json b/qapi-schema.json
index 8b1a423..d20c0ec 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -546,11 +546,15 @@
# been migrated, pulling the remaining pages along as needed. NOTE: If
# the migration fails during postcopy the VM will fail. (since 2.5)
#
+# @x-colo: If enabled, migration will never end, and the state of the VM on the
+# primary side will be migrated continuously to the VM on secondary
+# side. (since 2.6)
+#
# Since: 1.2
##
{ 'enum': 'MigrationCapability',
'data': ['xbzrle', 'rdma-pin-all', 'auto-converge', 'zero-blocks',
- 'compress', 'events', 'x-postcopy-ram'] }
+ 'compress', 'events', 'x-postcopy-ram', 'x-colo'] }
##
# @MigrationCapabilityStatus
diff --git a/qmp-commands.hx b/qmp-commands.hx
index e209688..92042d8 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -3625,6 +3625,7 @@ Query current migration capabilities
- "rdma-pin-all" : RDMA Pin Page state (json-bool)
- "auto-converge" : Auto Converge state (json-bool)
- "zero-blocks" : Zero Blocks state (json-bool)
+ - "x-colo" : COarse-Grain LOck Stepping for Non-stop Service (json-bool)
Arguments:
diff --git a/stubs/Makefile.objs b/stubs/Makefile.objs
index d7898a0..12eb4c6 100644
--- a/stubs/Makefile.objs
+++ b/stubs/Makefile.objs
@@ -38,3 +38,4 @@ stub-obj-y += qmp_pc_dimm_device_list.o
stub-obj-y += target-monitor-defs.o
stub-obj-y += target-get-monitor-def.o
stub-obj-y += vhost.o
+stub-obj-y += migration-colo.o
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
new file mode 100644
index 0000000..3d817df
--- /dev/null
+++ b/stubs/migration-colo.c
@@ -0,0 +1,18 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#include "migration/colo.h"
+
+bool colo_supported(void)
+{
+ return false;
+}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 03/39] COLO: migrate colo related info to secondary node
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 01/39] configure: Add parameter for configure to enable/disable COLO support zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 02/39] migration: Introduce capability 'x-colo' to migration zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 04/39] migration: Export migrate_set_state() zhanghailiang
` (35 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
We can know if VM in destination should go into COLO mode by refer to
the info that been migrated from PVM.
We skip this section if colo is not enabled (i.e.
migrate_set_capability colo off), so that, It not break compatibility with migration
however the --enable-colo/disable-colo on the source/destination;
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
v11:
- Add Reviewed-by tag
v10:
- Use VMSTATE_BOOL instead of VMSTATE_UNIT32 for 'colo_requested' (Dave's suggestion).
---
include/migration/colo.h | 2 ++
migration/Makefile.objs | 1 +
migration/colo-comm.c | 50 ++++++++++++++++++++++++++++++++++++++++++++++++
vl.c | 3 ++-
4 files changed, 55 insertions(+), 1 deletion(-)
create mode 100644 migration/colo-comm.c
diff --git a/include/migration/colo.h b/include/migration/colo.h
index c60a590..9b6662d 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -14,7 +14,9 @@
#define QEMU_COLO_H
#include "qemu-common.h"
+#include "migration/migration.h"
bool colo_supported(void);
+void colo_info_mig_init(void);
#endif
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index 65ecc35..81b5713 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -1,5 +1,6 @@
common-obj-y += migration.o tcp.o
common-obj-$(CONFIG_COLO) += colo.o
+common-obj-y += colo-comm.o
common-obj-y += vmstate.o
common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
common-obj-y += xbzrle.o postcopy-ram.o
diff --git a/migration/colo-comm.c b/migration/colo-comm.c
new file mode 100644
index 0000000..fb407e0
--- /dev/null
+++ b/migration/colo-comm.c
@@ -0,0 +1,50 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ *
+ */
+
+#include <migration/colo.h>
+#include "trace.h"
+
+typedef struct {
+ bool colo_requested;
+} COLOInfo;
+
+static COLOInfo colo_info;
+
+static void colo_info_pre_save(void *opaque)
+{
+ COLOInfo *s = opaque;
+
+ s->colo_requested = migrate_colo_enabled();
+}
+
+static bool colo_info_need(void *opaque)
+{
+ return migrate_colo_enabled();
+}
+
+static const VMStateDescription colo_state = {
+ .name = "COLOState",
+ .version_id = 1,
+ .minimum_version_id = 1,
+ .pre_save = colo_info_pre_save,
+ .needed = colo_info_need,
+ .fields = (VMStateField[]) {
+ VMSTATE_BOOL(colo_requested, COLOInfo),
+ VMSTATE_END_OF_LIST()
+ },
+};
+
+void colo_info_mig_init(void)
+{
+ vmstate_register(NULL, 0, &colo_state, &colo_info);
+}
diff --git a/vl.c b/vl.c
index 525929b..122199f 100644
--- a/vl.c
+++ b/vl.c
@@ -91,6 +91,7 @@ int main(int argc, char **argv)
#include "sysemu/dma.h"
#include "audio/audio.h"
#include "migration/migration.h"
+#include "migration/colo.h"
#include "sysemu/kvm.h"
#include "qapi/qmp/qjson.h"
#include "qemu/option.h"
@@ -4450,7 +4451,7 @@ int main(int argc, char **argv, char **envp)
blk_mig_init();
ram_mig_init();
-
+ colo_info_mig_init();
/* If the currently selected machine wishes to override the units-per-bus
* property of its default HBA interface type, do so now. */
if (machine_class->units_per_default_bus) {
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 04/39] migration: Export migrate_set_state()
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (2 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 03/39] COLO: migrate colo related info to secondary node zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 17:31 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 05/39] migration: Add state records for migration incoming zhanghailiang
` (34 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
Fix the first parameter of migrate_set_state(), and export it.
We will use it in later.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
---
v11:
- New patch which is split from patch
'migration: Add state records for migration incoming' (Juan's suggestion)
---
include/migration/migration.h | 2 ++
migration/migration.c | 36 +++++++++++++++++++++---------------
2 files changed, 23 insertions(+), 15 deletions(-)
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 1f004e4..4b19e80 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -169,6 +169,8 @@ struct MigrationState
RAMBlock *last_req_rb;
};
+void migrate_set_state(int *state, int old_state, int new_state);
+
void process_incoming_migration(QEMUFile *f);
void qemu_start_incoming_migration(const char *uri, Error **errp);
diff --git a/migration/migration.c b/migration/migration.c
index efacf52..7d98a3e 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -798,9 +798,9 @@ void qmp_migrate_start_postcopy(Error **errp)
/* shared migration helpers */
-static void migrate_set_state(MigrationState *s, int old_state, int new_state)
+void migrate_set_state(int *state, int old_state, int new_state)
{
- if (atomic_cmpxchg(&s->state, old_state, new_state) == old_state) {
+ if (atomic_cmpxchg(state, old_state, new_state) == old_state) {
trace_migrate_set_state(new_state);
migrate_generate_event(new_state);
}
@@ -833,7 +833,7 @@ static void migrate_fd_cleanup(void *opaque)
(s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE));
if (s->state == MIGRATION_STATUS_CANCELLING) {
- migrate_set_state(s, MIGRATION_STATUS_CANCELLING,
+ migrate_set_state(&s->state, MIGRATION_STATUS_CANCELLING,
MIGRATION_STATUS_CANCELLED);
}
@@ -844,7 +844,8 @@ void migrate_fd_error(MigrationState *s)
{
trace_migrate_fd_error();
assert(s->file == NULL);
- migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
+ migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
+ MIGRATION_STATUS_FAILED);
notifier_list_notify(&migration_state_notifiers, s);
}
@@ -864,7 +865,7 @@ static void migrate_fd_cancel(MigrationState *s)
if (!migration_is_setup_or_active(old_state)) {
break;
}
- migrate_set_state(s, old_state, MIGRATION_STATUS_CANCELLING);
+ migrate_set_state(&s->state, old_state, MIGRATION_STATUS_CANCELLING);
} while (s->state != MIGRATION_STATUS_CANCELLING);
/*
@@ -938,7 +939,7 @@ MigrationState *migrate_init(const MigrationParams *params)
s->migration_thread_running = false;
s->last_req_rb = NULL;
- migrate_set_state(s, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
+ migrate_set_state(&s->state, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
QSIMPLEQ_INIT(&s->src_page_requests);
@@ -1037,7 +1038,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
} else {
error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "uri",
"a valid migration protocol");
- migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
+ migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
+ MIGRATION_STATUS_FAILED);
return;
}
@@ -1416,7 +1418,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
int ret;
const QEMUSizedBuffer *qsb;
int64_t time_at_stop = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
- migrate_set_state(ms, MIGRATION_STATUS_ACTIVE,
+ migrate_set_state(&ms->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_POSTCOPY_ACTIVE);
trace_postcopy_start();
@@ -1507,7 +1509,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
ret = qemu_file_get_error(ms->file);
if (ret) {
error_report("postcopy_start: Migration stream errored");
- migrate_set_state(ms, MIGRATION_STATUS_POSTCOPY_ACTIVE,
+ migrate_set_state(&ms->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
MIGRATION_STATUS_FAILED);
}
@@ -1516,7 +1518,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
fail_closefb:
qemu_fclose(fb);
fail:
- migrate_set_state(ms, MIGRATION_STATUS_POSTCOPY_ACTIVE,
+ migrate_set_state(&ms->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
MIGRATION_STATUS_FAILED);
qemu_mutex_unlock_iothread();
return -1;
@@ -1585,11 +1587,13 @@ static void migration_completion(MigrationState *s, int current_active_state,
goto fail;
}
- migrate_set_state(s, current_active_state, MIGRATION_STATUS_COMPLETED);
+ migrate_set_state(&s->state, current_active_state,
+ MIGRATION_STATUS_COMPLETED);
return;
fail:
- migrate_set_state(s, current_active_state, MIGRATION_STATUS_FAILED);
+ migrate_set_state(&s->state, current_active_state,
+ MIGRATION_STATUS_FAILED);
}
bool migrate_colo_enabled(void)
@@ -1640,7 +1644,8 @@ static void *migration_thread(void *opaque)
s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
current_active_state = MIGRATION_STATUS_ACTIVE;
- migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_ACTIVE);
+ migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
+ MIGRATION_STATUS_ACTIVE);
trace_migration_thread_setup_complete();
@@ -1683,7 +1688,8 @@ static void *migration_thread(void *opaque)
}
if (qemu_file_get_error(s->file)) {
- migrate_set_state(s, current_active_state, MIGRATION_STATUS_FAILED);
+ migrate_set_state(&s->state, current_active_state,
+ MIGRATION_STATUS_FAILED);
trace_migration_thread_file_err();
break;
}
@@ -1764,7 +1770,7 @@ void migrate_fd_connect(MigrationState *s)
if (migrate_postcopy_ram()) {
if (open_return_path_on_source(s)) {
error_report("Unable to open return-path for postcopy");
- migrate_set_state(s, MIGRATION_STATUS_SETUP,
+ migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
MIGRATION_STATUS_FAILED);
migrate_fd_cleanup(s);
return;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 05/39] migration: Add state records for migration incoming
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (3 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 04/39] migration: Export migrate_set_state() zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 06/39] migration: Integrate COLO checkpoint process into migration zhanghailiang
` (33 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
For migration destination, we also need to know its state,
we will use it in COLO.
Here we add a new member 'state' for MigrationIncomingState,
and also use migrate_set_state() to modify its value.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
v11:
- Split exporting migrate_set_state() part into a new patch (Juan's suggestion)
---
include/migration/migration.h | 1 +
migration/migration.c | 14 +++++++++-----
2 files changed, 10 insertions(+), 5 deletions(-)
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 4b19e80..99dfa92 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -105,6 +105,7 @@ struct MigrationIncomingState {
QemuMutex rp_mutex; /* We send replies from multiple threads */
void *postcopy_tmp_page;
+ int state;
/* See savevm.c */
LoadStateEntry_Head loadvm_handlers;
};
diff --git a/migration/migration.c b/migration/migration.c
index 7d98a3e..390381d 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -112,6 +112,7 @@ MigrationIncomingState *migration_incoming_state_new(QEMUFile* f)
{
mis_current = g_new0(MigrationIncomingState, 1);
mis_current->from_src_file = f;
+ mis_current->state = MIGRATION_STATUS_NONE;
QLIST_INIT(&mis_current->loadvm_handlers);
qemu_mutex_init(&mis_current->rp_mutex);
qemu_event_init(&mis_current->main_thread_load_event, false);
@@ -332,8 +333,8 @@ static void process_incoming_migration_co(void *opaque)
mis = migration_incoming_state_new(f);
postcopy_state_set(POSTCOPY_INCOMING_NONE);
- migrate_generate_event(MIGRATION_STATUS_ACTIVE);
-
+ migrate_set_state(&mis->state, MIGRATION_STATUS_NONE,
+ MIGRATION_STATUS_ACTIVE);
ret = qemu_loadvm_state(f);
ps = postcopy_state_get();
@@ -362,7 +363,8 @@ static void process_incoming_migration_co(void *opaque)
migration_incoming_state_destroy();
if (ret < 0) {
- migrate_generate_event(MIGRATION_STATUS_FAILED);
+ migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+ MIGRATION_STATUS_FAILED);
error_report("load of migration failed: %s", strerror(-ret));
migrate_decompress_threads_join();
exit(EXIT_FAILURE);
@@ -371,7 +373,8 @@ static void process_incoming_migration_co(void *opaque)
/* Make sure all file formats flush their mutable metadata */
bdrv_invalidate_cache_all(&local_err);
if (local_err) {
- migrate_generate_event(MIGRATION_STATUS_FAILED);
+ migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+ MIGRATION_STATUS_FAILED);
error_report_err(local_err);
migrate_decompress_threads_join();
exit(EXIT_FAILURE);
@@ -403,7 +406,8 @@ static void process_incoming_migration_co(void *opaque)
* observer sees this event they might start to prod at the VM assuming
* it's ready to use.
*/
- migrate_generate_event(MIGRATION_STATUS_COMPLETED);
+ migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+ MIGRATION_STATUS_COMPLETED);
}
void process_incoming_migration(QEMUFile *f)
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 06/39] migration: Integrate COLO checkpoint process into migration
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (4 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 05/39] migration: Add state records for migration incoming zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 07/39] migration: Integrate COLO checkpoint process into loadvm zhanghailiang
` (32 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
Add a migrate state: MIGRATION_STATUS_COLO, enter this migration state
after the first live migration successfully finished.
We reuse migration thread, so if colo is enabled by user, migration thread will
go into the process of colo.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
v11:
- Rebase to master
- Add Reviewed-by tag
v10:
- Simplify process by dropping colo thread and reusing migration thread.
(Dave's suggestion)
---
include/migration/colo.h | 3 +++
migration/colo.c | 31 +++++++++++++++++++++++++++++++
migration/migration.c | 30 ++++++++++++++++++++++++++----
qapi-schema.json | 4 +++-
stubs/migration-colo.c | 9 +++++++++
trace-events | 3 +++
6 files changed, 75 insertions(+), 5 deletions(-)
diff --git a/include/migration/colo.h b/include/migration/colo.h
index 9b6662d..f462f06 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -19,4 +19,7 @@
bool colo_supported(void);
void colo_info_mig_init(void);
+void migrate_start_colo_process(MigrationState *s);
+bool migration_in_colo_state(void);
+
#endif
diff --git a/migration/colo.c b/migration/colo.c
index 2c40d2e..cf0ccb8 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -10,9 +10,40 @@
* later. See the COPYING file in the top-level directory.
*/
+#include "sysemu/sysemu.h"
#include "migration/colo.h"
+#include "trace.h"
bool colo_supported(void)
{
return true;
}
+
+bool migration_in_colo_state(void)
+{
+ MigrationState *s = migrate_get_current();
+
+ return (s->state == MIGRATION_STATUS_COLO);
+}
+
+static void colo_process_checkpoint(MigrationState *s)
+{
+ qemu_mutex_lock_iothread();
+ vm_start();
+ qemu_mutex_unlock_iothread();
+ trace_colo_vm_state_change("stop", "run");
+
+ /*TODO: COLO checkpoint savevm loop*/
+
+ migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
+ MIGRATION_STATUS_COMPLETED);
+}
+
+void migrate_start_colo_process(MigrationState *s)
+{
+ qemu_mutex_unlock_iothread();
+ migrate_set_state(&s->state, MIGRATION_STATUS_ACTIVE,
+ MIGRATION_STATUS_COLO);
+ colo_process_checkpoint(s);
+ qemu_mutex_lock_iothread();
+}
diff --git a/migration/migration.c b/migration/migration.c
index 390381d..46fe8a9 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -640,6 +640,10 @@ MigrationInfo *qmp_query_migrate(Error **errp)
get_xbzrle_cache_stats(info);
break;
+ case MIGRATION_STATUS_COLO:
+ info->has_status = true;
+ /* TODO: display COLO specific information (checkpoint info etc.) */
+ break;
case MIGRATION_STATUS_COMPLETED:
get_xbzrle_cache_stats(info);
@@ -999,7 +1003,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
params.shared = has_inc && inc;
if (migration_is_setup_or_active(s->state) ||
- s->state == MIGRATION_STATUS_CANCELLING) {
+ s->state == MIGRATION_STATUS_CANCELLING ||
+ s->state == MIGRATION_STATUS_COLO) {
error_setg(errp, QERR_MIGRATION_ACTIVE);
return;
}
@@ -1591,8 +1596,11 @@ static void migration_completion(MigrationState *s, int current_active_state,
goto fail;
}
- migrate_set_state(&s->state, current_active_state,
- MIGRATION_STATUS_COMPLETED);
+ if (!migrate_colo_enabled()) {
+ migrate_set_state(&s->state, current_active_state,
+ MIGRATION_STATUS_COMPLETED);
+ }
+
return;
fail:
@@ -1624,6 +1632,7 @@ static void *migration_thread(void *opaque)
bool entered_postcopy = false;
/* The active state we expect to be in; ACTIVE or POSTCOPY_ACTIVE */
enum MigrationStatus current_active_state = MIGRATION_STATUS_ACTIVE;
+ bool enable_colo = migrate_colo_enabled();
rcu_register_thread();
@@ -1731,7 +1740,11 @@ static void *migration_thread(void *opaque)
end_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
qemu_mutex_lock_iothread();
- qemu_savevm_state_cleanup();
+ /* The resource has been allocated by migration will be reused in COLO
+ process, so don't release them. */
+ if (!enable_colo) {
+ qemu_savevm_state_cleanup();
+ }
if (s->state == MIGRATION_STATUS_COMPLETED) {
uint64_t transferred_bytes = qemu_ftell(s->file);
s->total_time = end_time - s->total_time;
@@ -1744,6 +1757,15 @@ static void *migration_thread(void *opaque)
}
runstate_set(RUN_STATE_POSTMIGRATE);
} else {
+ if (s->state == MIGRATION_STATUS_ACTIVE && enable_colo) {
+ migrate_start_colo_process(s);
+ qemu_savevm_state_cleanup();
+ /*
+ * Fixme: we will run VM in COLO no matter its old running state.
+ * After exited COLO, we will keep running.
+ */
+ old_vm_running = true;
+ }
if (old_vm_running && !entered_postcopy) {
vm_start();
}
diff --git a/qapi-schema.json b/qapi-schema.json
index d20c0ec..24b35f3 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -434,6 +434,8 @@
#
# @completed: migration is finished.
#
+# @colo: VM is in the process of fault tolerance. (since 2.6)
+#
# @failed: some error occurred during migration process.
#
# Since: 2.3
@@ -441,7 +443,7 @@
##
{ 'enum': 'MigrationStatus',
'data': [ 'none', 'setup', 'cancelling', 'cancelled',
- 'active', 'postcopy-active', 'completed', 'failed' ] }
+ 'active', 'postcopy-active', 'completed', 'failed', 'colo' ] }
##
# @MigrationInfo
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index 3d817df..acddca6 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -16,3 +16,12 @@ bool colo_supported(void)
{
return false;
}
+
+bool migration_in_colo_state(void)
+{
+ return false;
+}
+
+void migrate_start_colo_process(MigrationState *s)
+{
+}
diff --git a/trace-events b/trace-events
index 0b0ff02..c98d473 100644
--- a/trace-events
+++ b/trace-events
@@ -1577,6 +1577,9 @@ postcopy_ram_incoming_cleanup_entry(void) ""
postcopy_ram_incoming_cleanup_exit(void) ""
postcopy_ram_incoming_cleanup_join(void) ""
+# migration/colo.c
+colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
+
# kvm-all.c
kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
kvm_vm_ioctl(int type, void *arg) "type 0x%x, arg %p"
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 07/39] migration: Integrate COLO checkpoint process into loadvm
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (5 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 06/39] migration: Integrate COLO checkpoint process into migration zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 18:14 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 08/39] migration: Rename the'file' member of MigrationState zhanghailiang
` (31 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
Switch from normal migration loadvm process into COLO checkpoint process if
COLO mode is enabled.
We add three new members to struct MigrationIncomingState, 'have_colo_incoming_thread'
and 'colo_incoming_thread' record the colo related threads for secondary VM,
'migration_incoming_co' records the original migration incoming coroutine.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
v11:
- We moved the place of bdrv_invalidate_cache_all(), but done the deleting work
in other patch. Fix it.
- Add documentation for colo in 'MigrationStatus' (Eric's review comment)
v10:
- fix a bug about fd leak which is found by Dave.
---
include/migration/colo.h | 7 +++++++
include/migration/migration.h | 7 +++++++
migration/colo-comm.c | 10 ++++++++++
migration/colo.c | 22 ++++++++++++++++++++++
migration/migration.c | 31 +++++++++++++++++++++----------
qapi-schema.json | 2 ++
stubs/migration-colo.c | 10 ++++++++++
7 files changed, 79 insertions(+), 10 deletions(-)
diff --git a/include/migration/colo.h b/include/migration/colo.h
index f462f06..2676c4a 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -15,6 +15,8 @@
#include "qemu-common.h"
#include "migration/migration.h"
+#include "qemu/coroutine_int.h"
+#include "qemu/thread.h"
bool colo_supported(void);
void colo_info_mig_init(void);
@@ -22,4 +24,9 @@ void colo_info_mig_init(void);
void migrate_start_colo_process(MigrationState *s);
bool migration_in_colo_state(void);
+/* loadvm */
+bool migration_incoming_enable_colo(void);
+void migration_incoming_exit_colo(void);
+void *colo_process_incoming_thread(void *opaque);
+bool migration_incoming_in_colo_state(void);
#endif
diff --git a/include/migration/migration.h b/include/migration/migration.h
index 99dfa92..a57a734 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -22,6 +22,7 @@
#include "migration/vmstate.h"
#include "qapi-types.h"
#include "exec/cpu-common.h"
+#include "qemu/coroutine_int.h"
#define QEMU_VM_FILE_MAGIC 0x5145564d
#define QEMU_VM_FILE_VERSION_COMPAT 0x00000002
@@ -106,6 +107,12 @@ struct MigrationIncomingState {
void *postcopy_tmp_page;
int state;
+
+ bool have_colo_incoming_thread;
+ QemuThread colo_incoming_thread;
+ /* The coroutine we should enter (back) after failover */
+ Coroutine *migration_incoming_co;
+
/* See savevm.c */
LoadStateEntry_Head loadvm_handlers;
};
diff --git a/migration/colo-comm.c b/migration/colo-comm.c
index fb407e0..30df3d3 100644
--- a/migration/colo-comm.c
+++ b/migration/colo-comm.c
@@ -48,3 +48,13 @@ void colo_info_mig_init(void)
{
vmstate_register(NULL, 0, &colo_state, &colo_info);
}
+
+bool migration_incoming_enable_colo(void)
+{
+ return colo_info.colo_requested;
+}
+
+void migration_incoming_exit_colo(void)
+{
+ colo_info.colo_requested = 0;
+}
diff --git a/migration/colo.c b/migration/colo.c
index cf0ccb8..6880aa0 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -13,6 +13,7 @@
#include "sysemu/sysemu.h"
#include "migration/colo.h"
#include "trace.h"
+#include "qemu/error-report.h"
bool colo_supported(void)
{
@@ -26,6 +27,13 @@ bool migration_in_colo_state(void)
return (s->state == MIGRATION_STATUS_COLO);
}
+bool migration_incoming_in_colo_state(void)
+{
+ MigrationIncomingState *mis = migration_incoming_get_current();
+
+ return mis && (mis->state == MIGRATION_STATUS_COLO);
+}
+
static void colo_process_checkpoint(MigrationState *s)
{
qemu_mutex_lock_iothread();
@@ -47,3 +55,17 @@ void migrate_start_colo_process(MigrationState *s)
colo_process_checkpoint(s);
qemu_mutex_lock_iothread();
}
+
+void *colo_process_incoming_thread(void *opaque)
+{
+ MigrationIncomingState *mis = opaque;
+
+ migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
+ MIGRATION_STATUS_COLO);
+
+ /* TODO: COLO checkpoint restore loop */
+
+ migration_incoming_exit_colo();
+
+ return NULL;
+}
diff --git a/migration/migration.c b/migration/migration.c
index 46fe8a9..41eac0d 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -358,6 +358,27 @@ static void process_incoming_migration_co(void *opaque)
/* Else if something went wrong then just fall out of the normal exit */
}
+ if (!ret) {
+ /* Make sure all file formats flush their mutable metadata */
+ bdrv_invalidate_cache_all(&local_err);
+ if (local_err) {
+ error_report_err(local_err);
+ migrate_decompress_threads_join();
+ exit(EXIT_FAILURE);
+ }
+ }
+ /* we get colo info, and know if we are in colo mode */
+ if (!ret && migration_incoming_enable_colo()) {
+ mis->migration_incoming_co = qemu_coroutine_self();
+ qemu_thread_create(&mis->colo_incoming_thread, "colo incoming",
+ colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE);
+ mis->have_colo_incoming_thread = true;
+ qemu_coroutine_yield();
+
+ /* Wait checkpoint incoming thread exit before free resource */
+ qemu_thread_join(&mis->colo_incoming_thread);
+ }
+
qemu_fclose(f);
free_xbzrle_decoded_buf();
migration_incoming_state_destroy();
@@ -370,16 +391,6 @@ static void process_incoming_migration_co(void *opaque)
exit(EXIT_FAILURE);
}
- /* Make sure all file formats flush their mutable metadata */
- bdrv_invalidate_cache_all(&local_err);
- if (local_err) {
- migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
- MIGRATION_STATUS_FAILED);
- error_report_err(local_err);
- migrate_decompress_threads_join();
- exit(EXIT_FAILURE);
- }
-
/*
* This must happen after all error conditions are dealt with and
* we're sure the VM is going to be running on this host.
diff --git a/qapi-schema.json b/qapi-schema.json
index 24b35f3..c2f3b63 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -438,6 +438,8 @@
#
# @failed: some error occurred during migration process.
#
+# @colo: VM is in the process of fault tolerance. (since 2.6)
+#
# Since: 2.3
#
##
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index acddca6..c12516e 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -22,6 +22,16 @@ bool migration_in_colo_state(void)
return false;
}
+bool migration_incoming_in_colo_state(void)
+{
+ return false;
+}
+
void migrate_start_colo_process(MigrationState *s)
{
}
+
+void *colo_process_incoming_thread(void *opaque)
+{
+ return NULL;
+}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 08/39] migration: Rename the'file' member of MigrationState
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (6 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 07/39] migration: Integrate COLO checkpoint process into loadvm zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 18:26 ` Dr. David Alan Gilbert
2015-12-10 6:41 ` Wen Congyang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 09/39] COLO/migration: Create a new communication path from destination to source zhanghailiang
` (30 subsequent siblings)
38 siblings, 2 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
Rename the 'file' member of MigrationState to 'to_dst_file'.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
v11:
- Only rename 'file' member of MigrationState
---
include/migration/migration.h | 2 +-
migration/exec.c | 4 +--
migration/fd.c | 4 +--
migration/migration.c | 72 ++++++++++++++++++++++---------------------
migration/postcopy-ram.c | 6 ++--
migration/savevm.c | 2 +-
migration/tcp.c | 4 +--
migration/unix.c | 4 +--
8 files changed, 51 insertions(+), 47 deletions(-)
diff --git a/include/migration/migration.h b/include/migration/migration.h
index a57a734..ba5bcec 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -140,7 +140,7 @@ struct MigrationState
size_t xfer_limit;
QemuThread thread;
QEMUBH *cleanup_bh;
- QEMUFile *file;
+ QEMUFile *to_dst_file;
int parameters[MIGRATION_PARAMETER_MAX];
int state;
diff --git a/migration/exec.c b/migration/exec.c
index 8406d2b..9037109 100644
--- a/migration/exec.c
+++ b/migration/exec.c
@@ -36,8 +36,8 @@
void exec_start_outgoing_migration(MigrationState *s, const char *command, Error **errp)
{
- s->file = qemu_popen_cmd(command, "w");
- if (s->file == NULL) {
+ s->to_dst_file = qemu_popen_cmd(command, "w");
+ if (s->to_dst_file == NULL) {
error_setg_errno(errp, errno, "failed to popen the migration target");
return;
}
diff --git a/migration/fd.c b/migration/fd.c
index 3e4bed0..9a9d6c5 100644
--- a/migration/fd.c
+++ b/migration/fd.c
@@ -50,9 +50,9 @@ void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **
}
if (fd_is_socket(fd)) {
- s->file = qemu_fopen_socket(fd, "wb");
+ s->to_dst_file = qemu_fopen_socket(fd, "wb");
} else {
- s->file = qemu_fdopen(fd, "wb");
+ s->to_dst_file = qemu_fdopen(fd, "wb");
}
migrate_fd_connect(s);
diff --git a/migration/migration.c b/migration/migration.c
index 41eac0d..a4c690d 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -834,7 +834,7 @@ static void migrate_fd_cleanup(void *opaque)
flush_page_queue(s);
- if (s->file) {
+ if (s->to_dst_file) {
trace_migrate_fd_cleanup();
qemu_mutex_unlock_iothread();
if (s->migration_thread_running) {
@@ -844,8 +844,8 @@ static void migrate_fd_cleanup(void *opaque)
qemu_mutex_lock_iothread();
migrate_compress_threads_join();
- qemu_fclose(s->file);
- s->file = NULL;
+ qemu_fclose(s->to_dst_file);
+ s->to_dst_file = NULL;
}
assert((s->state != MIGRATION_STATUS_ACTIVE) &&
@@ -862,7 +862,7 @@ static void migrate_fd_cleanup(void *opaque)
void migrate_fd_error(MigrationState *s)
{
trace_migrate_fd_error();
- assert(s->file == NULL);
+ assert(s->to_dst_file == NULL);
migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
MIGRATION_STATUS_FAILED);
notifier_list_notify(&migration_state_notifiers, s);
@@ -871,7 +871,7 @@ void migrate_fd_error(MigrationState *s)
static void migrate_fd_cancel(MigrationState *s)
{
int old_state ;
- QEMUFile *f = migrate_get_current()->file;
+ QEMUFile *f = migrate_get_current()->to_dst_file;
trace_migrate_fd_cancel();
if (s->rp_state.from_dst_file) {
@@ -942,7 +942,7 @@ MigrationState *migrate_init(const MigrationParams *params)
s->bytes_xfer = 0;
s->xfer_limit = 0;
s->cleanup_bh = 0;
- s->file = NULL;
+ s->to_dst_file = NULL;
s->state = MIGRATION_STATUS_NONE;
s->params = *params;
s->rp_state.from_dst_file = NULL;
@@ -1122,8 +1122,9 @@ void qmp_migrate_set_speed(int64_t value, Error **errp)
s = migrate_get_current();
s->bandwidth_limit = value;
- if (s->file) {
- qemu_file_set_rate_limit(s->file, s->bandwidth_limit / XFER_LIMIT_RATIO);
+ if (s->to_dst_file) {
+ qemu_file_set_rate_limit(s->to_dst_file,
+ s->bandwidth_limit / XFER_LIMIT_RATIO);
}
}
@@ -1393,7 +1394,7 @@ out:
static int open_return_path_on_source(MigrationState *ms)
{
- ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->file);
+ ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->to_dst_file);
if (!ms->rp_state.from_dst_file) {
return -1;
}
@@ -1415,7 +1416,7 @@ static int await_return_path_close_on_source(MigrationState *ms)
* rp_thread will exit, however if there's an error we need to cause
* it to exit.
*/
- if (qemu_file_get_error(ms->file) && ms->rp_state.from_dst_file) {
+ if (qemu_file_get_error(ms->to_dst_file) && ms->rp_state.from_dst_file) {
/*
* shutdown(2), if we have it, will cause it to unblock if it's stuck
* waiting for the destination.
@@ -1458,7 +1459,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
* Cause any non-postcopiable, but iterative devices to
* send out their final data.
*/
- qemu_savevm_state_complete_precopy(ms->file, true);
+ qemu_savevm_state_complete_precopy(ms->to_dst_file, true);
/*
* in Finish migrate and with the io-lock held everything should
@@ -1476,9 +1477,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
* will notice we're in POSTCOPY_ACTIVE and not actually
* wrap their state up here
*/
- qemu_file_set_rate_limit(ms->file, INT64_MAX);
+ qemu_file_set_rate_limit(ms->to_dst_file, INT64_MAX);
/* Ping just for debugging, helps line traces up */
- qemu_savevm_send_ping(ms->file, 2);
+ qemu_savevm_send_ping(ms->to_dst_file, 2);
/*
* While loading the device state we may trigger page transfer
@@ -1512,7 +1513,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
qsb = qemu_buf_get(fb);
/* Now send that blob */
- if (qemu_savevm_send_packaged(ms->file, qsb)) {
+ if (qemu_savevm_send_packaged(ms->to_dst_file, qsb)) {
goto fail_closefb;
}
qemu_fclose(fb);
@@ -1524,9 +1525,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
* Although this ping is just for debug, it could potentially be
* used for getting a better measurement of downtime at the source.
*/
- qemu_savevm_send_ping(ms->file, 4);
+ qemu_savevm_send_ping(ms->to_dst_file, 4);
- ret = qemu_file_get_error(ms->file);
+ ret = qemu_file_get_error(ms->to_dst_file);
if (ret) {
error_report("postcopy_start: Migration stream errored");
migrate_set_state(&ms->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
@@ -1569,8 +1570,8 @@ static void migration_completion(MigrationState *s, int current_active_state,
if (!ret) {
ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
if (ret >= 0) {
- qemu_file_set_rate_limit(s->file, INT64_MAX);
- qemu_savevm_state_complete_precopy(s->file, false);
+ qemu_file_set_rate_limit(s->to_dst_file, INT64_MAX);
+ qemu_savevm_state_complete_precopy(s->to_dst_file, false);
}
}
qemu_mutex_unlock_iothread();
@@ -1581,7 +1582,7 @@ static void migration_completion(MigrationState *s, int current_active_state,
} else if (s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) {
trace_migration_completion_postcopy_end();
- qemu_savevm_state_complete_postcopy(s->file);
+ qemu_savevm_state_complete_postcopy(s->to_dst_file);
trace_migration_completion_postcopy_end_after_complete();
}
@@ -1602,7 +1603,7 @@ static void migration_completion(MigrationState *s, int current_active_state,
}
}
- if (qemu_file_get_error(s->file)) {
+ if (qemu_file_get_error(s->to_dst_file)) {
trace_migration_completion_file_err();
goto fail;
}
@@ -1647,24 +1648,24 @@ static void *migration_thread(void *opaque)
rcu_register_thread();
- qemu_savevm_state_header(s->file);
+ qemu_savevm_state_header(s->to_dst_file);
if (migrate_postcopy_ram()) {
/* Now tell the dest that it should open its end so it can reply */
- qemu_savevm_send_open_return_path(s->file);
+ qemu_savevm_send_open_return_path(s->to_dst_file);
/* And do a ping that will make stuff easier to debug */
- qemu_savevm_send_ping(s->file, 1);
+ qemu_savevm_send_ping(s->to_dst_file, 1);
/*
* Tell the destination that we *might* want to do postcopy later;
* if the other end can't do postcopy it should fail now, nice and
* early.
*/
- qemu_savevm_send_postcopy_advise(s->file);
+ qemu_savevm_send_postcopy_advise(s->to_dst_file);
}
- qemu_savevm_state_begin(s->file, &s->params);
+ qemu_savevm_state_begin(s->to_dst_file, &s->params);
s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
current_active_state = MIGRATION_STATUS_ACTIVE;
@@ -1678,10 +1679,10 @@ static void *migration_thread(void *opaque)
int64_t current_time;
uint64_t pending_size;
- if (!qemu_file_rate_limit(s->file)) {
+ if (!qemu_file_rate_limit(s->to_dst_file)) {
uint64_t pend_post, pend_nonpost;
- qemu_savevm_state_pending(s->file, max_size, &pend_nonpost,
+ qemu_savevm_state_pending(s->to_dst_file, max_size, &pend_nonpost,
&pend_post);
pending_size = pend_nonpost + pend_post;
trace_migrate_pending(pending_size, max_size,
@@ -1702,7 +1703,7 @@ static void *migration_thread(void *opaque)
continue;
}
/* Just another iteration step */
- qemu_savevm_state_iterate(s->file, entered_postcopy);
+ qemu_savevm_state_iterate(s->to_dst_file, entered_postcopy);
} else {
trace_migration_thread_low_pending(pending_size);
migration_completion(s, current_active_state,
@@ -1711,7 +1712,7 @@ static void *migration_thread(void *opaque)
}
}
- if (qemu_file_get_error(s->file)) {
+ if (qemu_file_get_error(s->to_dst_file)) {
migrate_set_state(&s->state, current_active_state,
MIGRATION_STATUS_FAILED);
trace_migration_thread_file_err();
@@ -1719,7 +1720,8 @@ static void *migration_thread(void *opaque)
}
current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
if (current_time >= initial_time + BUFFER_DELAY) {
- uint64_t transferred_bytes = qemu_ftell(s->file) - initial_bytes;
+ uint64_t transferred_bytes = qemu_ftell(s->to_dst_file) -
+ initial_bytes;
uint64_t time_spent = current_time - initial_time;
double bandwidth = transferred_bytes / time_spent;
max_size = bandwidth * migrate_max_downtime() / 1000000;
@@ -1735,11 +1737,11 @@ static void *migration_thread(void *opaque)
s->expected_downtime = s->dirty_bytes_rate / bandwidth;
}
- qemu_file_reset_rate_limit(s->file);
+ qemu_file_reset_rate_limit(s->to_dst_file);
initial_time = current_time;
- initial_bytes = qemu_ftell(s->file);
+ initial_bytes = qemu_ftell(s->to_dst_file);
}
- if (qemu_file_rate_limit(s->file)) {
+ if (qemu_file_rate_limit(s->to_dst_file)) {
/* usleep expects microseconds */
g_usleep((initial_time + BUFFER_DELAY - current_time)*1000);
}
@@ -1757,7 +1759,7 @@ static void *migration_thread(void *opaque)
qemu_savevm_state_cleanup();
}
if (s->state == MIGRATION_STATUS_COMPLETED) {
- uint64_t transferred_bytes = qemu_ftell(s->file);
+ uint64_t transferred_bytes = qemu_ftell(s->to_dst_file);
s->total_time = end_time - s->total_time;
if (!entered_postcopy) {
s->downtime = end_time - start_time;
@@ -1794,7 +1796,7 @@ void migrate_fd_connect(MigrationState *s)
s->expected_downtime = max_downtime/1000000;
s->cleanup_bh = qemu_bh_new(migrate_fd_cleanup, s);
- qemu_file_set_rate_limit(s->file,
+ qemu_file_set_rate_limit(s->to_dst_file,
s->bandwidth_limit / XFER_LIMIT_RATIO);
/* Notify before starting migration thread */
diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
index 22d6b18..f74016d 100644
--- a/migration/postcopy-ram.c
+++ b/migration/postcopy-ram.c
@@ -733,7 +733,8 @@ void postcopy_discard_send_range(MigrationState *ms, PostcopyDiscardState *pds,
if (pds->cur_entry == MAX_DISCARDS_PER_COMMAND) {
/* Full set, ship it! */
- qemu_savevm_send_postcopy_ram_discard(ms->file, pds->ramblock_name,
+ qemu_savevm_send_postcopy_ram_discard(ms->to_dst_file,
+ pds->ramblock_name,
pds->cur_entry,
pds->start_list,
pds->length_list);
@@ -753,7 +754,8 @@ void postcopy_discard_send_finish(MigrationState *ms, PostcopyDiscardState *pds)
{
/* Anything unsent? */
if (pds->cur_entry) {
- qemu_savevm_send_postcopy_ram_discard(ms->file, pds->ramblock_name,
+ qemu_savevm_send_postcopy_ram_discard(ms->to_dst_file,
+ pds->ramblock_name,
pds->cur_entry,
pds->start_list,
pds->length_list);
diff --git a/migration/savevm.c b/migration/savevm.c
index 0ad1b93..f102870 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1163,7 +1163,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
.shared = 0
};
MigrationState *ms = migrate_init(¶ms);
- ms->file = f;
+ ms->to_dst_file = f;
if (qemu_savevm_state_blocked(errp)) {
return -EINVAL;
diff --git a/migration/tcp.c b/migration/tcp.c
index ae89172..e083d68 100644
--- a/migration/tcp.c
+++ b/migration/tcp.c
@@ -39,11 +39,11 @@ static void tcp_wait_for_connect(int fd, Error *err, void *opaque)
if (fd < 0) {
DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
- s->file = NULL;
+ s->to_dst_file = NULL;
migrate_fd_error(s);
} else {
DPRINTF("migrate connect success\n");
- s->file = qemu_fopen_socket(fd, "wb");
+ s->to_dst_file = qemu_fopen_socket(fd, "wb");
migrate_fd_connect(s);
}
}
diff --git a/migration/unix.c b/migration/unix.c
index b591813..5492dd6 100644
--- a/migration/unix.c
+++ b/migration/unix.c
@@ -39,11 +39,11 @@ static void unix_wait_for_connect(int fd, Error *err, void *opaque)
if (fd < 0) {
DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
- s->file = NULL;
+ s->to_dst_file = NULL;
migrate_fd_error(s);
} else {
DPRINTF("migrate connect success\n");
- s->file = qemu_fopen_socket(fd, "wb");
+ s->to_dst_file = qemu_fopen_socket(fd, "wb");
migrate_fd_connect(s);
}
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 09/39] COLO/migration: Create a new communication path from destination to source
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (7 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 08/39] migration: Rename the'file' member of MigrationState zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 18:40 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 10/39] COLO: Implement colo checkpoint protocol zhanghailiang
` (29 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
This new communication path will be used for returning messages
from destination to source.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
v11:
- Rebase master to use qemu_file_get_return_path() for opening return path
v10:
- fix the the error log (Dave's suggestion).
---
migration/colo.c | 39 +++++++++++++++++++++++++++++++++++++++
1 file changed, 39 insertions(+)
diff --git a/migration/colo.c b/migration/colo.c
index 6880aa0..0ab9618 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -36,6 +36,15 @@ bool migration_incoming_in_colo_state(void)
static void colo_process_checkpoint(MigrationState *s)
{
+ int ret = 0;
+
+ s->rp_state.from_dst_file = qemu_file_get_return_path(s->to_dst_file);
+ if (!s->rp_state.from_dst_file) {
+ ret = -EINVAL;
+ error_report("Open QEMUFile from_dst_file failed");
+ goto out;
+ }
+
qemu_mutex_lock_iothread();
vm_start();
qemu_mutex_unlock_iothread();
@@ -43,8 +52,16 @@ static void colo_process_checkpoint(MigrationState *s)
/*TODO: COLO checkpoint savevm loop*/
+out:
+ if (ret < 0) {
+ error_report("%s: %s", __func__, strerror(-ret));
+ }
migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
MIGRATION_STATUS_COMPLETED);
+
+ if (s->rp_state.from_dst_file) {
+ qemu_fclose(s->rp_state.from_dst_file);
+ }
}
void migrate_start_colo_process(MigrationState *s)
@@ -59,12 +76,34 @@ void migrate_start_colo_process(MigrationState *s)
void *colo_process_incoming_thread(void *opaque)
{
MigrationIncomingState *mis = opaque;
+ int ret = 0;
migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_COLO);
+ mis->to_src_file = qemu_file_get_return_path(mis->from_src_file);
+ if (!mis->to_src_file) {
+ ret = -EINVAL;
+ error_report("colo incoming thread: Open QEMUFile to_src_file failed");
+ goto out;
+ }
+ /* Note: We set the fd to unblocked in migration incoming coroutine,
+ * But here we are in the colo incoming thread, so it is ok to set the
+ * fd back to blocked.
+ */
+ qemu_set_block(qemu_get_fd(mis->from_src_file));
+
/* TODO: COLO checkpoint restore loop */
+out:
+ if (ret < 0) {
+ error_report("colo incoming thread will exit, detect error: %s",
+ strerror(-ret));
+ }
+
+ if (mis->to_src_file) {
+ qemu_fclose(mis->to_src_file);
+ }
migration_incoming_exit_colo();
return NULL;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 10/39] COLO: Implement colo checkpoint protocol
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (8 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 09/39] COLO/migration: Create a new communication path from destination to source zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 19:00 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 11/39] COLO: Add a new RunState RUN_STATE_COLO zhanghailiang
` (28 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
We need communications protocol of user-defined to control the checkpoint
process.
The new checkpoint request is started by Primary VM, and the interactive process
like below:
Checkpoint synchronizing points,
Primary Secondary
initial work
'checkpoint-ready' <------------------------------ @
'checkpoint-request' @ ----------------------------->
Suspend (Only in hybrid mode)
'checkpoint-reply' <------------------------------ @
Suspend&Save state
'vmstate-send' @ ----------------------------->
Send state Receive state
'vmstate-received' <------------------------------ @
Release packets Load state
'vmstate-load' <------------------------------ @
Resume Resume (Only in hybrid mode)
Start Comparing (Only in hybrid mode)
NOTE:
1) '@' who sends the message
2) Every sync-point is synchronized by two sides with only
one handshake(single direction) for low-latency.
If more strict synchronization is required, a opposite direction
sync-point should be added.
3) Since sync-points are single direction, the remote side may
go forward a lot when this side just receives the sync-point.
4) For now, we only support 'periodic' checkpoint, for which
the Secondary VM is not running, later we will support 'hybrid' mode.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Cc: Eric Blake <eblake@redhat.com>
---
v11:
- Add missing 'checkpoint-ready' communication in comment.
- Use parameter to return 'value' for colo_ctl_get() (Dave's suggestion)
- Fix trace for colo_ctl_get() to trace command and value both
v10:
- Rename enum COLOCmd to COLOCommand (Eric's suggestion).
- Remove unused 'ram-steal'
---
migration/colo.c | 198 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
qapi-schema.json | 27 ++++++++
trace-events | 2 +
3 files changed, 225 insertions(+), 2 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index 0ab9618..c045d61 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -10,10 +10,12 @@
* later. See the COPYING file in the top-level directory.
*/
+#include <unistd.h>
#include "sysemu/sysemu.h"
#include "migration/colo.h"
#include "trace.h"
#include "qemu/error-report.h"
+#include "qemu/sockets.h"
bool colo_supported(void)
{
@@ -34,9 +36,107 @@ bool migration_incoming_in_colo_state(void)
return mis && (mis->state == MIGRATION_STATUS_COLO);
}
+/* colo checkpoint control helper */
+static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value)
+{
+ int ret = 0;
+
+ qemu_put_be32(f, cmd);
+ qemu_put_be64(f, value);
+ qemu_fflush(f);
+
+ ret = qemu_file_get_error(f);
+ trace_colo_ctl_put(COLOCommand_lookup[cmd], value);
+
+ return ret;
+}
+
+static int colo_ctl_get_cmd(QEMUFile *f, uint32_t *cmd)
+{
+ int ret = 0;
+
+ *cmd = qemu_get_be32(f);
+ ret = qemu_file_get_error(f);
+ if (ret < 0) {
+ return ret;
+ }
+ if (*cmd >= COLO_COMMAND_MAX) {
+ error_report("Invalid colo command, got cmd:%d", *cmd);
+ return -EINVAL;
+ }
+
+ return 0;
+}
+
+static int colo_ctl_get(QEMUFile *f, uint32_t require, uint64_t *value)
+{
+ int ret;
+ uint32_t cmd;
+
+ ret = colo_ctl_get_cmd(f, &cmd);
+ if (ret < 0) {
+ return ret;
+ }
+ if (cmd != require) {
+ error_report("Unexpect colo command, expect:%d, but got cmd:%d",
+ require, cmd);
+ return -EINVAL;
+ }
+
+ *value = qemu_get_be64(f);
+ trace_colo_ctl_get(COLOCommand_lookup[cmd], *value);
+ ret = qemu_file_get_error(f);
+
+ return ret;
+}
+
+static int colo_do_checkpoint_transaction(MigrationState *s)
+{
+ int ret;
+ uint64_t value;
+
+ ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_CHECKPOINT_REQUEST, 0);
+ if (ret < 0) {
+ goto out;
+ }
+
+ ret = colo_ctl_get(s->rp_state.from_dst_file,
+ COLO_COMMAND_CHECKPOINT_REPLY, &value);
+ if (ret < 0) {
+ goto out;
+ }
+
+ /* TODO: suspend and save vm state to colo buffer */
+
+ ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
+ if (ret < 0) {
+ goto out;
+ }
+
+ /* TODO: send vmstate to Secondary */
+
+ ret = colo_ctl_get(s->rp_state.from_dst_file,
+ COLO_COMMAND_VMSTATE_RECEIVED, &value);
+ if (ret < 0) {
+ goto out;
+ }
+
+ ret = colo_ctl_get(s->rp_state.from_dst_file,
+ COLO_COMMAND_VMSTATE_LOADED, &value);
+ if (ret < 0) {
+ goto out;
+ }
+
+ /* TODO: resume Primary */
+
+out:
+ return ret;
+}
+
static void colo_process_checkpoint(MigrationState *s)
{
int ret = 0;
+ uint64_t value;
s->rp_state.from_dst_file = qemu_file_get_return_path(s->to_dst_file);
if (!s->rp_state.from_dst_file) {
@@ -45,12 +145,28 @@ static void colo_process_checkpoint(MigrationState *s)
goto out;
}
+ /*
+ * Wait for Secondary finish loading vm states and enter COLO
+ * restore.
+ */
+ ret = colo_ctl_get(s->rp_state.from_dst_file,
+ COLO_COMMAND_CHECKPOINT_READY, &value);
+ if (ret < 0) {
+ goto out;
+ }
+
qemu_mutex_lock_iothread();
vm_start();
qemu_mutex_unlock_iothread();
trace_colo_vm_state_change("stop", "run");
- /*TODO: COLO checkpoint savevm loop*/
+ while (s->state == MIGRATION_STATUS_COLO) {
+ /* start a colo checkpoint */
+ ret = colo_do_checkpoint_transaction(s);
+ if (ret < 0) {
+ goto out;
+ }
+ }
out:
if (ret < 0) {
@@ -73,10 +189,46 @@ void migrate_start_colo_process(MigrationState *s)
qemu_mutex_lock_iothread();
}
+/*
+ * return:
+ * 0: start a checkpoint
+ * -1: some error happened, exit colo restore
+ */
+static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
+{
+ int ret;
+ uint32_t cmd;
+ uint64_t value;
+
+ ret = colo_ctl_get_cmd(f, &cmd);
+ if (ret < 0) {
+ /* do failover ? */
+ return ret;
+ }
+ /* Fix me: this value should be 0, which is not so good,
+ * should be used for checking ?
+ */
+ value = qemu_get_be64(f);
+ if (value != 0) {
+ error_report("Got unexpected value %" PRIu64 " for '%s' command",
+ value, COLOCommand_lookup[cmd]);
+ return -EINVAL;
+ }
+
+ switch (cmd) {
+ case COLO_COMMAND_CHECKPOINT_REQUEST:
+ *checkpoint_request = 1;
+ return 0;
+ default:
+ return -EINVAL;
+ }
+}
+
void *colo_process_incoming_thread(void *opaque)
{
MigrationIncomingState *mis = opaque;
int ret = 0;
+ uint64_t value;
migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_COLO);
@@ -93,7 +245,49 @@ void *colo_process_incoming_thread(void *opaque)
*/
qemu_set_block(qemu_get_fd(mis->from_src_file));
- /* TODO: COLO checkpoint restore loop */
+
+ ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_READY, 0);
+ if (ret < 0) {
+ goto out;
+ }
+
+ while (mis->state == MIGRATION_STATUS_COLO) {
+ int request = 0;
+ int ret = colo_wait_handle_cmd(mis->from_src_file, &request);
+
+ if (ret < 0) {
+ break;
+ } else {
+ if (!request) {
+ continue;
+ }
+ }
+ /* FIXME: This is unnecessary for periodic checkpoint mode */
+ ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_REPLY, 0);
+ if (ret < 0) {
+ goto out;
+ }
+
+ ret = colo_ctl_get(mis->from_src_file, COLO_COMMAND_VMSTATE_SEND,
+ &value);
+ if (ret < 0) {
+ goto out;
+ }
+
+ /* TODO: read migration data into colo buffer */
+
+ ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_RECEIVED, 0);
+ if (ret < 0) {
+ goto out;
+ }
+
+ /* TODO: load vm state */
+
+ ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_LOADED, 0);
+ if (ret < 0) {
+ goto out;
+ }
+ }
out:
if (ret < 0) {
diff --git a/qapi-schema.json b/qapi-schema.json
index c2f3b63..48a87af 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -722,6 +722,33 @@
{ 'command': 'migrate-start-postcopy' }
##
+# @COLOCommand
+#
+# The commands for COLO fault tolerance
+#
+# @invalid: unknown command
+#
+# @checkpoint-ready: SVM is ready for checkpointing
+#
+# @checkpoint-request: PVM tells SVM to prepare for new checkpointing
+#
+# @checkpoint-reply: SVM gets PVM's checkpoint request
+#
+# @vmstate-send: VM's state will be sent by PVM.
+#
+# @vmstate-size: The total size of VMstate.
+#
+# @vmstate-received: VM's state has been received by SVM
+#
+# @vmstate-loaded: VM's state has been loaded by SVM
+#
+# Since: 2.6
+##
+{ 'enum': 'COLOCommand',
+ 'data': [ 'invalid', 'checkpoint-ready', 'checkpoint-request',
+ 'checkpoint-reply', 'vmstate-send', 'vmstate-size',
+ 'vmstate-received', 'vmstate-loaded' ] }
+
# @MouseInfo:
#
# Information about a mouse device.
diff --git a/trace-events b/trace-events
index c98d473..f8a0959 100644
--- a/trace-events
+++ b/trace-events
@@ -1579,6 +1579,8 @@ postcopy_ram_incoming_cleanup_join(void) ""
# migration/colo.c
colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
+colo_ctl_put(const char *msg, uint64_t value) "Send '%s' cmd, value: %" PRIu64""
+colo_ctl_get(const char *msg, uint64_t value) "Receive '%s' cmd, value: %" PRIu64""
# kvm-all.c
kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 11/39] COLO: Add a new RunState RUN_STATE_COLO
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (9 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 10/39] COLO: Implement colo checkpoint protocol zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 12/39] QEMUSizedBuffer: Introduce two help functions for qsb zhanghailiang
` (27 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, zhanghailiang,
arei.gonglei, stefanha, amit.shah, hongyang.yang
Guest will enter this state when paused to save/restore VM state
under colo checkpoint.
Cc: Eric Blake <eblake@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Eric Blake <eblake@redhat.com>
---
qapi-schema.json | 5 ++++-
vl.c | 8 ++++++++
2 files changed, 12 insertions(+), 1 deletion(-)
diff --git a/qapi-schema.json b/qapi-schema.json
index 48a87af..5ee089d 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -154,12 +154,15 @@
# @watchdog: the watchdog action is configured to pause and has been triggered
#
# @guest-panicked: guest has been panicked as a result of guest OS panic
+#
+# @colo: guest is paused to save/restore VM state under colo checkpoint (since
+# 2.6)
##
{ 'enum': 'RunState',
'data': [ 'debug', 'inmigrate', 'internal-error', 'io-error', 'paused',
'postmigrate', 'prelaunch', 'finish-migrate', 'restore-vm',
'running', 'save-vm', 'shutdown', 'suspended', 'watchdog',
- 'guest-panicked' ] }
+ 'guest-panicked', 'colo' ] }
##
# @StatusInfo:
diff --git a/vl.c b/vl.c
index 122199f..083a60c 100644
--- a/vl.c
+++ b/vl.c
@@ -594,6 +594,7 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_INMIGRATE, RUN_STATE_WATCHDOG },
{ RUN_STATE_INMIGRATE, RUN_STATE_GUEST_PANICKED },
{ RUN_STATE_INMIGRATE, RUN_STATE_FINISH_MIGRATE },
+ { RUN_STATE_INMIGRATE, RUN_STATE_COLO },
{ RUN_STATE_INTERNAL_ERROR, RUN_STATE_PAUSED },
{ RUN_STATE_INTERNAL_ERROR, RUN_STATE_FINISH_MIGRATE },
@@ -603,6 +604,7 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_PAUSED, RUN_STATE_RUNNING },
{ RUN_STATE_PAUSED, RUN_STATE_FINISH_MIGRATE },
+ { RUN_STATE_PAUSED, RUN_STATE_COLO},
{ RUN_STATE_POSTMIGRATE, RUN_STATE_RUNNING },
{ RUN_STATE_POSTMIGRATE, RUN_STATE_FINISH_MIGRATE },
@@ -613,9 +615,12 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_FINISH_MIGRATE, RUN_STATE_RUNNING },
{ RUN_STATE_FINISH_MIGRATE, RUN_STATE_POSTMIGRATE },
+ { RUN_STATE_FINISH_MIGRATE, RUN_STATE_COLO},
{ RUN_STATE_RESTORE_VM, RUN_STATE_RUNNING },
+ { RUN_STATE_COLO, RUN_STATE_RUNNING },
+
{ RUN_STATE_RUNNING, RUN_STATE_DEBUG },
{ RUN_STATE_RUNNING, RUN_STATE_INTERNAL_ERROR },
{ RUN_STATE_RUNNING, RUN_STATE_IO_ERROR },
@@ -626,6 +631,7 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_RUNNING, RUN_STATE_SHUTDOWN },
{ RUN_STATE_RUNNING, RUN_STATE_WATCHDOG },
{ RUN_STATE_RUNNING, RUN_STATE_GUEST_PANICKED },
+ { RUN_STATE_RUNNING, RUN_STATE_COLO},
{ RUN_STATE_SAVE_VM, RUN_STATE_RUNNING },
@@ -636,9 +642,11 @@ static const RunStateTransition runstate_transitions_def[] = {
{ RUN_STATE_RUNNING, RUN_STATE_SUSPENDED },
{ RUN_STATE_SUSPENDED, RUN_STATE_RUNNING },
{ RUN_STATE_SUSPENDED, RUN_STATE_FINISH_MIGRATE },
+ { RUN_STATE_SUSPENDED, RUN_STATE_COLO},
{ RUN_STATE_WATCHDOG, RUN_STATE_RUNNING },
{ RUN_STATE_WATCHDOG, RUN_STATE_FINISH_MIGRATE },
+ { RUN_STATE_WATCHDOG, RUN_STATE_COLO},
{ RUN_STATE_GUEST_PANICKED, RUN_STATE_RUNNING },
{ RUN_STATE_GUEST_PANICKED, RUN_STATE_FINISH_MIGRATE },
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 12/39] QEMUSizedBuffer: Introduce two help functions for qsb
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (10 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 11/39] COLO: Add a new RunState RUN_STATE_COLO zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 13/39] COLO: Save PVM state to secondary side when do checkpoint zhanghailiang
` (26 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
Introduce two new QEMUSizedBuffer APIs which will be used by COLO to buffer
VM state:
One is qsb_put_buffer(), which put the content of a given QEMUSizedBuffer
into QEMUFile, this is used to send buffered VM state to secondary.
Another is qsb_fill_buffer(), read 'size' bytes of data from the file into
qsb, this is used to get VM state from socket into a buffer.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
v11:
- size_t'ify these two help functions (Dave's suggestion)
---
include/migration/qemu-file.h | 3 ++-
migration/qemu-file-buf.c | 61 +++++++++++++++++++++++++++++++++++++++++++
2 files changed, 63 insertions(+), 1 deletion(-)
diff --git a/include/migration/qemu-file.h b/include/migration/qemu-file.h
index b5d08d2..ca6a582 100644
--- a/include/migration/qemu-file.h
+++ b/include/migration/qemu-file.h
@@ -150,7 +150,8 @@ ssize_t qsb_get_buffer(const QEMUSizedBuffer *, off_t start, size_t count,
uint8_t *buf);
ssize_t qsb_write_at(QEMUSizedBuffer *qsb, const uint8_t *buf,
off_t pos, size_t count);
-
+void qsb_put_buffer(QEMUFile *f, QEMUSizedBuffer *qsb, size_t size);
+size_t qsb_fill_buffer(QEMUSizedBuffer *qsb, QEMUFile *f, size_t size);
/*
* For use on files opened with qemu_bufopen
diff --git a/migration/qemu-file-buf.c b/migration/qemu-file-buf.c
index 49516b8..c50a495 100644
--- a/migration/qemu-file-buf.c
+++ b/migration/qemu-file-buf.c
@@ -366,6 +366,67 @@ ssize_t qsb_write_at(QEMUSizedBuffer *qsb, const uint8_t *source,
return count;
}
+/**
+ * Put the content of a given QEMUSizedBuffer into QEMUFile.
+ *
+ * @f: A QEMUFile
+ * @qsb: A QEMUSizedBuffer
+ * @size: size of content to write
+ */
+void qsb_put_buffer(QEMUFile *f, QEMUSizedBuffer *qsb, size_t size)
+{
+ size_t l;
+ int i;
+
+ for (i = 0; i < qsb->n_iov && size > 0; i++) {
+ l = MIN(qsb->iov[i].iov_len, size);
+ qemu_put_buffer(f, qsb->iov[i].iov_base, l);
+ size -= l;
+ }
+}
+
+/*
+ * Read 'size' bytes of data from the file into qsb.
+ * always fill from pos 0 and used after qsb_create().
+ *
+ * It will return size bytes unless there was an error, in which case it will
+ * return as many as it managed to read (assuming blocking fd's which
+ * all current QEMUFile are)
+ */
+size_t qsb_fill_buffer(QEMUSizedBuffer *qsb, QEMUFile *f, size_t size)
+{
+ ssize_t rc = qsb_grow(qsb, size);
+ ssize_t pending = size;
+ int i;
+ uint8_t *buf = NULL;
+
+ qsb->used = 0;
+
+ if (rc < 0) {
+ return rc;
+ }
+
+ for (i = 0; i < qsb->n_iov && pending > 0; i++) {
+ size_t doneone = 0;
+ /* read until iov full */
+ while (doneone < qsb->iov[i].iov_len && pending > 0) {
+ size_t readone = 0;
+
+ buf = qsb->iov[i].iov_base;
+ readone = qemu_get_buffer(f, buf,
+ MIN(qsb->iov[i].iov_len - doneone, pending));
+ if (readone == 0) {
+ return qsb->used;
+ }
+ buf += readone;
+ doneone += readone;
+ pending -= readone;
+ qsb->used += readone;
+ }
+ }
+ return qsb->used;
+}
+
typedef struct QEMUBuffer {
QEMUSizedBuffer *qsb;
QEMUFile *file;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 13/39] COLO: Save PVM state to secondary side when do checkpoint
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (11 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 12/39] QEMUSizedBuffer: Introduce two help functions for qsb zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 14/39] ram: Split host_from_stream_offset() into two helper functions zhanghailiang
` (25 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
The main process of checkpoint is to synchronize SVM with PVM.
VM's state includes ram and device state. So we will migrate PVM's
state to SVM when do checkpoint, just like migration does.
We will cache PVM's state in slave, we use QEMUSizedBuffer
to store the data, we need to know the size of VM state, so in master,
we use qsb to store VM state temporarily, get the data size by call qsb_get_length()
and then migrate the data to the qsb in the secondary side.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
v11:
- Add Reviewed-by tag
---
migration/colo.c | 68 +++++++++++++++++++++++++++++++++++++++++++++++++++-----
migration/ram.c | 39 +++++++++++++++++++++++---------
2 files changed, 92 insertions(+), 15 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index c045d61..012d8e5 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -17,6 +17,9 @@
#include "qemu/error-report.h"
#include "qemu/sockets.h"
+/* colo buffer */
+#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
+
bool colo_supported(void)
{
return true;
@@ -90,10 +93,13 @@ static int colo_ctl_get(QEMUFile *f, uint32_t require, uint64_t *value)
return ret;
}
-static int colo_do_checkpoint_transaction(MigrationState *s)
+static int colo_do_checkpoint_transaction(MigrationState *s,
+ QEMUSizedBuffer *buffer)
{
int ret;
uint64_t value;
+ size_t size;
+ QEMUFile *trans = NULL;
ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_CHECKPOINT_REQUEST, 0);
if (ret < 0) {
@@ -105,15 +111,47 @@ static int colo_do_checkpoint_transaction(MigrationState *s)
if (ret < 0) {
goto out;
}
+ /* Reset colo buffer and open it for write */
+ qsb_set_length(buffer, 0);
+ trans = qemu_bufopen("w", buffer);
+ if (!trans) {
+ error_report("Open colo buffer for write failed");
+ goto out;
+ }
- /* TODO: suspend and save vm state to colo buffer */
+ qemu_mutex_lock_iothread();
+ vm_stop_force_state(RUN_STATE_COLO);
+ qemu_mutex_unlock_iothread();
+ trace_colo_vm_state_change("run", "stop");
+
+ /* Disable block migration */
+ s->params.blk = 0;
+ s->params.shared = 0;
+ qemu_savevm_state_header(trans);
+ qemu_savevm_state_begin(trans, &s->params);
+ qemu_mutex_lock_iothread();
+ qemu_savevm_state_complete_precopy(trans, false);
+ qemu_mutex_unlock_iothread();
+
+ qemu_fflush(trans);
ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
if (ret < 0) {
goto out;
}
+ /* we send the total size of the vmstate first */
+ size = qsb_get_length(buffer);
+ ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SIZE, size);
+ if (ret < 0) {
+ goto out;
+ }
- /* TODO: send vmstate to Secondary */
+ qsb_put_buffer(s->to_dst_file, buffer, size);
+ qemu_fflush(s->to_dst_file);
+ ret = qemu_file_get_error(s->to_dst_file);
+ if (ret < 0) {
+ goto out;
+ }
ret = colo_ctl_get(s->rp_state.from_dst_file,
COLO_COMMAND_VMSTATE_RECEIVED, &value);
@@ -127,14 +165,24 @@ static int colo_do_checkpoint_transaction(MigrationState *s)
goto out;
}
- /* TODO: resume Primary */
+ ret = 0;
+ /* Resume primary guest */
+ qemu_mutex_lock_iothread();
+ vm_start();
+ qemu_mutex_unlock_iothread();
+ trace_colo_vm_state_change("stop", "run");
out:
+ if (trans) {
+ qemu_fclose(trans);
+ }
+
return ret;
}
static void colo_process_checkpoint(MigrationState *s)
{
+ QEMUSizedBuffer *buffer = NULL;
int ret = 0;
uint64_t value;
@@ -155,6 +203,13 @@ static void colo_process_checkpoint(MigrationState *s)
goto out;
}
+ buffer = qsb_create(NULL, COLO_BUFFER_BASE_SIZE);
+ if (buffer == NULL) {
+ ret = -ENOMEM;
+ error_report("Failed to allocate colo buffer!");
+ goto out;
+ }
+
qemu_mutex_lock_iothread();
vm_start();
qemu_mutex_unlock_iothread();
@@ -162,7 +217,7 @@ static void colo_process_checkpoint(MigrationState *s)
while (s->state == MIGRATION_STATUS_COLO) {
/* start a colo checkpoint */
- ret = colo_do_checkpoint_transaction(s);
+ ret = colo_do_checkpoint_transaction(s, buffer);
if (ret < 0) {
goto out;
}
@@ -175,6 +230,9 @@ out:
migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
MIGRATION_STATUS_COMPLETED);
+ qsb_free(buffer);
+ buffer = NULL;
+
if (s->rp_state.from_dst_file) {
qemu_fclose(s->rp_state.from_dst_file);
}
diff --git a/migration/ram.c b/migration/ram.c
index 1eb155a..cfe78aa 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -39,6 +39,7 @@
#include "trace.h"
#include "exec/ram_addr.h"
#include "qemu/rcu_queue.h"
+#include "migration/colo.h"
#ifdef DEBUG_MIGRATION_RAM
#define DPRINTF(fmt, ...) \
@@ -1857,16 +1858,8 @@ err:
return ret;
}
-
-/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has
- * long-running RCU critical section. When rcu-reclaims in the code
- * start to become numerous it will be necessary to reduce the
- * granularity of these critical sections.
- */
-
-static int ram_save_setup(QEMUFile *f, void *opaque)
+static int ram_save_init_globals(void)
{
- RAMBlock *block;
int64_t ram_bitmap_pages; /* Size of bitmap in pages, including gaps */
dirty_rate_high_cnt = 0;
@@ -1931,6 +1924,31 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
migration_bitmap_sync();
qemu_mutex_unlock_ramlist();
qemu_mutex_unlock_iothread();
+ rcu_read_unlock();
+
+ return 0;
+}
+
+/* Each of ram_save_setup, ram_save_iterate and ram_save_complete has
+ * long-running RCU critical section. When rcu-reclaims in the code
+ * start to become numerous it will be necessary to reduce the
+ * granularity of these critical sections.
+ */
+
+static int ram_save_setup(QEMUFile *f, void *opaque)
+{
+ RAMBlock *block;
+
+ /*
+ * migration has already setup the bitmap, reuse it.
+ */
+ if (!migration_in_colo_state()) {
+ if (ram_save_init_globals() < 0) {
+ return -1;
+ }
+ }
+
+ rcu_read_lock();
qemu_put_be64(f, ram_bytes_total() | RAM_SAVE_FLAG_MEM_SIZE);
@@ -2032,7 +2050,8 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
while (true) {
int pages;
- pages = ram_find_and_save_block(f, true, &bytes_transferred);
+ pages = ram_find_and_save_block(f, !migration_in_colo_state(),
+ &bytes_transferred);
/* no more blocks to sent */
if (pages == 0) {
break;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 14/39] ram: Split host_from_stream_offset() into two helper functions
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (12 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 13/39] COLO: Save PVM state to secondary side when do checkpoint zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-12-01 18:19 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 15/39] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily zhanghailiang
` (24 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
Split host_from_stream_offset() into two parts:
One is to get ram block, which the block idstr may be get from migration
stream, the other is to get hva (host) address from block and the offset.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
---
v11:
- New patch
---
migration/ram.c | 29 +++++++++++++++++++++--------
1 file changed, 21 insertions(+), 8 deletions(-)
diff --git a/migration/ram.c b/migration/ram.c
index cfe78aa..a161620 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2136,9 +2136,9 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
* offset: Offset within the block
* flags: Page flags (mostly to see if it's a continuation of previous block)
*/
-static inline void *host_from_stream_offset(QEMUFile *f,
- ram_addr_t offset,
- int flags)
+static inline RAMBlock *ram_block_from_stream(QEMUFile *f,
+ ram_addr_t offset,
+ int flags)
{
static RAMBlock *block = NULL;
char id[256];
@@ -2150,22 +2150,31 @@ static inline void *host_from_stream_offset(QEMUFile *f,
return NULL;
}
- return block->host + offset;
+ return block;
}
-
len = qemu_get_byte(f);
qemu_get_buffer(f, (uint8_t *)id, len);
id[len] = 0;
block = qemu_ram_block_by_name(id);
if (block && block->max_length > offset) {
- return block->host + offset;
+ return block;
}
error_report("Can't find block %s", id);
return NULL;
}
+static inline void *host_from_ram_block_offset(RAMBlock *block,
+ ram_addr_t offset)
+{
+ if (!block) {
+ return NULL;
+ }
+
+ return block->host + offset;
+}
+
/*
* If a page (or a whole RDMA chunk) has been
* determined to be zero, then zap it.
@@ -2310,7 +2319,9 @@ static int ram_load_postcopy(QEMUFile *f)
trace_ram_load_postcopy_loop((uint64_t)addr, flags);
place_needed = false;
if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE)) {
- host = host_from_stream_offset(f, addr, flags);
+ RAMBlock *block = ram_block_from_stream(f, addr, flags);
+
+ host = host_from_ram_block_offset(block, addr);
if (!host) {
error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
ret = -EINVAL;
@@ -2441,7 +2452,9 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE |
RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) {
- host = host_from_stream_offset(f, addr, flags);
+ RAMBlock *block = ram_block_from_stream(f, addr, flags);
+
+ host = host_from_ram_block_offset(block, addr);
if (!host) {
error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
ret = -EINVAL;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 15/39] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (13 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 14/39] ram: Split host_from_stream_offset() into two helper functions zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-12-01 19:02 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 16/39] ram/COLO: Record the dirty pages that SVM received zhanghailiang
` (23 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
We should not load PVM's state directly into SVM, because there maybe some
errors happen when SVM is receving data, which will break SVM.
We need to ensure receving all data before load the state into SVM. We use
an extra memory to cache these data (PVM's ram). The ram cache in secondary side
is initially the same as SVM/PVM's memory. And in the process of checkpoint,
we cache the dirty pages of PVM into this ram cache firstly, so this ram cache
always the same as PVM's memory at every checkpoint, then we flush this cached ram
to SVM after we receive all PVM's state.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
v11:
- Rename 'host_cache' to 'colo_cache' (Dave's suggestion)
v10:
- Split the process of dirty pages recording into a new patch
---
include/exec/ram_addr.h | 1 +
include/migration/migration.h | 4 +++
migration/colo.c | 10 +++++++
migration/ram.c | 69 ++++++++++++++++++++++++++++++++++++++++++-
4 files changed, 83 insertions(+), 1 deletion(-)
diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
index 7115154..bb44f66 100644
--- a/include/exec/ram_addr.h
+++ b/include/exec/ram_addr.h
@@ -26,6 +26,7 @@ struct RAMBlock {
struct rcu_head rcu;
struct MemoryRegion *mr;
uint8_t *host;
+ uint8_t *colo_cache; /* For colo, VM's ram cache */
ram_addr_t offset;
ram_addr_t used_length;
ram_addr_t max_length;
diff --git a/include/migration/migration.h b/include/migration/migration.h
index ba5bcec..e41372d 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -332,4 +332,8 @@ int ram_save_queue_pages(MigrationState *ms, const char *rbname,
PostcopyState postcopy_state_get(void);
/* Set the state and return the old state */
PostcopyState postcopy_state_set(PostcopyState new_state);
+
+/* ram cache */
+int colo_init_ram_cache(void);
+void colo_release_ram_cache(void);
#endif
diff --git a/migration/colo.c b/migration/colo.c
index 012d8e5..6e933fa 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -304,6 +304,12 @@ void *colo_process_incoming_thread(void *opaque)
qemu_set_block(qemu_get_fd(mis->from_src_file));
+ ret = colo_init_ram_cache();
+ if (ret < 0) {
+ error_report("Failed to initialize ram cache");
+ goto out;
+ }
+
ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_READY, 0);
if (ret < 0) {
goto out;
@@ -353,6 +359,10 @@ out:
strerror(-ret));
}
+ qemu_mutex_lock_iothread();
+ colo_release_ram_cache();
+ qemu_mutex_unlock_iothread();
+
if (mis->to_src_file) {
qemu_fclose(mis->to_src_file);
}
diff --git a/migration/ram.c b/migration/ram.c
index a161620..9d946a1 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -223,6 +223,7 @@ static RAMBlock *last_sent_block;
static ram_addr_t last_offset;
static QemuMutex migration_bitmap_mutex;
static uint64_t migration_dirty_pages;
+static bool ram_cache_enable;
static uint32_t last_version;
static bool ram_bulk_stage;
@@ -2175,6 +2176,16 @@ static inline void *host_from_ram_block_offset(RAMBlock *block,
return block->host + offset;
}
+static inline void *colo_cache_from_block_offset(RAMBlock *block,
+ ram_addr_t offset)
+{
+ if (!block) {
+ return NULL;
+ }
+
+ return block->colo_cache + offset;
+}
+
/*
* If a page (or a whole RDMA chunk) has been
* determined to be zero, then zap it.
@@ -2454,7 +2465,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) {
RAMBlock *block = ram_block_from_stream(f, addr, flags);
- host = host_from_ram_block_offset(block, addr);
+ /* After going into COLO, we should load the Page into colo_cache */
+ if (ram_cache_enable) {
+ host = colo_cache_from_block_offset(block, addr);
+ } else {
+ host = host_from_ram_block_offset(block, addr);
+ }
if (!host) {
error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
ret = -EINVAL;
@@ -2550,6 +2566,57 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
return ret;
}
+/*
+ * colo cache: this is for secondary VM, we cache the whole
+ * memory of the secondary VM, it will be called after first migration.
+ */
+int colo_init_ram_cache(void)
+{
+ RAMBlock *block;
+
+ rcu_read_lock();
+ QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
+ block->colo_cache = qemu_anon_ram_alloc(block->used_length, NULL);
+ if (!block->colo_cache) {
+ error_report("%s: Can't alloc memory for colo cache of block %s,"
+ "size %zu", __func__, block->idstr,
+ block->used_length);
+ goto out_locked;
+ }
+ memcpy(block->colo_cache, block->host, block->used_length);
+ }
+ rcu_read_unlock();
+ ram_cache_enable = true;
+ return 0;
+
+out_locked:
+ QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
+ if (block->colo_cache) {
+ qemu_anon_ram_free(block->colo_cache, block->used_length);
+ block->colo_cache = NULL;
+ }
+ }
+
+ rcu_read_unlock();
+ return -errno;
+}
+
+void colo_release_ram_cache(void)
+{
+ RAMBlock *block;
+
+ ram_cache_enable = false;
+
+ rcu_read_lock();
+ QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
+ if (block->colo_cache) {
+ qemu_anon_ram_free(block->colo_cache, block->used_length);
+ block->colo_cache = NULL;
+ }
+ }
+ rcu_read_unlock();
+}
+
static SaveVMHandlers savevm_ram_handlers = {
.save_live_setup = ram_save_setup,
.save_live_iterate = ram_save_iterate,
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 16/39] ram/COLO: Record the dirty pages that SVM received
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (14 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 15/39] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-12-01 19:36 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 17/39] COLO: Load VMState into qsb before restore it zhanghailiang
` (22 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
We record the address of the dirty pages that received,
it will help flushing pages that cached into SVM.
We record them by re-using migration dirty bitmap.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
---
v11:
- Split a new helper function from original
host_from_stream_offset() (Dave's suggestion)
- Only do recording work in this patch
v10:
- New patch split from v9's patch 13
- Rebase to master to use 'migration_bitmap_rcu'
---
migration/ram.c | 30 ++++++++++++++++++++++++++++++
1 file changed, 30 insertions(+)
diff --git a/migration/ram.c b/migration/ram.c
index 9d946a1..da6bbd6 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2153,6 +2153,7 @@ static inline RAMBlock *ram_block_from_stream(QEMUFile *f,
return block;
}
+
len = qemu_get_byte(f);
qemu_get_buffer(f, (uint8_t *)id, len);
id[len] = 0;
@@ -2179,10 +2180,23 @@ static inline void *host_from_ram_block_offset(RAMBlock *block,
static inline void *colo_cache_from_block_offset(RAMBlock *block,
ram_addr_t offset)
{
+ unsigned long *bitmap;
+ long k;
+
if (!block) {
return NULL;
}
+ k = (block->mr->ram_addr + offset) >> TARGET_PAGE_BITS;
+ bitmap = atomic_rcu_read(&migration_bitmap_rcu)->bmap;
+ /*
+ * During colo checkpoint, we need bitmap of these migrated pages.
+ * It help us to decide which pages in ram cache should be flushed
+ * into VM's RAM later.
+ */
+ if (!test_and_set_bit(k, bitmap)) {
+ migration_dirty_pages++;
+ }
return block->colo_cache + offset;
}
@@ -2573,6 +2587,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
int colo_init_ram_cache(void)
{
RAMBlock *block;
+ int64_t ram_cache_pages = last_ram_offset() >> TARGET_PAGE_BITS;
rcu_read_lock();
QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
@@ -2587,6 +2602,15 @@ int colo_init_ram_cache(void)
}
rcu_read_unlock();
ram_cache_enable = true;
+ /*
+ * Record the dirty pages that sent by PVM, we use this dirty bitmap together
+ * with to decide which page in cache should be flushed into SVM's RAM. Here
+ * we use the same name 'migration_bitmap_rcu' as for migration.
+ */
+ migration_bitmap_rcu = g_new0(struct BitmapRcu, 1);
+ migration_bitmap_rcu->bmap = bitmap_new(ram_cache_pages);
+ migration_dirty_pages = 0;
+
return 0;
out_locked:
@@ -2604,9 +2628,15 @@ out_locked:
void colo_release_ram_cache(void)
{
RAMBlock *block;
+ struct BitmapRcu *bitmap = migration_bitmap_rcu;
ram_cache_enable = false;
+ atomic_rcu_set(&migration_bitmap_rcu, NULL);
+ if (bitmap) {
+ call_rcu(bitmap, migration_bitmap_free, rcu);
+ }
+
rcu_read_lock();
QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
if (block->colo_cache) {
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 17/39] COLO: Load VMState into qsb before restore it
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (15 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 16/39] ram/COLO: Record the dirty pages that SVM received zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 18/39] COLO: Flush PVM's cached RAM into SVM's memory zhanghailiang
` (21 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
We should not destroy the state of SVM (Secondary VM) until we receive the whole
state from the PVM (Primary VM), in case the primary fails in the middle of sending
the state, so, here we cache the device state in Secondary before restore it.
Besides, we should call qemu_system_reset() before load VM state,
which can ensure the data is intact.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
migration/colo.c | 52 ++++++++++++++++++++++++++++++++++++++++++++++++++--
1 file changed, 50 insertions(+), 2 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index 6e933fa..5ac8ff2 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -285,6 +285,9 @@ static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
void *colo_process_incoming_thread(void *opaque)
{
MigrationIncomingState *mis = opaque;
+ QEMUFile *fb = NULL;
+ QEMUSizedBuffer *buffer = NULL; /* Cache incoming device state */
+ uint64_t total_size;
int ret = 0;
uint64_t value;
@@ -310,6 +313,12 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
+ buffer = qsb_create(NULL, COLO_BUFFER_BASE_SIZE);
+ if (buffer == NULL) {
+ error_report("Failed to allocate colo buffer!");
+ goto out;
+ }
+
ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_READY, 0);
if (ret < 0) {
goto out;
@@ -338,19 +347,53 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
- /* TODO: read migration data into colo buffer */
+ /* read the VM state total size first */
+ ret = colo_ctl_get(mis->from_src_file,
+ COLO_COMMAND_VMSTATE_SIZE, &value);
+ if (ret < 0) {
+ error_report("%s: Failed to get vmstate size", __func__);
+ goto out;
+ }
+
+ /* read vm device state into colo buffer */
+ total_size = qsb_fill_buffer(buffer, mis->from_src_file, value);
+ if (total_size != value) {
+ error_report("Got %lu VMState data, less than expected %lu",
+ total_size, value);
+ ret = -EINVAL;
+ goto out;
+ }
ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_RECEIVED, 0);
if (ret < 0) {
goto out;
}
- /* TODO: load vm state */
+ /* open colo buffer for read */
+ fb = qemu_bufopen("r", buffer);
+ if (!fb) {
+ error_report("can't open colo buffer for read");
+ goto out;
+ }
+
+ qemu_mutex_lock_iothread();
+ qemu_system_reset(VMRESET_SILENT);
+ if (qemu_loadvm_state(fb) < 0) {
+ error_report("COLO: loadvm failed");
+ qemu_mutex_unlock_iothread();
+ goto out;
+ }
+ qemu_mutex_unlock_iothread();
+
+ /* TODO: flush vm state */
ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_LOADED, 0);
if (ret < 0) {
goto out;
}
+
+ qemu_fclose(fb);
+ fb = NULL;
}
out:
@@ -359,6 +402,11 @@ out:
strerror(-ret));
}
+ if (fb) {
+ qemu_fclose(fb);
+ }
+ qsb_free(buffer);
+
qemu_mutex_lock_iothread();
colo_release_ram_cache();
qemu_mutex_unlock_iothread();
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 18/39] COLO: Flush PVM's cached RAM into SVM's memory
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (16 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 17/39] COLO: Load VMState into qsb before restore it zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-27 5:29 ` Li Zhijian
2015-12-01 20:06 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 19/39] COLO: Add checkpoint-delay parameter for migrate-set-parameters zhanghailiang
` (20 subsequent siblings)
38 siblings, 2 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
During the time of VM's running, PVM may dirty some pages, we will transfer
PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint
time. So, the content of SVM's RAM cache will always be some with PVM's memory
after checkpoint.
Instead of flushing all content of PVM's RAM cache into SVM's MEMORY,
we do this in a more efficient way:
Only flush any page that dirtied by PVM since last checkpoint.
In this way, we can ensure SVM's memory same with PVM's.
Besides, we must ensure flush RAM cache before load device state.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Signed-off-by: Gonglei <arei.gonglei@huawei.com>
---
v11:
- Move the place of 'need_flush' (Dave's suggestion)
- Remove unused 'DPRINTF("Flush ram_cache\n")'
v10:
- trace the number of dirty pages that be received.
---
include/migration/migration.h | 1 +
migration/colo.c | 2 --
migration/ram.c | 37 +++++++++++++++++++++++++++++++++++++
trace-events | 1 +
4 files changed, 39 insertions(+), 2 deletions(-)
diff --git a/include/migration/migration.h b/include/migration/migration.h
index e41372d..221176b 100644
--- a/include/migration/migration.h
+++ b/include/migration/migration.h
@@ -336,4 +336,5 @@ PostcopyState postcopy_state_set(PostcopyState new_state);
/* ram cache */
int colo_init_ram_cache(void);
void colo_release_ram_cache(void);
+void colo_flush_ram_cache(void);
#endif
diff --git a/migration/colo.c b/migration/colo.c
index 5ac8ff2..4095d97 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -385,8 +385,6 @@ void *colo_process_incoming_thread(void *opaque)
}
qemu_mutex_unlock_iothread();
- /* TODO: flush vm state */
-
ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_LOADED, 0);
if (ret < 0) {
goto out;
diff --git a/migration/ram.c b/migration/ram.c
index da6bbd6..4f37144 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2448,6 +2448,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
* be atomic
*/
bool postcopy_running = postcopy_state_get() >= POSTCOPY_INCOMING_LISTENING;
+ bool need_flush = false;
seq_iter++;
@@ -2482,6 +2483,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
/* After going into COLO, we should load the Page into colo_cache */
if (ram_cache_enable) {
host = colo_cache_from_block_offset(block, addr);
+ need_flush = true;
} else {
host = host_from_ram_block_offset(block, addr);
}
@@ -2575,6 +2577,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
}
rcu_read_unlock();
+
+ if (!ret && ram_cache_enable && need_flush) {
+ colo_flush_ram_cache();
+ }
DPRINTF("Completed load of VM with exit code %d seq iteration "
"%" PRIu64 "\n", ret, seq_iter);
return ret;
@@ -2647,6 +2653,37 @@ void colo_release_ram_cache(void)
rcu_read_unlock();
}
+/*
+ * Flush content of RAM cache into SVM's memory.
+ * Only flush the pages that be dirtied by PVM or SVM or both.
+ */
+void colo_flush_ram_cache(void)
+{
+ RAMBlock *block = NULL;
+ void *dst_host;
+ void *src_host;
+ ram_addr_t offset = 0;
+
+ trace_colo_flush_ram_cache(migration_dirty_pages);
+ rcu_read_lock();
+ block = QLIST_FIRST_RCU(&ram_list.blocks);
+ while (block) {
+ ram_addr_t ram_addr_abs;
+ offset = migration_bitmap_find_dirty(block, offset, &ram_addr_abs);
+ migration_bitmap_clear_dirty(ram_addr_abs);
+ if (offset >= block->used_length) {
+ offset = 0;
+ block = QLIST_NEXT_RCU(block, next);
+ } else {
+ dst_host = block->host + offset;
+ src_host = block->colo_cache + offset;
+ memcpy(dst_host, src_host, TARGET_PAGE_SIZE);
+ }
+ }
+ rcu_read_unlock();
+ assert(migration_dirty_pages == 0);
+}
+
static SaveVMHandlers savevm_ram_handlers = {
.save_live_setup = ram_save_setup,
.save_live_iterate = ram_save_iterate,
diff --git a/trace-events b/trace-events
index f8a0959..f158d2a 100644
--- a/trace-events
+++ b/trace-events
@@ -1264,6 +1264,7 @@ migration_throttle(void) ""
ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x"
ram_postcopy_send_discard_bitmap(void) ""
ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: %zx len: %zx"
+colo_flush_ram_cache(uint64_t dirty_pages) "dirty_pages %" PRIu64""
# hw/display/qxl.c
disable qxl_interface_set_mm_time(int qid, uint32_t mm_time) "%d %d"
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 19/39] COLO: Add checkpoint-delay parameter for migrate-set-parameters
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (17 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 18/39] COLO: Flush PVM's cached RAM into SVM's memory zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-12-09 18:50 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 20/39] COLO: synchronize PVM's state to SVM periodically zhanghailiang
` (19 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, zhanghailiang,
arei.gonglei, stefanha, amit.shah, Luiz Capitulino,
hongyang.yang
Add checkpoint-delay parameter for migrate-set-parameters, so that
we can control the checkpoint frequency when COLO is in periodic mode.
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
v11:
- Move this patch ahead of the patch where uses 'checkpoint_delay'
(Dave's suggestion)
v10:
- Fix related qmp command
---
hmp.c | 7 +++++++
migration/migration.c | 23 ++++++++++++++++++++++-
qapi-schema.json | 19 ++++++++++++++++---
qmp-commands.hx | 2 +-
4 files changed, 46 insertions(+), 5 deletions(-)
diff --git a/hmp.c b/hmp.c
index 2140605..ec038f1 100644
--- a/hmp.c
+++ b/hmp.c
@@ -284,6 +284,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
monitor_printf(mon, " %s: %" PRId64,
MigrationParameter_lookup[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT],
params->x_cpu_throttle_increment);
+ monitor_printf(mon, " %s: %" PRId64,
+ MigrationParameter_lookup[MIGRATION_PARAMETER_CHECKPOINT_DELAY],
+ params->checkpoint_delay);
monitor_printf(mon, "\n");
}
@@ -1237,6 +1240,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
bool has_decompress_threads = false;
bool has_x_cpu_throttle_initial = false;
bool has_x_cpu_throttle_increment = false;
+ bool has_checkpoint_delay = false;
int i;
for (i = 0; i < MIGRATION_PARAMETER_MAX; i++) {
@@ -1256,6 +1260,8 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
break;
case MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT:
has_x_cpu_throttle_increment = true;
+ case MIGRATION_PARAMETER_CHECKPOINT_DELAY:
+ has_checkpoint_delay = true;
break;
}
qmp_migrate_set_parameters(has_compress_level, value,
@@ -1263,6 +1269,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
has_decompress_threads, value,
has_x_cpu_throttle_initial, value,
has_x_cpu_throttle_increment, value,
+ has_checkpoint_delay, value,
&err);
break;
}
diff --git a/migration/migration.c b/migration/migration.c
index a4c690d..48545f3 100644
--- a/migration/migration.c
+++ b/migration/migration.c
@@ -56,6 +56,11 @@
/* Migration XBZRLE default cache size */
#define DEFAULT_MIGRATE_CACHE_SIZE (64 * 1024 * 1024)
+/* The delay time (in ms) between two COLO checkpoints
+ * Note: Please change this default value to 10000 when we support hybrid mode.
+ */
+#define DEFAULT_MIGRATE_CHECKPOINT_DELAY 200
+
static NotifierList migration_state_notifiers =
NOTIFIER_LIST_INITIALIZER(migration_state_notifiers);
@@ -91,6 +96,8 @@ MigrationState *migrate_get_current(void)
DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL,
.parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT,
+ .parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] =
+ DEFAULT_MIGRATE_CHECKPOINT_DELAY,
};
if (!once) {
@@ -530,6 +537,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
params->x_cpu_throttle_increment =
s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
+ params->checkpoint_delay =
+ s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY];
return params;
}
@@ -736,7 +745,10 @@ void qmp_migrate_set_parameters(bool has_compress_level,
bool has_x_cpu_throttle_initial,
int64_t x_cpu_throttle_initial,
bool has_x_cpu_throttle_increment,
- int64_t x_cpu_throttle_increment, Error **errp)
+ int64_t x_cpu_throttle_increment,
+ bool has_checkpoint_delay,
+ int64_t checkpoint_delay,
+ Error **errp)
{
MigrationState *s = migrate_get_current();
@@ -771,6 +783,11 @@ void qmp_migrate_set_parameters(bool has_compress_level,
"x_cpu_throttle_increment",
"an integer in the range of 1 to 99");
}
+ if (has_checkpoint_delay && (checkpoint_delay < 0)) {
+ error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
+ "checkpoint_delay",
+ "is invalid, it should be positive");
+ }
if (has_compress_level) {
s->parameters[MIGRATION_PARAMETER_COMPRESS_LEVEL] = compress_level;
@@ -791,6 +808,10 @@ void qmp_migrate_set_parameters(bool has_compress_level,
s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
x_cpu_throttle_increment;
}
+
+ if (has_checkpoint_delay) {
+ s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] = checkpoint_delay;
+ }
}
void qmp_migrate_start_postcopy(Error **errp)
diff --git a/qapi-schema.json b/qapi-schema.json
index 5ee089d..b4b47ed 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -625,11 +625,16 @@
# @x-cpu-throttle-increment: throttle percentage increase each time
# auto-converge detects that migration is not making
# progress. The default value is 10. (Since 2.5)
+#
+# @checkpoint-delay: The delay time (in ms) between two COLO checkpoints in
+# periodic mode. (Since 2.6)
+#
# Since: 2.4
##
{ 'enum': 'MigrationParameter',
'data': ['compress-level', 'compress-threads', 'decompress-threads',
- 'x-cpu-throttle-initial', 'x-cpu-throttle-increment'] }
+ 'x-cpu-throttle-initial', 'x-cpu-throttle-increment',
+ 'checkpoint-delay' ] }
#
# @migrate-set-parameters
@@ -649,6 +654,9 @@
# @x-cpu-throttle-increment: throttle percentage increase each time
# auto-converge detects that migration is not making
# progress. The default value is 10. (Since 2.5)
+#
+# @checkpoint-delay: the delay time between two checkpoints. (Since 2.6)
+#
# Since: 2.4
##
{ 'command': 'migrate-set-parameters',
@@ -656,7 +664,8 @@
'*compress-threads': 'int',
'*decompress-threads': 'int',
'*x-cpu-throttle-initial': 'int',
- '*x-cpu-throttle-increment': 'int'} }
+ '*x-cpu-throttle-increment': 'int',
+ '*checkpoint-delay': 'int' } }
#
# @MigrationParameters
@@ -675,6 +684,8 @@
# auto-converge detects that migration is not making
# progress. The default value is 10. (Since 2.5)
#
+# @checkpoint-delay: the delay time between two COLO checkpoints
+#
# Since: 2.4
##
{ 'struct': 'MigrationParameters',
@@ -682,7 +693,9 @@
'compress-threads': 'int',
'decompress-threads': 'int',
'x-cpu-throttle-initial': 'int',
- 'x-cpu-throttle-increment': 'int'} }
+ 'x-cpu-throttle-increment': 'int',
+ 'checkpoint-delay': 'int'} }
+
##
# @query-migrate-parameters
#
diff --git a/qmp-commands.hx b/qmp-commands.hx
index 92042d8..c39f2f0 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -3664,7 +3664,7 @@ EQMP
{
.name = "migrate-set-parameters",
.args_type =
- "compress-level:i?,compress-threads:i?,decompress-threads:i?",
+ "compress-level:i?,compress-threads:i?,decompress-threads:i?,checkpoint-delay:i?",
.mhandler.cmd_new = qmp_marshal_migrate_set_parameters,
},
SQMP
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 20/39] COLO: synchronize PVM's state to SVM periodically
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (18 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 19/39] COLO: Add checkpoint-delay parameter for migrate-set-parameters zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-12-09 18:53 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 21/39] COLO failover: Introduce a new command to trigger a failover zhanghailiang
` (18 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
Do checkpoint periodically, the default interval is 200ms.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
v11:
- Fix wrong sleep time for checkpoint period. (Dave's review comment)
---
migration/colo.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/migration/colo.c b/migration/colo.c
index 4095d97..5b7ff87 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -11,6 +11,7 @@
*/
#include <unistd.h>
+#include "qemu/timer.h"
#include "sysemu/sysemu.h"
#include "migration/colo.h"
#include "trace.h"
@@ -183,6 +184,7 @@ out:
static void colo_process_checkpoint(MigrationState *s)
{
QEMUSizedBuffer *buffer = NULL;
+ int64_t current_time, checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
int ret = 0;
uint64_t value;
@@ -216,11 +218,21 @@ static void colo_process_checkpoint(MigrationState *s)
trace_colo_vm_state_change("stop", "run");
while (s->state == MIGRATION_STATUS_COLO) {
+ current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ if (current_time - checkpoint_time <
+ s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) {
+ int64_t delay_ms;
+
+ delay_ms = s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] -
+ (current_time - checkpoint_time);
+ g_usleep(delay_ms * 1000);
+ }
/* start a colo checkpoint */
ret = colo_do_checkpoint_transaction(s, buffer);
if (ret < 0) {
goto out;
}
+ checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
}
out:
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 21/39] COLO failover: Introduce a new command to trigger a failover
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (19 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 20/39] COLO: synchronize PVM's state to SVM periodically zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 22/39] COLO failover: Introduce state to record failover process zhanghailiang
` (17 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, zhanghailiang,
arei.gonglei, stefanha, amit.shah, Luiz Capitulino,
hongyang.yang
We leave users to choose whatever heartbeat solution they want, if the heartbeat
is lost, or other errors they detect, they can use experimental command
'x_colo_lost_heartbeat' to tell COLO to do failover, COLO will do operations
accordingly.
For example, if the command is sent to the PVM, the Primary side will
exit COLO mode and take over operation. If sent to the Secondary, the
secondary will run failover work, then take over server operation to
become the new Primary.
Cc: Luiz Capitulino <lcapitulino@redhat.com>
Cc: Eric Blake <eblake@redhat.com>
Cc: Markus Armbruster <armbru@redhat.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
v11:
- Add more comments for x-colo-lost-heartbeat command (Eric's suggestion)
- Return 'enum' instead of 'int' for get_colo_mode() (Eric's suggestion)
v10:
- Rename command colo_lost_hearbeat to experimental 'x_colo_lost_heartbeat'
---
hmp-commands.hx | 15 +++++++++++++++
hmp.c | 8 ++++++++
hmp.h | 1 +
include/migration/colo.h | 3 +++
include/migration/failover.h | 20 ++++++++++++++++++++
migration/Makefile.objs | 2 +-
migration/colo-comm.c | 11 +++++++++++
migration/colo-failover.c | 41 +++++++++++++++++++++++++++++++++++++++++
migration/colo.c | 1 +
qapi-schema.json | 29 +++++++++++++++++++++++++++++
qmp-commands.hx | 19 +++++++++++++++++++
stubs/migration-colo.c | 8 ++++++++
12 files changed, 157 insertions(+), 1 deletion(-)
create mode 100644 include/migration/failover.h
create mode 100644 migration/colo-failover.c
diff --git a/hmp-commands.hx b/hmp-commands.hx
index bb52e4d..a381b0b 100644
--- a/hmp-commands.hx
+++ b/hmp-commands.hx
@@ -1039,6 +1039,21 @@ migration (or once already in postcopy).
ETEXI
{
+ .name = "x_colo_lost_heartbeat",
+ .args_type = "",
+ .params = "",
+ .help = "Tell COLO that heartbeat is lost,\n\t\t\t"
+ "a failover or takeover is needed.",
+ .mhandler.cmd = hmp_x_colo_lost_heartbeat,
+ },
+
+STEXI
+@item x_colo_lost_heartbeat
+@findex x_colo_lost_heartbeat
+Tell COLO that heartbeat is lost, a failover or takeover is needed.
+ETEXI
+
+ {
.name = "client_migrate_info",
.args_type = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
.params = "protocol hostname port tls-port cert-subject",
diff --git a/hmp.c b/hmp.c
index ec038f1..ce8a23c 100644
--- a/hmp.c
+++ b/hmp.c
@@ -1310,6 +1310,14 @@ void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict)
hmp_handle_error(mon, &err);
}
+void hmp_x_colo_lost_heartbeat(Monitor *mon, const QDict *qdict)
+{
+ Error *err = NULL;
+
+ qmp_x_colo_lost_heartbeat(&err);
+ hmp_handle_error(mon, &err);
+}
+
void hmp_set_password(Monitor *mon, const QDict *qdict)
{
const char *protocol = qdict_get_str(qdict, "protocol");
diff --git a/hmp.h b/hmp.h
index a8c5b5a..864a300 100644
--- a/hmp.h
+++ b/hmp.h
@@ -70,6 +70,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict);
void hmp_migrate_set_cache_size(Monitor *mon, const QDict *qdict);
void hmp_client_migrate_info(Monitor *mon, const QDict *qdict);
void hmp_migrate_start_postcopy(Monitor *mon, const QDict *qdict);
+void hmp_x_colo_lost_heartbeat(Monitor *mon, const QDict *qdict);
void hmp_set_password(Monitor *mon, const QDict *qdict);
void hmp_expire_password(Monitor *mon, const QDict *qdict);
void hmp_eject(Monitor *mon, const QDict *qdict);
diff --git a/include/migration/colo.h b/include/migration/colo.h
index 2676c4a..ba27719 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -17,6 +17,7 @@
#include "migration/migration.h"
#include "qemu/coroutine_int.h"
#include "qemu/thread.h"
+#include "qemu/main-loop.h"
bool colo_supported(void);
void colo_info_mig_init(void);
@@ -29,4 +30,6 @@ bool migration_incoming_enable_colo(void);
void migration_incoming_exit_colo(void);
void *colo_process_incoming_thread(void *opaque);
bool migration_incoming_in_colo_state(void);
+
+COLOMode get_colo_mode(void);
#endif
diff --git a/include/migration/failover.h b/include/migration/failover.h
new file mode 100644
index 0000000..1785b52
--- /dev/null
+++ b/include/migration/failover.h
@@ -0,0 +1,20 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO.,LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#ifndef QEMU_FAILOVER_H
+#define QEMU_FAILOVER_H
+
+#include "qemu-common.h"
+
+void failover_request_active(Error **errp);
+
+#endif
diff --git a/migration/Makefile.objs b/migration/Makefile.objs
index 81b5713..920d1e7 100644
--- a/migration/Makefile.objs
+++ b/migration/Makefile.objs
@@ -1,6 +1,6 @@
common-obj-y += migration.o tcp.o
-common-obj-$(CONFIG_COLO) += colo.o
common-obj-y += colo-comm.o
+common-obj-$(CONFIG_COLO) += colo.o colo-failover.o
common-obj-y += vmstate.o
common-obj-y += qemu-file.o qemu-file-buf.o qemu-file-unix.o qemu-file-stdio.o
common-obj-y += xbzrle.o postcopy-ram.o
diff --git a/migration/colo-comm.c b/migration/colo-comm.c
index 30df3d3..58a6488 100644
--- a/migration/colo-comm.c
+++ b/migration/colo-comm.c
@@ -20,6 +20,17 @@ typedef struct {
static COLOInfo colo_info;
+COLOMode get_colo_mode(void)
+{
+ if (migration_in_colo_state()) {
+ return COLO_MODE_PRIMARY;
+ } else if (migration_incoming_in_colo_state()) {
+ return COLO_MODE_SECONDARY;
+ } else {
+ return COLO_MODE_UNKNOWN;
+ }
+}
+
static void colo_info_pre_save(void *opaque)
{
COLOInfo *s = opaque;
diff --git a/migration/colo-failover.c b/migration/colo-failover.c
new file mode 100644
index 0000000..e3897c6
--- /dev/null
+++ b/migration/colo-failover.c
@@ -0,0 +1,41 @@
+/*
+ * COarse-grain LOck-stepping Virtual Machines for Non-stop Service (COLO)
+ * (a.k.a. Fault Tolerance or Continuous Replication)
+ *
+ * Copyright (c) 2015 HUAWEI TECHNOLOGIES CO., LTD.
+ * Copyright (c) 2015 FUJITSU LIMITED
+ * Copyright (c) 2015 Intel Corporation
+ *
+ * This work is licensed under the terms of the GNU GPL, version 2 or
+ * later. See the COPYING file in the top-level directory.
+ */
+
+#include "migration/colo.h"
+#include "migration/failover.h"
+#include "qmp-commands.h"
+#include "qapi/qmp/qerror.h"
+
+static QEMUBH *failover_bh;
+
+static void colo_failover_bh(void *opaque)
+{
+ qemu_bh_delete(failover_bh);
+ failover_bh = NULL;
+ /*TODO: Do failover work */
+}
+
+void failover_request_active(Error **errp)
+{
+ failover_bh = qemu_bh_new(colo_failover_bh, NULL);
+ qemu_bh_schedule(failover_bh);
+}
+
+void qmp_x_colo_lost_heartbeat(Error **errp)
+{
+ if (get_colo_mode() == COLO_MODE_UNKNOWN) {
+ error_setg(errp, QERR_FEATURE_DISABLED, "colo");
+ return;
+ }
+
+ failover_request_active(errp);
+}
diff --git a/migration/colo.c b/migration/colo.c
index 5b7ff87..2c4fc1c 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -17,6 +17,7 @@
#include "trace.h"
#include "qemu/error-report.h"
#include "qemu/sockets.h"
+#include "migration/failover.h"
/* colo buffer */
#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
diff --git a/qapi-schema.json b/qapi-schema.json
index b4b47ed..2ed63e0 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -765,6 +765,35 @@
'checkpoint-reply', 'vmstate-send', 'vmstate-size',
'vmstate-received', 'vmstate-loaded' ] }
+##
+# @COLOMode
+#
+# The colo mode
+#
+# @unknown: unknown mode
+#
+# @primary: master side
+#
+# @secondary: slave side
+#
+# Since: 2.6
+##
+{ 'enum': 'COLOMode',
+ 'data': [ 'unknown', 'primary', 'secondary'] }
+
+##
+# @x-colo-lost-heartbeat
+#
+# Tell qemu that heartbeat is lost, request it to do takeover procedures.
+# If this command is sent to the PVM, the Primary side will exit COLO mode.
+# If sent to the Secondary, the Secondary side will run failover work,
+# then takes over server operation to become the service VM.
+#
+# Since: 2.6
+##
+{ 'command': 'x-colo-lost-heartbeat' }
+
+##
# @MouseInfo:
#
# Information about a mouse device.
diff --git a/qmp-commands.hx b/qmp-commands.hx
index c39f2f0..6e79511 100644
--- a/qmp-commands.hx
+++ b/qmp-commands.hx
@@ -805,6 +805,25 @@ Example:
EQMP
{
+ .name = "x-colo-lost-heartbeat",
+ .args_type = "",
+ .mhandler.cmd_new = qmp_marshal_x_colo_lost_heartbeat,
+ },
+
+SQMP
+x-colo-lost-heartbeat
+--------------------
+
+Tell COLO that heartbeat is lost, a failover or takeover is needed.
+
+Example:
+
+-> { "execute": "x-colo-lost-heartbeat" }
+<- { "return": {} }
+
+EQMP
+
+ {
.name = "client_migrate_info",
.args_type = "protocol:s,hostname:s,port:i?,tls-port:i?,cert-subject:s?",
.params = "protocol hostname port tls-port cert-subject",
diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
index c12516e..5028f63 100644
--- a/stubs/migration-colo.c
+++ b/stubs/migration-colo.c
@@ -11,6 +11,7 @@
*/
#include "migration/colo.h"
+#include "qmp-commands.h"
bool colo_supported(void)
{
@@ -35,3 +36,10 @@ void *colo_process_incoming_thread(void *opaque)
{
return NULL;
}
+
+void qmp_x_colo_lost_heartbeat(Error **errp)
+{
+ error_setg(errp, "COLO is not supported, please rerun configure"
+ " with --enable-colo option in order to support"
+ " COLO feature");
+}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 22/39] COLO failover: Introduce state to record failover process
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (20 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 21/39] COLO failover: Introduce a new command to trigger a failover zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 23/39] COLO: Implement failover work for Primary VM zhanghailiang
` (16 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
When handling failover, we do different things according to the different stage
of failover process, here we introduce a global atomic variable to record the
status of failover.
We add four failover status to indicate the different stage of failover process.
You should use the helpers to get and set the value.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
---
v11:
- fix several typos found by Dave
- Add Reviewed-by tag
---
include/migration/failover.h | 10 ++++++++++
migration/colo-failover.c | 37 +++++++++++++++++++++++++++++++++++++
migration/colo.c | 4 ++++
trace-events | 1 +
4 files changed, 52 insertions(+)
diff --git a/include/migration/failover.h b/include/migration/failover.h
index 1785b52..882c625 100644
--- a/include/migration/failover.h
+++ b/include/migration/failover.h
@@ -15,6 +15,16 @@
#include "qemu-common.h"
+typedef enum COLOFailoverStatus {
+ FAILOVER_STATUS_NONE = 0,
+ FAILOVER_STATUS_REQUEST = 1, /* Request but not handled */
+ FAILOVER_STATUS_HANDLING = 2, /* In the process of handling failover */
+ FAILOVER_STATUS_COMPLETED = 3, /* Finish the failover process */
+} COLOFailoverStatus;
+
+void failover_init_state(void);
+int failover_set_state(int old_state, int new_state);
+int failover_get_state(void);
void failover_request_active(Error **errp);
#endif
diff --git a/migration/colo-failover.c b/migration/colo-failover.c
index e3897c6..1b1be24 100644
--- a/migration/colo-failover.c
+++ b/migration/colo-failover.c
@@ -14,22 +14,59 @@
#include "migration/failover.h"
#include "qmp-commands.h"
#include "qapi/qmp/qerror.h"
+#include "qemu/error-report.h"
+#include "trace.h"
static QEMUBH *failover_bh;
+static COLOFailoverStatus failover_state;
static void colo_failover_bh(void *opaque)
{
+ int old_state;
+
qemu_bh_delete(failover_bh);
failover_bh = NULL;
+ old_state = failover_set_state(FAILOVER_STATUS_REQUEST,
+ FAILOVER_STATUS_HANDLING);
+ if (old_state != FAILOVER_STATUS_REQUEST) {
+ error_report("Unkown error for failover, old_state=%d", old_state);
+ return;
+ }
/*TODO: Do failover work */
}
void failover_request_active(Error **errp)
{
+ if (failover_set_state(FAILOVER_STATUS_NONE, FAILOVER_STATUS_REQUEST)
+ != FAILOVER_STATUS_NONE) {
+ error_setg(errp, "COLO failover is already actived");
+ return;
+ }
failover_bh = qemu_bh_new(colo_failover_bh, NULL);
qemu_bh_schedule(failover_bh);
}
+void failover_init_state(void)
+{
+ failover_state = FAILOVER_STATUS_NONE;
+}
+
+int failover_set_state(int old_state, int new_state)
+{
+ int old;
+
+ old = atomic_cmpxchg(&failover_state, old_state, new_state);
+ if (old == old_state) {
+ trace_colo_failover_set_state(new_state);
+ }
+ return old;
+}
+
+int failover_get_state(void)
+{
+ return atomic_read(&failover_state);
+}
+
void qmp_x_colo_lost_heartbeat(Error **errp)
{
if (get_colo_mode() == COLO_MODE_UNKNOWN) {
diff --git a/migration/colo.c b/migration/colo.c
index 2c4fc1c..cedfc63 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -189,6 +189,8 @@ static void colo_process_checkpoint(MigrationState *s)
int ret = 0;
uint64_t value;
+ failover_init_state();
+
s->rp_state.from_dst_file = qemu_file_get_return_path(s->to_dst_file);
if (!s->rp_state.from_dst_file) {
ret = -EINVAL;
@@ -307,6 +309,8 @@ void *colo_process_incoming_thread(void *opaque)
migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_COLO);
+ failover_init_state();
+
mis->to_src_file = qemu_file_get_return_path(mis->from_src_file);
if (!mis->to_src_file) {
ret = -EINVAL;
diff --git a/trace-events b/trace-events
index f158d2a..b80c1e0 100644
--- a/trace-events
+++ b/trace-events
@@ -1582,6 +1582,7 @@ postcopy_ram_incoming_cleanup_join(void) ""
colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
colo_ctl_put(const char *msg, uint64_t value) "Send '%s' cmd, value: %" PRIu64""
colo_ctl_get(const char *msg, uint64_t value) "Receive '%s' cmd, value: %" PRIu64""
+colo_failover_set_state(int new_state) "new state %d"
# kvm-all.c
kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 23/39] COLO: Implement failover work for Primary VM
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (21 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 22/39] COLO failover: Introduce state to record failover process zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-12-10 18:34 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 24/39] COLO: Implement failover work for Secondary VM zhanghailiang
` (15 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
For PVM, if there is failover request from users.
The colo thread will exit the loop while the failover BH does the
cleanup work and resumes VM.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
v11:
- Don't call migration_end() in primary_vm_do_failover(),
The cleanup work will be done in migration_thread().
- Remove vm_start() in primary_vm_do_failover() which also been done
in migraiton_thread()
v10:
- Call migration_end() in primary_vm_do_failover()
---
include/migration/colo.h | 3 +++
include/migration/failover.h | 1 +
migration/colo-failover.c | 7 +++++-
migration/colo.c | 56 ++++++++++++++++++++++++++++++++++++++++++--
4 files changed, 64 insertions(+), 3 deletions(-)
diff --git a/include/migration/colo.h b/include/migration/colo.h
index ba27719..0b02e95 100644
--- a/include/migration/colo.h
+++ b/include/migration/colo.h
@@ -32,4 +32,7 @@ void *colo_process_incoming_thread(void *opaque);
bool migration_incoming_in_colo_state(void);
COLOMode get_colo_mode(void);
+
+/* failover */
+void colo_do_failover(MigrationState *s);
#endif
diff --git a/include/migration/failover.h b/include/migration/failover.h
index 882c625..fba3931 100644
--- a/include/migration/failover.h
+++ b/include/migration/failover.h
@@ -26,5 +26,6 @@ void failover_init_state(void);
int failover_set_state(int old_state, int new_state);
int failover_get_state(void);
void failover_request_active(Error **errp);
+bool failover_request_is_active(void);
#endif
diff --git a/migration/colo-failover.c b/migration/colo-failover.c
index 1b1be24..0c525da 100644
--- a/migration/colo-failover.c
+++ b/migration/colo-failover.c
@@ -32,7 +32,7 @@ static void colo_failover_bh(void *opaque)
error_report("Unkown error for failover, old_state=%d", old_state);
return;
}
- /*TODO: Do failover work */
+ colo_do_failover(NULL);
}
void failover_request_active(Error **errp)
@@ -67,6 +67,11 @@ int failover_get_state(void)
return atomic_read(&failover_state);
}
+bool failover_request_is_active(void)
+{
+ return ((failover_get_state() != FAILOVER_STATUS_NONE));
+}
+
void qmp_x_colo_lost_heartbeat(Error **errp)
{
if (get_colo_mode() == COLO_MODE_UNKNOWN) {
diff --git a/migration/colo.c b/migration/colo.c
index cedfc63..7a42fc6 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -41,6 +41,42 @@ bool migration_incoming_in_colo_state(void)
return mis && (mis->state == MIGRATION_STATUS_COLO);
}
+static bool colo_runstate_is_stopped(void)
+{
+ return runstate_check(RUN_STATE_COLO) || !runstate_is_running();
+}
+
+static void primary_vm_do_failover(void)
+{
+ MigrationState *s = migrate_get_current();
+ int old_state;
+
+ if (s->state != MIGRATION_STATUS_FAILED) {
+ migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
+ MIGRATION_STATUS_COMPLETED);
+ }
+
+ old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
+ FAILOVER_STATUS_COMPLETED);
+ if (old_state != FAILOVER_STATUS_HANDLING) {
+ error_report("Serious error while do failover for Primary VM,"
+ "old_state: %d", old_state);
+ return;
+ }
+}
+
+void colo_do_failover(MigrationState *s)
+{
+ /* Make sure vm stopped while failover */
+ if (!colo_runstate_is_stopped()) {
+ vm_stop_force_state(RUN_STATE_COLO);
+ }
+
+ if (get_colo_mode() == COLO_MODE_PRIMARY) {
+ primary_vm_do_failover();
+ }
+}
+
/* colo checkpoint control helper */
static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value)
{
@@ -122,9 +158,22 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
}
qemu_mutex_lock_iothread();
+ if (failover_request_is_active()) {
+ qemu_mutex_unlock_iothread();
+ ret = -1;
+ goto out;
+ }
vm_stop_force_state(RUN_STATE_COLO);
qemu_mutex_unlock_iothread();
trace_colo_vm_state_change("run", "stop");
+ /*
+ * failover request bh could be called after
+ * vm_stop_force_state so we check failover_request_is_active() again.
+ */
+ if (failover_request_is_active()) {
+ ret = -1;
+ goto out;
+ }
/* Disable block migration */
s->params.blk = 0;
@@ -221,6 +270,11 @@ static void colo_process_checkpoint(MigrationState *s)
trace_colo_vm_state_change("stop", "run");
while (s->state == MIGRATION_STATUS_COLO) {
+ if (failover_request_is_active()) {
+ error_report("failover request");
+ goto out;
+ }
+
current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
if (current_time - checkpoint_time <
s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) {
@@ -242,8 +296,6 @@ out:
if (ret < 0) {
error_report("%s: %s", __func__, strerror(-ret));
}
- migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
- MIGRATION_STATUS_COMPLETED);
qsb_free(buffer);
buffer = NULL;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 24/39] COLO: Implement failover work for Secondary VM
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (22 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 23/39] COLO: Implement failover work for Primary VM zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-12-10 18:50 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 25/39] COLO: implement default failover treatment zhanghailiang
` (14 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
If users require SVM to takeover work, colo incoming thread should
exit from loop while failover BH helps backing to migration incoming
coroutine.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 42 +++++++++++++++++++++++++++++++++++++++---
1 file changed, 39 insertions(+), 3 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index 7a42fc6..f31e957 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -46,6 +46,33 @@ static bool colo_runstate_is_stopped(void)
return runstate_check(RUN_STATE_COLO) || !runstate_is_running();
}
+static void secondary_vm_do_failover(void)
+{
+ int old_state;
+ MigrationIncomingState *mis = migration_incoming_get_current();
+
+ migrate_set_state(&mis->state, MIGRATION_STATUS_COLO,
+ MIGRATION_STATUS_COMPLETED);
+
+ if (!autostart) {
+ error_report("\"-S\" qemu option will be ignored in secondary side");
+ /* recover runstate to normal migration finish state */
+ autostart = true;
+ }
+
+ old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
+ FAILOVER_STATUS_COMPLETED);
+ if (old_state != FAILOVER_STATUS_HANDLING) {
+ error_report("Serious error while do failover for secondary VM,"
+ "old_state: %d", old_state);
+ return;
+ }
+ /* For Secondary VM, jump to incoming co */
+ if (mis->migration_incoming_co) {
+ qemu_coroutine_enter(mis->migration_incoming_co, NULL);
+ }
+}
+
static void primary_vm_do_failover(void)
{
MigrationState *s = migrate_get_current();
@@ -74,6 +101,8 @@ void colo_do_failover(MigrationState *s)
if (get_colo_mode() == COLO_MODE_PRIMARY) {
primary_vm_do_failover();
+ } else {
+ secondary_vm_do_failover();
}
}
@@ -404,6 +433,12 @@ void *colo_process_incoming_thread(void *opaque)
continue;
}
}
+
+ if (failover_request_is_active()) {
+ error_report("failover request");
+ goto out;
+ }
+
/* FIXME: This is unnecessary for periodic checkpoint mode */
ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_REPLY, 0);
if (ret < 0) {
@@ -473,10 +508,11 @@ out:
qemu_fclose(fb);
}
qsb_free(buffer);
-
- qemu_mutex_lock_iothread();
+ /* Here, we can ensure BH is hold the global lock, and will join colo
+ * incoming thread, so here it is not necessary to lock here again,
+ * or there will be a deadlock error.
+ */
colo_release_ram_cache();
- qemu_mutex_unlock_iothread();
if (mis->to_src_file) {
qemu_fclose(mis->to_src_file);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 25/39] COLO: implement default failover treatment
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (23 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 24/39] COLO: Implement failover work for Secondary VM zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-12-10 19:01 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 26/39] qmp event: Add event notification for COLO error zhanghailiang
` (13 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
If we detect some error in colo, we will wait for some time,
hoping users also detect it. If users don't issue failover command.
We will go into default failover procedure, which the PVM will takeover
work while SVM is exit in default.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 46 insertions(+)
diff --git a/migration/colo.c b/migration/colo.c
index f31e957..1e6d3dd 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -19,6 +19,14 @@
#include "qemu/sockets.h"
#include "migration/failover.h"
+/*
+ * The delay time before qemu begin the procedure of default failover treatment.
+ * Unit: ms
+ * Fix me: This value should be able to change by command
+ * 'migrate-set-parameters'
+ */
+#define DEFAULT_FAILOVER_DELAY 2000
+
/* colo buffer */
#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
@@ -264,6 +272,7 @@ static void colo_process_checkpoint(MigrationState *s)
{
QEMUSizedBuffer *buffer = NULL;
int64_t current_time, checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ int64_t error_time;
int ret = 0;
uint64_t value;
@@ -322,8 +331,25 @@ static void colo_process_checkpoint(MigrationState *s)
}
out:
+ current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
if (ret < 0) {
error_report("%s: %s", __func__, strerror(-ret));
+ /* Give users time to get involved in this verdict */
+ while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
+ if (failover_request_is_active()) {
+ error_report("Primary VM will take over work");
+ break;
+ }
+ usleep(100 * 1000);
+ current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ }
+
+ qemu_mutex_lock_iothread();
+ if (!failover_request_is_active()) {
+ error_report("Primary VM will take over work in default");
+ failover_request_active(NULL);
+ }
+ qemu_mutex_unlock_iothread();
}
qsb_free(buffer);
@@ -384,6 +410,7 @@ void *colo_process_incoming_thread(void *opaque)
QEMUFile *fb = NULL;
QEMUSizedBuffer *buffer = NULL; /* Cache incoming device state */
uint64_t total_size;
+ int64_t error_time, current_time;
int ret = 0;
uint64_t value;
@@ -499,9 +526,28 @@ void *colo_process_incoming_thread(void *opaque)
}
out:
+ current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
if (ret < 0) {
error_report("colo incoming thread will exit, detect error: %s",
strerror(-ret));
+ /* Give users time to get involved in this verdict */
+ while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
+ if (failover_request_is_active()) {
+ error_report("Secondary VM will take over work");
+ break;
+ }
+ usleep(100 * 1000);
+ current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
+ }
+ /* check flag again*/
+ if (!failover_request_is_active()) {
+ /*
+ * We assume that Primary VM is still alive according to
+ * heartbeat, just kill Secondary VM
+ */
+ error_report("SVM is going to exit in default!");
+ exit(1);
+ }
}
if (fb) {
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 26/39] qmp event: Add event notification for COLO error
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (24 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 25/39] COLO: implement default failover treatment zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 27/39] COLO failover: Shutdown related socket fd when do failover zhanghailiang
` (12 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, dgilbert, zhanghailiang,
arei.gonglei, stefanha, amit.shah, Michael Roth, hongyang.yang
If some errors happen during VM's COLO FT stage, it's important to notify the users
of this event. Together with 'colo_lost_heartbeat', users can intervene in COLO's
failover work immediately.
If users don't want to get involved in COLO's failover verdict,
it is still necessary to notify users that we exited COLO mode.
Cc: Markus Armbruster <armbru@redhat.com>
Cc: Michael Roth <mdroth@linux.vnet.ibm.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
v11:
- Fix several typos found by Eric
---
docs/qmp-events.txt | 17 +++++++++++++++++
migration/colo.c | 13 +++++++++++++
qapi-schema.json | 16 ++++++++++++++++
qapi/event.json | 17 +++++++++++++++++
4 files changed, 63 insertions(+)
diff --git a/docs/qmp-events.txt b/docs/qmp-events.txt
index d2f1ce4..19f68fc 100644
--- a/docs/qmp-events.txt
+++ b/docs/qmp-events.txt
@@ -184,6 +184,23 @@ Example:
Note: The "ready to complete" status is always reset by a BLOCK_JOB_ERROR
event.
+COLO_EXIT
+---------
+
+Emitted when VM finishes COLO mode due to some errors happening or
+at the request of users.
+
+Data:
+
+ - "mode": COLO mode, primary or secondary side (json-string)
+ - "reason": the exit reason, internal error or external request. (json-string)
+ - "error": error message (json-string, operation)
+
+Example:
+
+{"timestamp": {"seconds": 2032141960, "microseconds": 417172},
+ "event": "COLO_EXIT", "data": {"mode": "primary", "reason": "request" } }
+
DEVICE_DELETED
--------------
diff --git a/migration/colo.c b/migration/colo.c
index 1e6d3dd..4cd7b00 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -18,6 +18,7 @@
#include "qemu/error-report.h"
#include "qemu/sockets.h"
#include "migration/failover.h"
+#include "qapi-event.h"
/*
* The delay time before qemu begin the procedure of default failover treatment.
@@ -334,6 +335,9 @@ out:
current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
if (ret < 0) {
error_report("%s: %s", __func__, strerror(-ret));
+ qapi_event_send_colo_exit(COLO_MODE_PRIMARY, COLO_EXIT_REASON_ERROR,
+ true, strerror(-ret), NULL);
+
/* Give users time to get involved in this verdict */
while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
if (failover_request_is_active()) {
@@ -350,6 +354,9 @@ out:
failover_request_active(NULL);
}
qemu_mutex_unlock_iothread();
+ } else {
+ qapi_event_send_colo_exit(COLO_MODE_PRIMARY, COLO_EXIT_REASON_REQUEST,
+ false, NULL, NULL);
}
qsb_free(buffer);
@@ -530,6 +537,9 @@ out:
if (ret < 0) {
error_report("colo incoming thread will exit, detect error: %s",
strerror(-ret));
+ qapi_event_send_colo_exit(COLO_MODE_SECONDARY, COLO_EXIT_REASON_ERROR,
+ true, strerror(-ret), NULL);
+
/* Give users time to get involved in this verdict */
while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
if (failover_request_is_active()) {
@@ -548,6 +558,9 @@ out:
error_report("SVM is going to exit in default!");
exit(1);
}
+ } else {
+ qapi_event_send_colo_exit(COLO_MODE_SECONDARY, COLO_EXIT_REASON_REQUEST,
+ false, NULL, NULL);
}
if (fb) {
diff --git a/qapi-schema.json b/qapi-schema.json
index 2ed63e0..981e9b2 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -782,6 +782,22 @@
'data': [ 'unknown', 'primary', 'secondary'] }
##
+# @COLOExitReason
+#
+# The reason for a COLO exit
+#
+# @unknown: unknown reason
+#
+# @request: COLO exit is due to an external request
+#
+# @error: COLO exit is due to an internal error
+#
+# Since: 2.6
+##
+{ 'enum': 'COLOExitReason',
+ 'data': [ 'unknown', 'request', 'error'] }
+
+##
# @x-colo-lost-heartbeat
#
# Tell qemu that heartbeat is lost, request it to do takeover procedures.
diff --git a/qapi/event.json b/qapi/event.json
index f0cef01..f63d456 100644
--- a/qapi/event.json
+++ b/qapi/event.json
@@ -255,6 +255,23 @@
'data': {'status': 'MigrationStatus'}}
##
+# @COLO_EXIT
+#
+# Emitted when VM finishes COLO mode due to some errors happening or
+# at the request of users.
+#
+# @mode: which COLO mode the VM was in when it exited.
+#
+# @reason: describes the reason for the COLO exit.
+#
+# @error: #optional, error message. Only present on error happening.
+#
+# Since: 2.6
+##
+{ 'event': 'COLO_EXIT',
+ 'data': {'mode': 'COLOMode', 'reason': 'COLOExitReason', '*error': 'str' } }
+
+##
# @ACPI_DEVICE_OST
#
# Emitted when guest executes ACPI _OST method.
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 27/39] COLO failover: Shutdown related socket fd when do failover
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (25 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 26/39] qmp event: Add event notification for COLO error zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-12-10 20:03 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 28/39] COLO failover: Don't do failover during loading VM's state zhanghailiang
` (11 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
If the net connection between COLO's two sides is broken while colo/colo incoming
thread is blocked in 'read'/'write' socket fd. It will not detect this error until
connect timeout. It will be a long time.
Here we shutdown all the related socket file descriptors to wake up the blocking
operation in failover BH. Besides, we should close the corresponding file descriptors
after failvoer BH shutdown them, or there will be an error.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
v11:
- Only shutdown fd for once
---
migration/colo.c | 31 +++++++++++++++++++++++++++++--
1 file changed, 29 insertions(+), 2 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index 4cd7b00..994b80d 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -68,6 +68,14 @@ static void secondary_vm_do_failover(void)
/* recover runstate to normal migration finish state */
autostart = true;
}
+ /*
+ * Make sure colo incoming thread not block in recv,
+ * mis->from_src_file and mis->to_src_file use the same fd,
+ * so here we only need to shutdown it for once.
+ */
+ if (mis->from_src_file) {
+ qemu_file_shutdown(mis->from_src_file);
+ }
old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
FAILOVER_STATUS_COMPLETED);
@@ -92,6 +100,15 @@ static void primary_vm_do_failover(void)
MIGRATION_STATUS_COMPLETED);
}
+ /*
+ * Make sure colo thread no block in recv,
+ * Besides, s->rp_state.from_dst_file and s->to_dst_file use the
+ * same fd, so here we only need to shutdown it for once.
+ */
+ if (s->to_dst_file) {
+ qemu_file_shutdown(s->to_dst_file);
+ }
+
old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
FAILOVER_STATUS_COMPLETED);
if (old_state != FAILOVER_STATUS_HANDLING) {
@@ -333,7 +350,7 @@ static void colo_process_checkpoint(MigrationState *s)
out:
current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
- if (ret < 0) {
+ if (ret < 0 || (!ret && !failover_request_is_active())) {
error_report("%s: %s", __func__, strerror(-ret));
qapi_event_send_colo_exit(COLO_MODE_PRIMARY, COLO_EXIT_REASON_ERROR,
true, strerror(-ret), NULL);
@@ -362,6 +379,11 @@ out:
qsb_free(buffer);
buffer = NULL;
+ /* Hope this not to be too long to loop here */
+ while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
+ ;
+ }
+ /* Must be called after failover BH is completed */
if (s->rp_state.from_dst_file) {
qemu_fclose(s->rp_state.from_dst_file);
}
@@ -534,7 +556,7 @@ void *colo_process_incoming_thread(void *opaque)
out:
current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
- if (ret < 0) {
+ if (ret < 0 || (!ret && !failover_request_is_active())) {
error_report("colo incoming thread will exit, detect error: %s",
strerror(-ret));
qapi_event_send_colo_exit(COLO_MODE_SECONDARY, COLO_EXIT_REASON_ERROR,
@@ -573,6 +595,11 @@ out:
*/
colo_release_ram_cache();
+ /* Hope this not to be too long to loop here */
+ while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
+ ;
+ }
+ /* Must be called after failover BH is completed */
if (mis->to_src_file) {
qemu_fclose(mis->to_src_file);
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 28/39] COLO failover: Don't do failover during loading VM's state
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (26 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 27/39] COLO failover: Shutdown related socket fd when do failover zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 29/39] COLO: Process shutdown command for VM in COLO state zhanghailiang
` (10 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
We should not do failover work while the main thread is loading
VM's state, otherwise it will destroy the consistent of VM's memory and
device state.
Here we add a new failover status 'RELAUNCH' which means we should
relaunch the process of failover.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
include/migration/failover.h | 2 ++
migration/colo.c | 25 +++++++++++++++++++++++++
2 files changed, 27 insertions(+)
diff --git a/include/migration/failover.h b/include/migration/failover.h
index fba3931..e115d25 100644
--- a/include/migration/failover.h
+++ b/include/migration/failover.h
@@ -20,6 +20,8 @@ typedef enum COLOFailoverStatus {
FAILOVER_STATUS_REQUEST = 1, /* Request but not handled */
FAILOVER_STATUS_HANDLING = 2, /* In the process of handling failover */
FAILOVER_STATUS_COMPLETED = 3, /* Finish the failover process */
+ /* Optional, Relaunch the failover process, again 'NONE' -> 'COMPLETED' */
+ FAILOVER_STATUS_RELAUNCH = 4,
} COLOFailoverStatus;
void failover_init_state(void);
diff --git a/migration/colo.c b/migration/colo.c
index 994b80d..e247d70 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -28,6 +28,8 @@
*/
#define DEFAULT_FAILOVER_DELAY 2000
+static bool vmstate_loading;
+
/* colo buffer */
#define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
@@ -60,6 +62,19 @@ static void secondary_vm_do_failover(void)
int old_state;
MigrationIncomingState *mis = migration_incoming_get_current();
+ /* Can not do failover during the process of VM's loading VMstate, Or
+ * it will break the secondary VM.
+ */
+ if (vmstate_loading) {
+ old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
+ FAILOVER_STATUS_RELAUNCH);
+ if (old_state != FAILOVER_STATUS_HANDLING) {
+ error_report("Unknow error while do failover for secondary VM,"
+ "old_state: %d", old_state);
+ }
+ return;
+ }
+
migrate_set_state(&mis->state, MIGRATION_STATUS_COLO,
MIGRATION_STATUS_COMPLETED);
@@ -538,13 +553,23 @@ void *colo_process_incoming_thread(void *opaque)
qemu_mutex_lock_iothread();
qemu_system_reset(VMRESET_SILENT);
+ vmstate_loading = true;
if (qemu_loadvm_state(fb) < 0) {
error_report("COLO: loadvm failed");
+ vmstate_loading = false;
qemu_mutex_unlock_iothread();
goto out;
}
+
+ vmstate_loading = false;
qemu_mutex_unlock_iothread();
+ if (failover_get_state() == FAILOVER_STATUS_RELAUNCH) {
+ failover_set_state(FAILOVER_STATUS_RELAUNCH, FAILOVER_STATUS_NONE);
+ failover_request_active(NULL);
+ goto out;
+ }
+
ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_LOADED, 0);
if (ret < 0) {
goto out;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 29/39] COLO: Process shutdown command for VM in COLO state
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (27 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 28/39] COLO failover: Don't do failover during loading VM's state zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 30/39] COLO: Update the global runstate after going into colo state zhanghailiang
` (9 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
Paolo Bonzini, hongyang.yang
If VM is in COLO FT state, we should do some extra work before normal shutdown
process. SVM will ignore the shutdown command if this command is issued directly
to it, PVM will send the shutdown command to SVM if it gets this command.
Cc: Paolo Bonzini <pbonzini@redhat.com>
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
include/sysemu/sysemu.h | 3 +++
migration/colo.c | 25 +++++++++++++++++++++++--
qapi-schema.json | 4 +++-
vl.c | 26 ++++++++++++++++++++++++--
4 files changed, 53 insertions(+), 5 deletions(-)
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 3bb8897..91eeda3 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -52,6 +52,8 @@ typedef enum WakeupReason {
QEMU_WAKEUP_REASON_OTHER,
} WakeupReason;
+extern int colo_shutdown_requested;
+
void qemu_system_reset_request(void);
void qemu_system_suspend_request(void);
void qemu_register_suspend_notifier(Notifier *notifier);
@@ -59,6 +61,7 @@ void qemu_system_wakeup_request(WakeupReason reason);
void qemu_system_wakeup_enable(WakeupReason reason, bool enabled);
void qemu_register_wakeup_notifier(Notifier *notifier);
void qemu_system_shutdown_request(void);
+void qemu_system_shutdown_request_core(void);
void qemu_system_powerdown_request(void);
void qemu_register_powerdown_notifier(Notifier *notifier);
void qemu_system_debug_request(void);
diff --git a/migration/colo.c b/migration/colo.c
index e247d70..771ff42 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -206,6 +206,7 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
{
int ret;
uint64_t value;
+ int colo_shutdown;
size_t size;
QEMUFile *trans = NULL;
@@ -233,6 +234,7 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
ret = -1;
goto out;
}
+ colo_shutdown = colo_shutdown_requested;
vm_stop_force_state(RUN_STATE_COLO);
qemu_mutex_unlock_iothread();
trace_colo_vm_state_change("run", "stop");
@@ -286,6 +288,15 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
goto out;
}
+ if (colo_shutdown) {
+ colo_ctl_put(s->to_dst_file, COLO_COMMAND_GUEST_SHUTDOWN, 0);
+ qemu_fflush(s->to_dst_file);
+ colo_shutdown_requested = 0;
+ qemu_system_shutdown_request_core();
+ /* Fix me: Just let the colo thread exit ? */
+ qemu_thread_exit(0);
+ }
+
ret = 0;
/* Resume primary guest */
qemu_mutex_lock_iothread();
@@ -347,8 +358,9 @@ static void colo_process_checkpoint(MigrationState *s)
}
current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
- if (current_time - checkpoint_time <
- s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) {
+ if ((current_time - checkpoint_time <
+ s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) &&
+ !colo_shutdown_requested) {
int64_t delay_ms;
delay_ms = s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] -
@@ -443,6 +455,15 @@ static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
case COLO_COMMAND_CHECKPOINT_REQUEST:
*checkpoint_request = 1;
return 0;
+ case COLO_COMMAND_GUEST_SHUTDOWN:
+ qemu_mutex_lock_iothread();
+ vm_stop_force_state(RUN_STATE_COLO);
+ qemu_system_shutdown_request_core();
+ qemu_mutex_unlock_iothread();
+ /* the main thread will exit and termiante the whole
+ * process, do we need some cleanup?
+ */
+ qemu_thread_exit(0);
default:
return -EINVAL;
}
diff --git a/qapi-schema.json b/qapi-schema.json
index 981e9b2..cabca1e 100644
--- a/qapi-schema.json
+++ b/qapi-schema.json
@@ -758,12 +758,14 @@
#
# @vmstate-loaded: VM's state has been loaded by SVM
#
+# @guest-shutdown: shutdown require from PVM to SVM
+#
# Since: 2.6
##
{ 'enum': 'COLOCommand',
'data': [ 'invalid', 'checkpoint-ready', 'checkpoint-request',
'checkpoint-reply', 'vmstate-send', 'vmstate-size',
- 'vmstate-received', 'vmstate-loaded' ] }
+ 'vmstate-received', 'vmstate-loaded', 'guest-shutdown' ] }
##
# @COLOMode
diff --git a/vl.c b/vl.c
index 083a60c..8dc34ce 100644
--- a/vl.c
+++ b/vl.c
@@ -1636,6 +1636,8 @@ static NotifierList wakeup_notifiers =
NOTIFIER_LIST_INITIALIZER(wakeup_notifiers);
static uint32_t wakeup_reason_mask = ~(1 << QEMU_WAKEUP_REASON_NONE);
+int colo_shutdown_requested;
+
int qemu_shutdown_requested_get(void)
{
return shutdown_requested;
@@ -1767,6 +1769,10 @@ void qemu_system_guest_panicked(void)
void qemu_system_reset_request(void)
{
if (no_reboot) {
+ qemu_system_shutdown_request();
+ if (!shutdown_requested) {/* colo handle it ? */
+ return;
+ }
shutdown_requested = 1;
} else {
reset_requested = 1;
@@ -1840,14 +1846,30 @@ void qemu_system_killed(int signal, pid_t pid)
qemu_notify_event();
}
-void qemu_system_shutdown_request(void)
+void qemu_system_shutdown_request_core(void)
{
- trace_qemu_system_shutdown_request();
replay_shutdown_request();
shutdown_requested = 1;
qemu_notify_event();
}
+void qemu_system_shutdown_request(void)
+{
+ trace_qemu_system_shutdown_request();
+ /*
+ * if in colo mode, we need do some significant work before respond to the
+ * shutdown request.
+ */
+ if (migration_incoming_in_colo_state()) {
+ return ; /* primary's responsibility */
+ }
+ if (migration_in_colo_state()) {
+ colo_shutdown_requested = 1;
+ return;
+ }
+ qemu_system_shutdown_request_core();
+}
+
static void qemu_system_powerdown(void)
{
qapi_event_send_powerdown(&error_abort);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 30/39] COLO: Update the global runstate after going into colo state
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (28 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 29/39] COLO: Process shutdown command for VM in COLO state zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 31/39] savevm: Split load vm state function qemu_loadvm_state zhanghailiang
` (8 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
If we start qemu with -S, the runstate will change from 'prelaunch' to 'running'
after going into colo state.
So it is necessary to update the global runstate after going into colo state.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 5 +++++
1 file changed, 5 insertions(+)
diff --git a/migration/colo.c b/migration/colo.c
index 771ff42..3866e86 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -351,6 +351,11 @@ static void colo_process_checkpoint(MigrationState *s)
qemu_mutex_unlock_iothread();
trace_colo_vm_state_change("stop", "run");
+ ret = global_state_store();
+ if (ret < 0) {
+ goto out;
+ }
+
while (s->state == MIGRATION_STATUS_COLO) {
if (failover_request_is_active()) {
error_report("failover request");
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 31/39] savevm: Split load vm state function qemu_loadvm_state
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (29 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 30/39] COLO: Update the global runstate after going into colo state zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 32/39] COLO: Separate the process of saving/loading ram and device state zhanghailiang
` (7 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
qemu_loadvm_state is too long, and we can simplify it by splitting up
with three helper functions.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
---
migration/savevm.c | 161 ++++++++++++++++++++++++++++++++---------------------
1 file changed, 97 insertions(+), 64 deletions(-)
diff --git a/migration/savevm.c b/migration/savevm.c
index f102870..c7c26d8 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -1710,90 +1710,123 @@ void loadvm_free_handlers(MigrationIncomingState *mis)
}
}
+static int
+qemu_loadvm_section_start_full(QEMUFile *f, MigrationIncomingState *mis)
+{
+ uint32_t instance_id, version_id, section_id;
+ SaveStateEntry *se;
+ LoadStateEntry *le;
+ char idstr[256];
+ int ret;
+
+ /* Read section start */
+ section_id = qemu_get_be32(f);
+ if (!qemu_get_counted_string(f, idstr)) {
+ error_report("Unable to read ID string for section %u",
+ section_id);
+ return -EINVAL;
+ }
+ instance_id = qemu_get_be32(f);
+ version_id = qemu_get_be32(f);
+
+ trace_qemu_loadvm_state_section_startfull(section_id, idstr,
+ instance_id, version_id);
+ /* Find savevm section */
+ se = find_se(idstr, instance_id);
+ if (se == NULL) {
+ error_report("Unknown savevm section or instance '%s' %d",
+ idstr, instance_id);
+ ret = -EINVAL;
+ return ret;
+ }
+
+ /* Validate version */
+ if (version_id > se->version_id) {
+ error_report("savevm: unsupported version %d for '%s' v%d",
+ version_id, idstr, se->version_id);
+ ret = -EINVAL;
+ return ret;
+ }
+
+ /* Add entry */
+ le = g_malloc0(sizeof(*le));
+
+ le->se = se;
+ le->section_id = section_id;
+ le->version_id = version_id;
+ QLIST_INSERT_HEAD(&mis->loadvm_handlers, le, entry);
+
+ ret = vmstate_load(f, le->se, le->version_id);
+ if (ret < 0) {
+ error_report("error while loading state for instance 0x%x of"
+ " device '%s'", instance_id, idstr);
+ return ret;
+ }
+ if (!check_section_footer(f, le)) {
+ ret = -EINVAL;
+ return ret;
+ }
+
+ return 0;
+}
+
+static int
+qemu_loadvm_section_part_end(QEMUFile *f, MigrationIncomingState *mis)
+{
+ uint32_t section_id;
+ LoadStateEntry *le;
+ int ret;
+
+ section_id = qemu_get_be32(f);
+
+ trace_qemu_loadvm_state_section_partend(section_id);
+ QLIST_FOREACH(le, &mis->loadvm_handlers, entry) {
+ if (le->section_id == section_id) {
+ break;
+ }
+ }
+ if (le == NULL) {
+ error_report("Unknown savevm section %d", section_id);
+ ret = -EINVAL;
+ return ret;
+ }
+
+ ret = vmstate_load(f, le->se, le->version_id);
+ if (ret < 0) {
+ error_report("error while loading state section id %d(%s)",
+ section_id, le->se->idstr);
+ return ret;
+ }
+ if (!check_section_footer(f, le)) {
+ ret = -EINVAL;
+ return ret;
+ }
+
+ return 0;
+}
+
static int qemu_loadvm_state_main(QEMUFile *f, MigrationIncomingState *mis)
{
uint8_t section_type;
int ret;
while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
- uint32_t instance_id, version_id, section_id;
- SaveStateEntry *se;
- LoadStateEntry *le;
- char idstr[256];
trace_qemu_loadvm_state_section(section_type);
switch (section_type) {
case QEMU_VM_SECTION_START:
case QEMU_VM_SECTION_FULL:
- /* Read section start */
- section_id = qemu_get_be32(f);
- if (!qemu_get_counted_string(f, idstr)) {
- error_report("Unable to read ID string for section %u",
- section_id);
- return -EINVAL;
- }
- instance_id = qemu_get_be32(f);
- version_id = qemu_get_be32(f);
-
- trace_qemu_loadvm_state_section_startfull(section_id, idstr,
- instance_id, version_id);
- /* Find savevm section */
- se = find_se(idstr, instance_id);
- if (se == NULL) {
- error_report("Unknown savevm section or instance '%s' %d",
- idstr, instance_id);
- return -EINVAL;
- }
-
- /* Validate version */
- if (version_id > se->version_id) {
- error_report("savevm: unsupported version %d for '%s' v%d",
- version_id, idstr, se->version_id);
- return -EINVAL;
- }
-
- /* Add entry */
- le = g_malloc0(sizeof(*le));
-
- le->se = se;
- le->section_id = section_id;
- le->version_id = version_id;
- QLIST_INSERT_HEAD(&mis->loadvm_handlers, le, entry);
-
- ret = vmstate_load(f, le->se, le->version_id);
+ ret = qemu_loadvm_section_start_full(f, mis);
if (ret < 0) {
- error_report("error while loading state for instance 0x%x of"
- " device '%s'", instance_id, idstr);
return ret;
}
- if (!check_section_footer(f, le)) {
- return -EINVAL;
- }
break;
case QEMU_VM_SECTION_PART:
case QEMU_VM_SECTION_END:
- section_id = qemu_get_be32(f);
-
- trace_qemu_loadvm_state_section_partend(section_id);
- QLIST_FOREACH(le, &mis->loadvm_handlers, entry) {
- if (le->section_id == section_id) {
- break;
- }
- }
- if (le == NULL) {
- error_report("Unknown savevm section %d", section_id);
- return -EINVAL;
- }
-
- ret = vmstate_load(f, le->se, le->version_id);
+ ret = qemu_loadvm_section_part_end(f, mis);
if (ret < 0) {
- error_report("error while loading state section id %d(%s)",
- section_id, le->se->idstr);
return ret;
}
- if (!check_section_footer(f, le)) {
- return -EINVAL;
- }
break;
case QEMU_VM_COMMAND:
ret = loadvm_process_command(f);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 32/39] COLO: Separate the process of saving/loading ram and device state
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (30 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 31/39] savevm: Split load vm state function qemu_loadvm_state zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-27 5:10 ` Li Zhijian
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 33/39] COLO: Split qemu_savevm_state_begin out of checkpoint process zhanghailiang
` (6 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
We separate the process of saving/loading ram and device state when do checkpoint,
we add new helpers for save/load ram/device. With this change, we can directly
transfer ram from master to slave without using QEMUSizeBuffer as assistant,
which also reduce the size of extra memory been used during checkpoint.
Besides, we move the colo_flush_ram_cache to the proper position after the
above change.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
v11:
- Remove load configuration section in qemu_loadvm_state_begin()
---
include/sysemu/sysemu.h | 6 +++
migration/colo.c | 43 ++++++++++++----
migration/ram.c | 5 --
migration/savevm.c | 132 ++++++++++++++++++++++++++++++++++++++++++++++--
4 files changed, 168 insertions(+), 18 deletions(-)
diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
index 91eeda3..5deae53 100644
--- a/include/sysemu/sysemu.h
+++ b/include/sysemu/sysemu.h
@@ -133,7 +133,13 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
uint64_t *start_list,
uint64_t *length_list);
+int qemu_save_ram_precopy(QEMUFile *f);
+int qemu_save_device_state(QEMUFile *f);
+
int qemu_loadvm_state(QEMUFile *f);
+int qemu_loadvm_state_begin(QEMUFile *f);
+int qemu_load_ram_state(QEMUFile *f);
+int qemu_load_device_state(QEMUFile *f);
typedef enum DisplayType
{
diff --git a/migration/colo.c b/migration/colo.c
index 3866e86..f7f349b 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -247,21 +247,32 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
goto out;
}
+ ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
+ if (ret < 0) {
+ goto out;
+ }
/* Disable block migration */
s->params.blk = 0;
s->params.shared = 0;
- qemu_savevm_state_header(trans);
- qemu_savevm_state_begin(trans, &s->params);
- qemu_mutex_lock_iothread();
- qemu_savevm_state_complete_precopy(trans, false);
- qemu_mutex_unlock_iothread();
-
- qemu_fflush(trans);
+ qemu_savevm_state_begin(s->to_dst_file, &s->params);
+ ret = qemu_file_get_error(s->to_dst_file);
+ if (ret < 0) {
+ error_report("save vm state begin error\n");
+ goto out;
+ }
- ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
+ qemu_mutex_lock_iothread();
+ /* Note: device state is saved into buffer */
+ ret = qemu_save_device_state(trans);
if (ret < 0) {
+ error_report("save device state error\n");
+ qemu_mutex_unlock_iothread();
goto out;
}
+ qemu_fflush(trans);
+ qemu_save_ram_precopy(s->to_dst_file);
+ qemu_mutex_unlock_iothread();
+
/* we send the total size of the vmstate first */
size = qsb_get_length(buffer);
ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SIZE, size);
@@ -548,6 +559,16 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
+ ret = qemu_loadvm_state_begin(mis->from_src_file);
+ if (ret < 0) {
+ error_report("load vm state begin error, ret=%d", ret);
+ goto out;
+ }
+ ret = qemu_load_ram_state(mis->from_src_file);
+ if (ret < 0) {
+ error_report("load ram state error");
+ goto out;
+ }
/* read the VM state total size first */
ret = colo_ctl_get(mis->from_src_file,
COLO_COMMAND_VMSTATE_SIZE, &value);
@@ -580,8 +601,10 @@ void *colo_process_incoming_thread(void *opaque)
qemu_mutex_lock_iothread();
qemu_system_reset(VMRESET_SILENT);
vmstate_loading = true;
- if (qemu_loadvm_state(fb) < 0) {
- error_report("COLO: loadvm failed");
+ colo_flush_ram_cache();
+ ret = qemu_load_device_state(fb);
+ if (ret < 0) {
+ error_report("COLO: load device state failed\n");
vmstate_loading = false;
qemu_mutex_unlock_iothread();
goto out;
diff --git a/migration/ram.c b/migration/ram.c
index 4f37144..06a738b 100644
--- a/migration/ram.c
+++ b/migration/ram.c
@@ -2448,7 +2448,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
* be atomic
*/
bool postcopy_running = postcopy_state_get() >= POSTCOPY_INCOMING_LISTENING;
- bool need_flush = false;
seq_iter++;
@@ -2483,7 +2482,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
/* After going into COLO, we should load the Page into colo_cache */
if (ram_cache_enable) {
host = colo_cache_from_block_offset(block, addr);
- need_flush = true;
} else {
host = host_from_ram_block_offset(block, addr);
}
@@ -2578,9 +2576,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
rcu_read_unlock();
- if (!ret && ram_cache_enable && need_flush) {
- colo_flush_ram_cache();
- }
DPRINTF("Completed load of VM with exit code %d seq iteration "
"%" PRIu64 "\n", ret, seq_iter);
return ret;
diff --git a/migration/savevm.c b/migration/savevm.c
index c7c26d8..949caf0 100644
--- a/migration/savevm.c
+++ b/migration/savevm.c
@@ -50,6 +50,7 @@
#include "qemu/iov.h"
#include "block/snapshot.h"
#include "block/qapi.h"
+#include "migration/colo.h"
#ifndef ETH_P_RARP
@@ -923,6 +924,10 @@ void qemu_savevm_state_begin(QEMUFile *f,
break;
}
}
+ if (migration_in_colo_state()) {
+ qemu_put_byte(f, QEMU_VM_EOF);
+ qemu_fflush(f);
+ }
}
/*
@@ -1192,13 +1197,44 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
return ret;
}
-static int qemu_save_device_state(QEMUFile *f)
+int qemu_save_ram_precopy(QEMUFile *f)
{
SaveStateEntry *se;
+ int ret = 0;
- qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
- qemu_put_be32(f, QEMU_VM_FILE_VERSION);
+ QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
+ if (!se->ops || !se->ops->save_live_complete_precopy) {
+ continue;
+ }
+ if (se->ops && se->ops->is_active) {
+ if (!se->ops->is_active(se->opaque)) {
+ continue;
+ }
+ }
+ trace_savevm_section_start(se->idstr, se->section_id);
+ save_section_header(f, se, QEMU_VM_SECTION_END);
+
+ ret = se->ops->save_live_complete_precopy(f, se->opaque);
+ trace_savevm_section_end(se->idstr, se->section_id, ret);
+ save_section_footer(f, se);
+ if (ret < 0) {
+ qemu_file_set_error(f, ret);
+ return ret;
+ }
+ }
+ qemu_put_byte(f, QEMU_VM_EOF);
+
+ return 0;
+}
+
+int qemu_save_device_state(QEMUFile *f)
+{
+ SaveStateEntry *se;
+
+ if (!migration_in_colo_state()) {
+ qemu_savevm_state_header(f);
+ }
cpu_synchronize_all_states();
QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
@@ -1938,6 +1974,96 @@ int qemu_loadvm_state(QEMUFile *f)
return ret;
}
+int qemu_loadvm_state_begin(QEMUFile *f)
+{
+ uint8_t section_type;
+ int ret = -1;
+ MigrationIncomingState *mis = migration_incoming_get_current();
+
+ if (!mis) {
+ error_report("qemu_loadvm_state_begin");
+ return -EINVAL;
+ }
+ /* CleanUp */
+ loadvm_free_handlers(mis);
+
+ if (qemu_savevm_state_blocked(NULL)) {
+ return -EINVAL;
+ }
+
+ while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
+ if (section_type != QEMU_VM_SECTION_START) {
+ error_report("QEMU_VM_SECTION_START");
+ ret = -EINVAL;
+ goto out;
+ }
+ ret = qemu_loadvm_section_start_full(f, mis);
+ if (ret < 0) {
+ goto out;
+ }
+ }
+ ret = qemu_file_get_error(f);
+ if (ret == 0) {
+ return 0;
+ }
+out:
+ return ret;
+}
+
+int qemu_load_ram_state(QEMUFile *f)
+{
+ uint8_t section_type;
+ MigrationIncomingState *mis = migration_incoming_get_current();
+ int ret = -1;
+
+ while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
+ if (section_type != QEMU_VM_SECTION_PART &&
+ section_type != QEMU_VM_SECTION_END) {
+ error_report("load ram state, not get "
+ "QEMU_VM_SECTION_FULL or QEMU_VM_SECTION_END");
+ return -EINVAL;
+ }
+ ret = qemu_loadvm_section_part_end(f, mis);
+ if (ret < 0) {
+ goto out;
+ }
+ }
+ ret = qemu_file_get_error(f);
+ if (ret == 0) {
+ return 0;
+ }
+out:
+ return ret;
+}
+
+int qemu_load_device_state(QEMUFile *f)
+{
+ uint8_t section_type;
+ MigrationIncomingState *mis = migration_incoming_get_current();
+ int ret = -1;
+
+ while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
+ if (section_type != QEMU_VM_SECTION_FULL) {
+ error_report("load device state error: "
+ "Not get QEMU_VM_SECTION_FULL");
+ return -EINVAL;
+ }
+ ret = qemu_loadvm_section_start_full(f, mis);
+ if (ret < 0) {
+ goto out;
+ }
+ }
+
+ ret = qemu_file_get_error(f);
+
+ cpu_synchronize_all_post_init();
+ if (ret == 0) {
+ return 0;
+ }
+out:
+ return ret;
+}
+
void hmp_savevm(Monitor *mon, const QDict *qdict)
{
BlockDriverState *bs, *bs1;
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 33/39] COLO: Split qemu_savevm_state_begin out of checkpoint process
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (31 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 32/39] COLO: Separate the process of saving/loading ram and device state zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev zhanghailiang
` (5 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
It is unnecessary to call qemu_savevm_state_begin() in every checkponit process.
It mainly sets up devices and does the first device state pass. These data will
not change during the later checkpoint process. So, we split it out of
colo_do_checkpoint_transaction(), in this way, we can reduce these data
transferring in the later checkpoint.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 51 +++++++++++++++++++++++++++++++++++++--------------
1 file changed, 37 insertions(+), 14 deletions(-)
diff --git a/migration/colo.c b/migration/colo.c
index f7f349b..79a8d6b 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -251,15 +251,6 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
if (ret < 0) {
goto out;
}
- /* Disable block migration */
- s->params.blk = 0;
- s->params.shared = 0;
- qemu_savevm_state_begin(s->to_dst_file, &s->params);
- ret = qemu_file_get_error(s->to_dst_file);
- if (ret < 0) {
- error_report("save vm state begin error\n");
- goto out;
- }
qemu_mutex_lock_iothread();
/* Note: device state is saved into buffer */
@@ -323,6 +314,21 @@ out:
return ret;
}
+static int colo_prepare_before_save(MigrationState *s)
+{
+ int ret;
+ /* Disable block migration */
+ s->params.blk = 0;
+ s->params.shared = 0;
+ qemu_savevm_state_begin(s->to_dst_file, &s->params);
+ ret = qemu_file_get_error(s->to_dst_file);
+ if (ret < 0) {
+ error_report("save vm state begin error\n");
+ return ret;
+ }
+ return 0;
+}
+
static void colo_process_checkpoint(MigrationState *s)
{
QEMUSizedBuffer *buffer = NULL;
@@ -340,6 +346,11 @@ static void colo_process_checkpoint(MigrationState *s)
goto out;
}
+ ret = colo_prepare_before_save(s);
+ if (ret < 0) {
+ goto out;
+ }
+
/*
* Wait for Secondary finish loading vm states and enter COLO
* restore.
@@ -485,6 +496,18 @@ static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
}
}
+static int colo_prepare_before_load(QEMUFile *f)
+{
+ int ret;
+
+ ret = qemu_loadvm_state_begin(f);
+ if (ret < 0) {
+ error_report("load vm state begin error, ret=%d", ret);
+ return ret;
+ }
+ return 0;
+}
+
void *colo_process_incoming_thread(void *opaque)
{
MigrationIncomingState *mis = opaque;
@@ -525,6 +548,11 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
+ ret = colo_prepare_before_load(mis->from_src_file);
+ if (ret < 0) {
+ goto out;
+ }
+
ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_READY, 0);
if (ret < 0) {
goto out;
@@ -559,11 +587,6 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
- ret = qemu_loadvm_state_begin(mis->from_src_file);
- if (ret < 0) {
- error_report("load vm state begin error, ret=%d", ret);
- goto out;
- }
ret = qemu_load_ram_state(mis->from_src_file);
if (ret < 0) {
error_report("load ram state error");
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (32 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 33/39] COLO: Split qemu_savevm_state_begin out of checkpoint process zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-27 11:39 ` Yang Hongyang
` (2 more replies)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 35/39] filter-buffer: Accept zero interval zhanghailiang
` (4 subsequent siblings)
38 siblings, 3 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, zhanghailiang, arei.gonglei, stefanha,
amit.shah, hongyang.yang
We add each netdev a default filter-buffer, which will be used for COLO
or Micro-checkpoint to buffer VM's packets. The name of default filter-buffer
is 'nop'.
For the default filter-buffer, it will not buffer any packets in default.
So it has no side effect for the netdev.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Yang Hongyang <hongyang.yang@easystack.cn>
---
v11:
- New patch
---
include/net/filter.h | 3 +++
net/filter-buffer.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++
net/net.c | 8 ++++++
3 files changed, 85 insertions(+)
diff --git a/include/net/filter.h b/include/net/filter.h
index 2deda36..01a7e90 100644
--- a/include/net/filter.h
+++ b/include/net/filter.h
@@ -74,4 +74,7 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender,
int iovcnt,
void *opaque);
+void netdev_add_default_filter_buffer(const char *netdev_id,
+ NetFilterDirection direction,
+ Error **errp);
#endif /* QEMU_NET_FILTER_H */
diff --git a/net/filter-buffer.c b/net/filter-buffer.c
index 57be149..195af68 100644
--- a/net/filter-buffer.c
+++ b/net/filter-buffer.c
@@ -14,6 +14,12 @@
#include "qapi/qmp/qerror.h"
#include "qapi-visit.h"
#include "qom/object.h"
+#include "net/net.h"
+#include "qapi/qmp/qdict.h"
+#include "qapi/qmp-output-visitor.h"
+#include "qapi/qmp-input-visitor.h"
+#include "monitor/monitor.h"
+#include "qmp-commands.h"
#define TYPE_FILTER_BUFFER "filter-buffer"
@@ -26,6 +32,8 @@ typedef struct FilterBufferState {
NetQueue *incoming_queue;
uint32_t interval;
QEMUTimer release_timer;
+ bool is_default;
+ bool enable_buffer;
} FilterBufferState;
static void filter_buffer_flush(NetFilterState *nf)
@@ -65,6 +73,10 @@ static ssize_t filter_buffer_receive_iov(NetFilterState *nf,
{
FilterBufferState *s = FILTER_BUFFER(nf);
+ /* Don't buffer any packets if the filter is not enabled */
+ if (!s->enable_buffer) {
+ return 0;
+ }
/*
* We return size when buffer a packet, the sender will take it as
* a already sent packet, so sent_cb should not be called later.
@@ -102,6 +114,7 @@ static void filter_buffer_cleanup(NetFilterState *nf)
static void filter_buffer_setup(NetFilterState *nf, Error **errp)
{
FilterBufferState *s = FILTER_BUFFER(nf);
+ char *path = object_get_canonical_path_component(OBJECT(nf));
/*
* We may want to accept zero interval when VM FT solutions like MC
@@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
}
s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
+ s->is_default = !strcmp(path, "nop");
if (s->interval) {
timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
filter_buffer_release_timer, nf);
@@ -163,6 +177,66 @@ out:
error_propagate(errp, local_err);
}
+/*
+* This will be used by COLO or MC FT, for which they will need
+* to buffer the packets of VM's net devices, Here we add a default
+* buffer filter for each netdev. The name of default buffer filter is
+* 'nop'
+*/
+void netdev_add_default_filter_buffer(const char *netdev_id,
+ NetFilterDirection direction,
+ Error **errp)
+{
+ QmpOutputVisitor *qov;
+ QmpInputVisitor *qiv;
+ Visitor *ov, *iv;
+ QObject *obj = NULL;
+ QDict *qdict;
+ void *dummy = NULL;
+ const char *id = "nop";
+ char *queue = g_strdup(NetFilterDirection_lookup[direction]);
+ NetClientState *nc = qemu_find_netdev(netdev_id);
+ Error *err = NULL;
+
+ /* FIXME: Not support multiple queues */
+ if (!nc || nc->queue_index > 1) {
+ return;
+ }
+ qov = qmp_output_visitor_new();
+ ov = qmp_output_get_visitor(qov);
+ visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
+ if (err) {
+ goto out;
+ }
+ visit_type_str(ov, &nc->name, "netdev", &err);
+ if (err) {
+ goto out;
+ }
+ visit_type_str(ov, &queue, "queue", &err);
+ if (err) {
+ goto out;
+ }
+ visit_end_struct(ov, &err);
+ if (err) {
+ goto out;
+ }
+ obj = qmp_output_get_qobject(qov);
+ g_assert(obj != NULL);
+ qdict = qobject_to_qdict(obj);
+ qmp_output_visitor_cleanup(qov);
+
+ qiv = qmp_input_visitor_new(obj);
+ iv = qmp_input_get_visitor(qiv);
+ object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
+ qmp_input_visitor_cleanup(qiv);
+ qobject_decref(obj);
+out:
+ g_free(queue);
+ if (err) {
+ error_propagate(errp, err);
+ }
+}
+
static void filter_buffer_init(Object *obj)
{
object_property_add(obj, "interval", "int",
diff --git a/net/net.c b/net/net.c
index ade6051..b36d49f 100644
--- a/net/net.c
+++ b/net/net.c
@@ -1028,6 +1028,14 @@ static int net_client_init1(const void *object, int is_netdev, Error **errp)
}
return -1;
}
+
+ if (is_netdev) {
+ const Netdev *netdev = object;
+
+ netdev_add_default_filter_buffer(netdev->id,
+ NET_FILTER_DIRECTION_RX,
+ errp);
+ }
return 0;
}
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 35/39] filter-buffer: Accept zero interval
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (33 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-27 11:42 ` Yang Hongyang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 36/39] filter-buffer: Introduce a helper function to enable/disable default filter zhanghailiang
` (3 subsequent siblings)
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, zhanghailiang, arei.gonglei, stefanha,
amit.shah, hongyang.yang
For default buffer filter, its 'interval' value is zero,
so here we should accept zero interval.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Yang Hongyang <hongyang.yang@easystack.cn>
---
v11:
- Add comment
v10:
- new patch
---
net/filter-buffer.c | 10 ----------
1 file changed, 10 deletions(-)
diff --git a/net/filter-buffer.c b/net/filter-buffer.c
index 195af68..5cab4fc 100644
--- a/net/filter-buffer.c
+++ b/net/filter-buffer.c
@@ -116,16 +116,6 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
FilterBufferState *s = FILTER_BUFFER(nf);
char *path = object_get_canonical_path_component(OBJECT(nf));
- /*
- * We may want to accept zero interval when VM FT solutions like MC
- * or COLO use this filter to release packets on demand.
- */
- if (!s->interval) {
- error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "interval",
- "a non-zero interval");
- return;
- }
-
s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
s->is_default = !strcmp(path, "nop");
if (s->interval) {
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 36/39] filter-buffer: Introduce a helper function to enable/disable default filter
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (34 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 35/39] filter-buffer: Accept zero interval zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 37/39] filter-buffer: Introduce a helper function to release packets zhanghailiang
` (2 subsequent siblings)
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, zhanghailiang, arei.gonglei, stefanha,
amit.shah, hongyang.yang
The default buffer filter doesn't buffer packets in default,
but we need to buffer packets for COLO or Micro-checkpoint,
Here we add a helper function to enable/disable filter's buffer
capability.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Yang Hongyang <hongyang.yang@easystack.cn>
---
v11:
- New patch
---
include/net/filter.h | 1 +
include/net/net.h | 4 ++++
net/filter-buffer.c | 20 ++++++++++++++++++++
net/net.c | 29 +++++++++++++++++++++++++++++
4 files changed, 54 insertions(+)
diff --git a/include/net/filter.h b/include/net/filter.h
index 01a7e90..b57b29a 100644
--- a/include/net/filter.h
+++ b/include/net/filter.h
@@ -77,4 +77,5 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender,
void netdev_add_default_filter_buffer(const char *netdev_id,
NetFilterDirection direction,
Error **errp);
+void qemu_set_default_filter_buffers(bool enable_buffer);
#endif /* QEMU_NET_FILTER_H */
diff --git a/include/net/net.h b/include/net/net.h
index 7af3e15..5c65c45 100644
--- a/include/net/net.h
+++ b/include/net/net.h
@@ -125,6 +125,10 @@ NetClientState *qemu_find_vlan_client_by_name(Monitor *mon, int vlan_id,
const char *client_str);
typedef void (*qemu_nic_foreach)(NICState *nic, void *opaque);
void qemu_foreach_nic(qemu_nic_foreach func, void *opaque);
+typedef void (*qemu_netfilter_foreach)(NetFilterState *nf, void *opaque,
+ Error **errp);
+void qemu_foreach_netfilter(qemu_netfilter_foreach func, void *opaque,
+ Error **errp);
int qemu_can_send_packet(NetClientState *nc);
ssize_t qemu_sendv_packet(NetClientState *nc, const struct iovec *iov,
int iovcnt);
diff --git a/net/filter-buffer.c b/net/filter-buffer.c
index 5cab4fc..7115a2b 100644
--- a/net/filter-buffer.c
+++ b/net/filter-buffer.c
@@ -167,6 +167,26 @@ out:
error_propagate(errp, local_err);
}
+static void default_filter_set_buffer_flag(NetFilterState *nf,
+ void *opaque,
+ Error **errp)
+{
+ if (!strcmp(object_get_typename(OBJECT(nf)), TYPE_FILTER_BUFFER)) {
+ FilterBufferState *s = FILTER_BUFFER(nf);
+ bool *flag = opaque;
+
+ if (s->is_default) {
+ s->enable_buffer = *flag;
+ }
+ }
+}
+
+void qemu_set_default_filter_buffers(bool enable_buffer)
+{
+ qemu_foreach_netfilter(default_filter_set_buffer_flag,
+ &enable_buffer, NULL);
+}
+
/*
* This will be used by COLO or MC FT, for which they will need
* to buffer the packets of VM's net devices, Here we add a default
diff --git a/net/net.c b/net/net.c
index b36d49f..aa8e128 100644
--- a/net/net.c
+++ b/net/net.c
@@ -259,6 +259,35 @@ static char *assign_name(NetClientState *nc1, const char *model)
return g_strdup_printf("%s.%d", model, id);
}
+void qemu_foreach_netfilter(qemu_netfilter_foreach func, void *opaque,
+ Error **errp)
+{
+ NetClientState *nc;
+ NetFilterState *nf;
+
+ QTAILQ_FOREACH(nc, &net_clients, next) {
+ if (nc->info->type == NET_CLIENT_OPTIONS_KIND_NIC) {
+ continue;
+ }
+ /* FIXME: Not support multiqueue */
+ if (nc->queue_index > 1) {
+ error_setg(errp, "%s: multiqueue is not supported", __func__);
+ return;
+ }
+ QTAILQ_FOREACH(nf, &nc->filters, next) {
+ if (func) {
+ Error *local_err = NULL;
+
+ func(nf, opaque, &local_err);
+ if (local_err) {
+ error_propagate(errp, local_err);
+ return;
+ }
+ }
+ }
+ }
+}
+
static void qemu_net_client_destructor(NetClientState *nc)
{
g_free(nc);
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 37/39] filter-buffer: Introduce a helper function to release packets
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (35 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 36/39] filter-buffer: Introduce a helper function to enable/disable default filter zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 38/39] colo: Use default buffer-filter to buffer and " zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 39/39] COLO: Add block replication into colo process zhanghailiang
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, zhanghailiang, arei.gonglei, stefanha,
amit.shah, hongyang.yang
We need to release all the packets from VM in COLO or Micro-checkpoint,
here we add a new helper function to realse the packets that buffered
by default buffer-filter
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Yang Hongyang <hongyang.yang@easystack.cn>
---
v11:
- New patch
---
include/net/filter.h | 1 +
net/filter-buffer.c | 19 +++++++++++++++++++
2 files changed, 20 insertions(+)
diff --git a/include/net/filter.h b/include/net/filter.h
index b57b29a..e1e8d9b 100644
--- a/include/net/filter.h
+++ b/include/net/filter.h
@@ -77,5 +77,6 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender,
void netdev_add_default_filter_buffer(const char *netdev_id,
NetFilterDirection direction,
Error **errp);
+void qemu_release_default_filters_packets(void);
void qemu_set_default_filter_buffers(bool enable_buffer);
#endif /* QEMU_NET_FILTER_H */
diff --git a/net/filter-buffer.c b/net/filter-buffer.c
index 7115a2b..06f2190 100644
--- a/net/filter-buffer.c
+++ b/net/filter-buffer.c
@@ -167,6 +167,25 @@ out:
error_propagate(errp, local_err);
}
+static void default_filter_release_packets(NetFilterState *nf,
+ void *opaque,
+ Error **errp)
+{
+ if (!strcmp(object_get_typename(OBJECT(nf)), TYPE_FILTER_BUFFER)) {
+ FilterBufferState *s = FILTER_BUFFER(nf);
+
+ if (s->is_default) {
+ filter_buffer_flush(nf);
+ }
+ }
+}
+
+/* public APIs */
+void qemu_release_default_filters_packets(void)
+{
+ qemu_foreach_netfilter(default_filter_release_packets, NULL, NULL);
+}
+
static void default_filter_set_buffer_flag(NetFilterState *nf,
void *opaque,
Error **errp)
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 38/39] colo: Use default buffer-filter to buffer and release packets
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (36 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 37/39] filter-buffer: Introduce a helper function to release packets zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
2015-11-27 12:51 ` Yang Hongyang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 39/39] COLO: Add block replication into colo process zhanghailiang
38 siblings, 1 reply; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, zhanghailiang, arei.gonglei, stefanha,
amit.shah, hongyang.yang
Enable default filter to buffer packets and release the
packets after a checkpoint.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Cc: Jason Wang <jasowang@redhat.com>
Cc: Yang Hongyang <hongyang.yang@easystack.cn>
---
v11:
- Use new helper functions to buffer and release packets.
---
migration/colo.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/migration/colo.c b/migration/colo.c
index 79a8d6b..b1b7905 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -19,6 +19,7 @@
#include "qemu/sockets.h"
#include "migration/failover.h"
#include "qapi-event.h"
+#include "net/filter.h"
/*
* The delay time before qemu begin the procedure of default failover treatment.
@@ -131,6 +132,8 @@ static void primary_vm_do_failover(void)
"old_state: %d", old_state);
return;
}
+ /* Don't buffer any packets while exited COLO */
+ qemu_set_default_filter_buffers(false);
}
void colo_do_failover(MigrationState *s)
@@ -290,6 +293,8 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
goto out;
}
+ qemu_release_default_filters_packets();
+
if (colo_shutdown) {
colo_ctl_put(s->to_dst_file, COLO_COMMAND_GUEST_SHUTDOWN, 0);
qemu_fflush(s->to_dst_file);
@@ -367,6 +372,8 @@ static void colo_process_checkpoint(MigrationState *s)
error_report("Failed to allocate colo buffer!");
goto out;
}
+ /* Begin to buffer packets that sent by VM */
+ qemu_set_default_filter_buffers(true);
qemu_mutex_lock_iothread();
vm_start();
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* [Qemu-devel] [PATCH COLO-Frame v11 39/39] COLO: Add block replication into colo process
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
` (37 preceding siblings ...)
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 38/39] colo: Use default buffer-filter to buffer and " zhanghailiang
@ 2015-11-24 9:25 ` zhanghailiang
38 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-24 9:25 UTC (permalink / raw)
To: qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, zhanghailiang, arei.gonglei, stefanha, amit.shah,
hongyang.yang
Make sure master start block replication after slave's block replication started.
Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Signed-off-by: Wen Congyang <wency@cn.fujitsu.com>
Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
---
migration/colo.c | 60 ++++++++++++++++++++++++++++++++++++++++++++++++++++++++
trace-events | 2 ++
2 files changed, 62 insertions(+)
diff --git a/migration/colo.c b/migration/colo.c
index b1b7905..c534ff9 100644
--- a/migration/colo.c
+++ b/migration/colo.c
@@ -20,6 +20,7 @@
#include "migration/failover.h"
#include "qapi-event.h"
#include "net/filter.h"
+#include "block/block_int.h"
/*
* The delay time before qemu begin the procedure of default failover treatment.
@@ -62,6 +63,7 @@ static void secondary_vm_do_failover(void)
{
int old_state;
MigrationIncomingState *mis = migration_incoming_get_current();
+ Error *local_err = NULL;
/* Can not do failover during the process of VM's loading VMstate, Or
* it will break the secondary VM.
@@ -79,6 +81,12 @@ static void secondary_vm_do_failover(void)
migrate_set_state(&mis->state, MIGRATION_STATUS_COLO,
MIGRATION_STATUS_COMPLETED);
+ bdrv_stop_replication_all(true, &local_err);
+ if (local_err) {
+ error_report_err(local_err);
+ }
+ trace_colo_stop_block_replication("failover");
+
if (!autostart) {
error_report("\"-S\" qemu option will be ignored in secondary side");
/* recover runstate to normal migration finish state */
@@ -110,6 +118,7 @@ static void primary_vm_do_failover(void)
{
MigrationState *s = migrate_get_current();
int old_state;
+ Error *local_err = NULL;
if (s->state != MIGRATION_STATUS_FAILED) {
migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
@@ -134,6 +143,12 @@ static void primary_vm_do_failover(void)
}
/* Don't buffer any packets while exited COLO */
qemu_set_default_filter_buffers(false);
+
+ bdrv_stop_replication_all(true, &local_err);
+ if (local_err) {
+ error_report_err(local_err);
+ }
+ trace_colo_stop_block_replication("failover");
}
void colo_do_failover(MigrationState *s)
@@ -212,6 +227,7 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
int colo_shutdown;
size_t size;
QEMUFile *trans = NULL;
+ Error *local_err = NULL;
ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_CHECKPOINT_REQUEST, 0);
if (ret < 0) {
@@ -250,6 +266,16 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
goto out;
}
+ /* we call this api although this may do nothing on primary side */
+ qemu_mutex_lock_iothread();
+ bdrv_do_checkpoint_all(&local_err);
+ qemu_mutex_unlock_iothread();
+ if (local_err) {
+ error_report_err(local_err);
+ ret = -1;
+ goto out;
+ }
+
ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
if (ret < 0) {
goto out;
@@ -296,6 +322,10 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
qemu_release_default_filters_packets();
if (colo_shutdown) {
+ qemu_mutex_lock_iothread();
+ bdrv_stop_replication_all(false, NULL);
+ trace_colo_stop_block_replication("shutdown");
+ qemu_mutex_unlock_iothread();
colo_ctl_put(s->to_dst_file, COLO_COMMAND_GUEST_SHUTDOWN, 0);
qemu_fflush(s->to_dst_file);
colo_shutdown_requested = 0;
@@ -341,6 +371,7 @@ static void colo_process_checkpoint(MigrationState *s)
int64_t error_time;
int ret = 0;
uint64_t value;
+ Error *local_err = NULL;
failover_init_state();
@@ -376,6 +407,15 @@ static void colo_process_checkpoint(MigrationState *s)
qemu_set_default_filter_buffers(true);
qemu_mutex_lock_iothread();
+ /* start block replication */
+ bdrv_start_replication_all(REPLICATION_MODE_PRIMARY, &local_err);
+ if (local_err) {
+ qemu_mutex_unlock_iothread();
+ error_report_err(local_err);
+ ret = -EINVAL;
+ goto out;
+ }
+ trace_colo_start_block_replication();
vm_start();
qemu_mutex_unlock_iothread();
trace_colo_vm_state_change("stop", "run");
@@ -492,6 +532,8 @@ static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
case COLO_COMMAND_GUEST_SHUTDOWN:
qemu_mutex_lock_iothread();
vm_stop_force_state(RUN_STATE_COLO);
+ bdrv_stop_replication_all(false, NULL);
+ trace_colo_stop_block_replication("shutdown");
qemu_system_shutdown_request_core();
qemu_mutex_unlock_iothread();
/* the main thread will exit and termiante the whole
@@ -524,6 +566,7 @@ void *colo_process_incoming_thread(void *opaque)
int64_t error_time, current_time;
int ret = 0;
uint64_t value;
+ Error *local_err = NULL;
migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
MIGRATION_STATUS_COLO);
@@ -560,6 +603,16 @@ void *colo_process_incoming_thread(void *opaque)
goto out;
}
+ qemu_mutex_lock_iothread();
+ /* start block replication */
+ bdrv_start_replication_all(REPLICATION_MODE_SECONDARY, &local_err);
+ qemu_mutex_unlock_iothread();
+ if (local_err) {
+ error_report_err(local_err);
+ goto out;
+ }
+ trace_colo_start_block_replication();
+
ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_READY, 0);
if (ret < 0) {
goto out;
@@ -639,6 +692,13 @@ void *colo_process_incoming_thread(void *opaque)
qemu_mutex_unlock_iothread();
goto out;
}
+ /* discard colo disk buffer */
+ bdrv_do_checkpoint_all(&local_err);
+ qemu_mutex_unlock_iothread();
+ if (local_err) {
+ vmstate_loading = false;
+ goto out;
+ }
vmstate_loading = false;
qemu_mutex_unlock_iothread();
diff --git a/trace-events b/trace-events
index b80c1e0..5f95b3c 100644
--- a/trace-events
+++ b/trace-events
@@ -1583,6 +1583,8 @@ colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
colo_ctl_put(const char *msg, uint64_t value) "Send '%s' cmd, value: %" PRIu64""
colo_ctl_get(const char *msg, uint64_t value) "Receive '%s' cmd, value: %" PRIu64""
colo_failover_set_state(int new_state) "new state %d"
+colo_start_block_replication(void) "Block replication is started"
+colo_stop_block_replication(const char *reason) "Block replication is stopped(reason: '%s')"
# kvm-all.c
kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
--
1.8.3.1
^ permalink raw reply related [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 04/39] migration: Export migrate_set_state()
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 04/39] migration: Export migrate_set_state() zhanghailiang
@ 2015-11-24 17:31 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-11-24 17:31 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> Fix the first parameter of migrate_set_state(), and export it.
> We will use it in later.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> v11:
> - New patch which is split from patch
> 'migration: Add state records for migration incoming' (Juan's suggestion)
> ---
> include/migration/migration.h | 2 ++
> migration/migration.c | 36 +++++++++++++++++++++---------------
> 2 files changed, 23 insertions(+), 15 deletions(-)
>
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index 1f004e4..4b19e80 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -169,6 +169,8 @@ struct MigrationState
> RAMBlock *last_req_rb;
> };
>
> +void migrate_set_state(int *state, int old_state, int new_state);
> +
> void process_incoming_migration(QEMUFile *f);
>
> void qemu_start_incoming_migration(const char *uri, Error **errp);
> diff --git a/migration/migration.c b/migration/migration.c
> index efacf52..7d98a3e 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -798,9 +798,9 @@ void qmp_migrate_start_postcopy(Error **errp)
>
> /* shared migration helpers */
>
> -static void migrate_set_state(MigrationState *s, int old_state, int new_state)
> +void migrate_set_state(int *state, int old_state, int new_state)
> {
> - if (atomic_cmpxchg(&s->state, old_state, new_state) == old_state) {
> + if (atomic_cmpxchg(state, old_state, new_state) == old_state) {
> trace_migrate_set_state(new_state);
> migrate_generate_event(new_state);
> }
> @@ -833,7 +833,7 @@ static void migrate_fd_cleanup(void *opaque)
> (s->state != MIGRATION_STATUS_POSTCOPY_ACTIVE));
>
> if (s->state == MIGRATION_STATUS_CANCELLING) {
> - migrate_set_state(s, MIGRATION_STATUS_CANCELLING,
> + migrate_set_state(&s->state, MIGRATION_STATUS_CANCELLING,
> MIGRATION_STATUS_CANCELLED);
> }
>
> @@ -844,7 +844,8 @@ void migrate_fd_error(MigrationState *s)
> {
> trace_migrate_fd_error();
> assert(s->file == NULL);
> - migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
> + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> + MIGRATION_STATUS_FAILED);
> notifier_list_notify(&migration_state_notifiers, s);
> }
>
> @@ -864,7 +865,7 @@ static void migrate_fd_cancel(MigrationState *s)
> if (!migration_is_setup_or_active(old_state)) {
> break;
> }
> - migrate_set_state(s, old_state, MIGRATION_STATUS_CANCELLING);
> + migrate_set_state(&s->state, old_state, MIGRATION_STATUS_CANCELLING);
> } while (s->state != MIGRATION_STATUS_CANCELLING);
>
> /*
> @@ -938,7 +939,7 @@ MigrationState *migrate_init(const MigrationParams *params)
> s->migration_thread_running = false;
> s->last_req_rb = NULL;
>
> - migrate_set_state(s, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
> + migrate_set_state(&s->state, MIGRATION_STATUS_NONE, MIGRATION_STATUS_SETUP);
>
> QSIMPLEQ_INIT(&s->src_page_requests);
>
> @@ -1037,7 +1038,8 @@ void qmp_migrate(const char *uri, bool has_blk, bool blk,
> } else {
> error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "uri",
> "a valid migration protocol");
> - migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_FAILED);
> + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> + MIGRATION_STATUS_FAILED);
> return;
> }
>
> @@ -1416,7 +1418,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> int ret;
> const QEMUSizedBuffer *qsb;
> int64_t time_at_stop = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
> - migrate_set_state(ms, MIGRATION_STATUS_ACTIVE,
> + migrate_set_state(&ms->state, MIGRATION_STATUS_ACTIVE,
> MIGRATION_STATUS_POSTCOPY_ACTIVE);
>
> trace_postcopy_start();
> @@ -1507,7 +1509,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> ret = qemu_file_get_error(ms->file);
> if (ret) {
> error_report("postcopy_start: Migration stream errored");
> - migrate_set_state(ms, MIGRATION_STATUS_POSTCOPY_ACTIVE,
> + migrate_set_state(&ms->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
> MIGRATION_STATUS_FAILED);
> }
>
> @@ -1516,7 +1518,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> fail_closefb:
> qemu_fclose(fb);
> fail:
> - migrate_set_state(ms, MIGRATION_STATUS_POSTCOPY_ACTIVE,
> + migrate_set_state(&ms->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
> MIGRATION_STATUS_FAILED);
> qemu_mutex_unlock_iothread();
> return -1;
> @@ -1585,11 +1587,13 @@ static void migration_completion(MigrationState *s, int current_active_state,
> goto fail;
> }
>
> - migrate_set_state(s, current_active_state, MIGRATION_STATUS_COMPLETED);
> + migrate_set_state(&s->state, current_active_state,
> + MIGRATION_STATUS_COMPLETED);
> return;
>
> fail:
> - migrate_set_state(s, current_active_state, MIGRATION_STATUS_FAILED);
> + migrate_set_state(&s->state, current_active_state,
> + MIGRATION_STATUS_FAILED);
> }
>
> bool migrate_colo_enabled(void)
> @@ -1640,7 +1644,8 @@ static void *migration_thread(void *opaque)
>
> s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
> current_active_state = MIGRATION_STATUS_ACTIVE;
> - migrate_set_state(s, MIGRATION_STATUS_SETUP, MIGRATION_STATUS_ACTIVE);
> + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> + MIGRATION_STATUS_ACTIVE);
>
> trace_migration_thread_setup_complete();
>
> @@ -1683,7 +1688,8 @@ static void *migration_thread(void *opaque)
> }
>
> if (qemu_file_get_error(s->file)) {
> - migrate_set_state(s, current_active_state, MIGRATION_STATUS_FAILED);
> + migrate_set_state(&s->state, current_active_state,
> + MIGRATION_STATUS_FAILED);
> trace_migration_thread_file_err();
> break;
> }
> @@ -1764,7 +1770,7 @@ void migrate_fd_connect(MigrationState *s)
> if (migrate_postcopy_ram()) {
> if (open_return_path_on_source(s)) {
> error_report("Unable to open return-path for postcopy");
> - migrate_set_state(s, MIGRATION_STATUS_SETUP,
> + migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> MIGRATION_STATUS_FAILED);
> migrate_fd_cleanup(s);
> return;
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 07/39] migration: Integrate COLO checkpoint process into loadvm
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 07/39] migration: Integrate COLO checkpoint process into loadvm zhanghailiang
@ 2015-11-24 18:14 ` Dr. David Alan Gilbert
2015-11-25 6:39 ` zhanghailiang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-11-24 18:14 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> Switch from normal migration loadvm process into COLO checkpoint process if
> COLO mode is enabled.
> We add three new members to struct MigrationIncomingState, 'have_colo_incoming_thread'
> and 'colo_incoming_thread' record the colo related threads for secondary VM,
> 'migration_incoming_co' records the original migration incoming coroutine.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Minor comment that needs fixing, see below, but otherwise:
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> v11:
> - We moved the place of bdrv_invalidate_cache_all(), but done the deleting work
> in other patch. Fix it.
> - Add documentation for colo in 'MigrationStatus' (Eric's review comment)
> v10:
> - fix a bug about fd leak which is found by Dave.
> ---
> include/migration/colo.h | 7 +++++++
> include/migration/migration.h | 7 +++++++
> migration/colo-comm.c | 10 ++++++++++
> migration/colo.c | 22 ++++++++++++++++++++++
> migration/migration.c | 31 +++++++++++++++++++++----------
> qapi-schema.json | 2 ++
> stubs/migration-colo.c | 10 ++++++++++
> 7 files changed, 79 insertions(+), 10 deletions(-)
>
> diff --git a/include/migration/colo.h b/include/migration/colo.h
> index f462f06..2676c4a 100644
> --- a/include/migration/colo.h
> +++ b/include/migration/colo.h
> @@ -15,6 +15,8 @@
>
> #include "qemu-common.h"
> #include "migration/migration.h"
> +#include "qemu/coroutine_int.h"
> +#include "qemu/thread.h"
>
> bool colo_supported(void);
> void colo_info_mig_init(void);
> @@ -22,4 +24,9 @@ void colo_info_mig_init(void);
> void migrate_start_colo_process(MigrationState *s);
> bool migration_in_colo_state(void);
>
> +/* loadvm */
> +bool migration_incoming_enable_colo(void);
> +void migration_incoming_exit_colo(void);
> +void *colo_process_incoming_thread(void *opaque);
> +bool migration_incoming_in_colo_state(void);
> #endif
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index 99dfa92..a57a734 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -22,6 +22,7 @@
> #include "migration/vmstate.h"
> #include "qapi-types.h"
> #include "exec/cpu-common.h"
> +#include "qemu/coroutine_int.h"
>
> #define QEMU_VM_FILE_MAGIC 0x5145564d
> #define QEMU_VM_FILE_VERSION_COMPAT 0x00000002
> @@ -106,6 +107,12 @@ struct MigrationIncomingState {
> void *postcopy_tmp_page;
>
> int state;
> +
> + bool have_colo_incoming_thread;
> + QemuThread colo_incoming_thread;
> + /* The coroutine we should enter (back) after failover */
> + Coroutine *migration_incoming_co;
> +
> /* See savevm.c */
> LoadStateEntry_Head loadvm_handlers;
> };
> diff --git a/migration/colo-comm.c b/migration/colo-comm.c
> index fb407e0..30df3d3 100644
> --- a/migration/colo-comm.c
> +++ b/migration/colo-comm.c
> @@ -48,3 +48,13 @@ void colo_info_mig_init(void)
> {
> vmstate_register(NULL, 0, &colo_state, &colo_info);
> }
> +
> +bool migration_incoming_enable_colo(void)
> +{
> + return colo_info.colo_requested;
> +}
> +
> +void migration_incoming_exit_colo(void)
> +{
> + colo_info.colo_requested = 0;
> +}
> diff --git a/migration/colo.c b/migration/colo.c
> index cf0ccb8..6880aa0 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -13,6 +13,7 @@
> #include "sysemu/sysemu.h"
> #include "migration/colo.h"
> #include "trace.h"
> +#include "qemu/error-report.h"
>
> bool colo_supported(void)
> {
> @@ -26,6 +27,13 @@ bool migration_in_colo_state(void)
> return (s->state == MIGRATION_STATUS_COLO);
> }
>
> +bool migration_incoming_in_colo_state(void)
> +{
> + MigrationIncomingState *mis = migration_incoming_get_current();
> +
> + return mis && (mis->state == MIGRATION_STATUS_COLO);
> +}
> +
> static void colo_process_checkpoint(MigrationState *s)
> {
> qemu_mutex_lock_iothread();
> @@ -47,3 +55,17 @@ void migrate_start_colo_process(MigrationState *s)
> colo_process_checkpoint(s);
> qemu_mutex_lock_iothread();
> }
> +
> +void *colo_process_incoming_thread(void *opaque)
> +{
> + MigrationIncomingState *mis = opaque;
> +
> + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> + MIGRATION_STATUS_COLO);
> +
> + /* TODO: COLO checkpoint restore loop */
> +
> + migration_incoming_exit_colo();
> +
> + return NULL;
> +}
> diff --git a/migration/migration.c b/migration/migration.c
> index 46fe8a9..41eac0d 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -358,6 +358,27 @@ static void process_incoming_migration_co(void *opaque)
> /* Else if something went wrong then just fall out of the normal exit */
> }
>
> + if (!ret) {
> + /* Make sure all file formats flush their mutable metadata */
> + bdrv_invalidate_cache_all(&local_err);
> + if (local_err) {
> + error_report_err(local_err);
> + migrate_decompress_threads_join();
> + exit(EXIT_FAILURE);
> + }
> + }
> + /* we get colo info, and know if we are in colo mode */
> + if (!ret && migration_incoming_enable_colo()) {
> + mis->migration_incoming_co = qemu_coroutine_self();
> + qemu_thread_create(&mis->colo_incoming_thread, "colo incoming",
> + colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE);
> + mis->have_colo_incoming_thread = true;
> + qemu_coroutine_yield();
> +
> + /* Wait checkpoint incoming thread exit before free resource */
> + qemu_thread_join(&mis->colo_incoming_thread);
> + }
> +
> qemu_fclose(f);
> free_xbzrle_decoded_buf();
> migration_incoming_state_destroy();
> @@ -370,16 +391,6 @@ static void process_incoming_migration_co(void *opaque)
> exit(EXIT_FAILURE);
> }
>
> - /* Make sure all file formats flush their mutable metadata */
> - bdrv_invalidate_cache_all(&local_err);
> - if (local_err) {
> - migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> - MIGRATION_STATUS_FAILED);
> - error_report_err(local_err);
> - migrate_decompress_threads_join();
> - exit(EXIT_FAILURE);
> - }
> -
> /*
> * This must happen after all error conditions are dealt with and
> * we're sure the VM is going to be running on this host.
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 24b35f3..c2f3b63 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -438,6 +438,8 @@
> #
> # @failed: some error occurred during migration process.
> #
> +# @colo: VM is in the process of fault tolerance. (since 2.6)
> +#
> # Since: 2.3
> #
> ##
That belongs in the previous patch I think; the previous patch
added before the @failed, but this change is correct, it just
needs to go with the actual addition.
Dave
> diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
> index acddca6..c12516e 100644
> --- a/stubs/migration-colo.c
> +++ b/stubs/migration-colo.c
> @@ -22,6 +22,16 @@ bool migration_in_colo_state(void)
> return false;
> }
>
> +bool migration_incoming_in_colo_state(void)
> +{
> + return false;
> +}
> +
> void migrate_start_colo_process(MigrationState *s)
> {
> }
> +
> +void *colo_process_incoming_thread(void *opaque)
> +{
> + return NULL;
> +}
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 08/39] migration: Rename the'file' member of MigrationState
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 08/39] migration: Rename the'file' member of MigrationState zhanghailiang
@ 2015-11-24 18:26 ` Dr. David Alan Gilbert
2015-11-25 6:48 ` zhanghailiang
2015-12-10 6:41 ` Wen Congyang
1 sibling, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-11-24 18:26 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> Rename the 'file' member of MigrationState to 'to_dst_file'.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> v11:
> - Only rename 'file' member of MigrationState
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
You should be able to post this one separately.
Dave
> ---
> include/migration/migration.h | 2 +-
> migration/exec.c | 4 +--
> migration/fd.c | 4 +--
> migration/migration.c | 72 ++++++++++++++++++++++---------------------
> migration/postcopy-ram.c | 6 ++--
> migration/savevm.c | 2 +-
> migration/tcp.c | 4 +--
> migration/unix.c | 4 +--
> 8 files changed, 51 insertions(+), 47 deletions(-)
>
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index a57a734..ba5bcec 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -140,7 +140,7 @@ struct MigrationState
> size_t xfer_limit;
> QemuThread thread;
> QEMUBH *cleanup_bh;
> - QEMUFile *file;
> + QEMUFile *to_dst_file;
> int parameters[MIGRATION_PARAMETER_MAX];
>
> int state;
> diff --git a/migration/exec.c b/migration/exec.c
> index 8406d2b..9037109 100644
> --- a/migration/exec.c
> +++ b/migration/exec.c
> @@ -36,8 +36,8 @@
>
> void exec_start_outgoing_migration(MigrationState *s, const char *command, Error **errp)
> {
> - s->file = qemu_popen_cmd(command, "w");
> - if (s->file == NULL) {
> + s->to_dst_file = qemu_popen_cmd(command, "w");
> + if (s->to_dst_file == NULL) {
> error_setg_errno(errp, errno, "failed to popen the migration target");
> return;
> }
> diff --git a/migration/fd.c b/migration/fd.c
> index 3e4bed0..9a9d6c5 100644
> --- a/migration/fd.c
> +++ b/migration/fd.c
> @@ -50,9 +50,9 @@ void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **
> }
>
> if (fd_is_socket(fd)) {
> - s->file = qemu_fopen_socket(fd, "wb");
> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
> } else {
> - s->file = qemu_fdopen(fd, "wb");
> + s->to_dst_file = qemu_fdopen(fd, "wb");
> }
>
> migrate_fd_connect(s);
> diff --git a/migration/migration.c b/migration/migration.c
> index 41eac0d..a4c690d 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -834,7 +834,7 @@ static void migrate_fd_cleanup(void *opaque)
>
> flush_page_queue(s);
>
> - if (s->file) {
> + if (s->to_dst_file) {
> trace_migrate_fd_cleanup();
> qemu_mutex_unlock_iothread();
> if (s->migration_thread_running) {
> @@ -844,8 +844,8 @@ static void migrate_fd_cleanup(void *opaque)
> qemu_mutex_lock_iothread();
>
> migrate_compress_threads_join();
> - qemu_fclose(s->file);
> - s->file = NULL;
> + qemu_fclose(s->to_dst_file);
> + s->to_dst_file = NULL;
> }
>
> assert((s->state != MIGRATION_STATUS_ACTIVE) &&
> @@ -862,7 +862,7 @@ static void migrate_fd_cleanup(void *opaque)
> void migrate_fd_error(MigrationState *s)
> {
> trace_migrate_fd_error();
> - assert(s->file == NULL);
> + assert(s->to_dst_file == NULL);
> migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> MIGRATION_STATUS_FAILED);
> notifier_list_notify(&migration_state_notifiers, s);
> @@ -871,7 +871,7 @@ void migrate_fd_error(MigrationState *s)
> static void migrate_fd_cancel(MigrationState *s)
> {
> int old_state ;
> - QEMUFile *f = migrate_get_current()->file;
> + QEMUFile *f = migrate_get_current()->to_dst_file;
> trace_migrate_fd_cancel();
>
> if (s->rp_state.from_dst_file) {
> @@ -942,7 +942,7 @@ MigrationState *migrate_init(const MigrationParams *params)
> s->bytes_xfer = 0;
> s->xfer_limit = 0;
> s->cleanup_bh = 0;
> - s->file = NULL;
> + s->to_dst_file = NULL;
> s->state = MIGRATION_STATUS_NONE;
> s->params = *params;
> s->rp_state.from_dst_file = NULL;
> @@ -1122,8 +1122,9 @@ void qmp_migrate_set_speed(int64_t value, Error **errp)
>
> s = migrate_get_current();
> s->bandwidth_limit = value;
> - if (s->file) {
> - qemu_file_set_rate_limit(s->file, s->bandwidth_limit / XFER_LIMIT_RATIO);
> + if (s->to_dst_file) {
> + qemu_file_set_rate_limit(s->to_dst_file,
> + s->bandwidth_limit / XFER_LIMIT_RATIO);
> }
> }
>
> @@ -1393,7 +1394,7 @@ out:
> static int open_return_path_on_source(MigrationState *ms)
> {
>
> - ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->file);
> + ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->to_dst_file);
> if (!ms->rp_state.from_dst_file) {
> return -1;
> }
> @@ -1415,7 +1416,7 @@ static int await_return_path_close_on_source(MigrationState *ms)
> * rp_thread will exit, however if there's an error we need to cause
> * it to exit.
> */
> - if (qemu_file_get_error(ms->file) && ms->rp_state.from_dst_file) {
> + if (qemu_file_get_error(ms->to_dst_file) && ms->rp_state.from_dst_file) {
> /*
> * shutdown(2), if we have it, will cause it to unblock if it's stuck
> * waiting for the destination.
> @@ -1458,7 +1459,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> * Cause any non-postcopiable, but iterative devices to
> * send out their final data.
> */
> - qemu_savevm_state_complete_precopy(ms->file, true);
> + qemu_savevm_state_complete_precopy(ms->to_dst_file, true);
>
> /*
> * in Finish migrate and with the io-lock held everything should
> @@ -1476,9 +1477,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> * will notice we're in POSTCOPY_ACTIVE and not actually
> * wrap their state up here
> */
> - qemu_file_set_rate_limit(ms->file, INT64_MAX);
> + qemu_file_set_rate_limit(ms->to_dst_file, INT64_MAX);
> /* Ping just for debugging, helps line traces up */
> - qemu_savevm_send_ping(ms->file, 2);
> + qemu_savevm_send_ping(ms->to_dst_file, 2);
>
> /*
> * While loading the device state we may trigger page transfer
> @@ -1512,7 +1513,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> qsb = qemu_buf_get(fb);
>
> /* Now send that blob */
> - if (qemu_savevm_send_packaged(ms->file, qsb)) {
> + if (qemu_savevm_send_packaged(ms->to_dst_file, qsb)) {
> goto fail_closefb;
> }
> qemu_fclose(fb);
> @@ -1524,9 +1525,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> * Although this ping is just for debug, it could potentially be
> * used for getting a better measurement of downtime at the source.
> */
> - qemu_savevm_send_ping(ms->file, 4);
> + qemu_savevm_send_ping(ms->to_dst_file, 4);
>
> - ret = qemu_file_get_error(ms->file);
> + ret = qemu_file_get_error(ms->to_dst_file);
> if (ret) {
> error_report("postcopy_start: Migration stream errored");
> migrate_set_state(&ms->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
> @@ -1569,8 +1570,8 @@ static void migration_completion(MigrationState *s, int current_active_state,
> if (!ret) {
> ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
> if (ret >= 0) {
> - qemu_file_set_rate_limit(s->file, INT64_MAX);
> - qemu_savevm_state_complete_precopy(s->file, false);
> + qemu_file_set_rate_limit(s->to_dst_file, INT64_MAX);
> + qemu_savevm_state_complete_precopy(s->to_dst_file, false);
> }
> }
> qemu_mutex_unlock_iothread();
> @@ -1581,7 +1582,7 @@ static void migration_completion(MigrationState *s, int current_active_state,
> } else if (s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) {
> trace_migration_completion_postcopy_end();
>
> - qemu_savevm_state_complete_postcopy(s->file);
> + qemu_savevm_state_complete_postcopy(s->to_dst_file);
> trace_migration_completion_postcopy_end_after_complete();
> }
>
> @@ -1602,7 +1603,7 @@ static void migration_completion(MigrationState *s, int current_active_state,
> }
> }
>
> - if (qemu_file_get_error(s->file)) {
> + if (qemu_file_get_error(s->to_dst_file)) {
> trace_migration_completion_file_err();
> goto fail;
> }
> @@ -1647,24 +1648,24 @@ static void *migration_thread(void *opaque)
>
> rcu_register_thread();
>
> - qemu_savevm_state_header(s->file);
> + qemu_savevm_state_header(s->to_dst_file);
>
> if (migrate_postcopy_ram()) {
> /* Now tell the dest that it should open its end so it can reply */
> - qemu_savevm_send_open_return_path(s->file);
> + qemu_savevm_send_open_return_path(s->to_dst_file);
>
> /* And do a ping that will make stuff easier to debug */
> - qemu_savevm_send_ping(s->file, 1);
> + qemu_savevm_send_ping(s->to_dst_file, 1);
>
> /*
> * Tell the destination that we *might* want to do postcopy later;
> * if the other end can't do postcopy it should fail now, nice and
> * early.
> */
> - qemu_savevm_send_postcopy_advise(s->file);
> + qemu_savevm_send_postcopy_advise(s->to_dst_file);
> }
>
> - qemu_savevm_state_begin(s->file, &s->params);
> + qemu_savevm_state_begin(s->to_dst_file, &s->params);
>
> s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
> current_active_state = MIGRATION_STATUS_ACTIVE;
> @@ -1678,10 +1679,10 @@ static void *migration_thread(void *opaque)
> int64_t current_time;
> uint64_t pending_size;
>
> - if (!qemu_file_rate_limit(s->file)) {
> + if (!qemu_file_rate_limit(s->to_dst_file)) {
> uint64_t pend_post, pend_nonpost;
>
> - qemu_savevm_state_pending(s->file, max_size, &pend_nonpost,
> + qemu_savevm_state_pending(s->to_dst_file, max_size, &pend_nonpost,
> &pend_post);
> pending_size = pend_nonpost + pend_post;
> trace_migrate_pending(pending_size, max_size,
> @@ -1702,7 +1703,7 @@ static void *migration_thread(void *opaque)
> continue;
> }
> /* Just another iteration step */
> - qemu_savevm_state_iterate(s->file, entered_postcopy);
> + qemu_savevm_state_iterate(s->to_dst_file, entered_postcopy);
> } else {
> trace_migration_thread_low_pending(pending_size);
> migration_completion(s, current_active_state,
> @@ -1711,7 +1712,7 @@ static void *migration_thread(void *opaque)
> }
> }
>
> - if (qemu_file_get_error(s->file)) {
> + if (qemu_file_get_error(s->to_dst_file)) {
> migrate_set_state(&s->state, current_active_state,
> MIGRATION_STATUS_FAILED);
> trace_migration_thread_file_err();
> @@ -1719,7 +1720,8 @@ static void *migration_thread(void *opaque)
> }
> current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
> if (current_time >= initial_time + BUFFER_DELAY) {
> - uint64_t transferred_bytes = qemu_ftell(s->file) - initial_bytes;
> + uint64_t transferred_bytes = qemu_ftell(s->to_dst_file) -
> + initial_bytes;
> uint64_t time_spent = current_time - initial_time;
> double bandwidth = transferred_bytes / time_spent;
> max_size = bandwidth * migrate_max_downtime() / 1000000;
> @@ -1735,11 +1737,11 @@ static void *migration_thread(void *opaque)
> s->expected_downtime = s->dirty_bytes_rate / bandwidth;
> }
>
> - qemu_file_reset_rate_limit(s->file);
> + qemu_file_reset_rate_limit(s->to_dst_file);
> initial_time = current_time;
> - initial_bytes = qemu_ftell(s->file);
> + initial_bytes = qemu_ftell(s->to_dst_file);
> }
> - if (qemu_file_rate_limit(s->file)) {
> + if (qemu_file_rate_limit(s->to_dst_file)) {
> /* usleep expects microseconds */
> g_usleep((initial_time + BUFFER_DELAY - current_time)*1000);
> }
> @@ -1757,7 +1759,7 @@ static void *migration_thread(void *opaque)
> qemu_savevm_state_cleanup();
> }
> if (s->state == MIGRATION_STATUS_COMPLETED) {
> - uint64_t transferred_bytes = qemu_ftell(s->file);
> + uint64_t transferred_bytes = qemu_ftell(s->to_dst_file);
> s->total_time = end_time - s->total_time;
> if (!entered_postcopy) {
> s->downtime = end_time - start_time;
> @@ -1794,7 +1796,7 @@ void migrate_fd_connect(MigrationState *s)
> s->expected_downtime = max_downtime/1000000;
> s->cleanup_bh = qemu_bh_new(migrate_fd_cleanup, s);
>
> - qemu_file_set_rate_limit(s->file,
> + qemu_file_set_rate_limit(s->to_dst_file,
> s->bandwidth_limit / XFER_LIMIT_RATIO);
>
> /* Notify before starting migration thread */
> diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
> index 22d6b18..f74016d 100644
> --- a/migration/postcopy-ram.c
> +++ b/migration/postcopy-ram.c
> @@ -733,7 +733,8 @@ void postcopy_discard_send_range(MigrationState *ms, PostcopyDiscardState *pds,
>
> if (pds->cur_entry == MAX_DISCARDS_PER_COMMAND) {
> /* Full set, ship it! */
> - qemu_savevm_send_postcopy_ram_discard(ms->file, pds->ramblock_name,
> + qemu_savevm_send_postcopy_ram_discard(ms->to_dst_file,
> + pds->ramblock_name,
> pds->cur_entry,
> pds->start_list,
> pds->length_list);
> @@ -753,7 +754,8 @@ void postcopy_discard_send_finish(MigrationState *ms, PostcopyDiscardState *pds)
> {
> /* Anything unsent? */
> if (pds->cur_entry) {
> - qemu_savevm_send_postcopy_ram_discard(ms->file, pds->ramblock_name,
> + qemu_savevm_send_postcopy_ram_discard(ms->to_dst_file,
> + pds->ramblock_name,
> pds->cur_entry,
> pds->start_list,
> pds->length_list);
> diff --git a/migration/savevm.c b/migration/savevm.c
> index 0ad1b93..f102870 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -1163,7 +1163,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
> .shared = 0
> };
> MigrationState *ms = migrate_init(¶ms);
> - ms->file = f;
> + ms->to_dst_file = f;
>
> if (qemu_savevm_state_blocked(errp)) {
> return -EINVAL;
> diff --git a/migration/tcp.c b/migration/tcp.c
> index ae89172..e083d68 100644
> --- a/migration/tcp.c
> +++ b/migration/tcp.c
> @@ -39,11 +39,11 @@ static void tcp_wait_for_connect(int fd, Error *err, void *opaque)
>
> if (fd < 0) {
> DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
> - s->file = NULL;
> + s->to_dst_file = NULL;
> migrate_fd_error(s);
> } else {
> DPRINTF("migrate connect success\n");
> - s->file = qemu_fopen_socket(fd, "wb");
> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
> migrate_fd_connect(s);
> }
> }
> diff --git a/migration/unix.c b/migration/unix.c
> index b591813..5492dd6 100644
> --- a/migration/unix.c
> +++ b/migration/unix.c
> @@ -39,11 +39,11 @@ static void unix_wait_for_connect(int fd, Error *err, void *opaque)
>
> if (fd < 0) {
> DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
> - s->file = NULL;
> + s->to_dst_file = NULL;
> migrate_fd_error(s);
> } else {
> DPRINTF("migrate connect success\n");
> - s->file = qemu_fopen_socket(fd, "wb");
> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
> migrate_fd_connect(s);
> }
> }
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 09/39] COLO/migration: Create a new communication path from destination to source
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 09/39] COLO/migration: Create a new communication path from destination to source zhanghailiang
@ 2015-11-24 18:40 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-11-24 18:40 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> This new communication path will be used for returning messages
> from destination to source.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> v11:
> - Rebase master to use qemu_file_get_return_path() for opening return path
> v10:
> - fix the the error log (Dave's suggestion).
> ---
> migration/colo.c | 39 +++++++++++++++++++++++++++++++++++++++
> 1 file changed, 39 insertions(+)
>
> diff --git a/migration/colo.c b/migration/colo.c
> index 6880aa0..0ab9618 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -36,6 +36,15 @@ bool migration_incoming_in_colo_state(void)
>
> static void colo_process_checkpoint(MigrationState *s)
> {
> + int ret = 0;
> +
> + s->rp_state.from_dst_file = qemu_file_get_return_path(s->to_dst_file);
> + if (!s->rp_state.from_dst_file) {
> + ret = -EINVAL;
> + error_report("Open QEMUFile from_dst_file failed");
> + goto out;
> + }
> +
> qemu_mutex_lock_iothread();
> vm_start();
> qemu_mutex_unlock_iothread();
> @@ -43,8 +52,16 @@ static void colo_process_checkpoint(MigrationState *s)
>
> /*TODO: COLO checkpoint savevm loop*/
>
> +out:
> + if (ret < 0) {
> + error_report("%s: %s", __func__, strerror(-ret));
> + }
> migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
> MIGRATION_STATUS_COMPLETED);
> +
> + if (s->rp_state.from_dst_file) {
> + qemu_fclose(s->rp_state.from_dst_file);
> + }
> }
>
> void migrate_start_colo_process(MigrationState *s)
> @@ -59,12 +76,34 @@ void migrate_start_colo_process(MigrationState *s)
> void *colo_process_incoming_thread(void *opaque)
> {
> MigrationIncomingState *mis = opaque;
> + int ret = 0;
>
> migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> MIGRATION_STATUS_COLO);
>
> + mis->to_src_file = qemu_file_get_return_path(mis->from_src_file);
> + if (!mis->to_src_file) {
> + ret = -EINVAL;
> + error_report("colo incoming thread: Open QEMUFile to_src_file failed");
> + goto out;
> + }
> + /* Note: We set the fd to unblocked in migration incoming coroutine,
> + * But here we are in the colo incoming thread, so it is ok to set the
> + * fd back to blocked.
> + */
> + qemu_set_block(qemu_get_fd(mis->from_src_file));
> +
> /* TODO: COLO checkpoint restore loop */
>
> +out:
> + if (ret < 0) {
> + error_report("colo incoming thread will exit, detect error: %s",
> + strerror(-ret));
> + }
> +
> + if (mis->to_src_file) {
> + qemu_fclose(mis->to_src_file);
> + }
> migration_incoming_exit_colo();
>
> return NULL;
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 10/39] COLO: Implement colo checkpoint protocol
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 10/39] COLO: Implement colo checkpoint protocol zhanghailiang
@ 2015-11-24 19:00 ` Dr. David Alan Gilbert
2015-11-25 14:01 ` Eric Blake
2015-11-26 7:12 ` Hailiang Zhang
0 siblings, 2 replies; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-11-24 19:00 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> We need communications protocol of user-defined to control the checkpoint
> process.
>
> The new checkpoint request is started by Primary VM, and the interactive process
> like below:
> Checkpoint synchronizing points,
>
> Primary Secondary
> initial work
> 'checkpoint-ready' <------------------------------ @
>
> 'checkpoint-request' @ ----------------------------->
> Suspend (Only in hybrid mode)
> 'checkpoint-reply' <------------------------------ @
> Suspend&Save state
> 'vmstate-send' @ ----------------------------->
> Send state Receive state
> 'vmstate-received' <------------------------------ @
> Release packets Load state
> 'vmstate-load' <------------------------------ @
> Resume Resume (Only in hybrid mode)
>
> Start Comparing (Only in hybrid mode)
> NOTE:
> 1) '@' who sends the message
> 2) Every sync-point is synchronized by two sides with only
> one handshake(single direction) for low-latency.
> If more strict synchronization is required, a opposite direction
> sync-point should be added.
> 3) Since sync-points are single direction, the remote side may
> go forward a lot when this side just receives the sync-point.
> 4) For now, we only support 'periodic' checkpoint, for which
> the Secondary VM is not running, later we will support 'hybrid' mode.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> Cc: Eric Blake <eblake@redhat.com>
> ---
> v11:
> - Add missing 'checkpoint-ready' communication in comment.
> - Use parameter to return 'value' for colo_ctl_get() (Dave's suggestion)
> - Fix trace for colo_ctl_get() to trace command and value both
> v10:
> - Rename enum COLOCmd to COLOCommand (Eric's suggestion).
> - Remove unused 'ram-steal'
> ---
> migration/colo.c | 198 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
> qapi-schema.json | 27 ++++++++
> trace-events | 2 +
> 3 files changed, 225 insertions(+), 2 deletions(-)
>
> diff --git a/migration/colo.c b/migration/colo.c
> index 0ab9618..c045d61 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -10,10 +10,12 @@
> * later. See the COPYING file in the top-level directory.
> */
>
> +#include <unistd.h>
> #include "sysemu/sysemu.h"
> #include "migration/colo.h"
> #include "trace.h"
> #include "qemu/error-report.h"
> +#include "qemu/sockets.h"
>
> bool colo_supported(void)
> {
> @@ -34,9 +36,107 @@ bool migration_incoming_in_colo_state(void)
> return mis && (mis->state == MIGRATION_STATUS_COLO);
> }
>
> +/* colo checkpoint control helper */
> +static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value)
> +{
> + int ret = 0;
> +
> + qemu_put_be32(f, cmd);
> + qemu_put_be64(f, value);
> + qemu_fflush(f);
> +
> + ret = qemu_file_get_error(f);
> + trace_colo_ctl_put(COLOCommand_lookup[cmd], value);
> +
> + return ret;
> +}
> +
> +static int colo_ctl_get_cmd(QEMUFile *f, uint32_t *cmd)
> +{
> + int ret = 0;
> +
> + *cmd = qemu_get_be32(f);
> + ret = qemu_file_get_error(f);
> + if (ret < 0) {
> + return ret;
> + }
> + if (*cmd >= COLO_COMMAND_MAX) {
> + error_report("Invalid colo command, got cmd:%d", *cmd);
> + return -EINVAL;
> + }
> +
> + return 0;
> +}
> +
> +static int colo_ctl_get(QEMUFile *f, uint32_t require, uint64_t *value)
> +{
> + int ret;
> + uint32_t cmd;
> +
> + ret = colo_ctl_get_cmd(f, &cmd);
> + if (ret < 0) {
> + return ret;
> + }
> + if (cmd != require) {
> + error_report("Unexpect colo command, expect:%d, but got cmd:%d",
> + require, cmd);
I think you need to use PRIu32 rather than %d since they are uint32_t
(I doubt it will break on anything, but it's correct).
> + return -EINVAL;
> + }
> +
> + *value = qemu_get_be64(f);
> + trace_colo_ctl_get(COLOCommand_lookup[cmd], *value);
> + ret = qemu_file_get_error(f);
> +
> + return ret;
> +}
> +
> +static int colo_do_checkpoint_transaction(MigrationState *s)
> +{
> + int ret;
> + uint64_t value;
> +
> + ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_CHECKPOINT_REQUEST, 0);
> + if (ret < 0) {
> + goto out;
> + }
> +
> + ret = colo_ctl_get(s->rp_state.from_dst_file,
> + COLO_COMMAND_CHECKPOINT_REPLY, &value);
> + if (ret < 0) {
> + goto out;
> + }
> +
> + /* TODO: suspend and save vm state to colo buffer */
> +
> + ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
> + if (ret < 0) {
> + goto out;
> + }
> +
> + /* TODO: send vmstate to Secondary */
> +
> + ret = colo_ctl_get(s->rp_state.from_dst_file,
> + COLO_COMMAND_VMSTATE_RECEIVED, &value);
> + if (ret < 0) {
> + goto out;
> + }
> +
> + ret = colo_ctl_get(s->rp_state.from_dst_file,
> + COLO_COMMAND_VMSTATE_LOADED, &value);
> + if (ret < 0) {
> + goto out;
> + }
> +
> + /* TODO: resume Primary */
> +
> +out:
> + return ret;
> +}
> +
> static void colo_process_checkpoint(MigrationState *s)
> {
> int ret = 0;
> + uint64_t value;
>
> s->rp_state.from_dst_file = qemu_file_get_return_path(s->to_dst_file);
> if (!s->rp_state.from_dst_file) {
> @@ -45,12 +145,28 @@ static void colo_process_checkpoint(MigrationState *s)
> goto out;
> }
>
> + /*
> + * Wait for Secondary finish loading vm states and enter COLO
> + * restore.
> + */
> + ret = colo_ctl_get(s->rp_state.from_dst_file,
> + COLO_COMMAND_CHECKPOINT_READY, &value);
> + if (ret < 0) {
> + goto out;
> + }
> +
> qemu_mutex_lock_iothread();
> vm_start();
> qemu_mutex_unlock_iothread();
> trace_colo_vm_state_change("stop", "run");
>
> - /*TODO: COLO checkpoint savevm loop*/
> + while (s->state == MIGRATION_STATUS_COLO) {
> + /* start a colo checkpoint */
> + ret = colo_do_checkpoint_transaction(s);
> + if (ret < 0) {
> + goto out;
> + }
> + }
>
> out:
> if (ret < 0) {
> @@ -73,10 +189,46 @@ void migrate_start_colo_process(MigrationState *s)
> qemu_mutex_lock_iothread();
> }
>
> +/*
> + * return:
> + * 0: start a checkpoint
> + * -1: some error happened, exit colo restore
> + */
> +static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
> +{
> + int ret;
> + uint32_t cmd;
> + uint64_t value;
> +
> + ret = colo_ctl_get_cmd(f, &cmd);
> + if (ret < 0) {
> + /* do failover ? */
> + return ret;
> + }
> + /* Fix me: this value should be 0, which is not so good,
> + * should be used for checking ?
> + */
You could use something simple like an incrementing value, if you just
wanted to make it a little more robust against problems.
> + value = qemu_get_be64(f);
> + if (value != 0) {
> + error_report("Got unexpected value %" PRIu64 " for '%s' command",
> + value, COLOCommand_lookup[cmd]);
> + return -EINVAL;
> + }
> +
> + switch (cmd) {
> + case COLO_COMMAND_CHECKPOINT_REQUEST:
> + *checkpoint_request = 1;
> + return 0;
> + default:
> + return -EINVAL;
> + }
> +}
> +
> void *colo_process_incoming_thread(void *opaque)
> {
> MigrationIncomingState *mis = opaque;
> int ret = 0;
> + uint64_t value;
>
> migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
> MIGRATION_STATUS_COLO);
> @@ -93,7 +245,49 @@ void *colo_process_incoming_thread(void *opaque)
> */
> qemu_set_block(qemu_get_fd(mis->from_src_file));
>
> - /* TODO: COLO checkpoint restore loop */
> +
> + ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_READY, 0);
> + if (ret < 0) {
> + goto out;
> + }
> +
> + while (mis->state == MIGRATION_STATUS_COLO) {
> + int request = 0;
> + int ret = colo_wait_handle_cmd(mis->from_src_file, &request);
> +
> + if (ret < 0) {
> + break;
> + } else {
> + if (!request) {
> + continue;
> + }
> + }
> + /* FIXME: This is unnecessary for periodic checkpoint mode */
> + ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_REPLY, 0);
> + if (ret < 0) {
> + goto out;
> + }
> +
> + ret = colo_ctl_get(mis->from_src_file, COLO_COMMAND_VMSTATE_SEND,
> + &value);
> + if (ret < 0) {
> + goto out;
> + }
> +
> + /* TODO: read migration data into colo buffer */
> +
> + ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_RECEIVED, 0);
> + if (ret < 0) {
> + goto out;
> + }
> +
> + /* TODO: load vm state */
> +
> + ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_LOADED, 0);
> + if (ret < 0) {
> + goto out;
> + }
> + }
>
> out:
> if (ret < 0) {
> diff --git a/qapi-schema.json b/qapi-schema.json
> index c2f3b63..48a87af 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -722,6 +722,33 @@
> { 'command': 'migrate-start-postcopy' }
>
> ##
> +# @COLOCommand
> +#
> +# The commands for COLO fault tolerance
> +#
> +# @invalid: unknown command
> +#
> +# @checkpoint-ready: SVM is ready for checkpointing
> +#
> +# @checkpoint-request: PVM tells SVM to prepare for new checkpointing
> +#
> +# @checkpoint-reply: SVM gets PVM's checkpoint request
> +#
> +# @vmstate-send: VM's state will be sent by PVM.
> +#
> +# @vmstate-size: The total size of VMstate.
> +#
> +# @vmstate-received: VM's state has been received by SVM
> +#
> +# @vmstate-loaded: VM's state has been loaded by SVM
> +#
> +# Since: 2.6
> +##
> +{ 'enum': 'COLOCommand',
> + 'data': [ 'invalid', 'checkpoint-ready', 'checkpoint-request',
> + 'checkpoint-reply', 'vmstate-send', 'vmstate-size',
> + 'vmstate-received', 'vmstate-loaded' ] }
> +
> # @MouseInfo:
> #
> # Information about a mouse device.
> diff --git a/trace-events b/trace-events
> index c98d473..f8a0959 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -1579,6 +1579,8 @@ postcopy_ram_incoming_cleanup_join(void) ""
>
> # migration/colo.c
> colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
> +colo_ctl_put(const char *msg, uint64_t value) "Send '%s' cmd, value: %" PRIu64""
> +colo_ctl_get(const char *msg, uint64_t value) "Receive '%s' cmd, value: %" PRIu64""
I don't think you need the "" at the end of the line
Dave
>
> # kvm-all.c
> kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 07/39] migration: Integrate COLO checkpoint process into loadvm
2015-11-24 18:14 ` Dr. David Alan Gilbert
@ 2015-11-25 6:39 ` zhanghailiang
0 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-25 6:39 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/11/25 2:14, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> Switch from normal migration loadvm process into COLO checkpoint process if
>> COLO mode is enabled.
>> We add three new members to struct MigrationIncomingState, 'have_colo_incoming_thread'
>> and 'colo_incoming_thread' record the colo related threads for secondary VM,
>> 'migration_incoming_co' records the original migration incoming coroutine.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>
> Minor comment that needs fixing, see below, but otherwise:
>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
>> ---
>> v11:
>> - We moved the place of bdrv_invalidate_cache_all(), but done the deleting work
>> in other patch. Fix it.
>> - Add documentation for colo in 'MigrationStatus' (Eric's review comment)
>> v10:
>> - fix a bug about fd leak which is found by Dave.
>> ---
>> include/migration/colo.h | 7 +++++++
>> include/migration/migration.h | 7 +++++++
>> migration/colo-comm.c | 10 ++++++++++
>> migration/colo.c | 22 ++++++++++++++++++++++
>> migration/migration.c | 31 +++++++++++++++++++++----------
>> qapi-schema.json | 2 ++
>> stubs/migration-colo.c | 10 ++++++++++
>> 7 files changed, 79 insertions(+), 10 deletions(-)
>>
>> diff --git a/include/migration/colo.h b/include/migration/colo.h
>> index f462f06..2676c4a 100644
>> --- a/include/migration/colo.h
>> +++ b/include/migration/colo.h
>> @@ -15,6 +15,8 @@
>>
>> #include "qemu-common.h"
>> #include "migration/migration.h"
>> +#include "qemu/coroutine_int.h"
>> +#include "qemu/thread.h"
>>
>> bool colo_supported(void);
>> void colo_info_mig_init(void);
>> @@ -22,4 +24,9 @@ void colo_info_mig_init(void);
>> void migrate_start_colo_process(MigrationState *s);
>> bool migration_in_colo_state(void);
>>
>> +/* loadvm */
>> +bool migration_incoming_enable_colo(void);
>> +void migration_incoming_exit_colo(void);
>> +void *colo_process_incoming_thread(void *opaque);
>> +bool migration_incoming_in_colo_state(void);
>> #endif
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index 99dfa92..a57a734 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -22,6 +22,7 @@
>> #include "migration/vmstate.h"
>> #include "qapi-types.h"
>> #include "exec/cpu-common.h"
>> +#include "qemu/coroutine_int.h"
>>
>> #define QEMU_VM_FILE_MAGIC 0x5145564d
>> #define QEMU_VM_FILE_VERSION_COMPAT 0x00000002
>> @@ -106,6 +107,12 @@ struct MigrationIncomingState {
>> void *postcopy_tmp_page;
>>
>> int state;
>> +
>> + bool have_colo_incoming_thread;
>> + QemuThread colo_incoming_thread;
>> + /* The coroutine we should enter (back) after failover */
>> + Coroutine *migration_incoming_co;
>> +
>> /* See savevm.c */
>> LoadStateEntry_Head loadvm_handlers;
>> };
>> diff --git a/migration/colo-comm.c b/migration/colo-comm.c
>> index fb407e0..30df3d3 100644
>> --- a/migration/colo-comm.c
>> +++ b/migration/colo-comm.c
>> @@ -48,3 +48,13 @@ void colo_info_mig_init(void)
>> {
>> vmstate_register(NULL, 0, &colo_state, &colo_info);
>> }
>> +
>> +bool migration_incoming_enable_colo(void)
>> +{
>> + return colo_info.colo_requested;
>> +}
>> +
>> +void migration_incoming_exit_colo(void)
>> +{
>> + colo_info.colo_requested = 0;
>> +}
>> diff --git a/migration/colo.c b/migration/colo.c
>> index cf0ccb8..6880aa0 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -13,6 +13,7 @@
>> #include "sysemu/sysemu.h"
>> #include "migration/colo.h"
>> #include "trace.h"
>> +#include "qemu/error-report.h"
>>
>> bool colo_supported(void)
>> {
>> @@ -26,6 +27,13 @@ bool migration_in_colo_state(void)
>> return (s->state == MIGRATION_STATUS_COLO);
>> }
>>
>> +bool migration_incoming_in_colo_state(void)
>> +{
>> + MigrationIncomingState *mis = migration_incoming_get_current();
>> +
>> + return mis && (mis->state == MIGRATION_STATUS_COLO);
>> +}
>> +
>> static void colo_process_checkpoint(MigrationState *s)
>> {
>> qemu_mutex_lock_iothread();
>> @@ -47,3 +55,17 @@ void migrate_start_colo_process(MigrationState *s)
>> colo_process_checkpoint(s);
>> qemu_mutex_lock_iothread();
>> }
>> +
>> +void *colo_process_incoming_thread(void *opaque)
>> +{
>> + MigrationIncomingState *mis = opaque;
>> +
>> + migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>> + MIGRATION_STATUS_COLO);
>> +
>> + /* TODO: COLO checkpoint restore loop */
>> +
>> + migration_incoming_exit_colo();
>> +
>> + return NULL;
>> +}
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 46fe8a9..41eac0d 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -358,6 +358,27 @@ static void process_incoming_migration_co(void *opaque)
>> /* Else if something went wrong then just fall out of the normal exit */
>> }
>>
>> + if (!ret) {
>> + /* Make sure all file formats flush their mutable metadata */
>> + bdrv_invalidate_cache_all(&local_err);
>> + if (local_err) {
>> + error_report_err(local_err);
>> + migrate_decompress_threads_join();
>> + exit(EXIT_FAILURE);
>> + }
>> + }
>> + /* we get colo info, and know if we are in colo mode */
>> + if (!ret && migration_incoming_enable_colo()) {
>> + mis->migration_incoming_co = qemu_coroutine_self();
>> + qemu_thread_create(&mis->colo_incoming_thread, "colo incoming",
>> + colo_process_incoming_thread, mis, QEMU_THREAD_JOINABLE);
>> + mis->have_colo_incoming_thread = true;
>> + qemu_coroutine_yield();
>> +
>> + /* Wait checkpoint incoming thread exit before free resource */
>> + qemu_thread_join(&mis->colo_incoming_thread);
>> + }
>> +
>> qemu_fclose(f);
>> free_xbzrle_decoded_buf();
>> migration_incoming_state_destroy();
>> @@ -370,16 +391,6 @@ static void process_incoming_migration_co(void *opaque)
>> exit(EXIT_FAILURE);
>> }
>>
>> - /* Make sure all file formats flush their mutable metadata */
>> - bdrv_invalidate_cache_all(&local_err);
>> - if (local_err) {
>> - migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>> - MIGRATION_STATUS_FAILED);
>> - error_report_err(local_err);
>> - migrate_decompress_threads_join();
>> - exit(EXIT_FAILURE);
>> - }
>> -
>> /*
>> * This must happen after all error conditions are dealt with and
>> * we're sure the VM is going to be running on this host.
>> diff --git a/qapi-schema.json b/qapi-schema.json
>> index 24b35f3..c2f3b63 100644
>> --- a/qapi-schema.json
>> +++ b/qapi-schema.json
>> @@ -438,6 +438,8 @@
>> #
>> # @failed: some error occurred during migration process.
>> #
>> +# @colo: VM is in the process of fault tolerance. (since 2.6)
>> +#
>> # Since: 2.3
>> #
>> ##
>
> That belongs in the previous patch I think; the previous patch
> added before the @failed, but this change is correct, it just
> needs to go with the actual addition.
>
Good catch, i will fix it in next version.
Thanks.
zhanghailiang
>
>> diff --git a/stubs/migration-colo.c b/stubs/migration-colo.c
>> index acddca6..c12516e 100644
>> --- a/stubs/migration-colo.c
>> +++ b/stubs/migration-colo.c
>> @@ -22,6 +22,16 @@ bool migration_in_colo_state(void)
>> return false;
>> }
>>
>> +bool migration_incoming_in_colo_state(void)
>> +{
>> + return false;
>> +}
>> +
>> void migrate_start_colo_process(MigrationState *s)
>> {
>> }
>> +
>> +void *colo_process_incoming_thread(void *opaque)
>> +{
>> + return NULL;
>> +}
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 08/39] migration: Rename the'file' member of MigrationState
2015-11-24 18:26 ` Dr. David Alan Gilbert
@ 2015-11-25 6:48 ` zhanghailiang
0 siblings, 0 replies; 95+ messages in thread
From: zhanghailiang @ 2015-11-25 6:48 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/11/25 2:26, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> Rename the 'file' member of MigrationState to 'to_dst_file'.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> ---
>> v11:
>> - Only rename 'file' member of MigrationState
>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
> You should be able to post this one separately.
>
OK, i will post this patch together with other independent migration
patches as a prerequisite series. Thanks.
> Dave
>
>> ---
>> include/migration/migration.h | 2 +-
>> migration/exec.c | 4 +--
>> migration/fd.c | 4 +--
>> migration/migration.c | 72 ++++++++++++++++++++++---------------------
>> migration/postcopy-ram.c | 6 ++--
>> migration/savevm.c | 2 +-
>> migration/tcp.c | 4 +--
>> migration/unix.c | 4 +--
>> 8 files changed, 51 insertions(+), 47 deletions(-)
>>
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index a57a734..ba5bcec 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -140,7 +140,7 @@ struct MigrationState
>> size_t xfer_limit;
>> QemuThread thread;
>> QEMUBH *cleanup_bh;
>> - QEMUFile *file;
>> + QEMUFile *to_dst_file;
>> int parameters[MIGRATION_PARAMETER_MAX];
>>
>> int state;
>> diff --git a/migration/exec.c b/migration/exec.c
>> index 8406d2b..9037109 100644
>> --- a/migration/exec.c
>> +++ b/migration/exec.c
>> @@ -36,8 +36,8 @@
>>
>> void exec_start_outgoing_migration(MigrationState *s, const char *command, Error **errp)
>> {
>> - s->file = qemu_popen_cmd(command, "w");
>> - if (s->file == NULL) {
>> + s->to_dst_file = qemu_popen_cmd(command, "w");
>> + if (s->to_dst_file == NULL) {
>> error_setg_errno(errp, errno, "failed to popen the migration target");
>> return;
>> }
>> diff --git a/migration/fd.c b/migration/fd.c
>> index 3e4bed0..9a9d6c5 100644
>> --- a/migration/fd.c
>> +++ b/migration/fd.c
>> @@ -50,9 +50,9 @@ void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **
>> }
>>
>> if (fd_is_socket(fd)) {
>> - s->file = qemu_fopen_socket(fd, "wb");
>> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
>> } else {
>> - s->file = qemu_fdopen(fd, "wb");
>> + s->to_dst_file = qemu_fdopen(fd, "wb");
>> }
>>
>> migrate_fd_connect(s);
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 41eac0d..a4c690d 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -834,7 +834,7 @@ static void migrate_fd_cleanup(void *opaque)
>>
>> flush_page_queue(s);
>>
>> - if (s->file) {
>> + if (s->to_dst_file) {
>> trace_migrate_fd_cleanup();
>> qemu_mutex_unlock_iothread();
>> if (s->migration_thread_running) {
>> @@ -844,8 +844,8 @@ static void migrate_fd_cleanup(void *opaque)
>> qemu_mutex_lock_iothread();
>>
>> migrate_compress_threads_join();
>> - qemu_fclose(s->file);
>> - s->file = NULL;
>> + qemu_fclose(s->to_dst_file);
>> + s->to_dst_file = NULL;
>> }
>>
>> assert((s->state != MIGRATION_STATUS_ACTIVE) &&
>> @@ -862,7 +862,7 @@ static void migrate_fd_cleanup(void *opaque)
>> void migrate_fd_error(MigrationState *s)
>> {
>> trace_migrate_fd_error();
>> - assert(s->file == NULL);
>> + assert(s->to_dst_file == NULL);
>> migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
>> MIGRATION_STATUS_FAILED);
>> notifier_list_notify(&migration_state_notifiers, s);
>> @@ -871,7 +871,7 @@ void migrate_fd_error(MigrationState *s)
>> static void migrate_fd_cancel(MigrationState *s)
>> {
>> int old_state ;
>> - QEMUFile *f = migrate_get_current()->file;
>> + QEMUFile *f = migrate_get_current()->to_dst_file;
>> trace_migrate_fd_cancel();
>>
>> if (s->rp_state.from_dst_file) {
>> @@ -942,7 +942,7 @@ MigrationState *migrate_init(const MigrationParams *params)
>> s->bytes_xfer = 0;
>> s->xfer_limit = 0;
>> s->cleanup_bh = 0;
>> - s->file = NULL;
>> + s->to_dst_file = NULL;
>> s->state = MIGRATION_STATUS_NONE;
>> s->params = *params;
>> s->rp_state.from_dst_file = NULL;
>> @@ -1122,8 +1122,9 @@ void qmp_migrate_set_speed(int64_t value, Error **errp)
>>
>> s = migrate_get_current();
>> s->bandwidth_limit = value;
>> - if (s->file) {
>> - qemu_file_set_rate_limit(s->file, s->bandwidth_limit / XFER_LIMIT_RATIO);
>> + if (s->to_dst_file) {
>> + qemu_file_set_rate_limit(s->to_dst_file,
>> + s->bandwidth_limit / XFER_LIMIT_RATIO);
>> }
>> }
>>
>> @@ -1393,7 +1394,7 @@ out:
>> static int open_return_path_on_source(MigrationState *ms)
>> {
>>
>> - ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->file);
>> + ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->to_dst_file);
>> if (!ms->rp_state.from_dst_file) {
>> return -1;
>> }
>> @@ -1415,7 +1416,7 @@ static int await_return_path_close_on_source(MigrationState *ms)
>> * rp_thread will exit, however if there's an error we need to cause
>> * it to exit.
>> */
>> - if (qemu_file_get_error(ms->file) && ms->rp_state.from_dst_file) {
>> + if (qemu_file_get_error(ms->to_dst_file) && ms->rp_state.from_dst_file) {
>> /*
>> * shutdown(2), if we have it, will cause it to unblock if it's stuck
>> * waiting for the destination.
>> @@ -1458,7 +1459,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
>> * Cause any non-postcopiable, but iterative devices to
>> * send out their final data.
>> */
>> - qemu_savevm_state_complete_precopy(ms->file, true);
>> + qemu_savevm_state_complete_precopy(ms->to_dst_file, true);
>>
>> /*
>> * in Finish migrate and with the io-lock held everything should
>> @@ -1476,9 +1477,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
>> * will notice we're in POSTCOPY_ACTIVE and not actually
>> * wrap their state up here
>> */
>> - qemu_file_set_rate_limit(ms->file, INT64_MAX);
>> + qemu_file_set_rate_limit(ms->to_dst_file, INT64_MAX);
>> /* Ping just for debugging, helps line traces up */
>> - qemu_savevm_send_ping(ms->file, 2);
>> + qemu_savevm_send_ping(ms->to_dst_file, 2);
>>
>> /*
>> * While loading the device state we may trigger page transfer
>> @@ -1512,7 +1513,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
>> qsb = qemu_buf_get(fb);
>>
>> /* Now send that blob */
>> - if (qemu_savevm_send_packaged(ms->file, qsb)) {
>> + if (qemu_savevm_send_packaged(ms->to_dst_file, qsb)) {
>> goto fail_closefb;
>> }
>> qemu_fclose(fb);
>> @@ -1524,9 +1525,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
>> * Although this ping is just for debug, it could potentially be
>> * used for getting a better measurement of downtime at the source.
>> */
>> - qemu_savevm_send_ping(ms->file, 4);
>> + qemu_savevm_send_ping(ms->to_dst_file, 4);
>>
>> - ret = qemu_file_get_error(ms->file);
>> + ret = qemu_file_get_error(ms->to_dst_file);
>> if (ret) {
>> error_report("postcopy_start: Migration stream errored");
>> migrate_set_state(&ms->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
>> @@ -1569,8 +1570,8 @@ static void migration_completion(MigrationState *s, int current_active_state,
>> if (!ret) {
>> ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
>> if (ret >= 0) {
>> - qemu_file_set_rate_limit(s->file, INT64_MAX);
>> - qemu_savevm_state_complete_precopy(s->file, false);
>> + qemu_file_set_rate_limit(s->to_dst_file, INT64_MAX);
>> + qemu_savevm_state_complete_precopy(s->to_dst_file, false);
>> }
>> }
>> qemu_mutex_unlock_iothread();
>> @@ -1581,7 +1582,7 @@ static void migration_completion(MigrationState *s, int current_active_state,
>> } else if (s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) {
>> trace_migration_completion_postcopy_end();
>>
>> - qemu_savevm_state_complete_postcopy(s->file);
>> + qemu_savevm_state_complete_postcopy(s->to_dst_file);
>> trace_migration_completion_postcopy_end_after_complete();
>> }
>>
>> @@ -1602,7 +1603,7 @@ static void migration_completion(MigrationState *s, int current_active_state,
>> }
>> }
>>
>> - if (qemu_file_get_error(s->file)) {
>> + if (qemu_file_get_error(s->to_dst_file)) {
>> trace_migration_completion_file_err();
>> goto fail;
>> }
>> @@ -1647,24 +1648,24 @@ static void *migration_thread(void *opaque)
>>
>> rcu_register_thread();
>>
>> - qemu_savevm_state_header(s->file);
>> + qemu_savevm_state_header(s->to_dst_file);
>>
>> if (migrate_postcopy_ram()) {
>> /* Now tell the dest that it should open its end so it can reply */
>> - qemu_savevm_send_open_return_path(s->file);
>> + qemu_savevm_send_open_return_path(s->to_dst_file);
>>
>> /* And do a ping that will make stuff easier to debug */
>> - qemu_savevm_send_ping(s->file, 1);
>> + qemu_savevm_send_ping(s->to_dst_file, 1);
>>
>> /*
>> * Tell the destination that we *might* want to do postcopy later;
>> * if the other end can't do postcopy it should fail now, nice and
>> * early.
>> */
>> - qemu_savevm_send_postcopy_advise(s->file);
>> + qemu_savevm_send_postcopy_advise(s->to_dst_file);
>> }
>>
>> - qemu_savevm_state_begin(s->file, &s->params);
>> + qemu_savevm_state_begin(s->to_dst_file, &s->params);
>>
>> s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
>> current_active_state = MIGRATION_STATUS_ACTIVE;
>> @@ -1678,10 +1679,10 @@ static void *migration_thread(void *opaque)
>> int64_t current_time;
>> uint64_t pending_size;
>>
>> - if (!qemu_file_rate_limit(s->file)) {
>> + if (!qemu_file_rate_limit(s->to_dst_file)) {
>> uint64_t pend_post, pend_nonpost;
>>
>> - qemu_savevm_state_pending(s->file, max_size, &pend_nonpost,
>> + qemu_savevm_state_pending(s->to_dst_file, max_size, &pend_nonpost,
>> &pend_post);
>> pending_size = pend_nonpost + pend_post;
>> trace_migrate_pending(pending_size, max_size,
>> @@ -1702,7 +1703,7 @@ static void *migration_thread(void *opaque)
>> continue;
>> }
>> /* Just another iteration step */
>> - qemu_savevm_state_iterate(s->file, entered_postcopy);
>> + qemu_savevm_state_iterate(s->to_dst_file, entered_postcopy);
>> } else {
>> trace_migration_thread_low_pending(pending_size);
>> migration_completion(s, current_active_state,
>> @@ -1711,7 +1712,7 @@ static void *migration_thread(void *opaque)
>> }
>> }
>>
>> - if (qemu_file_get_error(s->file)) {
>> + if (qemu_file_get_error(s->to_dst_file)) {
>> migrate_set_state(&s->state, current_active_state,
>> MIGRATION_STATUS_FAILED);
>> trace_migration_thread_file_err();
>> @@ -1719,7 +1720,8 @@ static void *migration_thread(void *opaque)
>> }
>> current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
>> if (current_time >= initial_time + BUFFER_DELAY) {
>> - uint64_t transferred_bytes = qemu_ftell(s->file) - initial_bytes;
>> + uint64_t transferred_bytes = qemu_ftell(s->to_dst_file) -
>> + initial_bytes;
>> uint64_t time_spent = current_time - initial_time;
>> double bandwidth = transferred_bytes / time_spent;
>> max_size = bandwidth * migrate_max_downtime() / 1000000;
>> @@ -1735,11 +1737,11 @@ static void *migration_thread(void *opaque)
>> s->expected_downtime = s->dirty_bytes_rate / bandwidth;
>> }
>>
>> - qemu_file_reset_rate_limit(s->file);
>> + qemu_file_reset_rate_limit(s->to_dst_file);
>> initial_time = current_time;
>> - initial_bytes = qemu_ftell(s->file);
>> + initial_bytes = qemu_ftell(s->to_dst_file);
>> }
>> - if (qemu_file_rate_limit(s->file)) {
>> + if (qemu_file_rate_limit(s->to_dst_file)) {
>> /* usleep expects microseconds */
>> g_usleep((initial_time + BUFFER_DELAY - current_time)*1000);
>> }
>> @@ -1757,7 +1759,7 @@ static void *migration_thread(void *opaque)
>> qemu_savevm_state_cleanup();
>> }
>> if (s->state == MIGRATION_STATUS_COMPLETED) {
>> - uint64_t transferred_bytes = qemu_ftell(s->file);
>> + uint64_t transferred_bytes = qemu_ftell(s->to_dst_file);
>> s->total_time = end_time - s->total_time;
>> if (!entered_postcopy) {
>> s->downtime = end_time - start_time;
>> @@ -1794,7 +1796,7 @@ void migrate_fd_connect(MigrationState *s)
>> s->expected_downtime = max_downtime/1000000;
>> s->cleanup_bh = qemu_bh_new(migrate_fd_cleanup, s);
>>
>> - qemu_file_set_rate_limit(s->file,
>> + qemu_file_set_rate_limit(s->to_dst_file,
>> s->bandwidth_limit / XFER_LIMIT_RATIO);
>>
>> /* Notify before starting migration thread */
>> diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
>> index 22d6b18..f74016d 100644
>> --- a/migration/postcopy-ram.c
>> +++ b/migration/postcopy-ram.c
>> @@ -733,7 +733,8 @@ void postcopy_discard_send_range(MigrationState *ms, PostcopyDiscardState *pds,
>>
>> if (pds->cur_entry == MAX_DISCARDS_PER_COMMAND) {
>> /* Full set, ship it! */
>> - qemu_savevm_send_postcopy_ram_discard(ms->file, pds->ramblock_name,
>> + qemu_savevm_send_postcopy_ram_discard(ms->to_dst_file,
>> + pds->ramblock_name,
>> pds->cur_entry,
>> pds->start_list,
>> pds->length_list);
>> @@ -753,7 +754,8 @@ void postcopy_discard_send_finish(MigrationState *ms, PostcopyDiscardState *pds)
>> {
>> /* Anything unsent? */
>> if (pds->cur_entry) {
>> - qemu_savevm_send_postcopy_ram_discard(ms->file, pds->ramblock_name,
>> + qemu_savevm_send_postcopy_ram_discard(ms->to_dst_file,
>> + pds->ramblock_name,
>> pds->cur_entry,
>> pds->start_list,
>> pds->length_list);
>> diff --git a/migration/savevm.c b/migration/savevm.c
>> index 0ad1b93..f102870 100644
>> --- a/migration/savevm.c
>> +++ b/migration/savevm.c
>> @@ -1163,7 +1163,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
>> .shared = 0
>> };
>> MigrationState *ms = migrate_init(¶ms);
>> - ms->file = f;
>> + ms->to_dst_file = f;
>>
>> if (qemu_savevm_state_blocked(errp)) {
>> return -EINVAL;
>> diff --git a/migration/tcp.c b/migration/tcp.c
>> index ae89172..e083d68 100644
>> --- a/migration/tcp.c
>> +++ b/migration/tcp.c
>> @@ -39,11 +39,11 @@ static void tcp_wait_for_connect(int fd, Error *err, void *opaque)
>>
>> if (fd < 0) {
>> DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
>> - s->file = NULL;
>> + s->to_dst_file = NULL;
>> migrate_fd_error(s);
>> } else {
>> DPRINTF("migrate connect success\n");
>> - s->file = qemu_fopen_socket(fd, "wb");
>> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
>> migrate_fd_connect(s);
>> }
>> }
>> diff --git a/migration/unix.c b/migration/unix.c
>> index b591813..5492dd6 100644
>> --- a/migration/unix.c
>> +++ b/migration/unix.c
>> @@ -39,11 +39,11 @@ static void unix_wait_for_connect(int fd, Error *err, void *opaque)
>>
>> if (fd < 0) {
>> DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
>> - s->file = NULL;
>> + s->to_dst_file = NULL;
>> migrate_fd_error(s);
>> } else {
>> DPRINTF("migrate connect success\n");
>> - s->file = qemu_fopen_socket(fd, "wb");
>> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
>> migrate_fd_connect(s);
>> }
>> }
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 10/39] COLO: Implement colo checkpoint protocol
2015-11-24 19:00 ` Dr. David Alan Gilbert
@ 2015-11-25 14:01 ` Eric Blake
2015-11-26 6:52 ` Hailiang Zhang
2015-11-26 7:12 ` Hailiang Zhang
1 sibling, 1 reply; 95+ messages in thread
From: Eric Blake @ 2015-11-25 14:01 UTC (permalink / raw)
To: Dr. David Alan Gilbert, zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
[-- Attachment #1: Type: text/plain, Size: 2043 bytes --]
On 11/24/2015 12:00 PM, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> We need communications protocol of user-defined to control the checkpoint
>> process.
>>
>> +static int colo_ctl_get(QEMUFile *f, uint32_t require, uint64_t *value)
>> +{
>> + int ret;
>> + uint32_t cmd;
>> +
>> + ret = colo_ctl_get_cmd(f, &cmd);
>> + if (ret < 0) {
>> + return ret;
>> + }
>> + if (cmd != require) {
>> + error_report("Unexpect colo command, expect:%d, but got cmd:%d",
>> + require, cmd);
>
> I think you need to use PRIu32 rather than %d since they are uint32_t
> (I doubt it will break on anything, but it's correct).
32-bit cygwin uses 'long' for uint32_t, so there ARE platforms where it
is absolutely necessary to use PRIu32 for printing (even if qemu isn't
ported to cygwin).
>> +++ b/qapi-schema.json
>> @@ -722,6 +722,33 @@
>> { 'command': 'migrate-start-postcopy' }
>>
>> ##
>> +# @COLOCommand
>> +#
>> +# The commands for COLO fault tolerance
>> +#
>> +# @invalid: unknown command
>> +#
>> +# @checkpoint-ready: SVM is ready for checkpointing
>> +#
>> +# @checkpoint-request: PVM tells SVM to prepare for new checkpointing
>> +#
>> +# @checkpoint-reply: SVM gets PVM's checkpoint request
>> +#
>> +# @vmstate-send: VM's state will be sent by PVM.
>> +#
>> +# @vmstate-size: The total size of VMstate.
>> +#
>> +# @vmstate-received: VM's state has been received by SVM
>> +#
>> +# @vmstate-loaded: VM's state has been loaded by SVM
Inconsistent use of trailing '.'
Also, do we really need 'invalid'?
>> +#
>> +# Since: 2.6
>> +##
>> +{ 'enum': 'COLOCommand',
>> + 'data': [ 'invalid', 'checkpoint-ready', 'checkpoint-request',
>> + 'checkpoint-reply', 'vmstate-send', 'vmstate-size',
>> + 'vmstate-received', 'vmstate-loaded' ] }
>> +
--
Eric Blake eblake redhat com +1-919-301-3266
Libvirt virtualization library http://libvirt.org
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 604 bytes --]
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 10/39] COLO: Implement colo checkpoint protocol
2015-11-25 14:01 ` Eric Blake
@ 2015-11-26 6:52 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-11-26 6:52 UTC (permalink / raw)
To: Eric Blake, Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/11/25 22:01, Eric Blake wrote:
> On 11/24/2015 12:00 PM, Dr. David Alan Gilbert wrote:
>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>>> We need communications protocol of user-defined to control the checkpoint
>>> process.
>>>
>
>>> +static int colo_ctl_get(QEMUFile *f, uint32_t require, uint64_t *value)
>>> +{
>>> + int ret;
>>> + uint32_t cmd;
>>> +
>>> + ret = colo_ctl_get_cmd(f, &cmd);
>>> + if (ret < 0) {
>>> + return ret;
>>> + }
>>> + if (cmd != require) {
>>> + error_report("Unexpect colo command, expect:%d, but got cmd:%d",
>>> + require, cmd);
>>
>> I think you need to use PRIu32 rather than %d since they are uint32_t
>> (I doubt it will break on anything, but it's correct).
>
> 32-bit cygwin uses 'long' for uint32_t, so there ARE platforms where it
> is absolutely necessary to use PRIu32 for printing (even if qemu isn't
> ported to cygwin).
>
Got it, i will fix it.
>
>>> +++ b/qapi-schema.json
>>> @@ -722,6 +722,33 @@
>>> { 'command': 'migrate-start-postcopy' }
>>>
>>> ##
>>> +# @COLOCommand
>>> +#
>>> +# The commands for COLO fault tolerance
>>> +#
>>> +# @invalid: unknown command
>>> +#
>>> +# @checkpoint-ready: SVM is ready for checkpointing
>>> +#
>>> +# @checkpoint-request: PVM tells SVM to prepare for new checkpointing
>>> +#
>>> +# @checkpoint-reply: SVM gets PVM's checkpoint request
>>> +#
>>> +# @vmstate-send: VM's state will be sent by PVM.
>>> +#
>>> +# @vmstate-size: The total size of VMstate.
>>> +#
>>> +# @vmstate-received: VM's state has been received by SVM
>>> +#
>>> +# @vmstate-loaded: VM's state has been loaded by SVM
>
> Inconsistent use of trailing '.'
>
> Also, do we really need 'invalid'?
>
No, we can remove it. Thanks.
>>> +#
>>> +# Since: 2.6
>>> +##
>>> +{ 'enum': 'COLOCommand',
>>> + 'data': [ 'invalid', 'checkpoint-ready', 'checkpoint-request',
>>> + 'checkpoint-reply', 'vmstate-send', 'vmstate-size',
>>> + 'vmstate-received', 'vmstate-loaded' ] }
>>> +
>
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 10/39] COLO: Implement colo checkpoint protocol
2015-11-24 19:00 ` Dr. David Alan Gilbert
2015-11-25 14:01 ` Eric Blake
@ 2015-11-26 7:12 ` Hailiang Zhang
1 sibling, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-11-26 7:12 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/11/25 3:00, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> We need communications protocol of user-defined to control the checkpoint
>> process.
>>
>> The new checkpoint request is started by Primary VM, and the interactive process
>> like below:
>> Checkpoint synchronizing points,
>>
>> Primary Secondary
>> initial work
>> 'checkpoint-ready' <------------------------------ @
>>
>> 'checkpoint-request' @ ----------------------------->
>> Suspend (Only in hybrid mode)
>> 'checkpoint-reply' <------------------------------ @
>> Suspend&Save state
>> 'vmstate-send' @ ----------------------------->
>> Send state Receive state
>> 'vmstate-received' <------------------------------ @
>> Release packets Load state
>> 'vmstate-load' <------------------------------ @
>> Resume Resume (Only in hybrid mode)
>>
>> Start Comparing (Only in hybrid mode)
>> NOTE:
>> 1) '@' who sends the message
>> 2) Every sync-point is synchronized by two sides with only
>> one handshake(single direction) for low-latency.
>> If more strict synchronization is required, a opposite direction
>> sync-point should be added.
>> 3) Since sync-points are single direction, the remote side may
>> go forward a lot when this side just receives the sync-point.
>> 4) For now, we only support 'periodic' checkpoint, for which
>> the Secondary VM is not running, later we will support 'hybrid' mode.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> Cc: Eric Blake <eblake@redhat.com>
>> ---
>> v11:
>> - Add missing 'checkpoint-ready' communication in comment.
>> - Use parameter to return 'value' for colo_ctl_get() (Dave's suggestion)
>> - Fix trace for colo_ctl_get() to trace command and value both
>> v10:
>> - Rename enum COLOCmd to COLOCommand (Eric's suggestion).
>> - Remove unused 'ram-steal'
>> ---
>> migration/colo.c | 198 ++++++++++++++++++++++++++++++++++++++++++++++++++++++-
>> qapi-schema.json | 27 ++++++++
>> trace-events | 2 +
>> 3 files changed, 225 insertions(+), 2 deletions(-)
>>
>> diff --git a/migration/colo.c b/migration/colo.c
>> index 0ab9618..c045d61 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -10,10 +10,12 @@
>> * later. See the COPYING file in the top-level directory.
>> */
>>
>> +#include <unistd.h>
>> #include "sysemu/sysemu.h"
>> #include "migration/colo.h"
>> #include "trace.h"
>> #include "qemu/error-report.h"
>> +#include "qemu/sockets.h"
>>
>> bool colo_supported(void)
>> {
>> @@ -34,9 +36,107 @@ bool migration_incoming_in_colo_state(void)
>> return mis && (mis->state == MIGRATION_STATUS_COLO);
>> }
>>
>> +/* colo checkpoint control helper */
>> +static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value)
>> +{
>> + int ret = 0;
>> +
>> + qemu_put_be32(f, cmd);
>> + qemu_put_be64(f, value);
>> + qemu_fflush(f);
>> +
>> + ret = qemu_file_get_error(f);
>> + trace_colo_ctl_put(COLOCommand_lookup[cmd], value);
>> +
>> + return ret;
>> +}
>> +
>> +static int colo_ctl_get_cmd(QEMUFile *f, uint32_t *cmd)
>> +{
>> + int ret = 0;
>> +
>> + *cmd = qemu_get_be32(f);
>> + ret = qemu_file_get_error(f);
>> + if (ret < 0) {
>> + return ret;
>> + }
>> + if (*cmd >= COLO_COMMAND_MAX) {
>> + error_report("Invalid colo command, got cmd:%d", *cmd);
>> + return -EINVAL;
>> + }
>> +
>> + return 0;
>> +}
>> +
>> +static int colo_ctl_get(QEMUFile *f, uint32_t require, uint64_t *value)
>> +{
>> + int ret;
>> + uint32_t cmd;
>> +
>> + ret = colo_ctl_get_cmd(f, &cmd);
>> + if (ret < 0) {
>> + return ret;
>> + }
>> + if (cmd != require) {
>> + error_report("Unexpect colo command, expect:%d, but got cmd:%d",
>> + require, cmd);
>
> I think you need to use PRIu32 rather than %d since they are uint32_t
> (I doubt it will break on anything, but it's correct).
>
>> + return -EINVAL;
>> + }
>> +
>> + *value = qemu_get_be64(f);
>> + trace_colo_ctl_get(COLOCommand_lookup[cmd], *value);
>> + ret = qemu_file_get_error(f);
>> +
>> + return ret;
>> +}
>> +
>> +static int colo_do_checkpoint_transaction(MigrationState *s)
>> +{
>> + int ret;
>> + uint64_t value;
>> +
>> + ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_CHECKPOINT_REQUEST, 0);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> +
>> + ret = colo_ctl_get(s->rp_state.from_dst_file,
>> + COLO_COMMAND_CHECKPOINT_REPLY, &value);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> +
>> + /* TODO: suspend and save vm state to colo buffer */
>> +
>> + ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> +
>> + /* TODO: send vmstate to Secondary */
>> +
>> + ret = colo_ctl_get(s->rp_state.from_dst_file,
>> + COLO_COMMAND_VMSTATE_RECEIVED, &value);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> +
>> + ret = colo_ctl_get(s->rp_state.from_dst_file,
>> + COLO_COMMAND_VMSTATE_LOADED, &value);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> +
>> + /* TODO: resume Primary */
>> +
>> +out:
>> + return ret;
>> +}
>> +
>> static void colo_process_checkpoint(MigrationState *s)
>> {
>> int ret = 0;
>> + uint64_t value;
>>
>> s->rp_state.from_dst_file = qemu_file_get_return_path(s->to_dst_file);
>> if (!s->rp_state.from_dst_file) {
>> @@ -45,12 +145,28 @@ static void colo_process_checkpoint(MigrationState *s)
>> goto out;
>> }
>>
>> + /*
>> + * Wait for Secondary finish loading vm states and enter COLO
>> + * restore.
>> + */
>> + ret = colo_ctl_get(s->rp_state.from_dst_file,
>> + COLO_COMMAND_CHECKPOINT_READY, &value);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> +
>> qemu_mutex_lock_iothread();
>> vm_start();
>> qemu_mutex_unlock_iothread();
>> trace_colo_vm_state_change("stop", "run");
>>
>> - /*TODO: COLO checkpoint savevm loop*/
>> + while (s->state == MIGRATION_STATUS_COLO) {
>> + /* start a colo checkpoint */
>> + ret = colo_do_checkpoint_transaction(s);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> + }
>>
>> out:
>> if (ret < 0) {
>> @@ -73,10 +189,46 @@ void migrate_start_colo_process(MigrationState *s)
>> qemu_mutex_lock_iothread();
>> }
>>
>> +/*
>> + * return:
>> + * 0: start a checkpoint
>> + * -1: some error happened, exit colo restore
>> + */
>> +static int colo_wait_handle_cmd(QEMUFile *f, int *checkpoint_request)
>> +{
>> + int ret;
>> + uint32_t cmd;
>> + uint64_t value;
>> +
>> + ret = colo_ctl_get_cmd(f, &cmd);
>> + if (ret < 0) {
>> + /* do failover ? */
>> + return ret;
>> + }
>> + /* Fix me: this value should be 0, which is not so good,
>> + * should be used for checking ?
>> + */
>
> You could use something simple like an incrementing value, if you just
> wanted to make it a little more robust against problems.
>
Hmm, this a good choice, but we have to hold a variable to record the increasing
value.
Checking the length of the corresponding commands' string seems better. :)
>> + value = qemu_get_be64(f);
>> + if (value != 0) {
>> + error_report("Got unexpected value %" PRIu64 " for '%s' command",
>> + value, COLOCommand_lookup[cmd]);
>> + return -EINVAL;
>> + }
>> +
>> + switch (cmd) {
>> + case COLO_COMMAND_CHECKPOINT_REQUEST:
>> + *checkpoint_request = 1;
>> + return 0;
>> + default:
>> + return -EINVAL;
>> + }
>> +}
>> +
>> void *colo_process_incoming_thread(void *opaque)
>> {
>> MigrationIncomingState *mis = opaque;
>> int ret = 0;
>> + uint64_t value;
>>
>> migrate_set_state(&mis->state, MIGRATION_STATUS_ACTIVE,
>> MIGRATION_STATUS_COLO);
>> @@ -93,7 +245,49 @@ void *colo_process_incoming_thread(void *opaque)
>> */
>> qemu_set_block(qemu_get_fd(mis->from_src_file));
>>
>> - /* TODO: COLO checkpoint restore loop */
>> +
>> + ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_READY, 0);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> +
>> + while (mis->state == MIGRATION_STATUS_COLO) {
>> + int request = 0;
>> + int ret = colo_wait_handle_cmd(mis->from_src_file, &request);
>> +
>> + if (ret < 0) {
>> + break;
>> + } else {
>> + if (!request) {
>> + continue;
>> + }
>> + }
>> + /* FIXME: This is unnecessary for periodic checkpoint mode */
>> + ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_REPLY, 0);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> +
>> + ret = colo_ctl_get(mis->from_src_file, COLO_COMMAND_VMSTATE_SEND,
>> + &value);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> +
>> + /* TODO: read migration data into colo buffer */
>> +
>> + ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_RECEIVED, 0);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> +
>> + /* TODO: load vm state */
>> +
>> + ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_LOADED, 0);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> + }
>>
>> out:
>> if (ret < 0) {
>> diff --git a/qapi-schema.json b/qapi-schema.json
>> index c2f3b63..48a87af 100644
>> --- a/qapi-schema.json
>> +++ b/qapi-schema.json
>> @@ -722,6 +722,33 @@
>> { 'command': 'migrate-start-postcopy' }
>>
>> ##
>> +# @COLOCommand
>> +#
>> +# The commands for COLO fault tolerance
>> +#
>> +# @invalid: unknown command
>> +#
>> +# @checkpoint-ready: SVM is ready for checkpointing
>> +#
>> +# @checkpoint-request: PVM tells SVM to prepare for new checkpointing
>> +#
>> +# @checkpoint-reply: SVM gets PVM's checkpoint request
>> +#
>> +# @vmstate-send: VM's state will be sent by PVM.
>> +#
>> +# @vmstate-size: The total size of VMstate.
>> +#
>> +# @vmstate-received: VM's state has been received by SVM
>> +#
>> +# @vmstate-loaded: VM's state has been loaded by SVM
>> +#
>> +# Since: 2.6
>> +##
>> +{ 'enum': 'COLOCommand',
>> + 'data': [ 'invalid', 'checkpoint-ready', 'checkpoint-request',
>> + 'checkpoint-reply', 'vmstate-send', 'vmstate-size',
>> + 'vmstate-received', 'vmstate-loaded' ] }
>> +
>> # @MouseInfo:
>> #
>> # Information about a mouse device.
>> diff --git a/trace-events b/trace-events
>> index c98d473..f8a0959 100644
>> --- a/trace-events
>> +++ b/trace-events
>> @@ -1579,6 +1579,8 @@ postcopy_ram_incoming_cleanup_join(void) ""
>>
>> # migration/colo.c
>> colo_vm_state_change(const char *old, const char *new) "Change '%s' => '%s'"
>> +colo_ctl_put(const char *msg, uint64_t value) "Send '%s' cmd, value: %" PRIu64""
>> +colo_ctl_get(const char *msg, uint64_t value) "Receive '%s' cmd, value: %" PRIu64""
>
> I don't think you need the "" at the end of the line
>
OK, will fix in next version.
Thanks,
Hailiang
> Dave
>
>>
>> # kvm-all.c
>> kvm_ioctl(int type, void *arg) "type 0x%x, arg %p"
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 32/39] COLO: Separate the process of saving/loading ram and device state
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 32/39] COLO: Separate the process of saving/loading ram and device state zhanghailiang
@ 2015-11-27 5:10 ` Li Zhijian
2015-12-01 12:07 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Li Zhijian @ 2015-11-27 5:10 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: quintela, yunhong.jiang, eddie.dong, peter.huangpeng, dgilbert,
arei.gonglei, stefanha, amit.shah, hongyang.yang
On 11/24/2015 05:25 PM, zhanghailiang wrote:
> We separate the process of saving/loading ram and device state when do checkpoint,
> we add new helpers for save/load ram/device. With this change, we can directly
> transfer ram from master to slave without using QEMUSizeBuffer as assistant,
> which also reduce the size of extra memory been used during checkpoint.
>
> Besides, we move the colo_flush_ram_cache to the proper position after the
> above change.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> ---
> v11:
> - Remove load configuration section in qemu_loadvm_state_begin()
> ---
> include/sysemu/sysemu.h | 6 +++
> migration/colo.c | 43 ++++++++++++----
> migration/ram.c | 5 --
> migration/savevm.c | 132 ++++++++++++++++++++++++++++++++++++++++++++++--
> 4 files changed, 168 insertions(+), 18 deletions(-)
>
> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
> index 91eeda3..5deae53 100644
> --- a/include/sysemu/sysemu.h
> +++ b/include/sysemu/sysemu.h
> @@ -133,7 +133,13 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
> uint64_t *start_list,
> uint64_t *length_list);
>
> +int qemu_save_ram_precopy(QEMUFile *f);
> +int qemu_save_device_state(QEMUFile *f);
> +
> int qemu_loadvm_state(QEMUFile *f);
> +int qemu_loadvm_state_begin(QEMUFile *f);
> +int qemu_load_ram_state(QEMUFile *f);
> +int qemu_load_device_state(QEMUFile *f);
>
> typedef enum DisplayType
> {
> diff --git a/migration/colo.c b/migration/colo.c
> index 3866e86..f7f349b 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -247,21 +247,32 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
> goto out;
> }
>
> + ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
> + if (ret < 0) {
> + goto out;
> + }
> /* Disable block migration */
> s->params.blk = 0;
> s->params.shared = 0;
> - qemu_savevm_state_header(trans);
> - qemu_savevm_state_begin(trans, &s->params);
> - qemu_mutex_lock_iothread();
> - qemu_savevm_state_complete_precopy(trans, false);
> - qemu_mutex_unlock_iothread();
> -
> - qemu_fflush(trans);
> + qemu_savevm_state_begin(s->to_dst_file, &s->params);
> + ret = qemu_file_get_error(s->to_dst_file);
> + if (ret < 0) {
> + error_report("save vm state begin error\n");
> + goto out;
> + }
>
> - ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
> + qemu_mutex_lock_iothread();
> + /* Note: device state is saved into buffer */
> + ret = qemu_save_device_state(trans);
> if (ret < 0) {
> + error_report("save device state error\n");
> + qemu_mutex_unlock_iothread();
> goto out;
> }
> + qemu_fflush(trans);
> + qemu_save_ram_precopy(s->to_dst_file);
> + qemu_mutex_unlock_iothread();
> +
> /* we send the total size of the vmstate first */
> size = qsb_get_length(buffer);
> ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SIZE, size);
> @@ -548,6 +559,16 @@ void *colo_process_incoming_thread(void *opaque)
> goto out;
> }
>
> + ret = qemu_loadvm_state_begin(mis->from_src_file);
> + if (ret < 0) {
> + error_report("load vm state begin error, ret=%d", ret);
> + goto out;
> + }
> + ret = qemu_load_ram_state(mis->from_src_file);
> + if (ret < 0) {
> + error_report("load ram state error");
> + goto out;
> + }
> /* read the VM state total size first */
> ret = colo_ctl_get(mis->from_src_file,
> COLO_COMMAND_VMSTATE_SIZE, &value);
> @@ -580,8 +601,10 @@ void *colo_process_incoming_thread(void *opaque)
> qemu_mutex_lock_iothread();
> qemu_system_reset(VMRESET_SILENT);
> vmstate_loading = true;
> - if (qemu_loadvm_state(fb) < 0) {
> - error_report("COLO: loadvm failed");
> + colo_flush_ram_cache();
> + ret = qemu_load_device_state(fb);
> + if (ret < 0) {
> + error_report("COLO: load device state failed\n");
> vmstate_loading = false;
> qemu_mutex_unlock_iothread();
> goto out;
> diff --git a/migration/ram.c b/migration/ram.c
> index 4f37144..06a738b 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -2448,7 +2448,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> * be atomic
> */
> bool postcopy_running = postcopy_state_get() >= POSTCOPY_INCOMING_LISTENING;
> - bool need_flush = false;
>
> seq_iter++;
>
> @@ -2483,7 +2482,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> /* After going into COLO, we should load the Page into colo_cache */
> if (ram_cache_enable) {
> host = colo_cache_from_block_offset(block, addr);
> - need_flush = true;
> } else {
> host = host_from_ram_block_offset(block, addr);
> }
> @@ -2578,9 +2576,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>
> rcu_read_unlock();
>
> - if (!ret && ram_cache_enable && need_flush) {
> - colo_flush_ram_cache();
> - }
> DPRINTF("Completed load of VM with exit code %d seq iteration "
> "%" PRIu64 "\n", ret, seq_iter);
> return ret;
> diff --git a/migration/savevm.c b/migration/savevm.c
> index c7c26d8..949caf0 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -50,6 +50,7 @@
> #include "qemu/iov.h"
> #include "block/snapshot.h"
> #include "block/qapi.h"
> +#include "migration/colo.h"
>
>
> #ifndef ETH_P_RARP
> @@ -923,6 +924,10 @@ void qemu_savevm_state_begin(QEMUFile *f,
> break;
> }
> }
> + if (migration_in_colo_state()) {
> + qemu_put_byte(f, QEMU_VM_EOF);
> + qemu_fflush(f);
> + }
> }
>
> /*
> @@ -1192,13 +1197,44 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
> return ret;
> }
>
> -static int qemu_save_device_state(QEMUFile *f)
> +int qemu_save_ram_precopy(QEMUFile *f)
> {
> SaveStateEntry *se;
> + int ret = 0;
>
> - qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
> - qemu_put_be32(f, QEMU_VM_FILE_VERSION);
> + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
> + if (!se->ops || !se->ops->save_live_complete_precopy) {
> + continue;
> + }
> + if (se->ops && se->ops->is_active) {
> + if (!se->ops->is_active(se->opaque)) {
> + continue;
> + }
> + }
> + trace_savevm_section_start(se->idstr, se->section_id);
>
> + save_section_header(f, se, QEMU_VM_SECTION_END);
> +
> + ret = se->ops->save_live_complete_precopy(f, se->opaque);
> + trace_savevm_section_end(se->idstr, se->section_id, ret);
> + save_section_footer(f, se);
> + if (ret < 0) {
> + qemu_file_set_error(f, ret);
> + return ret;
> + }
> + }
> + qemu_put_byte(f, QEMU_VM_EOF);
> +
> + return 0;
> +}
> +
> +int qemu_save_device_state(QEMUFile *f)
> +{
> + SaveStateEntry *se;
> +
> + if (!migration_in_colo_state()) {
> + qemu_savevm_state_header(f);
> + }
> cpu_synchronize_all_states();
>
> QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
> @@ -1938,6 +1974,96 @@ int qemu_loadvm_state(QEMUFile *f)
> return ret;
> }
>
> +int qemu_loadvm_state_begin(QEMUFile *f)
> +{
> + uint8_t section_type;
> + int ret = -1;
> + MigrationIncomingState *mis = migration_incoming_get_current();
> +
> + if (!mis) {
> + error_report("qemu_loadvm_state_begin");
> + return -EINVAL;
> + }
> + /* CleanUp */
> + loadvm_free_handlers(mis);
> +
> + if (qemu_savevm_state_blocked(NULL)) {
> + return -EINVAL;
> + }
> +
> + while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
> + if (section_type != QEMU_VM_SECTION_START) {
> + error_report("QEMU_VM_SECTION_START");
> + ret = -EINVAL;
> + goto out;
> + }
> + ret = qemu_loadvm_section_start_full(f, mis);
> + if (ret < 0) {
> + goto out;
> + }
> + }
> + ret = qemu_file_get_error(f);
> + if (ret == 0) {
> + return 0;
> + }
> +out:
> + return ret;
> +}
> +
> +int qemu_load_ram_state(QEMUFile *f)
> +{
> + uint8_t section_type;
> + MigrationIncomingState *mis = migration_incoming_get_current();
> + int ret = -1;
> +
> + while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
> + if (section_type != QEMU_VM_SECTION_PART &&
> + section_type != QEMU_VM_SECTION_END) {
> + error_report("load ram state, not get "
> + "QEMU_VM_SECTION_FULL or QEMU_VM_SECTION_END");
> + return -EINVAL;
> + }
> + ret = qemu_loadvm_section_part_end(f, mis);
> + if (ret < 0) {
> + goto out;
> + }
> + }
> + ret = qemu_file_get_error(f);
> + if (ret == 0) {
> + return 0;
> + }
> +out:
> + return ret;
> +}
> +
> +int qemu_load_device_state(QEMUFile *f)
> +{
> + uint8_t section_type;
> + MigrationIncomingState *mis = migration_incoming_get_current();
> + int ret = -1;
> +
> + while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
> + if (section_type != QEMU_VM_SECTION_FULL) {
> + error_report("load device state error: "
> + "Not get QEMU_VM_SECTION_FULL");
> + return -EINVAL;
> + }
> + ret = qemu_loadvm_section_start_full(f, mis);
> + if (ret < 0) {
> + goto out;
> + }
coding sytle：ａbove 4 lines are not aligned
> + }
> +
> + ret = qemu_file_get_error(f);
> +
> + cpu_synchronize_all_post_init();
> + if (ret == 0) {
> + return 0;
> + }
aligned issue
thanks
Li
> +out:
> + return ret;
> +}
> +
> void hmp_savevm(Monitor *mon, const QDict *qdict)
> {
> BlockDriverState *bs, *bs1;
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 18/39] COLO: Flush PVM's cached RAM into SVM's memory
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 18/39] COLO: Flush PVM's cached RAM into SVM's memory zhanghailiang
@ 2015-11-27 5:29 ` Li Zhijian
2015-12-01 12:02 ` Hailiang Zhang
2015-12-01 20:06 ` Dr. David Alan Gilbert
1 sibling, 1 reply; 95+ messages in thread
From: Li Zhijian @ 2015-11-27 5:29 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: quintela, yunhong.jiang, eddie.dong, peter.huangpeng, dgilbert,
arei.gonglei, stefanha, amit.shah, hongyang.yang
On 11/24/2015 05:25 PM, zhanghailiang wrote:
> During the time of VM's running, PVM may dirty some pages, we will transfer
> PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint
> time. So, the content of SVM's RAM cache will always be some with PVM's memory
> after checkpoint.
>
> Instead of flushing all content of PVM's RAM cache into SVM's MEMORY,
> we do this in a more efficient way:
> Only flush any page that dirtied by PVM since last checkpoint.
> In this way, we can ensure SVM's memory same with PVM's.
>
> Besides, we must ensure flush RAM cache before load device state.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
> v11:
> - Move the place of 'need_flush' (Dave's suggestion)
> - Remove unused 'DPRINTF("Flush ram_cache\n")'
> v10:
> - trace the number of dirty pages that be received.
> ---
> include/migration/migration.h | 1 +
> migration/colo.c | 2 --
> migration/ram.c | 37 +++++++++++++++++++++++++++++++++++++
> trace-events | 1 +
> 4 files changed, 39 insertions(+), 2 deletions(-)
>
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index e41372d..221176b 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -336,4 +336,5 @@ PostcopyState postcopy_state_set(PostcopyState new_state);
> /* ram cache */
> int colo_init_ram_cache(void);
> void colo_release_ram_cache(void);
> +void colo_flush_ram_cache(void);
> #endif
> diff --git a/migration/colo.c b/migration/colo.c
> index 5ac8ff2..4095d97 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -385,8 +385,6 @@ void *colo_process_incoming_thread(void *opaque)
> }
> qemu_mutex_unlock_iothread();
>
> - /* TODO: flush vm state */
> -
> ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_LOADED, 0);
> if (ret < 0) {
> goto out;
> diff --git a/migration/ram.c b/migration/ram.c
> index da6bbd6..4f37144 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -2448,6 +2448,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> * be atomic
> */
> bool postcopy_running = postcopy_state_get() >= POSTCOPY_INCOMING_LISTENING;
> + bool need_flush = false;
>
> seq_iter++;
>
> @@ -2482,6 +2483,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> /* After going into COLO, we should load the Page into colo_cache */
> if (ram_cache_enable) {
> host = colo_cache_from_block_offset(block, addr);
> + need_flush = true;
> } else {
> host = host_from_ram_block_offset(block, addr);
> }
> @@ -2575,6 +2577,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> }
>
> rcu_read_unlock();
> +
> + if (!ret && ram_cache_enable && need_flush) {
> + colo_flush_ram_cache();
> + }
> DPRINTF("Completed load of VM with exit code %d seq iteration "
> "%" PRIu64 "\n", ret, seq_iter);
> return ret;
> @@ -2647,6 +2653,37 @@ void colo_release_ram_cache(void)
> rcu_read_unlock();
> }
>
> +/*
> + * Flush content of RAM cache into SVM's memory.
> + * Only flush the pages that be dirtied by PVM or SVM or both.
> + */
> +void colo_flush_ram_cache(void)
> +{
> + RAMBlock *block = NULL;
> + void *dst_host;
> + void *src_host;
> + ram_addr_t offset = 0;
coding sytle: double space
thanks
Li
> +
> + trace_colo_flush_ram_cache(migration_dirty_pages);
> + rcu_read_lock();
> + block = QLIST_FIRST_RCU(&ram_list.blocks);
> + while (block) {
> + ram_addr_t ram_addr_abs;
> + offset = migration_bitmap_find_dirty(block, offset, &ram_addr_abs);
> + migration_bitmap_clear_dirty(ram_addr_abs);
> + if (offset >= block->used_length) {
> + offset = 0;
> + block = QLIST_NEXT_RCU(block, next);
> + } else {
> + dst_host = block->host + offset;
> + src_host = block->colo_cache + offset;
> + memcpy(dst_host, src_host, TARGET_PAGE_SIZE);
> + }
> + }
> + rcu_read_unlock();
> + assert(migration_dirty_pages == 0);
> +}
> +
> static SaveVMHandlers savevm_ram_handlers = {
> .save_live_setup = ram_save_setup,
> .save_live_iterate = ram_save_iterate,
> diff --git a/trace-events b/trace-events
> index f8a0959..f158d2a 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -1264,6 +1264,7 @@ migration_throttle(void) ""
> ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x"
> ram_postcopy_send_discard_bitmap(void) ""
> ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: %zx len: %zx"
> +colo_flush_ram_cache(uint64_t dirty_pages) "dirty_pages %" PRIu64""
>
> # hw/display/qxl.c
> disable qxl_interface_set_mm_time(int qid, uint32_t mm_time) "%d %d"
--
Best regards.
Li Zhijian (8555)
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev zhanghailiang
@ 2015-11-27 11:39 ` Yang Hongyang
2015-11-28 5:55 ` Hailiang Zhang
2015-11-30 1:19 ` Li Zhijian
2015-12-03 1:17 ` Wen Congyang
2 siblings, 1 reply; 95+ messages in thread
From: Yang Hongyang @ 2015-11-27 11:39 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah
On 2015年11月24日 17:25, zhanghailiang wrote:
> We add each netdev a default filter-buffer, which will be used for COLO
> or Micro-checkpoint to buffer VM's packets. The name of default filter-buffer
> is 'nop'.
> For the default filter-buffer, it will not buffer any packets in default.
> So it has no side effect for the netdev.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
> ---
> v11:
> - New patch
> ---
[...]
> static void filter_buffer_flush(NetFilterState *nf)
> @@ -65,6 +73,10 @@ static ssize_t filter_buffer_receive_iov(NetFilterState *nf,
> {
> FilterBufferState *s = FILTER_BUFFER(nf);
>
> + /* Don't buffer any packets if the filter is not enabled */
> + if (!s->enable_buffer) {
> + return 0;
> + }
> /*
> * We return size when buffer a packet, the sender will take it as
> * a already sent packet, so sent_cb should not be called later.
> @@ -102,6 +114,7 @@ static void filter_buffer_cleanup(NetFilterState *nf)
> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
> {
> FilterBufferState *s = FILTER_BUFFER(nf);
> + char *path = object_get_canonical_path_component(OBJECT(nf));
path should be freed after use.
>
> /*
> * We may want to accept zero interval when VM FT solutions like MC
> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
> }
>
> s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
> + s->is_default = !strcmp(path, "nop");
free(path);
> if (s->interval) {
> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
> filter_buffer_release_timer, nf);
> @@ -163,6 +177,66 @@ out:
> error_propagate(errp, local_err);
> }
>
> +/*
> +* This will be used by COLO or MC FT, for which they will need
> +* to buffer the packets of VM's net devices, Here we add a default
> +* buffer filter for each netdev. The name of default buffer filter is
> +* 'nop'
> +*/
> +void netdev_add_default_filter_buffer(const char *netdev_id,
> + NetFilterDirection direction,
> + Error **errp)
> +{
> + QmpOutputVisitor *qov;
> + QmpInputVisitor *qiv;
> + Visitor *ov, *iv;
> + QObject *obj = NULL;
> + QDict *qdict;
> + void *dummy = NULL;
> + const char *id = "nop";
> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
> + NetClientState *nc = qemu_find_netdev(netdev_id);
> + Error *err = NULL;
> +
> + /* FIXME: Not support multiple queues */
> + if (!nc || nc->queue_index > 1) {
> + return;
> + }
> + qov = qmp_output_visitor_new();
> + ov = qmp_output_get_visitor(qov);
> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
> + if (err) {
> + goto out;
> + }
> + visit_type_str(ov, &nc->name, "netdev", &err);
> + if (err) {
> + goto out;
> + }
> + visit_type_str(ov, &queue, "queue", &err);
> + if (err) {
> + goto out;
> + }
> + visit_end_struct(ov, &err);
> + if (err) {
> + goto out;
> + }
> + obj = qmp_output_get_qobject(qov);
> + g_assert(obj != NULL);
> + qdict = qobject_to_qdict(obj);
> + qmp_output_visitor_cleanup(qov);
> +
> + qiv = qmp_input_visitor_new(obj);
> + iv = qmp_input_get_visitor(qiv);
> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
> + qmp_input_visitor_cleanup(qiv);
> + qobject_decref(obj);
> +out:
> + g_free(queue);
> + if (err) {
> + error_propagate(errp, err);
> + }
> +}
> +
> static void filter_buffer_init(Object *obj)
> {
> object_property_add(obj, "interval", "int",
> diff --git a/net/net.c b/net/net.c
> index ade6051..b36d49f 100644
> --- a/net/net.c
> +++ b/net/net.c
> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void *object, int is_netdev, Error **errp)
> }
> return -1;
> }
> +
> + if (is_netdev) {
> + const Netdev *netdev = object;
> +
> + netdev_add_default_filter_buffer(netdev->id,
> + NET_FILTER_DIRECTION_RX,
> + errp);
> + }
I'm not sure if add this default to all netdev is a good idea, because
in most cases, they do not need the default filter, although it is a
nop, it still go though the filter. Maybe we can add a argument to
netdev to easily turn if on, for example,
-netdev default-filter=true
if not supplied, default to off, that is, by default no default filter
will be attached to the netdev.
> return 0;
> }
>
>
--
Thanks,
Yang
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 35/39] filter-buffer: Accept zero interval
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 35/39] filter-buffer: Accept zero interval zhanghailiang
@ 2015-11-27 11:42 ` Yang Hongyang
0 siblings, 0 replies; 95+ messages in thread
From: Yang Hongyang @ 2015-11-27 11:42 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah
On 2015年11月24日 17:25, zhanghailiang wrote:
> For default buffer filter, its 'interval' value is zero,
> so here we should accept zero interval.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
Reviewed-by: Yang Hongyang <hongyang.yang@easystack.cn>
> ---
> v11:
> - Add comment
> v10:
> - new patch
> ---
> net/filter-buffer.c | 10 ----------
> 1 file changed, 10 deletions(-)
>
> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
> index 195af68..5cab4fc 100644
> --- a/net/filter-buffer.c
> +++ b/net/filter-buffer.c
> @@ -116,16 +116,6 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
> FilterBufferState *s = FILTER_BUFFER(nf);
> char *path = object_get_canonical_path_component(OBJECT(nf));
>
> - /*
> - * We may want to accept zero interval when VM FT solutions like MC
> - * or COLO use this filter to release packets on demand.
> - */
> - if (!s->interval) {
> - error_setg(errp, QERR_INVALID_PARAMETER_VALUE, "interval",
> - "a non-zero interval");
> - return;
> - }
> -
> s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
> s->is_default = !strcmp(path, "nop");
> if (s->interval) {
>
--
Thanks,
Yang
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 38/39] colo: Use default buffer-filter to buffer and release packets
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 38/39] colo: Use default buffer-filter to buffer and " zhanghailiang
@ 2015-11-27 12:51 ` Yang Hongyang
2015-11-28 6:15 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Yang Hongyang @ 2015-11-27 12:51 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah
On 2015年11月24日 17:25, zhanghailiang wrote:
> Enable default filter to buffer packets and release the
> packets after a checkpoint.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
> ---
> v11:
> - Use new helper functions to buffer and release packets.
> ---
> migration/colo.c | 7 +++++++
> 1 file changed, 7 insertions(+)
>
> diff --git a/migration/colo.c b/migration/colo.c
> index 79a8d6b..b1b7905 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -19,6 +19,7 @@
> #include "qemu/sockets.h"
> #include "migration/failover.h"
> #include "qapi-event.h"
> +#include "net/filter.h"
>
> /*
> * The delay time before qemu begin the procedure of default failover treatment.
> @@ -131,6 +132,8 @@ static void primary_vm_do_failover(void)
> "old_state: %d", old_state);
> return;
> }
> + /* Don't buffer any packets while exited COLO */
> + qemu_set_default_filter_buffers(false);
You might need to release packets before exited COLO ?
> }
>
> void colo_do_failover(MigrationState *s)
> @@ -290,6 +293,8 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
> goto out;
> }
>
> + qemu_release_default_filters_packets();
> +
> if (colo_shutdown) {
> colo_ctl_put(s->to_dst_file, COLO_COMMAND_GUEST_SHUTDOWN, 0);
> qemu_fflush(s->to_dst_file);
> @@ -367,6 +372,8 @@ static void colo_process_checkpoint(MigrationState *s)
> error_report("Failed to allocate colo buffer!");
> goto out;
> }
> + /* Begin to buffer packets that sent by VM */
> + qemu_set_default_filter_buffers(true);
>
> qemu_mutex_lock_iothread();
> vm_start();
>
--
Thanks,
Yang
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-11-27 11:39 ` Yang Hongyang
@ 2015-11-28 5:55 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-11-28 5:55 UTC (permalink / raw)
To: Yang Hongyang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah
On 2015/11/27 19:39, Yang Hongyang wrote:
> On 2015年11月24日 17:25, zhanghailiang wrote:
>> We add each netdev a default filter-buffer, which will be used for COLO
>> or Micro-checkpoint to buffer VM's packets. The name of default filter-buffer
>> is 'nop'.
>> For the default filter-buffer, it will not buffer any packets in default.
>> So it has no side effect for the netdev.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Cc: Jason Wang <jasowang@redhat.com>
>> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
>> ---
>> v11:
>> - New patch
>> ---
> [...]
>> static void filter_buffer_flush(NetFilterState *nf)
>> @@ -65,6 +73,10 @@ static ssize_t filter_buffer_receive_iov(NetFilterState *nf,
>> {
>> FilterBufferState *s = FILTER_BUFFER(nf);
>>
>> + /* Don't buffer any packets if the filter is not enabled */
>> + if (!s->enable_buffer) {
>> + return 0;
>> + }
>> /*
>> * We return size when buffer a packet, the sender will take it as
>> * a already sent packet, so sent_cb should not be called later.
>> @@ -102,6 +114,7 @@ static void filter_buffer_cleanup(NetFilterState *nf)
>> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>> {
>> FilterBufferState *s = FILTER_BUFFER(nf);
>> + char *path = object_get_canonical_path_component(OBJECT(nf));
>
> path should be freed after use.
>
>>
>> /*
>> * We may want to accept zero interval when VM FT solutions like MC
>> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>> }
>>
>> s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
>> + s->is_default = !strcmp(path, "nop");
>
> free(path);
Good catch, :)
>
>> if (s->interval) {
>> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
>> filter_buffer_release_timer, nf);
>> @@ -163,6 +177,66 @@ out:
>> error_propagate(errp, local_err);
>> }
>>
>> +/*
>> +* This will be used by COLO or MC FT, for which they will need
>> +* to buffer the packets of VM's net devices, Here we add a default
>> +* buffer filter for each netdev. The name of default buffer filter is
>> +* 'nop'
>> +*/
>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>> + NetFilterDirection direction,
>> + Error **errp)
>> +{
>> + QmpOutputVisitor *qov;
>> + QmpInputVisitor *qiv;
>> + Visitor *ov, *iv;
>> + QObject *obj = NULL;
>> + QDict *qdict;
>> + void *dummy = NULL;
>> + const char *id = "nop";
>> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
>> + NetClientState *nc = qemu_find_netdev(netdev_id);
>> + Error *err = NULL;
>> +
>> + /* FIXME: Not support multiple queues */
>> + if (!nc || nc->queue_index > 1) {
>> + return;
>> + }
>> + qov = qmp_output_visitor_new();
>> + ov = qmp_output_get_visitor(qov);
>> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
>> + if (err) {
>> + goto out;
>> + }
>> + visit_type_str(ov, &nc->name, "netdev", &err);
>> + if (err) {
>> + goto out;
>> + }
>> + visit_type_str(ov, &queue, "queue", &err);
>> + if (err) {
>> + goto out;
>> + }
>> + visit_end_struct(ov, &err);
>> + if (err) {
>> + goto out;
>> + }
>> + obj = qmp_output_get_qobject(qov);
>> + g_assert(obj != NULL);
>> + qdict = qobject_to_qdict(obj);
>> + qmp_output_visitor_cleanup(qov);
>> +
>> + qiv = qmp_input_visitor_new(obj);
>> + iv = qmp_input_get_visitor(qiv);
>> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
>> + qmp_input_visitor_cleanup(qiv);
>> + qobject_decref(obj);
>> +out:
>> + g_free(queue);
>> + if (err) {
>> + error_propagate(errp, err);
>> + }
>> +}
>> +
>> static void filter_buffer_init(Object *obj)
>> {
>> object_property_add(obj, "interval", "int",
>> diff --git a/net/net.c b/net/net.c
>> index ade6051..b36d49f 100644
>> --- a/net/net.c
>> +++ b/net/net.c
>> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void *object, int is_netdev, Error **errp)
>> }
>> return -1;
>> }
>> +
>> + if (is_netdev) {
>> + const Netdev *netdev = object;
>> +
>> + netdev_add_default_filter_buffer(netdev->id,
>> + NET_FILTER_DIRECTION_RX,
>> + errp);
>> + }
>
> I'm not sure if add this default to all netdev is a good idea, because
> in most cases, they do not need the default filter, although it is a
> nop, it still go though the filter. Maybe we can add a argument to
> netdev to easily turn if on, for example,
> -netdev default-filter=true
>
Hmm, Adding default filter is suggested by Jason, and yes, it still go
through the default filter though it does nothing, which is not graceful.
> if not supplied, default to off, that is, by default no default filter
> will be attached to the netdev.
>
That's a good idea, and we should also show the off/on info for 'info network' command.
Thanks,
zhanghailiang
>> return 0;
>> }
>>
>>
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 38/39] colo: Use default buffer-filter to buffer and release packets
2015-11-27 12:51 ` Yang Hongyang
@ 2015-11-28 6:15 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-11-28 6:15 UTC (permalink / raw)
To: Yang Hongyang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah
On 2015/11/27 20:51, Yang Hongyang wrote:
>
>
> On 2015年11月24日 17:25, zhanghailiang wrote:
>> Enable default filter to buffer packets and release the
>> packets after a checkpoint.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Cc: Jason Wang <jasowang@redhat.com>
>> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
>> ---
>> v11:
>> - Use new helper functions to buffer and release packets.
>> ---
>> migration/colo.c | 7 +++++++
>> 1 file changed, 7 insertions(+)
>>
>> diff --git a/migration/colo.c b/migration/colo.c
>> index 79a8d6b..b1b7905 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -19,6 +19,7 @@
>> #include "qemu/sockets.h"
>> #include "migration/failover.h"
>> #include "qapi-event.h"
>> +#include "net/filter.h"
>>
>> /*
>> * The delay time before qemu begin the procedure of default failover treatment.
>> @@ -131,6 +132,8 @@ static void primary_vm_do_failover(void)
>> "old_state: %d", old_state);
>> return;
>> }
>> + /* Don't buffer any packets while exited COLO */
>> + qemu_set_default_filter_buffers(false);
>
> You might need to release packets before exited COLO ?
>
Yes, we should flush the queue and release the buffered packets.
I will fix this it in next version, thanks.
>> }
>>
>> void colo_do_failover(MigrationState *s)
>> @@ -290,6 +293,8 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
>> goto out;
>> }
>>
>> + qemu_release_default_filters_packets();
>> +
>> if (colo_shutdown) {
>> colo_ctl_put(s->to_dst_file, COLO_COMMAND_GUEST_SHUTDOWN, 0);
>> qemu_fflush(s->to_dst_file);
>> @@ -367,6 +372,8 @@ static void colo_process_checkpoint(MigrationState *s)
>> error_report("Failed to allocate colo buffer!");
>> goto out;
>> }
>> + /* Begin to buffer packets that sent by VM */
>> + qemu_set_default_filter_buffers(true);
>>
>> qemu_mutex_lock_iothread();
>> vm_start();
>>
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev zhanghailiang
2015-11-27 11:39 ` Yang Hongyang
@ 2015-11-30 1:19 ` Li Zhijian
2015-12-01 8:56 ` Hailiang Zhang
2015-12-03 1:17 ` Wen Congyang
2 siblings, 1 reply; 95+ messages in thread
From: Li Zhijian @ 2015-11-30 1:19 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: quintela, Jason Wang, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 11/24/2015 05:25 PM, zhanghailiang wrote:
> We add each netdev a default filter-buffer, which will be used for COLO
> or Micro-checkpoint to buffer VM's packets. The name of default filter-buffer
> is 'nop'.
> For the default filter-buffer, it will not buffer any packets in default.
> So it has no side effect for the netdev.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
> ---
> v11:
> - New patch
> ---
> include/net/filter.h | 3 +++
> net/filter-buffer.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> net/net.c | 8 ++++++
> 3 files changed, 85 insertions(+)
>
> diff --git a/include/net/filter.h b/include/net/filter.h
> index 2deda36..01a7e90 100644
> --- a/include/net/filter.h
> +++ b/include/net/filter.h
> @@ -74,4 +74,7 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender,
> int iovcnt,
> void *opaque);
>
> +void netdev_add_default_filter_buffer(const char *netdev_id,
> + NetFilterDirection direction,
> + Error **errp);
> #endif /* QEMU_NET_FILTER_H */
> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
> index 57be149..195af68 100644
> --- a/net/filter-buffer.c
> +++ b/net/filter-buffer.c
> @@ -14,6 +14,12 @@
> #include "qapi/qmp/qerror.h"
> #include "qapi-visit.h"
> #include "qom/object.h"
> +#include "net/net.h"
> +#include "qapi/qmp/qdict.h"
> +#include "qapi/qmp-output-visitor.h"
> +#include "qapi/qmp-input-visitor.h"
> +#include "monitor/monitor.h"
> +#include "qmp-commands.h"
>
> #define TYPE_FILTER_BUFFER "filter-buffer"
>
> @@ -26,6 +32,8 @@ typedef struct FilterBufferState {
> NetQueue *incoming_queue;
> uint32_t interval;
> QEMUTimer release_timer;
> + bool is_default;
> + bool enable_buffer;
> } FilterBufferState;
>
> static void filter_buffer_flush(NetFilterState *nf)
> @@ -65,6 +73,10 @@ static ssize_t filter_buffer_receive_iov(NetFilterState *nf,
> {
> FilterBufferState *s = FILTER_BUFFER(nf);
>
> + /* Don't buffer any packets if the filter is not enabled */
> + if (!s->enable_buffer) {
> + return 0;
> + }
> /*
> * We return size when buffer a packet, the sender will take it as
> * a already sent packet, so sent_cb should not be called later.
> @@ -102,6 +114,7 @@ static void filter_buffer_cleanup(NetFilterState *nf)
> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
> {
> FilterBufferState *s = FILTER_BUFFER(nf);
> + char *path = object_get_canonical_path_component(OBJECT(nf));
>
> /*
> * We may want to accept zero interval when VM FT solutions like MC
> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
> }
>
> s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
> + s->is_default = !strcmp(path, "nop");
> if (s->interval) {
> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
> filter_buffer_release_timer, nf);
> @@ -163,6 +177,66 @@ out:
> error_propagate(errp, local_err);
> }
>
> +/*
> +* This will be used by COLO or MC FT, for which they will need
> +* to buffer the packets of VM's net devices, Here we add a default
> +* buffer filter for each netdev. The name of default buffer filter is
> +* 'nop'
> +*/
> +void netdev_add_default_filter_buffer(const char *netdev_id,
> + NetFilterDirection direction,
> + Error **errp)
> +{
> + QmpOutputVisitor *qov;
> + QmpInputVisitor *qiv;
> + Visitor *ov, *iv;
> + QObject *obj = NULL;
> + QDict *qdict;
> + void *dummy = NULL;
> + const char *id = "nop";
> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
> + NetClientState *nc = qemu_find_netdev(netdev_id);
> + Error *err = NULL;
> +
> + /* FIXME: Not support multiple queues */
> + if (!nc || nc->queue_index > 1) {
queue memory leaks ?
Thanks
Li
> + return;
> + }
> + qov = qmp_output_visitor_new();
> + ov = qmp_output_get_visitor(qov);
> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
> + if (err) {
> + goto out;
> + }
> + visit_type_str(ov, &nc->name, "netdev", &err);
> + if (err) {
> + goto out;
> + }
> + visit_type_str(ov, &queue, "queue", &err);
> + if (err) {
> + goto out;
> + }
> + visit_end_struct(ov, &err);
> + if (err) {
> + goto out;
> + }
> + obj = qmp_output_get_qobject(qov);
> + g_assert(obj != NULL);
> + qdict = qobject_to_qdict(obj);
> + qmp_output_visitor_cleanup(qov);
> +
> + qiv = qmp_input_visitor_new(obj);
> + iv = qmp_input_get_visitor(qiv);
> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
> + qmp_input_visitor_cleanup(qiv);
> + qobject_decref(obj);
> +out:
> + g_free(queue);
> + if (err) {
> + error_propagate(errp, err);
> + }
> +}
> +
> static void filter_buffer_init(Object *obj)
> {
> object_property_add(obj, "interval", "int",
> diff --git a/net/net.c b/net/net.c
> index ade6051..b36d49f 100644
> --- a/net/net.c
> +++ b/net/net.c
> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void *object, int is_netdev, Error **errp)
> }
> return -1;
> }
> +
> + if (is_netdev) {
> + const Netdev *netdev = object;
> +
> + netdev_add_default_filter_buffer(netdev->id,
> + NET_FILTER_DIRECTION_RX,
> + errp);
> + }
> return 0;
> }
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-11-30 1:19 ` Li Zhijian
@ 2015-12-01 8:56 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-01 8:56 UTC (permalink / raw)
To: Li Zhijian, qemu-devel
Cc: quintela, Jason Wang, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/11/30 9:19, Li Zhijian wrote:
> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>> We add each netdev a default filter-buffer, which will be used for COLO
>> or Micro-checkpoint to buffer VM's packets. The name of default filter-buffer
>> is 'nop'.
>> For the default filter-buffer, it will not buffer any packets in default.
>> So it has no side effect for the netdev.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Cc: Jason Wang <jasowang@redhat.com>
>> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
>> ---
>> v11:
>> - New patch
>> ---
>> include/net/filter.h | 3 +++
>> net/filter-buffer.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>> net/net.c | 8 ++++++
>> 3 files changed, 85 insertions(+)
>>
>> diff --git a/include/net/filter.h b/include/net/filter.h
>> index 2deda36..01a7e90 100644
>> --- a/include/net/filter.h
>> +++ b/include/net/filter.h
>> @@ -74,4 +74,7 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender,
>> int iovcnt,
>> void *opaque);
>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>> + NetFilterDirection direction,
>> + Error **errp);
>> #endif /* QEMU_NET_FILTER_H */
>> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
>> index 57be149..195af68 100644
>> --- a/net/filter-buffer.c
>> +++ b/net/filter-buffer.c
>> @@ -14,6 +14,12 @@
>> #include "qapi/qmp/qerror.h"
>> #include "qapi-visit.h"
>> #include "qom/object.h"
>> +#include "net/net.h"
>> +#include "qapi/qmp/qdict.h"
>> +#include "qapi/qmp-output-visitor.h"
>> +#include "qapi/qmp-input-visitor.h"
>> +#include "monitor/monitor.h"
>> +#include "qmp-commands.h"
>> #define TYPE_FILTER_BUFFER "filter-buffer"
>> @@ -26,6 +32,8 @@ typedef struct FilterBufferState {
>> NetQueue *incoming_queue;
>> uint32_t interval;
>> QEMUTimer release_timer;
>> + bool is_default;
>> + bool enable_buffer;
>> } FilterBufferState;
>> static void filter_buffer_flush(NetFilterState *nf)
>> @@ -65,6 +73,10 @@ static ssize_t filter_buffer_receive_iov(NetFilterState *nf,
>> {
>> FilterBufferState *s = FILTER_BUFFER(nf);
>> + /* Don't buffer any packets if the filter is not enabled */
>> + if (!s->enable_buffer) {
>> + return 0;
>> + }
>> /*
>> * We return size when buffer a packet, the sender will take it as
>> * a already sent packet, so sent_cb should not be called later.
>> @@ -102,6 +114,7 @@ static void filter_buffer_cleanup(NetFilterState *nf)
>> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>> {
>> FilterBufferState *s = FILTER_BUFFER(nf);
>> + char *path = object_get_canonical_path_component(OBJECT(nf));
>> /*
>> * We may want to accept zero interval when VM FT solutions like MC
>> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>> }
>> s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
>> + s->is_default = !strcmp(path, "nop");
>> if (s->interval) {
>> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
>> filter_buffer_release_timer, nf);
>> @@ -163,6 +177,66 @@ out:
>> error_propagate(errp, local_err);
>> }
>> +/*
>> +* This will be used by COLO or MC FT, for which they will need
>> +* to buffer the packets of VM's net devices, Here we add a default
>> +* buffer filter for each netdev. The name of default buffer filter is
>> +* 'nop'
>> +*/
>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>> + NetFilterDirection direction,
>> + Error **errp)
>> +{
>> + QmpOutputVisitor *qov;
>> + QmpInputVisitor *qiv;
>> + Visitor *ov, *iv;
>> + QObject *obj = NULL;
>> + QDict *qdict;
>> + void *dummy = NULL;
>> + const char *id = "nop";
>> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
>> + NetClientState *nc = qemu_find_netdev(netdev_id);
>> + Error *err = NULL;
>> +
>> + /* FIXME: Not support multiple queues */
>> + if (!nc || nc->queue_index > 1) {
>
> queue memory leaks ?
>
Good catch, i will fix it in next version. Thanks.
>
> Thanks
> Li
>
>> + return;
>> + }
>> + qov = qmp_output_visitor_new();
>> + ov = qmp_output_get_visitor(qov);
>> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
>> + if (err) {
>> + goto out;
>> + }
>> + visit_type_str(ov, &nc->name, "netdev", &err);
>> + if (err) {
>> + goto out;
>> + }
>> + visit_type_str(ov, &queue, "queue", &err);
>> + if (err) {
>> + goto out;
>> + }
>> + visit_end_struct(ov, &err);
>> + if (err) {
>> + goto out;
>> + }
>> + obj = qmp_output_get_qobject(qov);
>> + g_assert(obj != NULL);
>> + qdict = qobject_to_qdict(obj);
>> + qmp_output_visitor_cleanup(qov);
>> +
>> + qiv = qmp_input_visitor_new(obj);
>> + iv = qmp_input_get_visitor(qiv);
>> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
>> + qmp_input_visitor_cleanup(qiv);
>> + qobject_decref(obj);
>> +out:
>> + g_free(queue);
>> + if (err) {
>> + error_propagate(errp, err);
>> + }
>> +}
>> +
>> static void filter_buffer_init(Object *obj)
>> {
>> object_property_add(obj, "interval", "int",
>> diff --git a/net/net.c b/net/net.c
>> index ade6051..b36d49f 100644
>> --- a/net/net.c
>> +++ b/net/net.c
>> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void *object, int is_netdev, Error **errp)
>> }
>> return -1;
>> }
>> +
>> + if (is_netdev) {
>> + const Netdev *netdev = object;
>> +
>> + netdev_add_default_filter_buffer(netdev->id,
>> + NET_FILTER_DIRECTION_RX,
>> + errp);
>> + }
>> return 0;
>> }
>
>
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 18/39] COLO: Flush PVM's cached RAM into SVM's memory
2015-11-27 5:29 ` Li Zhijian
@ 2015-12-01 12:02 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-01 12:02 UTC (permalink / raw)
To: Li Zhijian, qemu-devel
Cc: quintela, yunhong.jiang, eddie.dong, peter.huangpeng, dgilbert,
arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/11/27 13:29, Li Zhijian wrote:
> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>
>> During the time of VM's running, PVM may dirty some pages, we will transfer
>> PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint
>> time. So, the content of SVM's RAM cache will always be some with PVM's memory
>> after checkpoint.
>>
>> Instead of flushing all content of PVM's RAM cache into SVM's MEMORY,
>> we do this in a more efficient way:
>> Only flush any page that dirtied by PVM since last checkpoint.
>> In this way, we can ensure SVM's memory same with PVM's.
>>
>> Besides, we must ensure flush RAM cache before load device state.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>> v11:
>> - Move the place of 'need_flush' (Dave's suggestion)
>> - Remove unused 'DPRINTF("Flush ram_cache\n")'
>> v10:
>> - trace the number of dirty pages that be received.
>> ---
>> include/migration/migration.h | 1 +
>> migration/colo.c | 2 --
>> migration/ram.c | 37 +++++++++++++++++++++++++++++++++++++
>> trace-events | 1 +
>> 4 files changed, 39 insertions(+), 2 deletions(-)
>>
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index e41372d..221176b 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -336,4 +336,5 @@ PostcopyState postcopy_state_set(PostcopyState new_state);
>> /* ram cache */
>> int colo_init_ram_cache(void);
>> void colo_release_ram_cache(void);
>> +void colo_flush_ram_cache(void);
>> #endif
>> diff --git a/migration/colo.c b/migration/colo.c
>> index 5ac8ff2..4095d97 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -385,8 +385,6 @@ void *colo_process_incoming_thread(void *opaque)
>> }
>> qemu_mutex_unlock_iothread();
>> - /* TODO: flush vm state */
>> -
>> ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_LOADED, 0);
>> if (ret < 0) {
>> goto out;
>> diff --git a/migration/ram.c b/migration/ram.c
>> index da6bbd6..4f37144 100644
>> --- a/migration/ram.c
>> +++ b/migration/ram.c
>> @@ -2448,6 +2448,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> * be atomic
>> */
>> bool postcopy_running = postcopy_state_get() >= POSTCOPY_INCOMING_LISTENING;
>> + bool need_flush = false;
>> seq_iter++;
>> @@ -2482,6 +2483,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> /* After going into COLO, we should load the Page into colo_cache */
>> if (ram_cache_enable) {
>> host = colo_cache_from_block_offset(block, addr);
>> + need_flush = true;
>> } else {
>> host = host_from_ram_block_offset(block, addr);
>> }
>> @@ -2575,6 +2577,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> }
>> rcu_read_unlock();
>> +
>> + if (!ret && ram_cache_enable && need_flush) {
>> + colo_flush_ram_cache();
>> + }
>> DPRINTF("Completed load of VM with exit code %d seq iteration "
>> "%" PRIu64 "\n", ret, seq_iter);
>> return ret;
>> @@ -2647,6 +2653,37 @@ void colo_release_ram_cache(void)
>> rcu_read_unlock();
>> }
>> +/*
>> + * Flush content of RAM cache into SVM's memory.
>> + * Only flush the pages that be dirtied by PVM or SVM or both.
>> + */
>> +void colo_flush_ram_cache(void)
>> +{
>> + RAMBlock *block = NULL;
>> + void *dst_host;
>> + void *src_host;
>> + ram_addr_t offset = 0;
>
> coding sytle: double space
>
Fixed, thanks.
> thanks
> Li
>
>> +
>> + trace_colo_flush_ram_cache(migration_dirty_pages);
>> + rcu_read_lock();
>> + block = QLIST_FIRST_RCU(&ram_list.blocks);
>> + while (block) {
>> + ram_addr_t ram_addr_abs;
>> + offset = migration_bitmap_find_dirty(block, offset, &ram_addr_abs);
>> + migration_bitmap_clear_dirty(ram_addr_abs);
>> + if (offset >= block->used_length) {
>> + offset = 0;
>> + block = QLIST_NEXT_RCU(block, next);
>> + } else {
>> + dst_host = block->host + offset;
>> + src_host = block->colo_cache + offset;
>> + memcpy(dst_host, src_host, TARGET_PAGE_SIZE);
>> + }
>> + }
>> + rcu_read_unlock();
>> + assert(migration_dirty_pages == 0);
>> +}
>> +
>> static SaveVMHandlers savevm_ram_handlers = {
>> .save_live_setup = ram_save_setup,
>> .save_live_iterate = ram_save_iterate,
>> diff --git a/trace-events b/trace-events
>> index f8a0959..f158d2a 100644
>> --- a/trace-events
>> +++ b/trace-events
>> @@ -1264,6 +1264,7 @@ migration_throttle(void) ""
>> ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x"
>> ram_postcopy_send_discard_bitmap(void) ""
>> ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: %zx len: %zx"
>> +colo_flush_ram_cache(uint64_t dirty_pages) "dirty_pages %" PRIu64""
>> # hw/display/qxl.c
>> disable qxl_interface_set_mm_time(int qid, uint32_t mm_time) "%d %d"
>
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 32/39] COLO: Separate the process of saving/loading ram and device state
2015-11-27 5:10 ` Li Zhijian
@ 2015-12-01 12:07 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-01 12:07 UTC (permalink / raw)
To: Li Zhijian, qemu-devel
Cc: quintela, yunhong.jiang, eddie.dong, peter.huangpeng, dgilbert,
arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/11/27 13:10, Li Zhijian wrote:
> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>> We separate the process of saving/loading ram and device state when do checkpoint,
>> we add new helpers for save/load ram/device. With this change, we can directly
>> transfer ram from master to slave without using QEMUSizeBuffer as assistant,
>> which also reduce the size of extra memory been used during checkpoint.
>>
>> Besides, we move the colo_flush_ram_cache to the proper position after the
>> above change.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> ---
>> v11:
>> - Remove load configuration section in qemu_loadvm_state_begin()
>> ---
>> include/sysemu/sysemu.h | 6 +++
>> migration/colo.c | 43 ++++++++++++----
>> migration/ram.c | 5 --
>> migration/savevm.c | 132 ++++++++++++++++++++++++++++++++++++++++++++++--
>> 4 files changed, 168 insertions(+), 18 deletions(-)
>>
>> diff --git a/include/sysemu/sysemu.h b/include/sysemu/sysemu.h
>> index 91eeda3..5deae53 100644
>> --- a/include/sysemu/sysemu.h
>> +++ b/include/sysemu/sysemu.h
>> @@ -133,7 +133,13 @@ void qemu_savevm_send_postcopy_ram_discard(QEMUFile *f, const char *name,
>> uint64_t *start_list,
>> uint64_t *length_list);
>> +int qemu_save_ram_precopy(QEMUFile *f);
>> +int qemu_save_device_state(QEMUFile *f);
>> +
>> int qemu_loadvm_state(QEMUFile *f);
>> +int qemu_loadvm_state_begin(QEMUFile *f);
>> +int qemu_load_ram_state(QEMUFile *f);
>> +int qemu_load_device_state(QEMUFile *f);
>> typedef enum DisplayType
>> {
>> diff --git a/migration/colo.c b/migration/colo.c
>> index 3866e86..f7f349b 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -247,21 +247,32 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
>> goto out;
>> }
>> + ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> /* Disable block migration */
>> s->params.blk = 0;
>> s->params.shared = 0;
>> - qemu_savevm_state_header(trans);
>> - qemu_savevm_state_begin(trans, &s->params);
>> - qemu_mutex_lock_iothread();
>> - qemu_savevm_state_complete_precopy(trans, false);
>> - qemu_mutex_unlock_iothread();
>> -
>> - qemu_fflush(trans);
>> + qemu_savevm_state_begin(s->to_dst_file, &s->params);
>> + ret = qemu_file_get_error(s->to_dst_file);
>> + if (ret < 0) {
>> + error_report("save vm state begin error\n");
>> + goto out;
>> + }
>> - ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SEND, 0);
>> + qemu_mutex_lock_iothread();
>> + /* Note: device state is saved into buffer */
>> + ret = qemu_save_device_state(trans);
>> if (ret < 0) {
>> + error_report("save device state error\n");
>> + qemu_mutex_unlock_iothread();
>> goto out;
>> }
>> + qemu_fflush(trans);
>> + qemu_save_ram_precopy(s->to_dst_file);
>> + qemu_mutex_unlock_iothread();
>> +
>> /* we send the total size of the vmstate first */
>> size = qsb_get_length(buffer);
>> ret = colo_ctl_put(s->to_dst_file, COLO_COMMAND_VMSTATE_SIZE, size);
>> @@ -548,6 +559,16 @@ void *colo_process_incoming_thread(void *opaque)
>> goto out;
>> }
>> + ret = qemu_loadvm_state_begin(mis->from_src_file);
>> + if (ret < 0) {
>> + error_report("load vm state begin error, ret=%d", ret);
>> + goto out;
>> + }
>> + ret = qemu_load_ram_state(mis->from_src_file);
>> + if (ret < 0) {
>> + error_report("load ram state error");
>> + goto out;
>> + }
>> /* read the VM state total size first */
>> ret = colo_ctl_get(mis->from_src_file,
>> COLO_COMMAND_VMSTATE_SIZE, &value);
>> @@ -580,8 +601,10 @@ void *colo_process_incoming_thread(void *opaque)
>> qemu_mutex_lock_iothread();
>> qemu_system_reset(VMRESET_SILENT);
>> vmstate_loading = true;
>> - if (qemu_loadvm_state(fb) < 0) {
>> - error_report("COLO: loadvm failed");
>> + colo_flush_ram_cache();
>> + ret = qemu_load_device_state(fb);
>> + if (ret < 0) {
>> + error_report("COLO: load device state failed\n");
>> vmstate_loading = false;
>> qemu_mutex_unlock_iothread();
>> goto out;
>> diff --git a/migration/ram.c b/migration/ram.c
>> index 4f37144..06a738b 100644
>> --- a/migration/ram.c
>> +++ b/migration/ram.c
>> @@ -2448,7 +2448,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> * be atomic
>> */
>> bool postcopy_running = postcopy_state_get() >= POSTCOPY_INCOMING_LISTENING;
>> - bool need_flush = false;
>> seq_iter++;
>> @@ -2483,7 +2482,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> /* After going into COLO, we should load the Page into colo_cache */
>> if (ram_cache_enable) {
>> host = colo_cache_from_block_offset(block, addr);
>> - need_flush = true;
>> } else {
>> host = host_from_ram_block_offset(block, addr);
>> }
>> @@ -2578,9 +2576,6 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> rcu_read_unlock();
>> - if (!ret && ram_cache_enable && need_flush) {
>> - colo_flush_ram_cache();
>> - }
>> DPRINTF("Completed load of VM with exit code %d seq iteration "
>> "%" PRIu64 "\n", ret, seq_iter);
>> return ret;
>> diff --git a/migration/savevm.c b/migration/savevm.c
>> index c7c26d8..949caf0 100644
>> --- a/migration/savevm.c
>> +++ b/migration/savevm.c
>> @@ -50,6 +50,7 @@
>> #include "qemu/iov.h"
>> #include "block/snapshot.h"
>> #include "block/qapi.h"
>> +#include "migration/colo.h"
>> #ifndef ETH_P_RARP
>> @@ -923,6 +924,10 @@ void qemu_savevm_state_begin(QEMUFile *f,
>> break;
>> }
>> }
>> + if (migration_in_colo_state()) {
>> + qemu_put_byte(f, QEMU_VM_EOF);
>> + qemu_fflush(f);
>> + }
>> }
>> /*
>> @@ -1192,13 +1197,44 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
>> return ret;
>> }
>> -static int qemu_save_device_state(QEMUFile *f)
>> +int qemu_save_ram_precopy(QEMUFile *f)
>> {
>> SaveStateEntry *se;
>> + int ret = 0;
>> - qemu_put_be32(f, QEMU_VM_FILE_MAGIC);
>> - qemu_put_be32(f, QEMU_VM_FILE_VERSION);
>> + QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
>> + if (!se->ops || !se->ops->save_live_complete_precopy) {
>> + continue;
>> + }
>> + if (se->ops && se->ops->is_active) {
>> + if (!se->ops->is_active(se->opaque)) {
>> + continue;
>> + }
>> + }
>> + trace_savevm_section_start(se->idstr, se->section_id);
>> + save_section_header(f, se, QEMU_VM_SECTION_END);
>> +
>> + ret = se->ops->save_live_complete_precopy(f, se->opaque);
>> + trace_savevm_section_end(se->idstr, se->section_id, ret);
>> + save_section_footer(f, se);
>> + if (ret < 0) {
>> + qemu_file_set_error(f, ret);
>> + return ret;
>> + }
>> + }
>> + qemu_put_byte(f, QEMU_VM_EOF);
>> +
>> + return 0;
>> +}
>> +
>> +int qemu_save_device_state(QEMUFile *f)
>> +{
>> + SaveStateEntry *se;
>> +
>> + if (!migration_in_colo_state()) {
>> + qemu_savevm_state_header(f);
>> + }
>> cpu_synchronize_all_states();
>> QTAILQ_FOREACH(se, &savevm_state.handlers, entry) {
>> @@ -1938,6 +1974,96 @@ int qemu_loadvm_state(QEMUFile *f)
>> return ret;
>> }
>> +int qemu_loadvm_state_begin(QEMUFile *f)
>> +{
>> + uint8_t section_type;
>> + int ret = -1;
>> + MigrationIncomingState *mis = migration_incoming_get_current();
>> +
>> + if (!mis) {
>> + error_report("qemu_loadvm_state_begin");
>> + return -EINVAL;
>> + }
>> + /* CleanUp */
>> + loadvm_free_handlers(mis);
>> +
>> + if (qemu_savevm_state_blocked(NULL)) {
>> + return -EINVAL;
>> + }
>> +
>> + while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
>> + if (section_type != QEMU_VM_SECTION_START) {
>> + error_report("QEMU_VM_SECTION_START");
>> + ret = -EINVAL;
>> + goto out;
>> + }
>> + ret = qemu_loadvm_section_start_full(f, mis);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> + }
>> + ret = qemu_file_get_error(f);
>> + if (ret == 0) {
>> + return 0;
>> + }
>> +out:
>> + return ret;
>> +}
>> +
>> +int qemu_load_ram_state(QEMUFile *f)
>> +{
>> + uint8_t section_type;
>> + MigrationIncomingState *mis = migration_incoming_get_current();
>> + int ret = -1;
>> +
>> + while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
>> + if (section_type != QEMU_VM_SECTION_PART &&
>> + section_type != QEMU_VM_SECTION_END) {
>> + error_report("load ram state, not get "
>> + "QEMU_VM_SECTION_FULL or QEMU_VM_SECTION_END");
>> + return -EINVAL;
>> + }
>> + ret = qemu_loadvm_section_part_end(f, mis);
>> + if (ret < 0) {
>> + goto out;
>> + }
>> + }
>> + ret = qemu_file_get_error(f);
>> + if (ret == 0) {
>> + return 0;
>> + }
>> +out:
>> + return ret;
>> +}
>> +
>> +int qemu_load_device_state(QEMUFile *f)
>> +{
>> + uint8_t section_type;
>> + MigrationIncomingState *mis = migration_incoming_get_current();
>> + int ret = -1;
>> +
>> + while ((section_type = qemu_get_byte(f)) != QEMU_VM_EOF) {
>> + if (section_type != QEMU_VM_SECTION_FULL) {
>> + error_report("load device state error: "
>> + "Not get QEMU_VM_SECTION_FULL");
>> + return -EINVAL;
>> + }
>> + ret = qemu_loadvm_section_start_full(f, mis);
>> + if (ret < 0) {
>> + goto out;
>> + }
>
> coding sytle：ａbove 4 lines are not aligned
>
>> + }
>> +
>> + ret = qemu_file_get_error(f);
>> +
>> + cpu_synchronize_all_post_init();
>> + if (ret == 0) {
>> + return 0;
>> + }
>
> aligned issue
>
I will fix them all in next version, thanks.
> thanks
> Li
>
>> +out:
>> + return ret;
>> +}
>> +
>> void hmp_savevm(Monitor *mon, const QDict *qdict)
>> {
>> BlockDriverState *bs, *bs1;
>
>
>
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 14/39] ram: Split host_from_stream_offset() into two helper functions
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 14/39] ram: Split host_from_stream_offset() into two helper functions zhanghailiang
@ 2015-12-01 18:19 ` Dr. David Alan Gilbert
2015-12-03 7:19 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-01 18:19 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> Split host_from_stream_offset() into two parts:
> One is to get ram block, which the block idstr may be get from migration
> stream, the other is to get hva (host) address from block and the offset.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
OK, I see why you're doing this from the next patch.
> ---
> v11:
> - New patch
> ---
> migration/ram.c | 29 +++++++++++++++++++++--------
> 1 file changed, 21 insertions(+), 8 deletions(-)
>
> diff --git a/migration/ram.c b/migration/ram.c
> index cfe78aa..a161620 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -2136,9 +2136,9 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
> * offset: Offset within the block
> * flags: Page flags (mostly to see if it's a continuation of previous block)
> */
> -static inline void *host_from_stream_offset(QEMUFile *f,
> - ram_addr_t offset,
> - int flags)
> +static inline RAMBlock *ram_block_from_stream(QEMUFile *f,
> + ram_addr_t offset,
> + int flags)
> {
> static RAMBlock *block = NULL;
> char id[256];
> @@ -2150,22 +2150,31 @@ static inline void *host_from_stream_offset(QEMUFile *f,
> return NULL;
> }
>
> - return block->host + offset;
> + return block;
> }
> -
> len = qemu_get_byte(f);
> qemu_get_buffer(f, (uint8_t *)id, len);
> id[len] = 0;
>
> block = qemu_ram_block_by_name(id);
> if (block && block->max_length > offset) {
> - return block->host + offset;
> + return block;
> }
>
> error_report("Can't find block %s", id);
> return NULL;
> }
>
> +static inline void *host_from_ram_block_offset(RAMBlock *block,
> + ram_addr_t offset)
> +{
> + if (!block) {
> + return NULL;
> + }
> +
> + return block->host + offset;
> +}
That's almost the same as ramblock_ptr in include/exec/ram_addr.h, but
it assert's rather than doing NULL on errors.
I'm not sure about this, but can I suggest:
ram_block_from_stream(QEMUFile *f, int flags)
doesn't have the offset; just finds the block and handles the CONT.
bool offset_in_ramblock(RAMBlock *b, ram_addr_t offset);
actually does the check; put this in exec.c, and declare it in include/exec/ram_addr.h
void *ramblock_ptr_try(RAMBlock *block, ram_addr_t offset)
which returns NULL if offset_in_ramblock fails, and otherwise returns the result
of ramblock_ptr - again put that in include/exec/ram_addr.h
(I'm not sure about this - I almost suggested changing ramblock_ptr to not do
the checks, and just add a call to assert(offset_in_ramblock) before each use, but
that sounded too painful).
Hmm - we check here for block->max_length > offset - where as the check in
ram_addr.h is used_length - I wonder if we should be using used_length?
Dave
> +
> /*
> * If a page (or a whole RDMA chunk) has been
> * determined to be zero, then zap it.
> @@ -2310,7 +2319,9 @@ static int ram_load_postcopy(QEMUFile *f)
> trace_ram_load_postcopy_loop((uint64_t)addr, flags);
> place_needed = false;
> if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE)) {
> - host = host_from_stream_offset(f, addr, flags);
> + RAMBlock *block = ram_block_from_stream(f, addr, flags);
> +
> + host = host_from_ram_block_offset(block, addr);
> if (!host) {
> error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
> ret = -EINVAL;
> @@ -2441,7 +2452,9 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>
> if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE |
> RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) {
> - host = host_from_stream_offset(f, addr, flags);
> + RAMBlock *block = ram_block_from_stream(f, addr, flags);
> +
> + host = host_from_ram_block_offset(block, addr);
> if (!host) {
> error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
> ret = -EINVAL;
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 15/39] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 15/39] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily zhanghailiang
@ 2015-12-01 19:02 ` Dr. David Alan Gilbert
2015-12-03 8:25 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-01 19:02 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> We should not load PVM's state directly into SVM, because there maybe some
> errors happen when SVM is receving data, which will break SVM.
>
> We need to ensure receving all data before load the state into SVM. We use
> an extra memory to cache these data (PVM's ram). The ram cache in secondary side
> is initially the same as SVM/PVM's memory. And in the process of checkpoint,
> we cache the dirty pages of PVM into this ram cache firstly, so this ram cache
> always the same as PVM's memory at every checkpoint, then we flush this cached ram
> to SVM after we receive all PVM's state.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
> v11:
> - Rename 'host_cache' to 'colo_cache' (Dave's suggestion)
> v10:
> - Split the process of dirty pages recording into a new patch
> ---
> include/exec/ram_addr.h | 1 +
> include/migration/migration.h | 4 +++
> migration/colo.c | 10 +++++++
> migration/ram.c | 69 ++++++++++++++++++++++++++++++++++++++++++-
> 4 files changed, 83 insertions(+), 1 deletion(-)
>
> diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
> index 7115154..bb44f66 100644
> --- a/include/exec/ram_addr.h
> +++ b/include/exec/ram_addr.h
> @@ -26,6 +26,7 @@ struct RAMBlock {
> struct rcu_head rcu;
> struct MemoryRegion *mr;
> uint8_t *host;
> + uint8_t *colo_cache; /* For colo, VM's ram cache */
> ram_addr_t offset;
> ram_addr_t used_length;
> ram_addr_t max_length;
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index ba5bcec..e41372d 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -332,4 +332,8 @@ int ram_save_queue_pages(MigrationState *ms, const char *rbname,
> PostcopyState postcopy_state_get(void);
> /* Set the state and return the old state */
> PostcopyState postcopy_state_set(PostcopyState new_state);
> +
> +/* ram cache */
> +int colo_init_ram_cache(void);
> +void colo_release_ram_cache(void);
> #endif
> diff --git a/migration/colo.c b/migration/colo.c
> index 012d8e5..6e933fa 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -304,6 +304,12 @@ void *colo_process_incoming_thread(void *opaque)
> qemu_set_block(qemu_get_fd(mis->from_src_file));
>
>
> + ret = colo_init_ram_cache();
> + if (ret < 0) {
> + error_report("Failed to initialize ram cache");
> + goto out;
> + }
> +
> ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_READY, 0);
> if (ret < 0) {
> goto out;
> @@ -353,6 +359,10 @@ out:
> strerror(-ret));
> }
>
> + qemu_mutex_lock_iothread();
> + colo_release_ram_cache();
> + qemu_mutex_unlock_iothread();
> +
> if (mis->to_src_file) {
> qemu_fclose(mis->to_src_file);
> }
> diff --git a/migration/ram.c b/migration/ram.c
> index a161620..9d946a1 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -223,6 +223,7 @@ static RAMBlock *last_sent_block;
> static ram_addr_t last_offset;
> static QemuMutex migration_bitmap_mutex;
> static uint64_t migration_dirty_pages;
> +static bool ram_cache_enable;
> static uint32_t last_version;
> static bool ram_bulk_stage;
>
> @@ -2175,6 +2176,16 @@ static inline void *host_from_ram_block_offset(RAMBlock *block,
> return block->host + offset;
> }
>
> +static inline void *colo_cache_from_block_offset(RAMBlock *block,
> + ram_addr_t offset)
> +{
> + if (!block) {
> + return NULL;
> + }
> +
> + return block->colo_cache + offset;
> +}
> +
> /*
> * If a page (or a whole RDMA chunk) has been
> * determined to be zero, then zap it.
> @@ -2454,7 +2465,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) {
> RAMBlock *block = ram_block_from_stream(f, addr, flags);
>
> - host = host_from_ram_block_offset(block, addr);
> + /* After going into COLO, we should load the Page into colo_cache */
> + if (ram_cache_enable) {
> + host = colo_cache_from_block_offset(block, addr);
> + } else {
> + host = host_from_ram_block_offset(block, addr);
> + }
> if (!host) {
> error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
> ret = -EINVAL;
> @@ -2550,6 +2566,57 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> return ret;
> }
>
> +/*
> + * colo cache: this is for secondary VM, we cache the whole
> + * memory of the secondary VM, it will be called after first migration.
> + */
> +int colo_init_ram_cache(void)
> +{
> + RAMBlock *block;
> +
> + rcu_read_lock();
> + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
> + block->colo_cache = qemu_anon_ram_alloc(block->used_length, NULL);
> + if (!block->colo_cache) {
> + error_report("%s: Can't alloc memory for colo cache of block %s,"
> + "size %zu", __func__, block->idstr,
> + block->used_length);
Minor one that I didn't spot before; I think that has to be RAM_ADDR_FMT instead of %zu
However, other than that;
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Dave
> + goto out_locked;
> + }
> + memcpy(block->colo_cache, block->host, block->used_length);
> + }
> + rcu_read_unlock();
> + ram_cache_enable = true;
> + return 0;
> +
> +out_locked:
> + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
> + if (block->colo_cache) {
> + qemu_anon_ram_free(block->colo_cache, block->used_length);
> + block->colo_cache = NULL;
> + }
> + }
> +
> + rcu_read_unlock();
> + return -errno;
> +}
> +
> +void colo_release_ram_cache(void)
> +{
> + RAMBlock *block;
> +
> + ram_cache_enable = false;
> +
> + rcu_read_lock();
> + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
> + if (block->colo_cache) {
> + qemu_anon_ram_free(block->colo_cache, block->used_length);
> + block->colo_cache = NULL;
> + }
> + }
> + rcu_read_unlock();
> +}
> +
> static SaveVMHandlers savevm_ram_handlers = {
> .save_live_setup = ram_save_setup,
> .save_live_iterate = ram_save_iterate,
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 16/39] ram/COLO: Record the dirty pages that SVM received
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 16/39] ram/COLO: Record the dirty pages that SVM received zhanghailiang
@ 2015-12-01 19:36 ` Dr. David Alan Gilbert
2015-12-03 8:29 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-01 19:36 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> We record the address of the dirty pages that received,
> it will help flushing pages that cached into SVM.
> We record them by re-using migration dirty bitmap.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> ---
> v11:
> - Split a new helper function from original
> host_from_stream_offset() (Dave's suggestion)
> - Only do recording work in this patch
> v10:
> - New patch split from v9's patch 13
> - Rebase to master to use 'migration_bitmap_rcu'
> ---
> migration/ram.c | 30 ++++++++++++++++++++++++++++++
> 1 file changed, 30 insertions(+)
>
> diff --git a/migration/ram.c b/migration/ram.c
> index 9d946a1..da6bbd6 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -2153,6 +2153,7 @@ static inline RAMBlock *ram_block_from_stream(QEMUFile *f,
>
> return block;
> }
> +
> len = qemu_get_byte(f);
> qemu_get_buffer(f, (uint8_t *)id, len);
> id[len] = 0;
That blank should probably go, but other than that:
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Dave
> @@ -2179,10 +2180,23 @@ static inline void *host_from_ram_block_offset(RAMBlock *block,
> static inline void *colo_cache_from_block_offset(RAMBlock *block,
> ram_addr_t offset)
> {
> + unsigned long *bitmap;
> + long k;
> +
> if (!block) {
> return NULL;
> }
>
> + k = (block->mr->ram_addr + offset) >> TARGET_PAGE_BITS;
> + bitmap = atomic_rcu_read(&migration_bitmap_rcu)->bmap;
> + /*
> + * During colo checkpoint, we need bitmap of these migrated pages.
> + * It help us to decide which pages in ram cache should be flushed
> + * into VM's RAM later.
> + */
> + if (!test_and_set_bit(k, bitmap)) {
> + migration_dirty_pages++;
> + }
> return block->colo_cache + offset;
> }
>
> @@ -2573,6 +2587,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> int colo_init_ram_cache(void)
> {
> RAMBlock *block;
> + int64_t ram_cache_pages = last_ram_offset() >> TARGET_PAGE_BITS;
>
> rcu_read_lock();
> QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
> @@ -2587,6 +2602,15 @@ int colo_init_ram_cache(void)
> }
> rcu_read_unlock();
> ram_cache_enable = true;
> + /*
> + * Record the dirty pages that sent by PVM, we use this dirty bitmap together
> + * with to decide which page in cache should be flushed into SVM's RAM. Here
> + * we use the same name 'migration_bitmap_rcu' as for migration.
> + */
> + migration_bitmap_rcu = g_new0(struct BitmapRcu, 1);
> + migration_bitmap_rcu->bmap = bitmap_new(ram_cache_pages);
> + migration_dirty_pages = 0;
> +
> return 0;
>
> out_locked:
> @@ -2604,9 +2628,15 @@ out_locked:
> void colo_release_ram_cache(void)
> {
> RAMBlock *block;
> + struct BitmapRcu *bitmap = migration_bitmap_rcu;
>
> ram_cache_enable = false;
>
> + atomic_rcu_set(&migration_bitmap_rcu, NULL);
> + if (bitmap) {
> + call_rcu(bitmap, migration_bitmap_free, rcu);
> + }
> +
> rcu_read_lock();
> QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
> if (block->colo_cache) {
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 18/39] COLO: Flush PVM's cached RAM into SVM's memory
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 18/39] COLO: Flush PVM's cached RAM into SVM's memory zhanghailiang
2015-11-27 5:29 ` Li Zhijian
@ 2015-12-01 20:06 ` Dr. David Alan Gilbert
2015-12-03 8:50 ` Hailiang Zhang
1 sibling, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-01 20:06 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> During the time of VM's running, PVM may dirty some pages, we will transfer
> PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint
> time. So, the content of SVM's RAM cache will always be some with PVM's memory
> after checkpoint.
>
> Instead of flushing all content of PVM's RAM cache into SVM's MEMORY,
> we do this in a more efficient way:
> Only flush any page that dirtied by PVM since last checkpoint.
> In this way, we can ensure SVM's memory same with PVM's.
>
> Besides, we must ensure flush RAM cache before load device state.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
> ---
> v11:
> - Move the place of 'need_flush' (Dave's suggestion)
> - Remove unused 'DPRINTF("Flush ram_cache\n")'
> v10:
> - trace the number of dirty pages that be received.
> ---
> include/migration/migration.h | 1 +
> migration/colo.c | 2 --
> migration/ram.c | 37 +++++++++++++++++++++++++++++++++++++
> trace-events | 1 +
> 4 files changed, 39 insertions(+), 2 deletions(-)
>
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index e41372d..221176b 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -336,4 +336,5 @@ PostcopyState postcopy_state_set(PostcopyState new_state);
> /* ram cache */
> int colo_init_ram_cache(void);
> void colo_release_ram_cache(void);
> +void colo_flush_ram_cache(void);
> #endif
> diff --git a/migration/colo.c b/migration/colo.c
> index 5ac8ff2..4095d97 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -385,8 +385,6 @@ void *colo_process_incoming_thread(void *opaque)
> }
> qemu_mutex_unlock_iothread();
>
> - /* TODO: flush vm state */
> -
Might have been better to put the TODO in a place that needed to be changed!
> ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_LOADED, 0);
> if (ret < 0) {
> goto out;
> diff --git a/migration/ram.c b/migration/ram.c
> index da6bbd6..4f37144 100644
> --- a/migration/ram.c
> +++ b/migration/ram.c
> @@ -2448,6 +2448,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> * be atomic
> */
> bool postcopy_running = postcopy_state_get() >= POSTCOPY_INCOMING_LISTENING;
> + bool need_flush = false;
>
> seq_iter++;
>
> @@ -2482,6 +2483,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> /* After going into COLO, we should load the Page into colo_cache */
> if (ram_cache_enable) {
> host = colo_cache_from_block_offset(block, addr);
> + need_flush = true;
> } else {
> host = host_from_ram_block_offset(block, addr);
> }
> @@ -2575,6 +2577,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
> }
>
> rcu_read_unlock();
> +
> + if (!ret && ram_cache_enable && need_flush) {
> + colo_flush_ram_cache();
> + }
> DPRINTF("Completed load of VM with exit code %d seq iteration "
> "%" PRIu64 "\n", ret, seq_iter);
> return ret;
> @@ -2647,6 +2653,37 @@ void colo_release_ram_cache(void)
> rcu_read_unlock();
> }
>
> +/*
> + * Flush content of RAM cache into SVM's memory.
> + * Only flush the pages that be dirtied by PVM or SVM or both.
> + */
> +void colo_flush_ram_cache(void)
> +{
> + RAMBlock *block = NULL;
> + void *dst_host;
> + void *src_host;
> + ram_addr_t offset = 0;
> +
> + trace_colo_flush_ram_cache(migration_dirty_pages);
> + rcu_read_lock();
> + block = QLIST_FIRST_RCU(&ram_list.blocks);
> + while (block) {
> + ram_addr_t ram_addr_abs;
> + offset = migration_bitmap_find_dirty(block, offset, &ram_addr_abs);
> + migration_bitmap_clear_dirty(ram_addr_abs);
> + if (offset >= block->used_length) {
> + offset = 0;
> + block = QLIST_NEXT_RCU(block, next);
> + } else {
> + dst_host = block->host + offset;
> + src_host = block->colo_cache + offset;
> + memcpy(dst_host, src_host, TARGET_PAGE_SIZE);
> + }
> + }
> + rcu_read_unlock();
If you added a trace point here as well, it would make it very easy
to measure how long the flush was taking.
> + assert(migration_dirty_pages == 0);
> +}
> +
> static SaveVMHandlers savevm_ram_handlers = {
> .save_live_setup = ram_save_setup,
> .save_live_iterate = ram_save_iterate,
> diff --git a/trace-events b/trace-events
> index f8a0959..f158d2a 100644
> --- a/trace-events
> +++ b/trace-events
> @@ -1264,6 +1264,7 @@ migration_throttle(void) ""
> ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x"
> ram_postcopy_send_discard_bitmap(void) ""
> ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: %zx len: %zx"
> +colo_flush_ram_cache(uint64_t dirty_pages) "dirty_pages %" PRIu64""
Minor; I think you can remove the "" at the end.
Other than those minor things (and those double-space!):
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Dave
>
> # hw/display/qxl.c
> disable qxl_interface_set_mm_time(int qid, uint32_t mm_time) "%d %d"
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev zhanghailiang
2015-11-27 11:39 ` Yang Hongyang
2015-11-30 1:19 ` Li Zhijian
@ 2015-12-03 1:17 ` Wen Congyang
2015-12-03 3:53 ` Hailiang Zhang
2 siblings, 1 reply; 95+ messages in thread
From: Wen Congyang @ 2015-12-03 1:17 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah,
hongyang.yang
On 11/24/2015 05:25 PM, zhanghailiang wrote:
> We add each netdev a default filter-buffer, which will be used for COLO
> or Micro-checkpoint to buffer VM's packets. The name of default filter-buffer
> is 'nop'.
> For the default filter-buffer, it will not buffer any packets in default.
> So it has no side effect for the netdev.
No, filter-buffer doesn't support vhost, so if you add default filter-buffer
for each netdev, you can't use vhost.
Thanks
Wen Congyang
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Cc: Jason Wang <jasowang@redhat.com>
> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
> ---
> v11:
> - New patch
> ---
> include/net/filter.h | 3 +++
> net/filter-buffer.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> net/net.c | 8 ++++++
> 3 files changed, 85 insertions(+)
>
> diff --git a/include/net/filter.h b/include/net/filter.h
> index 2deda36..01a7e90 100644
> --- a/include/net/filter.h
> +++ b/include/net/filter.h
> @@ -74,4 +74,7 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender,
> int iovcnt,
> void *opaque);
>
> +void netdev_add_default_filter_buffer(const char *netdev_id,
> + NetFilterDirection direction,
> + Error **errp);
> #endif /* QEMU_NET_FILTER_H */
> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
> index 57be149..195af68 100644
> --- a/net/filter-buffer.c
> +++ b/net/filter-buffer.c
> @@ -14,6 +14,12 @@
> #include "qapi/qmp/qerror.h"
> #include "qapi-visit.h"
> #include "qom/object.h"
> +#include "net/net.h"
> +#include "qapi/qmp/qdict.h"
> +#include "qapi/qmp-output-visitor.h"
> +#include "qapi/qmp-input-visitor.h"
> +#include "monitor/monitor.h"
> +#include "qmp-commands.h"
>
> #define TYPE_FILTER_BUFFER "filter-buffer"
>
> @@ -26,6 +32,8 @@ typedef struct FilterBufferState {
> NetQueue *incoming_queue;
> uint32_t interval;
> QEMUTimer release_timer;
> + bool is_default;
> + bool enable_buffer;
> } FilterBufferState;
>
> static void filter_buffer_flush(NetFilterState *nf)
> @@ -65,6 +73,10 @@ static ssize_t filter_buffer_receive_iov(NetFilterState *nf,
> {
> FilterBufferState *s = FILTER_BUFFER(nf);
>
> + /* Don't buffer any packets if the filter is not enabled */
> + if (!s->enable_buffer) {
> + return 0;
> + }
> /*
> * We return size when buffer a packet, the sender will take it as
> * a already sent packet, so sent_cb should not be called later.
> @@ -102,6 +114,7 @@ static void filter_buffer_cleanup(NetFilterState *nf)
> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
> {
> FilterBufferState *s = FILTER_BUFFER(nf);
> + char *path = object_get_canonical_path_component(OBJECT(nf));
>
> /*
> * We may want to accept zero interval when VM FT solutions like MC
> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
> }
>
> s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
> + s->is_default = !strcmp(path, "nop");
> if (s->interval) {
> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
> filter_buffer_release_timer, nf);
> @@ -163,6 +177,66 @@ out:
> error_propagate(errp, local_err);
> }
>
> +/*
> +* This will be used by COLO or MC FT, for which they will need
> +* to buffer the packets of VM's net devices, Here we add a default
> +* buffer filter for each netdev. The name of default buffer filter is
> +* 'nop'
> +*/
> +void netdev_add_default_filter_buffer(const char *netdev_id,
> + NetFilterDirection direction,
> + Error **errp)
> +{
> + QmpOutputVisitor *qov;
> + QmpInputVisitor *qiv;
> + Visitor *ov, *iv;
> + QObject *obj = NULL;
> + QDict *qdict;
> + void *dummy = NULL;
> + const char *id = "nop";
> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
> + NetClientState *nc = qemu_find_netdev(netdev_id);
> + Error *err = NULL;
> +
> + /* FIXME: Not support multiple queues */
> + if (!nc || nc->queue_index > 1) {
> + return;
> + }
> + qov = qmp_output_visitor_new();
> + ov = qmp_output_get_visitor(qov);
> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
> + if (err) {
> + goto out;
> + }
> + visit_type_str(ov, &nc->name, "netdev", &err);
> + if (err) {
> + goto out;
> + }
> + visit_type_str(ov, &queue, "queue", &err);
> + if (err) {
> + goto out;
> + }
> + visit_end_struct(ov, &err);
> + if (err) {
> + goto out;
> + }
> + obj = qmp_output_get_qobject(qov);
> + g_assert(obj != NULL);
> + qdict = qobject_to_qdict(obj);
> + qmp_output_visitor_cleanup(qov);
> +
> + qiv = qmp_input_visitor_new(obj);
> + iv = qmp_input_get_visitor(qiv);
> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
> + qmp_input_visitor_cleanup(qiv);
> + qobject_decref(obj);
> +out:
> + g_free(queue);
> + if (err) {
> + error_propagate(errp, err);
> + }
> +}
> +
> static void filter_buffer_init(Object *obj)
> {
> object_property_add(obj, "interval", "int",
> diff --git a/net/net.c b/net/net.c
> index ade6051..b36d49f 100644
> --- a/net/net.c
> +++ b/net/net.c
> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void *object, int is_netdev, Error **errp)
> }
> return -1;
> }
> +
> + if (is_netdev) {
> + const Netdev *netdev = object;
> +
> + netdev_add_default_filter_buffer(netdev->id,
> + NET_FILTER_DIRECTION_RX,
> + errp);
> + }
> return 0;
> }
>
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-12-03 1:17 ` Wen Congyang
@ 2015-12-03 3:53 ` Hailiang Zhang
2015-12-03 6:25 ` Wen Congyang
0 siblings, 1 reply; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-03 3:53 UTC (permalink / raw)
To: Wen Congyang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah,
hongyang.yang
On 2015/12/3 9:17, Wen Congyang wrote:
> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>> We add each netdev a default filter-buffer, which will be used for COLO
>> or Micro-checkpoint to buffer VM's packets. The name of default filter-buffer
>> is 'nop'.
>> For the default filter-buffer, it will not buffer any packets in default.
>> So it has no side effect for the netdev.
>
> No, filter-buffer doesn't support vhost, so if you add default filter-buffer
> for each netdev, you can't use vhost.
>
Have you tested it ? Did the default filter-buffer break vhost ?
It's not supposed to break vhost, I will look into it. Thanks.
> Thanks
> Wen Congyang
>
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Cc: Jason Wang <jasowang@redhat.com>
>> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
>> ---
>> v11:
>> - New patch
>> ---
>> include/net/filter.h | 3 +++
>> net/filter-buffer.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>> net/net.c | 8 ++++++
>> 3 files changed, 85 insertions(+)
>>
>> diff --git a/include/net/filter.h b/include/net/filter.h
>> index 2deda36..01a7e90 100644
>> --- a/include/net/filter.h
>> +++ b/include/net/filter.h
>> @@ -74,4 +74,7 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender,
>> int iovcnt,
>> void *opaque);
>>
>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>> + NetFilterDirection direction,
>> + Error **errp);
>> #endif /* QEMU_NET_FILTER_H */
>> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
>> index 57be149..195af68 100644
>> --- a/net/filter-buffer.c
>> +++ b/net/filter-buffer.c
>> @@ -14,6 +14,12 @@
>> #include "qapi/qmp/qerror.h"
>> #include "qapi-visit.h"
>> #include "qom/object.h"
>> +#include "net/net.h"
>> +#include "qapi/qmp/qdict.h"
>> +#include "qapi/qmp-output-visitor.h"
>> +#include "qapi/qmp-input-visitor.h"
>> +#include "monitor/monitor.h"
>> +#include "qmp-commands.h"
>>
>> #define TYPE_FILTER_BUFFER "filter-buffer"
>>
>> @@ -26,6 +32,8 @@ typedef struct FilterBufferState {
>> NetQueue *incoming_queue;
>> uint32_t interval;
>> QEMUTimer release_timer;
>> + bool is_default;
>> + bool enable_buffer;
>> } FilterBufferState;
>>
>> static void filter_buffer_flush(NetFilterState *nf)
>> @@ -65,6 +73,10 @@ static ssize_t filter_buffer_receive_iov(NetFilterState *nf,
>> {
>> FilterBufferState *s = FILTER_BUFFER(nf);
>>
>> + /* Don't buffer any packets if the filter is not enabled */
>> + if (!s->enable_buffer) {
>> + return 0;
>> + }
>> /*
>> * We return size when buffer a packet, the sender will take it as
>> * a already sent packet, so sent_cb should not be called later.
>> @@ -102,6 +114,7 @@ static void filter_buffer_cleanup(NetFilterState *nf)
>> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>> {
>> FilterBufferState *s = FILTER_BUFFER(nf);
>> + char *path = object_get_canonical_path_component(OBJECT(nf));
>>
>> /*
>> * We may want to accept zero interval when VM FT solutions like MC
>> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>> }
>>
>> s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
>> + s->is_default = !strcmp(path, "nop");
>> if (s->interval) {
>> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
>> filter_buffer_release_timer, nf);
>> @@ -163,6 +177,66 @@ out:
>> error_propagate(errp, local_err);
>> }
>>
>> +/*
>> +* This will be used by COLO or MC FT, for which they will need
>> +* to buffer the packets of VM's net devices, Here we add a default
>> +* buffer filter for each netdev. The name of default buffer filter is
>> +* 'nop'
>> +*/
>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>> + NetFilterDirection direction,
>> + Error **errp)
>> +{
>> + QmpOutputVisitor *qov;
>> + QmpInputVisitor *qiv;
>> + Visitor *ov, *iv;
>> + QObject *obj = NULL;
>> + QDict *qdict;
>> + void *dummy = NULL;
>> + const char *id = "nop";
>> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
>> + NetClientState *nc = qemu_find_netdev(netdev_id);
>> + Error *err = NULL;
>> +
>> + /* FIXME: Not support multiple queues */
>> + if (!nc || nc->queue_index > 1) {
>> + return;
>> + }
>> + qov = qmp_output_visitor_new();
>> + ov = qmp_output_get_visitor(qov);
>> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
>> + if (err) {
>> + goto out;
>> + }
>> + visit_type_str(ov, &nc->name, "netdev", &err);
>> + if (err) {
>> + goto out;
>> + }
>> + visit_type_str(ov, &queue, "queue", &err);
>> + if (err) {
>> + goto out;
>> + }
>> + visit_end_struct(ov, &err);
>> + if (err) {
>> + goto out;
>> + }
>> + obj = qmp_output_get_qobject(qov);
>> + g_assert(obj != NULL);
>> + qdict = qobject_to_qdict(obj);
>> + qmp_output_visitor_cleanup(qov);
>> +
>> + qiv = qmp_input_visitor_new(obj);
>> + iv = qmp_input_get_visitor(qiv);
>> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
>> + qmp_input_visitor_cleanup(qiv);
>> + qobject_decref(obj);
>> +out:
>> + g_free(queue);
>> + if (err) {
>> + error_propagate(errp, err);
>> + }
>> +}
>> +
>> static void filter_buffer_init(Object *obj)
>> {
>> object_property_add(obj, "interval", "int",
>> diff --git a/net/net.c b/net/net.c
>> index ade6051..b36d49f 100644
>> --- a/net/net.c
>> +++ b/net/net.c
>> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void *object, int is_netdev, Error **errp)
>> }
>> return -1;
>> }
>> +
>> + if (is_netdev) {
>> + const Netdev *netdev = object;
>> +
>> + netdev_add_default_filter_buffer(netdev->id,
>> + NET_FILTER_DIRECTION_RX,
>> + errp);
>> + }
>> return 0;
>> }
>>
>>
>
>
>
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-12-03 3:53 ` Hailiang Zhang
@ 2015-12-03 6:25 ` Wen Congyang
2015-12-03 6:48 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Wen Congyang @ 2015-12-03 6:25 UTC (permalink / raw)
To: Hailiang Zhang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah,
hongyang.yang
On 12/03/2015 11:53 AM, Hailiang Zhang wrote:
> On 2015/12/3 9:17, Wen Congyang wrote:
>> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>>> We add each netdev a default filter-buffer, which will be used for COLO
>>> or Micro-checkpoint to buffer VM's packets. The name of default filter-buffer
>>> is 'nop'.
>>> For the default filter-buffer, it will not buffer any packets in default.
>>> So it has no side effect for the netdev.
>>
>> No, filter-buffer doesn't support vhost, so if you add default filter-buffer
>> for each netdev, you can't use vhost.
>>
>
> Have you tested it ? Did the default filter-buffer break vhost ?
> It's not supposed to break vhost, I will look into it. Thanks.
Yes, I have tested it. When I want to start a normal vm with vhost, I get
the following error messages:
qemu-system-x86_64: -netdev tap,id=hn0,queues=1,vhost=on: Vhost is not supported
Thanks
Wen Congyang
>
>> Thanks
>> Wen Congyang
>>
>>>
>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>> Cc: Jason Wang <jasowang@redhat.com>
>>> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
>>> ---
>>> v11:
>>> - New patch
>>> ---
>>> include/net/filter.h | 3 +++
>>> net/filter-buffer.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>> net/net.c | 8 ++++++
>>> 3 files changed, 85 insertions(+)
>>>
>>> diff --git a/include/net/filter.h b/include/net/filter.h
>>> index 2deda36..01a7e90 100644
>>> --- a/include/net/filter.h
>>> +++ b/include/net/filter.h
>>> @@ -74,4 +74,7 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender,
>>> int iovcnt,
>>> void *opaque);
>>>
>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>> + NetFilterDirection direction,
>>> + Error **errp);
>>> #endif /* QEMU_NET_FILTER_H */
>>> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
>>> index 57be149..195af68 100644
>>> --- a/net/filter-buffer.c
>>> +++ b/net/filter-buffer.c
>>> @@ -14,6 +14,12 @@
>>> #include "qapi/qmp/qerror.h"
>>> #include "qapi-visit.h"
>>> #include "qom/object.h"
>>> +#include "net/net.h"
>>> +#include "qapi/qmp/qdict.h"
>>> +#include "qapi/qmp-output-visitor.h"
>>> +#include "qapi/qmp-input-visitor.h"
>>> +#include "monitor/monitor.h"
>>> +#include "qmp-commands.h"
>>>
>>> #define TYPE_FILTER_BUFFER "filter-buffer"
>>>
>>> @@ -26,6 +32,8 @@ typedef struct FilterBufferState {
>>> NetQueue *incoming_queue;
>>> uint32_t interval;
>>> QEMUTimer release_timer;
>>> + bool is_default;
>>> + bool enable_buffer;
>>> } FilterBufferState;
>>>
>>> static void filter_buffer_flush(NetFilterState *nf)
>>> @@ -65,6 +73,10 @@ static ssize_t filter_buffer_receive_iov(NetFilterState *nf,
>>> {
>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>
>>> + /* Don't buffer any packets if the filter is not enabled */
>>> + if (!s->enable_buffer) {
>>> + return 0;
>>> + }
>>> /*
>>> * We return size when buffer a packet, the sender will take it as
>>> * a already sent packet, so sent_cb should not be called later.
>>> @@ -102,6 +114,7 @@ static void filter_buffer_cleanup(NetFilterState *nf)
>>> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>>> {
>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>> + char *path = object_get_canonical_path_component(OBJECT(nf));
>>>
>>> /*
>>> * We may want to accept zero interval when VM FT solutions like MC
>>> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>>> }
>>>
>>> s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
>>> + s->is_default = !strcmp(path, "nop");
>>> if (s->interval) {
>>> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
>>> filter_buffer_release_timer, nf);
>>> @@ -163,6 +177,66 @@ out:
>>> error_propagate(errp, local_err);
>>> }
>>>
>>> +/*
>>> +* This will be used by COLO or MC FT, for which they will need
>>> +* to buffer the packets of VM's net devices, Here we add a default
>>> +* buffer filter for each netdev. The name of default buffer filter is
>>> +* 'nop'
>>> +*/
>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>> + NetFilterDirection direction,
>>> + Error **errp)
>>> +{
>>> + QmpOutputVisitor *qov;
>>> + QmpInputVisitor *qiv;
>>> + Visitor *ov, *iv;
>>> + QObject *obj = NULL;
>>> + QDict *qdict;
>>> + void *dummy = NULL;
>>> + const char *id = "nop";
>>> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
>>> + NetClientState *nc = qemu_find_netdev(netdev_id);
>>> + Error *err = NULL;
>>> +
>>> + /* FIXME: Not support multiple queues */
>>> + if (!nc || nc->queue_index > 1) {
>>> + return;
>>> + }
>>> + qov = qmp_output_visitor_new();
>>> + ov = qmp_output_get_visitor(qov);
>>> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
>>> + if (err) {
>>> + goto out;
>>> + }
>>> + visit_type_str(ov, &nc->name, "netdev", &err);
>>> + if (err) {
>>> + goto out;
>>> + }
>>> + visit_type_str(ov, &queue, "queue", &err);
>>> + if (err) {
>>> + goto out;
>>> + }
>>> + visit_end_struct(ov, &err);
>>> + if (err) {
>>> + goto out;
>>> + }
>>> + obj = qmp_output_get_qobject(qov);
>>> + g_assert(obj != NULL);
>>> + qdict = qobject_to_qdict(obj);
>>> + qmp_output_visitor_cleanup(qov);
>>> +
>>> + qiv = qmp_input_visitor_new(obj);
>>> + iv = qmp_input_get_visitor(qiv);
>>> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
>>> + qmp_input_visitor_cleanup(qiv);
>>> + qobject_decref(obj);
>>> +out:
>>> + g_free(queue);
>>> + if (err) {
>>> + error_propagate(errp, err);
>>> + }
>>> +}
>>> +
>>> static void filter_buffer_init(Object *obj)
>>> {
>>> object_property_add(obj, "interval", "int",
>>> diff --git a/net/net.c b/net/net.c
>>> index ade6051..b36d49f 100644
>>> --- a/net/net.c
>>> +++ b/net/net.c
>>> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void *object, int is_netdev, Error **errp)
>>> }
>>> return -1;
>>> }
>>> +
>>> + if (is_netdev) {
>>> + const Netdev *netdev = object;
>>> +
>>> + netdev_add_default_filter_buffer(netdev->id,
>>> + NET_FILTER_DIRECTION_RX,
>>> + errp);
>>> + }
>>> return 0;
>>> }
>>>
>>>
>>
>>
>>
>>
>> .
>>
>
>
>
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-12-03 6:25 ` Wen Congyang
@ 2015-12-03 6:48 ` Hailiang Zhang
2015-12-03 7:21 ` Yang Hongyang
0 siblings, 1 reply; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-03 6:48 UTC (permalink / raw)
To: Wen Congyang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah,
hongyang.yang
On 2015/12/3 14:25, Wen Congyang wrote:
> On 12/03/2015 11:53 AM, Hailiang Zhang wrote:
>> On 2015/12/3 9:17, Wen Congyang wrote:
>>> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>>>> We add each netdev a default filter-buffer, which will be used for COLO
>>>> or Micro-checkpoint to buffer VM's packets. The name of default filter-buffer
>>>> is 'nop'.
>>>> For the default filter-buffer, it will not buffer any packets in default.
>>>> So it has no side effect for the netdev.
>>>
>>> No, filter-buffer doesn't support vhost, so if you add default filter-buffer
>>> for each netdev, you can't use vhost.
>>>
>>
>> Have you tested it ? Did the default filter-buffer break vhost ?
>> It's not supposed to break vhost, I will look into it. Thanks.
>
> Yes, I have tested it. When I want to start a normal vm with vhost, I get
> the following error messages:
>
> qemu-system-x86_64: -netdev tap,id=hn0,queues=1,vhost=on: Vhost is not supported
>
Hmm, that is reported by filter in nextfilter_complete(), i will investigate it.
We hope the default buffer filter has no side effect on netdev.
Thanks,
Hailiang
>>
>>> Thanks
>>> Wen Congyang
>>>
>>>>
>>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>> Cc: Jason Wang <jasowang@redhat.com>
>>>> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
>>>> ---
>>>> v11:
>>>> - New patch
>>>> ---
>>>> include/net/filter.h | 3 +++
>>>> net/filter-buffer.c | 74 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>> net/net.c | 8 ++++++
>>>> 3 files changed, 85 insertions(+)
>>>>
>>>> diff --git a/include/net/filter.h b/include/net/filter.h
>>>> index 2deda36..01a7e90 100644
>>>> --- a/include/net/filter.h
>>>> +++ b/include/net/filter.h
>>>> @@ -74,4 +74,7 @@ ssize_t qemu_netfilter_pass_to_next(NetClientState *sender,
>>>> int iovcnt,
>>>> void *opaque);
>>>>
>>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>>> + NetFilterDirection direction,
>>>> + Error **errp);
>>>> #endif /* QEMU_NET_FILTER_H */
>>>> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
>>>> index 57be149..195af68 100644
>>>> --- a/net/filter-buffer.c
>>>> +++ b/net/filter-buffer.c
>>>> @@ -14,6 +14,12 @@
>>>> #include "qapi/qmp/qerror.h"
>>>> #include "qapi-visit.h"
>>>> #include "qom/object.h"
>>>> +#include "net/net.h"
>>>> +#include "qapi/qmp/qdict.h"
>>>> +#include "qapi/qmp-output-visitor.h"
>>>> +#include "qapi/qmp-input-visitor.h"
>>>> +#include "monitor/monitor.h"
>>>> +#include "qmp-commands.h"
>>>>
>>>> #define TYPE_FILTER_BUFFER "filter-buffer"
>>>>
>>>> @@ -26,6 +32,8 @@ typedef struct FilterBufferState {
>>>> NetQueue *incoming_queue;
>>>> uint32_t interval;
>>>> QEMUTimer release_timer;
>>>> + bool is_default;
>>>> + bool enable_buffer;
>>>> } FilterBufferState;
>>>>
>>>> static void filter_buffer_flush(NetFilterState *nf)
>>>> @@ -65,6 +73,10 @@ static ssize_t filter_buffer_receive_iov(NetFilterState *nf,
>>>> {
>>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>>
>>>> + /* Don't buffer any packets if the filter is not enabled */
>>>> + if (!s->enable_buffer) {
>>>> + return 0;
>>>> + }
>>>> /*
>>>> * We return size when buffer a packet, the sender will take it as
>>>> * a already sent packet, so sent_cb should not be called later.
>>>> @@ -102,6 +114,7 @@ static void filter_buffer_cleanup(NetFilterState *nf)
>>>> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>>>> {
>>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>> + char *path = object_get_canonical_path_component(OBJECT(nf));
>>>>
>>>> /*
>>>> * We may want to accept zero interval when VM FT solutions like MC
>>>> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>>>> }
>>>>
>>>> s->incoming_queue = qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
>>>> + s->is_default = !strcmp(path, "nop");
>>>> if (s->interval) {
>>>> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
>>>> filter_buffer_release_timer, nf);
>>>> @@ -163,6 +177,66 @@ out:
>>>> error_propagate(errp, local_err);
>>>> }
>>>>
>>>> +/*
>>>> +* This will be used by COLO or MC FT, for which they will need
>>>> +* to buffer the packets of VM's net devices, Here we add a default
>>>> +* buffer filter for each netdev. The name of default buffer filter is
>>>> +* 'nop'
>>>> +*/
>>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>>> + NetFilterDirection direction,
>>>> + Error **errp)
>>>> +{
>>>> + QmpOutputVisitor *qov;
>>>> + QmpInputVisitor *qiv;
>>>> + Visitor *ov, *iv;
>>>> + QObject *obj = NULL;
>>>> + QDict *qdict;
>>>> + void *dummy = NULL;
>>>> + const char *id = "nop";
>>>> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
>>>> + NetClientState *nc = qemu_find_netdev(netdev_id);
>>>> + Error *err = NULL;
>>>> +
>>>> + /* FIXME: Not support multiple queues */
>>>> + if (!nc || nc->queue_index > 1) {
>>>> + return;
>>>> + }
>>>> + qov = qmp_output_visitor_new();
>>>> + ov = qmp_output_get_visitor(qov);
>>>> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
>>>> + if (err) {
>>>> + goto out;
>>>> + }
>>>> + visit_type_str(ov, &nc->name, "netdev", &err);
>>>> + if (err) {
>>>> + goto out;
>>>> + }
>>>> + visit_type_str(ov, &queue, "queue", &err);
>>>> + if (err) {
>>>> + goto out;
>>>> + }
>>>> + visit_end_struct(ov, &err);
>>>> + if (err) {
>>>> + goto out;
>>>> + }
>>>> + obj = qmp_output_get_qobject(qov);
>>>> + g_assert(obj != NULL);
>>>> + qdict = qobject_to_qdict(obj);
>>>> + qmp_output_visitor_cleanup(qov);
>>>> +
>>>> + qiv = qmp_input_visitor_new(obj);
>>>> + iv = qmp_input_get_visitor(qiv);
>>>> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
>>>> + qmp_input_visitor_cleanup(qiv);
>>>> + qobject_decref(obj);
>>>> +out:
>>>> + g_free(queue);
>>>> + if (err) {
>>>> + error_propagate(errp, err);
>>>> + }
>>>> +}
>>>> +
>>>> static void filter_buffer_init(Object *obj)
>>>> {
>>>> object_property_add(obj, "interval", "int",
>>>> diff --git a/net/net.c b/net/net.c
>>>> index ade6051..b36d49f 100644
>>>> --- a/net/net.c
>>>> +++ b/net/net.c
>>>> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void *object, int is_netdev, Error **errp)
>>>> }
>>>> return -1;
>>>> }
>>>> +
>>>> + if (is_netdev) {
>>>> + const Netdev *netdev = object;
>>>> +
>>>> + netdev_add_default_filter_buffer(netdev->id,
>>>> + NET_FILTER_DIRECTION_RX,
>>>> + errp);
>>>> + }
>>>> return 0;
>>>> }
>>>>
>>>>
>>>
>>>
>>>
>>>
>>> .
>>>
>>
>>
>>
>>
>> .
>>
>
>
>
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 14/39] ram: Split host_from_stream_offset() into two helper functions
2015-12-01 18:19 ` Dr. David Alan Gilbert
@ 2015-12-03 7:19 ` Hailiang Zhang
2015-12-03 7:29 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-03 7:19 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/2 2:19, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> Split host_from_stream_offset() into two parts:
>> One is to get ram block, which the block idstr may be get from migration
>> stream, the other is to get hva (host) address from block and the offset.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>
> OK, I see why you're doing this from the next patch.
>
>> ---
>> v11:
>> - New patch
>> ---
>> migration/ram.c | 29 +++++++++++++++++++++--------
>> 1 file changed, 21 insertions(+), 8 deletions(-)
>>
>> diff --git a/migration/ram.c b/migration/ram.c
>> index cfe78aa..a161620 100644
>> --- a/migration/ram.c
>> +++ b/migration/ram.c
>> @@ -2136,9 +2136,9 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
>> * offset: Offset within the block
>> * flags: Page flags (mostly to see if it's a continuation of previous block)
>> */
>> -static inline void *host_from_stream_offset(QEMUFile *f,
>> - ram_addr_t offset,
>> - int flags)
>> +static inline RAMBlock *ram_block_from_stream(QEMUFile *f,
>> + ram_addr_t offset,
>> + int flags)
>> {
>> static RAMBlock *block = NULL;
>> char id[256];
>> @@ -2150,22 +2150,31 @@ static inline void *host_from_stream_offset(QEMUFile *f,
>> return NULL;
>> }
>>
>> - return block->host + offset;
>> + return block;
>> }
>> -
>> len = qemu_get_byte(f);
>> qemu_get_buffer(f, (uint8_t *)id, len);
>> id[len] = 0;
>>
>> block = qemu_ram_block_by_name(id);
>> if (block && block->max_length > offset) {
>> - return block->host + offset;
>> + return block;
>> }
>>
>> error_report("Can't find block %s", id);
>> return NULL;
>> }
>>
>> +static inline void *host_from_ram_block_offset(RAMBlock *block,
>> + ram_addr_t offset)
>> +{
>> + if (!block) {
>> + return NULL;
>> + }
>> +
>> + return block->host + offset;
>> +}
>
> That's almost the same as ramblock_ptr in include/exec/ram_addr.h, but
> it assert's rather than doing NULL on errors.
>
Hmm, i didn't notice that before.
> I'm not sure about this, but can I suggest:
>
> ram_block_from_stream(QEMUFile *f, int flags)
>
> doesn't have the offset; just finds the block and handles the CONT.
>
> bool offset_in_ramblock(RAMBlock *b, ram_addr_t offset);
>
> actually does the check; put this in exec.c, and declare it in include/exec/ram_addr.h
>
> void *ramblock_ptr_try(RAMBlock *block, ram_addr_t offset)
> which returns NULL if offset_in_ramblock fails, and otherwise returns the result
> of ramblock_ptr - again put that in include/exec/ram_addr.h
>
That sounds good. I will add the ramblock_ptr_try() and keep the ramblock_ptr(). They
have different behavior when the check failed. I'm not supposed to break it.
> (I'm not sure about this - I almost suggested changing ramblock_ptr to not do
> the checks, and just add a call to assert(offset_in_ramblock) before each use, but
> that sounded too painful).
>
Yes, that's not so good.
> Hmm - we check here for block->max_length > offset - where as the check in
> ram_addr.h is used_length - I wonder if we should be using used_length?
>
Er, i think we should use used_length instead of max_length.
Thanks,
zhanghailiang
> Dave
>
>> +
>> /*
>> * If a page (or a whole RDMA chunk) has been
>> * determined to be zero, then zap it.
>> @@ -2310,7 +2319,9 @@ static int ram_load_postcopy(QEMUFile *f)
>> trace_ram_load_postcopy_loop((uint64_t)addr, flags);
>> place_needed = false;
>> if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE)) {
>> - host = host_from_stream_offset(f, addr, flags);
>> + RAMBlock *block = ram_block_from_stream(f, addr, flags);
>> +
>> + host = host_from_ram_block_offset(block, addr);
>> if (!host) {
>> error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
>> ret = -EINVAL;
>> @@ -2441,7 +2452,9 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>>
>> if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE |
>> RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) {
>> - host = host_from_stream_offset(f, addr, flags);
>> + RAMBlock *block = ram_block_from_stream(f, addr, flags);
>> +
>> + host = host_from_ram_block_offset(block, addr);
>> if (!host) {
>> error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
>> ret = -EINVAL;
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-12-03 6:48 ` Hailiang Zhang
@ 2015-12-03 7:21 ` Yang Hongyang
2015-12-03 8:37 ` Hailiang Zhang
2015-12-07 7:38 ` Hailiang Zhang
0 siblings, 2 replies; 95+ messages in thread
From: Yang Hongyang @ 2015-12-03 7:21 UTC (permalink / raw)
To: Hailiang Zhang, Wen Congyang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah
On 2015年12月03日 14:48, Hailiang Zhang wrote:
> On 2015/12/3 14:25, Wen Congyang wrote:
>> On 12/03/2015 11:53 AM, Hailiang Zhang wrote:
>>> On 2015/12/3 9:17, Wen Congyang wrote:
>>>> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>>>>> We add each netdev a default filter-buffer, which will be used for
>>>>> COLO
>>>>> or Micro-checkpoint to buffer VM's packets. The name of default
>>>>> filter-buffer
>>>>> is 'nop'.
>>>>> For the default filter-buffer, it will not buffer any packets in
>>>>> default.
>>>>> So it has no side effect for the netdev.
>>>>
>>>> No, filter-buffer doesn't support vhost, so if you add default
>>>> filter-buffer
>>>> for each netdev, you can't use vhost.
>>>>
>>>
>>> Have you tested it ? Did the default filter-buffer break vhost ?
>>> It's not supposed to break vhost, I will look into it. Thanks.
>>
>> Yes, I have tested it. When I want to start a normal vm with vhost, I get
>> the following error messages:
>>
>> qemu-system-x86_64: -netdev tap,id=hn0,queues=1,vhost=on: Vhost is not
>> supported
>>
>
> Hmm, that is reported by filter in nextfilter_complete(), i will
> investigate it.
> We hope the default buffer filter has no side effect on netdev.
I think you'd better add an option to netdev to turn on/off default
filter, that will solve the problem.
>
> Thanks,
> Hailiang
>
>>>
>>>> Thanks
>>>> Wen Congyang
>>>>
>>>>>
>>>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>>> Cc: Jason Wang <jasowang@redhat.com>
>>>>> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
>>>>> ---
>>>>> v11:
>>>>> - New patch
>>>>> ---
>>>>> include/net/filter.h | 3 +++
>>>>> net/filter-buffer.c | 74
>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>> net/net.c | 8 ++++++
>>>>> 3 files changed, 85 insertions(+)
>>>>>
>>>>> diff --git a/include/net/filter.h b/include/net/filter.h
>>>>> index 2deda36..01a7e90 100644
>>>>> --- a/include/net/filter.h
>>>>> +++ b/include/net/filter.h
>>>>> @@ -74,4 +74,7 @@ ssize_t
>>>>> qemu_netfilter_pass_to_next(NetClientState *sender,
>>>>> int iovcnt,
>>>>> void *opaque);
>>>>>
>>>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>>>> + NetFilterDirection direction,
>>>>> + Error **errp);
>>>>> #endif /* QEMU_NET_FILTER_H */
>>>>> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
>>>>> index 57be149..195af68 100644
>>>>> --- a/net/filter-buffer.c
>>>>> +++ b/net/filter-buffer.c
>>>>> @@ -14,6 +14,12 @@
>>>>> #include "qapi/qmp/qerror.h"
>>>>> #include "qapi-visit.h"
>>>>> #include "qom/object.h"
>>>>> +#include "net/net.h"
>>>>> +#include "qapi/qmp/qdict.h"
>>>>> +#include "qapi/qmp-output-visitor.h"
>>>>> +#include "qapi/qmp-input-visitor.h"
>>>>> +#include "monitor/monitor.h"
>>>>> +#include "qmp-commands.h"
>>>>>
>>>>> #define TYPE_FILTER_BUFFER "filter-buffer"
>>>>>
>>>>> @@ -26,6 +32,8 @@ typedef struct FilterBufferState {
>>>>> NetQueue *incoming_queue;
>>>>> uint32_t interval;
>>>>> QEMUTimer release_timer;
>>>>> + bool is_default;
>>>>> + bool enable_buffer;
>>>>> } FilterBufferState;
>>>>>
>>>>> static void filter_buffer_flush(NetFilterState *nf)
>>>>> @@ -65,6 +73,10 @@ static ssize_t
>>>>> filter_buffer_receive_iov(NetFilterState *nf,
>>>>> {
>>>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>>>
>>>>> + /* Don't buffer any packets if the filter is not enabled */
>>>>> + if (!s->enable_buffer) {
>>>>> + return 0;
>>>>> + }
>>>>> /*
>>>>> * We return size when buffer a packet, the sender will take
>>>>> it as
>>>>> * a already sent packet, so sent_cb should not be called
>>>>> later.
>>>>> @@ -102,6 +114,7 @@ static void
>>>>> filter_buffer_cleanup(NetFilterState *nf)
>>>>> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>>>>> {
>>>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>>> + char *path = object_get_canonical_path_component(OBJECT(nf));
>>>>>
>>>>> /*
>>>>> * We may want to accept zero interval when VM FT solutions
>>>>> like MC
>>>>> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState
>>>>> *nf, Error **errp)
>>>>> }
>>>>>
>>>>> s->incoming_queue =
>>>>> qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
>>>>> + s->is_default = !strcmp(path, "nop");
>>>>> if (s->interval) {
>>>>> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
>>>>> filter_buffer_release_timer, nf);
>>>>> @@ -163,6 +177,66 @@ out:
>>>>> error_propagate(errp, local_err);
>>>>> }
>>>>>
>>>>> +/*
>>>>> +* This will be used by COLO or MC FT, for which they will need
>>>>> +* to buffer the packets of VM's net devices, Here we add a default
>>>>> +* buffer filter for each netdev. The name of default buffer filter is
>>>>> +* 'nop'
>>>>> +*/
>>>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>>>> + NetFilterDirection direction,
>>>>> + Error **errp)
>>>>> +{
>>>>> + QmpOutputVisitor *qov;
>>>>> + QmpInputVisitor *qiv;
>>>>> + Visitor *ov, *iv;
>>>>> + QObject *obj = NULL;
>>>>> + QDict *qdict;
>>>>> + void *dummy = NULL;
>>>>> + const char *id = "nop";
>>>>> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
>>>>> + NetClientState *nc = qemu_find_netdev(netdev_id);
>>>>> + Error *err = NULL;
>>>>> +
>>>>> + /* FIXME: Not support multiple queues */
>>>>> + if (!nc || nc->queue_index > 1) {
>>>>> + return;
>>>>> + }
>>>>> + qov = qmp_output_visitor_new();
>>>>> + ov = qmp_output_get_visitor(qov);
>>>>> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
>>>>> + if (err) {
>>>>> + goto out;
>>>>> + }
>>>>> + visit_type_str(ov, &nc->name, "netdev", &err);
>>>>> + if (err) {
>>>>> + goto out;
>>>>> + }
>>>>> + visit_type_str(ov, &queue, "queue", &err);
>>>>> + if (err) {
>>>>> + goto out;
>>>>> + }
>>>>> + visit_end_struct(ov, &err);
>>>>> + if (err) {
>>>>> + goto out;
>>>>> + }
>>>>> + obj = qmp_output_get_qobject(qov);
>>>>> + g_assert(obj != NULL);
>>>>> + qdict = qobject_to_qdict(obj);
>>>>> + qmp_output_visitor_cleanup(qov);
>>>>> +
>>>>> + qiv = qmp_input_visitor_new(obj);
>>>>> + iv = qmp_input_get_visitor(qiv);
>>>>> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
>>>>> + qmp_input_visitor_cleanup(qiv);
>>>>> + qobject_decref(obj);
>>>>> +out:
>>>>> + g_free(queue);
>>>>> + if (err) {
>>>>> + error_propagate(errp, err);
>>>>> + }
>>>>> +}
>>>>> +
>>>>> static void filter_buffer_init(Object *obj)
>>>>> {
>>>>> object_property_add(obj, "interval", "int",
>>>>> diff --git a/net/net.c b/net/net.c
>>>>> index ade6051..b36d49f 100644
>>>>> --- a/net/net.c
>>>>> +++ b/net/net.c
>>>>> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void
>>>>> *object, int is_netdev, Error **errp)
>>>>> }
>>>>> return -1;
>>>>> }
>>>>> +
>>>>> + if (is_netdev) {
>>>>> + const Netdev *netdev = object;
>>>>> +
>>>>> + netdev_add_default_filter_buffer(netdev->id,
>>>>> + NET_FILTER_DIRECTION_RX,
>>>>> + errp);
>>>>> + }
>>>>> return 0;
>>>>> }
>>>>>
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> .
>>>>
>>>
>>>
>>>
>>>
>>> .
>>>
>>
>>
>>
>>
>> .
>>
>
>
>
--
Thanks,
Yang
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 14/39] ram: Split host_from_stream_offset() into two helper functions
2015-12-03 7:19 ` Hailiang Zhang
@ 2015-12-03 7:29 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-03 7:29 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/3 15:19, Hailiang Zhang wrote:
> On 2015/12/2 2:19, Dr. David Alan Gilbert wrote:
>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>>> Split host_from_stream_offset() into two parts:
>>> One is to get ram block, which the block idstr may be get from migration
>>> stream, the other is to get hva (host) address from block and the offset.
>>>
>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>
>> OK, I see why you're doing this from the next patch.
>>
>>> ---
>>> v11:
>>> - New patch
>>> ---
>>> migration/ram.c | 29 +++++++++++++++++++++--------
>>> 1 file changed, 21 insertions(+), 8 deletions(-)
>>>
>>> diff --git a/migration/ram.c b/migration/ram.c
>>> index cfe78aa..a161620 100644
>>> --- a/migration/ram.c
>>> +++ b/migration/ram.c
>>> @@ -2136,9 +2136,9 @@ static int load_xbzrle(QEMUFile *f, ram_addr_t addr, void *host)
>>> * offset: Offset within the block
>>> * flags: Page flags (mostly to see if it's a continuation of previous block)
>>> */
>>> -static inline void *host_from_stream_offset(QEMUFile *f,
>>> - ram_addr_t offset,
>>> - int flags)
>>> +static inline RAMBlock *ram_block_from_stream(QEMUFile *f,
>>> + ram_addr_t offset,
>>> + int flags)
>>> {
>>> static RAMBlock *block = NULL;
>>> char id[256];
>>> @@ -2150,22 +2150,31 @@ static inline void *host_from_stream_offset(QEMUFile *f,
>>> return NULL;
>>> }
>>>
>>> - return block->host + offset;
>>> + return block;
>>> }
>>> -
>>> len = qemu_get_byte(f);
>>> qemu_get_buffer(f, (uint8_t *)id, len);
>>> id[len] = 0;
>>>
>>> block = qemu_ram_block_by_name(id);
>>> if (block && block->max_length > offset) {
>>> - return block->host + offset;
>>> + return block;
>>> }
>>>
>>> error_report("Can't find block %s", id);
>>> return NULL;
>>> }
>>>
>>> +static inline void *host_from_ram_block_offset(RAMBlock *block,
>>> + ram_addr_t offset)
>>> +{
>>> + if (!block) {
>>> + return NULL;
>>> + }
>>> +
>>> + return block->host + offset;
>>> +}
>>
>> That's almost the same as ramblock_ptr in include/exec/ram_addr.h, but
>> it assert's rather than doing NULL on errors.
>>
>
> Hmm, i didn't notice that before.
>
>> I'm not sure about this, but can I suggest:
>>
>> ram_block_from_stream(QEMUFile *f, int flags)
>>
>> doesn't have the offset; just finds the block and handles the CONT.
>>
>> bool offset_in_ramblock(RAMBlock *b, ram_addr_t offset);
>>
>> actually does the check; put this in exec.c, and declare it in include/exec/ram_addr.h
>>
>> void *ramblock_ptr_try(RAMBlock *block, ram_addr_t offset)
>> which returns NULL if offset_in_ramblock fails, and otherwise returns the result
>> of ramblock_ptr - again put that in include/exec/ram_addr.h
>>
>
> That sounds good. I will add the ramblock_ptr_try() and keep the ramblock_ptr(). They
Keeping the name host_from_ram_block_offset() seems better, it corresponds with the name
colo_cache_from_block_offset() in next patch. ;)
> have different behavior when the check failed. I'm not supposed to break it.
>
>> (I'm not sure about this - I almost suggested changing ramblock_ptr to not do
>> the checks, and just add a call to assert(offset_in_ramblock) before each use, but
>> that sounded too painful).
>>
>
> Yes, that's not so good.
>
>> Hmm - we check here for block->max_length > offset - where as the check in
>> ram_addr.h is used_length - I wonder if we should be using used_length?
>>
>
> Er, i think we should use used_length instead of max_length.
>
> Thanks,
> zhanghailiang
>
>> Dave
>>
>>> +
>>> /*
>>> * If a page (or a whole RDMA chunk) has been
>>> * determined to be zero, then zap it.
>>> @@ -2310,7 +2319,9 @@ static int ram_load_postcopy(QEMUFile *f)
>>> trace_ram_load_postcopy_loop((uint64_t)addr, flags);
>>> place_needed = false;
>>> if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE)) {
>>> - host = host_from_stream_offset(f, addr, flags);
>>> + RAMBlock *block = ram_block_from_stream(f, addr, flags);
>>> +
>>> + host = host_from_ram_block_offset(block, addr);
>>> if (!host) {
>>> error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
>>> ret = -EINVAL;
>>> @@ -2441,7 +2452,9 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>>>
>>> if (flags & (RAM_SAVE_FLAG_COMPRESS | RAM_SAVE_FLAG_PAGE |
>>> RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) {
>>> - host = host_from_stream_offset(f, addr, flags);
>>> + RAMBlock *block = ram_block_from_stream(f, addr, flags);
>>> +
>>> + host = host_from_ram_block_offset(block, addr);
>>> if (!host) {
>>> error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
>>> ret = -EINVAL;
>>> --
>>> 1.8.3.1
>>>
>>>
>> --
>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>
>> .
>>
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 15/39] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily
2015-12-01 19:02 ` Dr. David Alan Gilbert
@ 2015-12-03 8:25 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-03 8:25 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/2 3:02, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> We should not load PVM's state directly into SVM, because there maybe some
>> errors happen when SVM is receving data, which will break SVM.
>>
>> We need to ensure receving all data before load the state into SVM. We use
>> an extra memory to cache these data (PVM's ram). The ram cache in secondary side
>> is initially the same as SVM/PVM's memory. And in the process of checkpoint,
>> we cache the dirty pages of PVM into this ram cache firstly, so this ram cache
>> always the same as PVM's memory at every checkpoint, then we flush this cached ram
>> to SVM after we receive all PVM's state.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>> v11:
>> - Rename 'host_cache' to 'colo_cache' (Dave's suggestion)
>> v10:
>> - Split the process of dirty pages recording into a new patch
>> ---
>> include/exec/ram_addr.h | 1 +
>> include/migration/migration.h | 4 +++
>> migration/colo.c | 10 +++++++
>> migration/ram.c | 69 ++++++++++++++++++++++++++++++++++++++++++-
>> 4 files changed, 83 insertions(+), 1 deletion(-)
>>
>> diff --git a/include/exec/ram_addr.h b/include/exec/ram_addr.h
>> index 7115154..bb44f66 100644
>> --- a/include/exec/ram_addr.h
>> +++ b/include/exec/ram_addr.h
>> @@ -26,6 +26,7 @@ struct RAMBlock {
>> struct rcu_head rcu;
>> struct MemoryRegion *mr;
>> uint8_t *host;
>> + uint8_t *colo_cache; /* For colo, VM's ram cache */
>> ram_addr_t offset;
>> ram_addr_t used_length;
>> ram_addr_t max_length;
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index ba5bcec..e41372d 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -332,4 +332,8 @@ int ram_save_queue_pages(MigrationState *ms, const char *rbname,
>> PostcopyState postcopy_state_get(void);
>> /* Set the state and return the old state */
>> PostcopyState postcopy_state_set(PostcopyState new_state);
>> +
>> +/* ram cache */
>> +int colo_init_ram_cache(void);
>> +void colo_release_ram_cache(void);
>> #endif
>> diff --git a/migration/colo.c b/migration/colo.c
>> index 012d8e5..6e933fa 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -304,6 +304,12 @@ void *colo_process_incoming_thread(void *opaque)
>> qemu_set_block(qemu_get_fd(mis->from_src_file));
>>
>>
>> + ret = colo_init_ram_cache();
>> + if (ret < 0) {
>> + error_report("Failed to initialize ram cache");
>> + goto out;
>> + }
>> +
>> ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_READY, 0);
>> if (ret < 0) {
>> goto out;
>> @@ -353,6 +359,10 @@ out:
>> strerror(-ret));
>> }
>>
>> + qemu_mutex_lock_iothread();
>> + colo_release_ram_cache();
>> + qemu_mutex_unlock_iothread();
>> +
>> if (mis->to_src_file) {
>> qemu_fclose(mis->to_src_file);
>> }
>> diff --git a/migration/ram.c b/migration/ram.c
>> index a161620..9d946a1 100644
>> --- a/migration/ram.c
>> +++ b/migration/ram.c
>> @@ -223,6 +223,7 @@ static RAMBlock *last_sent_block;
>> static ram_addr_t last_offset;
>> static QemuMutex migration_bitmap_mutex;
>> static uint64_t migration_dirty_pages;
>> +static bool ram_cache_enable;
>> static uint32_t last_version;
>> static bool ram_bulk_stage;
>>
>> @@ -2175,6 +2176,16 @@ static inline void *host_from_ram_block_offset(RAMBlock *block,
>> return block->host + offset;
>> }
>>
>> +static inline void *colo_cache_from_block_offset(RAMBlock *block,
>> + ram_addr_t offset)
>> +{
>> + if (!block) {
>> + return NULL;
>> + }
>> +
>> + return block->colo_cache + offset;
>> +}
>> +
>> /*
>> * If a page (or a whole RDMA chunk) has been
>> * determined to be zero, then zap it.
>> @@ -2454,7 +2465,12 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> RAM_SAVE_FLAG_COMPRESS_PAGE | RAM_SAVE_FLAG_XBZRLE)) {
>> RAMBlock *block = ram_block_from_stream(f, addr, flags);
>>
>> - host = host_from_ram_block_offset(block, addr);
>> + /* After going into COLO, we should load the Page into colo_cache */
>> + if (ram_cache_enable) {
>> + host = colo_cache_from_block_offset(block, addr);
>> + } else {
>> + host = host_from_ram_block_offset(block, addr);
>> + }
>> if (!host) {
>> error_report("Illegal RAM offset " RAM_ADDR_FMT, addr);
>> ret = -EINVAL;
>> @@ -2550,6 +2566,57 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> return ret;
>> }
>>
>> +/*
>> + * colo cache: this is for secondary VM, we cache the whole
>> + * memory of the secondary VM, it will be called after first migration.
>> + */
>> +int colo_init_ram_cache(void)
>> +{
>> + RAMBlock *block;
>> +
>> + rcu_read_lock();
>> + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
>> + block->colo_cache = qemu_anon_ram_alloc(block->used_length, NULL);
>> + if (!block->colo_cache) {
>> + error_report("%s: Can't alloc memory for colo cache of block %s,"
>> + "size %zu", __func__, block->idstr,
>> + block->used_length);
>
> Minor one that I didn't spot before; I think that has to be RAM_ADDR_FMT instead of %zu
>
OK, i will fix it, thanks.
> However, other than that;
>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
> Dave
>
>> + goto out_locked;
>> + }
>> + memcpy(block->colo_cache, block->host, block->used_length);
>> + }
>> + rcu_read_unlock();
>> + ram_cache_enable = true;
>> + return 0;
>> +
>> +out_locked:
>> + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
>> + if (block->colo_cache) {
>> + qemu_anon_ram_free(block->colo_cache, block->used_length);
>> + block->colo_cache = NULL;
>> + }
>> + }
>> +
>> + rcu_read_unlock();
>> + return -errno;
>> +}
>> +
>> +void colo_release_ram_cache(void)
>> +{
>> + RAMBlock *block;
>> +
>> + ram_cache_enable = false;
>> +
>> + rcu_read_lock();
>> + QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
>> + if (block->colo_cache) {
>> + qemu_anon_ram_free(block->colo_cache, block->used_length);
>> + block->colo_cache = NULL;
>> + }
>> + }
>> + rcu_read_unlock();
>> +}
>> +
>> static SaveVMHandlers savevm_ram_handlers = {
>> .save_live_setup = ram_save_setup,
>> .save_live_iterate = ram_save_iterate,
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 16/39] ram/COLO: Record the dirty pages that SVM received
2015-12-01 19:36 ` Dr. David Alan Gilbert
@ 2015-12-03 8:29 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-03 8:29 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/2 3:36, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> We record the address of the dirty pages that received,
>> it will help flushing pages that cached into SVM.
>> We record them by re-using migration dirty bitmap.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> ---
>> v11:
>> - Split a new helper function from original
>> host_from_stream_offset() (Dave's suggestion)
>> - Only do recording work in this patch
>> v10:
>> - New patch split from v9's patch 13
>> - Rebase to master to use 'migration_bitmap_rcu'
>> ---
>> migration/ram.c | 30 ++++++++++++++++++++++++++++++
>> 1 file changed, 30 insertions(+)
>>
>> diff --git a/migration/ram.c b/migration/ram.c
>> index 9d946a1..da6bbd6 100644
>> --- a/migration/ram.c
>> +++ b/migration/ram.c
>> @@ -2153,6 +2153,7 @@ static inline RAMBlock *ram_block_from_stream(QEMUFile *f,
>>
>> return block;
>> }
>> +
>> len = qemu_get_byte(f);
>> qemu_get_buffer(f, (uint8_t *)id, len);
>> id[len] = 0;
>
> That blank should probably go, but other than that:
>
I will remove it, thanks.
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
> Dave
>
>> @@ -2179,10 +2180,23 @@ static inline void *host_from_ram_block_offset(RAMBlock *block,
>> static inline void *colo_cache_from_block_offset(RAMBlock *block,
>> ram_addr_t offset)
>> {
>> + unsigned long *bitmap;
>> + long k;
>> +
>> if (!block) {
>> return NULL;
>> }
>>
>> + k = (block->mr->ram_addr + offset) >> TARGET_PAGE_BITS;
>> + bitmap = atomic_rcu_read(&migration_bitmap_rcu)->bmap;
>> + /*
>> + * During colo checkpoint, we need bitmap of these migrated pages.
>> + * It help us to decide which pages in ram cache should be flushed
>> + * into VM's RAM later.
>> + */
>> + if (!test_and_set_bit(k, bitmap)) {
>> + migration_dirty_pages++;
>> + }
>> return block->colo_cache + offset;
>> }
>>
>> @@ -2573,6 +2587,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> int colo_init_ram_cache(void)
>> {
>> RAMBlock *block;
>> + int64_t ram_cache_pages = last_ram_offset() >> TARGET_PAGE_BITS;
>>
>> rcu_read_lock();
>> QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
>> @@ -2587,6 +2602,15 @@ int colo_init_ram_cache(void)
>> }
>> rcu_read_unlock();
>> ram_cache_enable = true;
>> + /*
>> + * Record the dirty pages that sent by PVM, we use this dirty bitmap together
>> + * with to decide which page in cache should be flushed into SVM's RAM. Here
>> + * we use the same name 'migration_bitmap_rcu' as for migration.
>> + */
>> + migration_bitmap_rcu = g_new0(struct BitmapRcu, 1);
>> + migration_bitmap_rcu->bmap = bitmap_new(ram_cache_pages);
>> + migration_dirty_pages = 0;
>> +
>> return 0;
>>
>> out_locked:
>> @@ -2604,9 +2628,15 @@ out_locked:
>> void colo_release_ram_cache(void)
>> {
>> RAMBlock *block;
>> + struct BitmapRcu *bitmap = migration_bitmap_rcu;
>>
>> ram_cache_enable = false;
>>
>> + atomic_rcu_set(&migration_bitmap_rcu, NULL);
>> + if (bitmap) {
>> + call_rcu(bitmap, migration_bitmap_free, rcu);
>> + }
>> +
>> rcu_read_lock();
>> QLIST_FOREACH_RCU(block, &ram_list.blocks, next) {
>> if (block->colo_cache) {
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-12-03 7:21 ` Yang Hongyang
@ 2015-12-03 8:37 ` Hailiang Zhang
2015-12-07 7:38 ` Hailiang Zhang
1 sibling, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-03 8:37 UTC (permalink / raw)
To: Yang Hongyang, Wen Congyang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah
On 2015/12/3 15:21, Yang Hongyang wrote:
>
>
> On 2015年12月03日 14:48, Hailiang Zhang wrote:
>> On 2015/12/3 14:25, Wen Congyang wrote:
>>> On 12/03/2015 11:53 AM, Hailiang Zhang wrote:
>>>> On 2015/12/3 9:17, Wen Congyang wrote:
>>>>> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>>>>>> We add each netdev a default filter-buffer, which will be used for
>>>>>> COLO
>>>>>> or Micro-checkpoint to buffer VM's packets. The name of default
>>>>>> filter-buffer
>>>>>> is 'nop'.
>>>>>> For the default filter-buffer, it will not buffer any packets in
>>>>>> default.
>>>>>> So it has no side effect for the netdev.
>>>>>
>>>>> No, filter-buffer doesn't support vhost, so if you add default
>>>>> filter-buffer
>>>>> for each netdev, you can't use vhost.
>>>>>
>>>>
>>>> Have you tested it ? Did the default filter-buffer break vhost ?
>>>> It's not supposed to break vhost, I will look into it. Thanks.
>>>
>>> Yes, I have tested it. When I want to start a normal vm with vhost, I get
>>> the following error messages:
>>>
>>> qemu-system-x86_64: -netdev tap,id=hn0,queues=1,vhost=on: Vhost is not
>>> supported
>>>
>>
>> Hmm, that is reported by filter in nextfilter_complete(), i will
>> investigate it.
>> We hope the default buffer filter has no side effect on netdev.
>
> I think you'd better add an option to netdev to turn on/off default
> filter, that will solve the problem.
>
OK, i will try to fix it like that. Thanks.
>>
>> Thanks,
>> Hailiang
>>
>>>>
>>>>> Thanks
>>>>> Wen Congyang
>>>>>
>>>>>>
>>>>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>>>> Cc: Jason Wang <jasowang@redhat.com>
>>>>>> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
>>>>>> ---
>>>>>> v11:
>>>>>> - New patch
>>>>>> ---
>>>>>> include/net/filter.h | 3 +++
>>>>>> net/filter-buffer.c | 74
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> net/net.c | 8 ++++++
>>>>>> 3 files changed, 85 insertions(+)
>>>>>>
>>>>>> diff --git a/include/net/filter.h b/include/net/filter.h
>>>>>> index 2deda36..01a7e90 100644
>>>>>> --- a/include/net/filter.h
>>>>>> +++ b/include/net/filter.h
>>>>>> @@ -74,4 +74,7 @@ ssize_t
>>>>>> qemu_netfilter_pass_to_next(NetClientState *sender,
>>>>>> int iovcnt,
>>>>>> void *opaque);
>>>>>>
>>>>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>>>>> + NetFilterDirection direction,
>>>>>> + Error **errp);
>>>>>> #endif /* QEMU_NET_FILTER_H */
>>>>>> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
>>>>>> index 57be149..195af68 100644
>>>>>> --- a/net/filter-buffer.c
>>>>>> +++ b/net/filter-buffer.c
>>>>>> @@ -14,6 +14,12 @@
>>>>>> #include "qapi/qmp/qerror.h"
>>>>>> #include "qapi-visit.h"
>>>>>> #include "qom/object.h"
>>>>>> +#include "net/net.h"
>>>>>> +#include "qapi/qmp/qdict.h"
>>>>>> +#include "qapi/qmp-output-visitor.h"
>>>>>> +#include "qapi/qmp-input-visitor.h"
>>>>>> +#include "monitor/monitor.h"
>>>>>> +#include "qmp-commands.h"
>>>>>>
>>>>>> #define TYPE_FILTER_BUFFER "filter-buffer"
>>>>>>
>>>>>> @@ -26,6 +32,8 @@ typedef struct FilterBufferState {
>>>>>> NetQueue *incoming_queue;
>>>>>> uint32_t interval;
>>>>>> QEMUTimer release_timer;
>>>>>> + bool is_default;
>>>>>> + bool enable_buffer;
>>>>>> } FilterBufferState;
>>>>>>
>>>>>> static void filter_buffer_flush(NetFilterState *nf)
>>>>>> @@ -65,6 +73,10 @@ static ssize_t
>>>>>> filter_buffer_receive_iov(NetFilterState *nf,
>>>>>> {
>>>>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>>>>
>>>>>> + /* Don't buffer any packets if the filter is not enabled */
>>>>>> + if (!s->enable_buffer) {
>>>>>> + return 0;
>>>>>> + }
>>>>>> /*
>>>>>> * We return size when buffer a packet, the sender will take
>>>>>> it as
>>>>>> * a already sent packet, so sent_cb should not be called
>>>>>> later.
>>>>>> @@ -102,6 +114,7 @@ static void
>>>>>> filter_buffer_cleanup(NetFilterState *nf)
>>>>>> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>>>>>> {
>>>>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>>>> + char *path = object_get_canonical_path_component(OBJECT(nf));
>>>>>>
>>>>>> /*
>>>>>> * We may want to accept zero interval when VM FT solutions
>>>>>> like MC
>>>>>> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState
>>>>>> *nf, Error **errp)
>>>>>> }
>>>>>>
>>>>>> s->incoming_queue =
>>>>>> qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
>>>>>> + s->is_default = !strcmp(path, "nop");
>>>>>> if (s->interval) {
>>>>>> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
>>>>>> filter_buffer_release_timer, nf);
>>>>>> @@ -163,6 +177,66 @@ out:
>>>>>> error_propagate(errp, local_err);
>>>>>> }
>>>>>>
>>>>>> +/*
>>>>>> +* This will be used by COLO or MC FT, for which they will need
>>>>>> +* to buffer the packets of VM's net devices, Here we add a default
>>>>>> +* buffer filter for each netdev. The name of default buffer filter is
>>>>>> +* 'nop'
>>>>>> +*/
>>>>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>>>>> + NetFilterDirection direction,
>>>>>> + Error **errp)
>>>>>> +{
>>>>>> + QmpOutputVisitor *qov;
>>>>>> + QmpInputVisitor *qiv;
>>>>>> + Visitor *ov, *iv;
>>>>>> + QObject *obj = NULL;
>>>>>> + QDict *qdict;
>>>>>> + void *dummy = NULL;
>>>>>> + const char *id = "nop";
>>>>>> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
>>>>>> + NetClientState *nc = qemu_find_netdev(netdev_id);
>>>>>> + Error *err = NULL;
>>>>>> +
>>>>>> + /* FIXME: Not support multiple queues */
>>>>>> + if (!nc || nc->queue_index > 1) {
>>>>>> + return;
>>>>>> + }
>>>>>> + qov = qmp_output_visitor_new();
>>>>>> + ov = qmp_output_get_visitor(qov);
>>>>>> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
>>>>>> + if (err) {
>>>>>> + goto out;
>>>>>> + }
>>>>>> + visit_type_str(ov, &nc->name, "netdev", &err);
>>>>>> + if (err) {
>>>>>> + goto out;
>>>>>> + }
>>>>>> + visit_type_str(ov, &queue, "queue", &err);
>>>>>> + if (err) {
>>>>>> + goto out;
>>>>>> + }
>>>>>> + visit_end_struct(ov, &err);
>>>>>> + if (err) {
>>>>>> + goto out;
>>>>>> + }
>>>>>> + obj = qmp_output_get_qobject(qov);
>>>>>> + g_assert(obj != NULL);
>>>>>> + qdict = qobject_to_qdict(obj);
>>>>>> + qmp_output_visitor_cleanup(qov);
>>>>>> +
>>>>>> + qiv = qmp_input_visitor_new(obj);
>>>>>> + iv = qmp_input_get_visitor(qiv);
>>>>>> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
>>>>>> + qmp_input_visitor_cleanup(qiv);
>>>>>> + qobject_decref(obj);
>>>>>> +out:
>>>>>> + g_free(queue);
>>>>>> + if (err) {
>>>>>> + error_propagate(errp, err);
>>>>>> + }
>>>>>> +}
>>>>>> +
>>>>>> static void filter_buffer_init(Object *obj)
>>>>>> {
>>>>>> object_property_add(obj, "interval", "int",
>>>>>> diff --git a/net/net.c b/net/net.c
>>>>>> index ade6051..b36d49f 100644
>>>>>> --- a/net/net.c
>>>>>> +++ b/net/net.c
>>>>>> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void
>>>>>> *object, int is_netdev, Error **errp)
>>>>>> }
>>>>>> return -1;
>>>>>> }
>>>>>> +
>>>>>> + if (is_netdev) {
>>>>>> + const Netdev *netdev = object;
>>>>>> +
>>>>>> + netdev_add_default_filter_buffer(netdev->id,
>>>>>> + NET_FILTER_DIRECTION_RX,
>>>>>> + errp);
>>>>>> + }
>>>>>> return 0;
>>>>>> }
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> .
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> .
>>>>
>>>
>>>
>>>
>>>
>>> .
>>>
>>
>>
>>
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 18/39] COLO: Flush PVM's cached RAM into SVM's memory
2015-12-01 20:06 ` Dr. David Alan Gilbert
@ 2015-12-03 8:50 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-03 8:50 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/2 4:06, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> During the time of VM's running, PVM may dirty some pages, we will transfer
>> PVM's dirty pages to SVM and store them into SVM's RAM cache at next checkpoint
>> time. So, the content of SVM's RAM cache will always be some with PVM's memory
>> after checkpoint.
>>
>> Instead of flushing all content of PVM's RAM cache into SVM's MEMORY,
>> we do this in a more efficient way:
>> Only flush any page that dirtied by PVM since last checkpoint.
>> In this way, we can ensure SVM's memory same with PVM's.
>>
>> Besides, we must ensure flush RAM cache before load device state.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> Signed-off-by: Gonglei <arei.gonglei@huawei.com>
>> ---
>> v11:
>> - Move the place of 'need_flush' (Dave's suggestion)
>> - Remove unused 'DPRINTF("Flush ram_cache\n")'
>> v10:
>> - trace the number of dirty pages that be received.
>> ---
>> include/migration/migration.h | 1 +
>> migration/colo.c | 2 --
>> migration/ram.c | 37 +++++++++++++++++++++++++++++++++++++
>> trace-events | 1 +
>> 4 files changed, 39 insertions(+), 2 deletions(-)
>>
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index e41372d..221176b 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -336,4 +336,5 @@ PostcopyState postcopy_state_set(PostcopyState new_state);
>> /* ram cache */
>> int colo_init_ram_cache(void);
>> void colo_release_ram_cache(void);
>> +void colo_flush_ram_cache(void);
>> #endif
>> diff --git a/migration/colo.c b/migration/colo.c
>> index 5ac8ff2..4095d97 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -385,8 +385,6 @@ void *colo_process_incoming_thread(void *opaque)
>> }
>> qemu_mutex_unlock_iothread();
>>
>> - /* TODO: flush vm state */
>> -
>
> Might have been better to put the TODO in a place that needed to be changed!
>
OK~
>> ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_VMSTATE_LOADED, 0);
>> if (ret < 0) {
>> goto out;
>> diff --git a/migration/ram.c b/migration/ram.c
>> index da6bbd6..4f37144 100644
>> --- a/migration/ram.c
>> +++ b/migration/ram.c
>> @@ -2448,6 +2448,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> * be atomic
>> */
>> bool postcopy_running = postcopy_state_get() >= POSTCOPY_INCOMING_LISTENING;
>> + bool need_flush = false;
>>
>> seq_iter++;
>>
>> @@ -2482,6 +2483,7 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> /* After going into COLO, we should load the Page into colo_cache */
>> if (ram_cache_enable) {
>> host = colo_cache_from_block_offset(block, addr);
>> + need_flush = true;
>> } else {
>> host = host_from_ram_block_offset(block, addr);
>> }
>> @@ -2575,6 +2577,10 @@ static int ram_load(QEMUFile *f, void *opaque, int version_id)
>> }
>>
>> rcu_read_unlock();
>> +
>> + if (!ret && ram_cache_enable && need_flush) {
>> + colo_flush_ram_cache();
>> + }
>> DPRINTF("Completed load of VM with exit code %d seq iteration "
>> "%" PRIu64 "\n", ret, seq_iter);
>> return ret;
>> @@ -2647,6 +2653,37 @@ void colo_release_ram_cache(void)
>> rcu_read_unlock();
>> }
>>
>> +/*
>> + * Flush content of RAM cache into SVM's memory.
>> + * Only flush the pages that be dirtied by PVM or SVM or both.
>> + */
>> +void colo_flush_ram_cache(void)
>> +{
>> + RAMBlock *block = NULL;
>> + void *dst_host;
>> + void *src_host;
>> + ram_addr_t offset = 0;
>> +
>> + trace_colo_flush_ram_cache(migration_dirty_pages);
>> + rcu_read_lock();
>> + block = QLIST_FIRST_RCU(&ram_list.blocks);
>> + while (block) {
>> + ram_addr_t ram_addr_abs;
>> + offset = migration_bitmap_find_dirty(block, offset, &ram_addr_abs);
>> + migration_bitmap_clear_dirty(ram_addr_abs);
>> + if (offset >= block->used_length) {
>> + offset = 0;
>> + block = QLIST_NEXT_RCU(block, next);
>> + } else {
>> + dst_host = block->host + offset;
>> + src_host = block->colo_cache + offset;
>> + memcpy(dst_host, src_host, TARGET_PAGE_SIZE);
>> + }
>> + }
>> + rcu_read_unlock();
>
> If you added a trace point here as well, it would make it very easy
> to measure how long the flush was taking.
>
Good idea, i will add it in next version.
>> + assert(migration_dirty_pages == 0);
>> +}
>> +
>> static SaveVMHandlers savevm_ram_handlers = {
>> .save_live_setup = ram_save_setup,
>> .save_live_iterate = ram_save_iterate,
>> diff --git a/trace-events b/trace-events
>> index f8a0959..f158d2a 100644
>> --- a/trace-events
>> +++ b/trace-events
>> @@ -1264,6 +1264,7 @@ migration_throttle(void) ""
>> ram_load_postcopy_loop(uint64_t addr, int flags) "@%" PRIx64 " %x"
>> ram_postcopy_send_discard_bitmap(void) ""
>> ram_save_queue_pages(const char *rbname, size_t start, size_t len) "%s: start: %zx len: %zx"
>> +colo_flush_ram_cache(uint64_t dirty_pages) "dirty_pages %" PRIu64""
>
> Minor; I think you can remove the "" at the end.
>
> Other than those minor things (and those double-space!):
>
OK, i will fix them all in next version, thanks.
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
> Dave
>
>>
>> # hw/display/qxl.c
>> disable qxl_interface_set_mm_time(int qid, uint32_t mm_time) "%d %d"
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-12-03 7:21 ` Yang Hongyang
2015-12-03 8:37 ` Hailiang Zhang
@ 2015-12-07 7:38 ` Hailiang Zhang
2015-12-08 1:49 ` Yang Hongyang
1 sibling, 1 reply; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-07 7:38 UTC (permalink / raw)
To: Yang Hongyang, Wen Congyang, qemu-devel
Cc: lizhijian, quintela, Jason Wang, yunhong.jiang, eddie.dong,
peter.huangpeng, dgilbert, arei.gonglei, stefanha, amit.shah
On 2015/12/3 15:21, Yang Hongyang wrote:
>
>
> On 2015年12月03日 14:48, Hailiang Zhang wrote:
>> On 2015/12/3 14:25, Wen Congyang wrote:
>>> On 12/03/2015 11:53 AM, Hailiang Zhang wrote:
>>>> On 2015/12/3 9:17, Wen Congyang wrote:
>>>>> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>>>>>> We add each netdev a default filter-buffer, which will be used for
>>>>>> COLO
>>>>>> or Micro-checkpoint to buffer VM's packets. The name of default
>>>>>> filter-buffer
>>>>>> is 'nop'.
>>>>>> For the default filter-buffer, it will not buffer any packets in
>>>>>> default.
>>>>>> So it has no side effect for the netdev.
>>>>>
>>>>> No, filter-buffer doesn't support vhost, so if you add default
>>>>> filter-buffer
>>>>> for each netdev, you can't use vhost.
>>>>>
>>>>
>>>> Have you tested it ? Did the default filter-buffer break vhost ?
>>>> It's not supposed to break vhost, I will look into it. Thanks.
>>>
>>> Yes, I have tested it. When I want to start a normal vm with vhost, I get
>>> the following error messages:
>>>
>>> qemu-system-x86_64: -netdev tap,id=hn0,queues=1,vhost=on: Vhost is not
>>> supported
>>>
>>
>> Hmm, that is reported by filter in nextfilter_complete(), i will
>> investigate it.
>> We hope the default buffer filter has no side effect on netdev.
>
> I think you'd better add an option to netdev to turn on/off default
> filter, that will solve the problem.
>
Hi Hongyang,
I have fix this problem in another way, just skip add filter for vhost-user
netdev. And by the way, I didn't add any option to netdev to control
the default filter's on/off, since i'd like to control the default filter
just in filter layer, i add an 'enabled' and an 'is-default' flag in
struct NetFilterState, and for the default filter buffer, its status is 'off' (disabled),
so it will not buffer any packets, we will skip it in filter_receive_iov() to ensure
the packets will not go through the process of filter.
Besides, for hmp command 'info network', we will show the on/off status for filter, just
like: (Or, maybe we don't show the filter that is disabled ?)
(qemu) info network
net-pci0: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:56
\ bn0: index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
filters:
- nop: type=filter-buffer,interval=0,netdev=bn0,queue=rx,status=off
- f0: type=filter-buffer,interval=1000,netdev=bn0,queue=rx,status=on
Is this acceptable?
Thanks.
Hailiang
>>
>> Thanks,
>> Hailiang
>>
>>>>
>>>>> Thanks
>>>>> Wen Congyang
>>>>>
>>>>>>
>>>>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>>>> Cc: Jason Wang <jasowang@redhat.com>
>>>>>> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
>>>>>> ---
>>>>>> v11:
>>>>>> - New patch
>>>>>> ---
>>>>>> include/net/filter.h | 3 +++
>>>>>> net/filter-buffer.c | 74
>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>> net/net.c | 8 ++++++
>>>>>> 3 files changed, 85 insertions(+)
>>>>>>
>>>>>> diff --git a/include/net/filter.h b/include/net/filter.h
>>>>>> index 2deda36..01a7e90 100644
>>>>>> --- a/include/net/filter.h
>>>>>> +++ b/include/net/filter.h
>>>>>> @@ -74,4 +74,7 @@ ssize_t
>>>>>> qemu_netfilter_pass_to_next(NetClientState *sender,
>>>>>> int iovcnt,
>>>>>> void *opaque);
>>>>>>
>>>>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>>>>> + NetFilterDirection direction,
>>>>>> + Error **errp);
>>>>>> #endif /* QEMU_NET_FILTER_H */
>>>>>> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
>>>>>> index 57be149..195af68 100644
>>>>>> --- a/net/filter-buffer.c
>>>>>> +++ b/net/filter-buffer.c
>>>>>> @@ -14,6 +14,12 @@
>>>>>> #include "qapi/qmp/qerror.h"
>>>>>> #include "qapi-visit.h"
>>>>>> #include "qom/object.h"
>>>>>> +#include "net/net.h"
>>>>>> +#include "qapi/qmp/qdict.h"
>>>>>> +#include "qapi/qmp-output-visitor.h"
>>>>>> +#include "qapi/qmp-input-visitor.h"
>>>>>> +#include "monitor/monitor.h"
>>>>>> +#include "qmp-commands.h"
>>>>>>
>>>>>> #define TYPE_FILTER_BUFFER "filter-buffer"
>>>>>>
>>>>>> @@ -26,6 +32,8 @@ typedef struct FilterBufferState {
>>>>>> NetQueue *incoming_queue;
>>>>>> uint32_t interval;
>>>>>> QEMUTimer release_timer;
>>>>>> + bool is_default;
>>>>>> + bool enable_buffer;
>>>>>> } FilterBufferState;
>>>>>>
>>>>>> static void filter_buffer_flush(NetFilterState *nf)
>>>>>> @@ -65,6 +73,10 @@ static ssize_t
>>>>>> filter_buffer_receive_iov(NetFilterState *nf,
>>>>>> {
>>>>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>>>>
>>>>>> + /* Don't buffer any packets if the filter is not enabled */
>>>>>> + if (!s->enable_buffer) {
>>>>>> + return 0;
>>>>>> + }
>>>>>> /*
>>>>>> * We return size when buffer a packet, the sender will take
>>>>>> it as
>>>>>> * a already sent packet, so sent_cb should not be called
>>>>>> later.
>>>>>> @@ -102,6 +114,7 @@ static void
>>>>>> filter_buffer_cleanup(NetFilterState *nf)
>>>>>> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>>>>>> {
>>>>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>>>> + char *path = object_get_canonical_path_component(OBJECT(nf));
>>>>>>
>>>>>> /*
>>>>>> * We may want to accept zero interval when VM FT solutions
>>>>>> like MC
>>>>>> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState
>>>>>> *nf, Error **errp)
>>>>>> }
>>>>>>
>>>>>> s->incoming_queue =
>>>>>> qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
>>>>>> + s->is_default = !strcmp(path, "nop");
>>>>>> if (s->interval) {
>>>>>> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
>>>>>> filter_buffer_release_timer, nf);
>>>>>> @@ -163,6 +177,66 @@ out:
>>>>>> error_propagate(errp, local_err);
>>>>>> }
>>>>>>
>>>>>> +/*
>>>>>> +* This will be used by COLO or MC FT, for which they will need
>>>>>> +* to buffer the packets of VM's net devices, Here we add a default
>>>>>> +* buffer filter for each netdev. The name of default buffer filter is
>>>>>> +* 'nop'
>>>>>> +*/
>>>>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>>>>> + NetFilterDirection direction,
>>>>>> + Error **errp)
>>>>>> +{
>>>>>> + QmpOutputVisitor *qov;
>>>>>> + QmpInputVisitor *qiv;
>>>>>> + Visitor *ov, *iv;
>>>>>> + QObject *obj = NULL;
>>>>>> + QDict *qdict;
>>>>>> + void *dummy = NULL;
>>>>>> + const char *id = "nop";
>>>>>> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
>>>>>> + NetClientState *nc = qemu_find_netdev(netdev_id);
>>>>>> + Error *err = NULL;
>>>>>> +
>>>>>> + /* FIXME: Not support multiple queues */
>>>>>> + if (!nc || nc->queue_index > 1) {
>>>>>> + return;
>>>>>> + }
>>>>>> + qov = qmp_output_visitor_new();
>>>>>> + ov = qmp_output_get_visitor(qov);
>>>>>> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
>>>>>> + if (err) {
>>>>>> + goto out;
>>>>>> + }
>>>>>> + visit_type_str(ov, &nc->name, "netdev", &err);
>>>>>> + if (err) {
>>>>>> + goto out;
>>>>>> + }
>>>>>> + visit_type_str(ov, &queue, "queue", &err);
>>>>>> + if (err) {
>>>>>> + goto out;
>>>>>> + }
>>>>>> + visit_end_struct(ov, &err);
>>>>>> + if (err) {
>>>>>> + goto out;
>>>>>> + }
>>>>>> + obj = qmp_output_get_qobject(qov);
>>>>>> + g_assert(obj != NULL);
>>>>>> + qdict = qobject_to_qdict(obj);
>>>>>> + qmp_output_visitor_cleanup(qov);
>>>>>> +
>>>>>> + qiv = qmp_input_visitor_new(obj);
>>>>>> + iv = qmp_input_get_visitor(qiv);
>>>>>> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
>>>>>> + qmp_input_visitor_cleanup(qiv);
>>>>>> + qobject_decref(obj);
>>>>>> +out:
>>>>>> + g_free(queue);
>>>>>> + if (err) {
>>>>>> + error_propagate(errp, err);
>>>>>> + }
>>>>>> +}
>>>>>> +
>>>>>> static void filter_buffer_init(Object *obj)
>>>>>> {
>>>>>> object_property_add(obj, "interval", "int",
>>>>>> diff --git a/net/net.c b/net/net.c
>>>>>> index ade6051..b36d49f 100644
>>>>>> --- a/net/net.c
>>>>>> +++ b/net/net.c
>>>>>> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void
>>>>>> *object, int is_netdev, Error **errp)
>>>>>> }
>>>>>> return -1;
>>>>>> }
>>>>>> +
>>>>>> + if (is_netdev) {
>>>>>> + const Netdev *netdev = object;
>>>>>> +
>>>>>> + netdev_add_default_filter_buffer(netdev->id,
>>>>>> + NET_FILTER_DIRECTION_RX,
>>>>>> + errp);
>>>>>> + }
>>>>>> return 0;
>>>>>> }
>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> .
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> .
>>>>
>>>
>>>
>>>
>>>
>>> .
>>>
>>
>>
>>
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev
2015-12-07 7:38 ` Hailiang Zhang
@ 2015-12-08 1:49 ` Yang Hongyang
0 siblings, 0 replies; 95+ messages in thread
From: Yang Hongyang @ 2015-12-08 1:49 UTC (permalink / raw)
To: Hailiang Zhang, qemu-devel, Jason Wang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, dgilbert,
peter.huangpeng, arei.gonglei, stefanha, amit.shah
On 2015年12月07日 15:38, Hailiang Zhang wrote:
> On 2015/12/3 15:21, Yang Hongyang wrote:
>>
>>
>> On 2015年12月03日 14:48, Hailiang Zhang wrote:
>>> On 2015/12/3 14:25, Wen Congyang wrote:
>>>> On 12/03/2015 11:53 AM, Hailiang Zhang wrote:
>>>>> On 2015/12/3 9:17, Wen Congyang wrote:
>>>>>> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>>>>>>> We add each netdev a default filter-buffer, which will be used for
>>>>>>> COLO
>>>>>>> or Micro-checkpoint to buffer VM's packets. The name of default
>>>>>>> filter-buffer
>>>>>>> is 'nop'.
>>>>>>> For the default filter-buffer, it will not buffer any packets in
>>>>>>> default.
>>>>>>> So it has no side effect for the netdev.
>>>>>>
>>>>>> No, filter-buffer doesn't support vhost, so if you add default
>>>>>> filter-buffer
>>>>>> for each netdev, you can't use vhost.
>>>>>>
>>>>>
>>>>> Have you tested it ? Did the default filter-buffer break vhost ?
>>>>> It's not supposed to break vhost, I will look into it. Thanks.
>>>>
>>>> Yes, I have tested it. When I want to start a normal vm with vhost,
>>>> I get
>>>> the following error messages:
>>>>
>>>> qemu-system-x86_64: -netdev tap,id=hn0,queues=1,vhost=on: Vhost is not
>>>> supported
>>>>
>>>
>>> Hmm, that is reported by filter in nextfilter_complete(), i will
>>> investigate it.
>>> We hope the default buffer filter has no side effect on netdev.
>>
>> I think you'd better add an option to netdev to turn on/off default
>> filter, that will solve the problem.
>>
>
> Hi Hongyang,
>
> I have fix this problem in another way, just skip add filter for vhost-user
> netdev. And by the way, I didn't add any option to netdev to control
> the default filter's on/off, since i'd like to control the default filter
> just in filter layer, i add an 'enabled' and an 'is-default' flag in
> struct NetFilterState, and for the default filter buffer, its status is
> 'off' (disabled),
> so it will not buffer any packets, we will skip it in
> filter_receive_iov() to ensure
> the packets will not go through the process of filter.
>
> Besides, for hmp command 'info network', we will show the on/off status
> for filter, just
> like: (Or, maybe we don't show the filter that is disabled ?)
> (qemu) info network
> net-pci0: index=0,type=nic,model=virtio-net-pci,macaddr=52:54:00:12:34:56
> \ bn0:
> index=0,type=tap,ifname=tap0,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown
>
> filters:
> - nop: type=filter-buffer,interval=0,netdev=bn0,queue=rx,status=off
> - f0: type=filter-buffer,interval=1000,netdev=bn0,queue=rx,status=on
>
> Is this acceptable?
Hi Hailiang, thanks for the explanation.
It's up to Jason though, cause he's the network Maintainer :)
>
> Thanks.
> Hailiang
>
>>>
>>> Thanks,
>>> Hailiang
>>>
>>>>>
>>>>>> Thanks
>>>>>> Wen Congyang
>>>>>>
>>>>>>>
>>>>>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>>>>> Cc: Jason Wang <jasowang@redhat.com>
>>>>>>> Cc: Yang Hongyang <hongyang.yang@easystack.cn>
>>>>>>> ---
>>>>>>> v11:
>>>>>>> - New patch
>>>>>>> ---
>>>>>>> include/net/filter.h | 3 +++
>>>>>>> net/filter-buffer.c | 74
>>>>>>> ++++++++++++++++++++++++++++++++++++++++++++++++++++
>>>>>>> net/net.c | 8 ++++++
>>>>>>> 3 files changed, 85 insertions(+)
>>>>>>>
>>>>>>> diff --git a/include/net/filter.h b/include/net/filter.h
>>>>>>> index 2deda36..01a7e90 100644
>>>>>>> --- a/include/net/filter.h
>>>>>>> +++ b/include/net/filter.h
>>>>>>> @@ -74,4 +74,7 @@ ssize_t
>>>>>>> qemu_netfilter_pass_to_next(NetClientState *sender,
>>>>>>> int iovcnt,
>>>>>>> void *opaque);
>>>>>>>
>>>>>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>>>>>> + NetFilterDirection direction,
>>>>>>> + Error **errp);
>>>>>>> #endif /* QEMU_NET_FILTER_H */
>>>>>>> diff --git a/net/filter-buffer.c b/net/filter-buffer.c
>>>>>>> index 57be149..195af68 100644
>>>>>>> --- a/net/filter-buffer.c
>>>>>>> +++ b/net/filter-buffer.c
>>>>>>> @@ -14,6 +14,12 @@
>>>>>>> #include "qapi/qmp/qerror.h"
>>>>>>> #include "qapi-visit.h"
>>>>>>> #include "qom/object.h"
>>>>>>> +#include "net/net.h"
>>>>>>> +#include "qapi/qmp/qdict.h"
>>>>>>> +#include "qapi/qmp-output-visitor.h"
>>>>>>> +#include "qapi/qmp-input-visitor.h"
>>>>>>> +#include "monitor/monitor.h"
>>>>>>> +#include "qmp-commands.h"
>>>>>>>
>>>>>>> #define TYPE_FILTER_BUFFER "filter-buffer"
>>>>>>>
>>>>>>> @@ -26,6 +32,8 @@ typedef struct FilterBufferState {
>>>>>>> NetQueue *incoming_queue;
>>>>>>> uint32_t interval;
>>>>>>> QEMUTimer release_timer;
>>>>>>> + bool is_default;
>>>>>>> + bool enable_buffer;
>>>>>>> } FilterBufferState;
>>>>>>>
>>>>>>> static void filter_buffer_flush(NetFilterState *nf)
>>>>>>> @@ -65,6 +73,10 @@ static ssize_t
>>>>>>> filter_buffer_receive_iov(NetFilterState *nf,
>>>>>>> {
>>>>>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>>>>>
>>>>>>> + /* Don't buffer any packets if the filter is not enabled */
>>>>>>> + if (!s->enable_buffer) {
>>>>>>> + return 0;
>>>>>>> + }
>>>>>>> /*
>>>>>>> * We return size when buffer a packet, the sender will take
>>>>>>> it as
>>>>>>> * a already sent packet, so sent_cb should not be called
>>>>>>> later.
>>>>>>> @@ -102,6 +114,7 @@ static void
>>>>>>> filter_buffer_cleanup(NetFilterState *nf)
>>>>>>> static void filter_buffer_setup(NetFilterState *nf, Error **errp)
>>>>>>> {
>>>>>>> FilterBufferState *s = FILTER_BUFFER(nf);
>>>>>>> + char *path = object_get_canonical_path_component(OBJECT(nf));
>>>>>>>
>>>>>>> /*
>>>>>>> * We may want to accept zero interval when VM FT solutions
>>>>>>> like MC
>>>>>>> @@ -114,6 +127,7 @@ static void filter_buffer_setup(NetFilterState
>>>>>>> *nf, Error **errp)
>>>>>>> }
>>>>>>>
>>>>>>> s->incoming_queue =
>>>>>>> qemu_new_net_queue(qemu_netfilter_pass_to_next, nf);
>>>>>>> + s->is_default = !strcmp(path, "nop");
>>>>>>> if (s->interval) {
>>>>>>> timer_init_us(&s->release_timer, QEMU_CLOCK_VIRTUAL,
>>>>>>> filter_buffer_release_timer, nf);
>>>>>>> @@ -163,6 +177,66 @@ out:
>>>>>>> error_propagate(errp, local_err);
>>>>>>> }
>>>>>>>
>>>>>>> +/*
>>>>>>> +* This will be used by COLO or MC FT, for which they will need
>>>>>>> +* to buffer the packets of VM's net devices, Here we add a default
>>>>>>> +* buffer filter for each netdev. The name of default buffer
>>>>>>> filter is
>>>>>>> +* 'nop'
>>>>>>> +*/
>>>>>>> +void netdev_add_default_filter_buffer(const char *netdev_id,
>>>>>>> + NetFilterDirection direction,
>>>>>>> + Error **errp)
>>>>>>> +{
>>>>>>> + QmpOutputVisitor *qov;
>>>>>>> + QmpInputVisitor *qiv;
>>>>>>> + Visitor *ov, *iv;
>>>>>>> + QObject *obj = NULL;
>>>>>>> + QDict *qdict;
>>>>>>> + void *dummy = NULL;
>>>>>>> + const char *id = "nop";
>>>>>>> + char *queue = g_strdup(NetFilterDirection_lookup[direction]);
>>>>>>> + NetClientState *nc = qemu_find_netdev(netdev_id);
>>>>>>> + Error *err = NULL;
>>>>>>> +
>>>>>>> + /* FIXME: Not support multiple queues */
>>>>>>> + if (!nc || nc->queue_index > 1) {
>>>>>>> + return;
>>>>>>> + }
>>>>>>> + qov = qmp_output_visitor_new();
>>>>>>> + ov = qmp_output_get_visitor(qov);
>>>>>>> + visit_start_struct(ov, &dummy, NULL, NULL, 0, &err);
>>>>>>> + if (err) {
>>>>>>> + goto out;
>>>>>>> + }
>>>>>>> + visit_type_str(ov, &nc->name, "netdev", &err);
>>>>>>> + if (err) {
>>>>>>> + goto out;
>>>>>>> + }
>>>>>>> + visit_type_str(ov, &queue, "queue", &err);
>>>>>>> + if (err) {
>>>>>>> + goto out;
>>>>>>> + }
>>>>>>> + visit_end_struct(ov, &err);
>>>>>>> + if (err) {
>>>>>>> + goto out;
>>>>>>> + }
>>>>>>> + obj = qmp_output_get_qobject(qov);
>>>>>>> + g_assert(obj != NULL);
>>>>>>> + qdict = qobject_to_qdict(obj);
>>>>>>> + qmp_output_visitor_cleanup(qov);
>>>>>>> +
>>>>>>> + qiv = qmp_input_visitor_new(obj);
>>>>>>> + iv = qmp_input_get_visitor(qiv);
>>>>>>> + object_add(TYPE_FILTER_BUFFER, id, qdict, iv, &err);
>>>>>>> + qmp_input_visitor_cleanup(qiv);
>>>>>>> + qobject_decref(obj);
>>>>>>> +out:
>>>>>>> + g_free(queue);
>>>>>>> + if (err) {
>>>>>>> + error_propagate(errp, err);
>>>>>>> + }
>>>>>>> +}
>>>>>>> +
>>>>>>> static void filter_buffer_init(Object *obj)
>>>>>>> {
>>>>>>> object_property_add(obj, "interval", "int",
>>>>>>> diff --git a/net/net.c b/net/net.c
>>>>>>> index ade6051..b36d49f 100644
>>>>>>> --- a/net/net.c
>>>>>>> +++ b/net/net.c
>>>>>>> @@ -1028,6 +1028,14 @@ static int net_client_init1(const void
>>>>>>> *object, int is_netdev, Error **errp)
>>>>>>> }
>>>>>>> return -1;
>>>>>>> }
>>>>>>> +
>>>>>>> + if (is_netdev) {
>>>>>>> + const Netdev *netdev = object;
>>>>>>> +
>>>>>>> + netdev_add_default_filter_buffer(netdev->id,
>>>>>>> + NET_FILTER_DIRECTION_RX,
>>>>>>> + errp);
>>>>>>> + }
>>>>>>> return 0;
>>>>>>> }
>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> .
>>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> .
>>>>>
>>>>
>>>>
>>>>
>>>>
>>>> .
>>>>
>>>
>>>
>>>
>>
>
>
>
--
Thanks,
Yang
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 19/39] COLO: Add checkpoint-delay parameter for migrate-set-parameters
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 19/39] COLO: Add checkpoint-delay parameter for migrate-set-parameters zhanghailiang
@ 2015-12-09 18:50 ` Dr. David Alan Gilbert
2015-12-11 3:20 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-09 18:50 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, qemu-devel, arei.gonglei, stefanha,
amit.shah, Luiz Capitulino, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> Add checkpoint-delay parameter for migrate-set-parameters, so that
> we can control the checkpoint frequency when COLO is in periodic mode.
>
> Cc: Luiz Capitulino <lcapitulino@redhat.com>
> Cc: Eric Blake <eblake@redhat.com>
> Cc: Markus Armbruster <armbru@redhat.com>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
It's probably a good idea to make this x- but other than that:
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> v11:
> - Move this patch ahead of the patch where uses 'checkpoint_delay'
> (Dave's suggestion)
> v10:
> - Fix related qmp command
> ---
> hmp.c | 7 +++++++
> migration/migration.c | 23 ++++++++++++++++++++++-
> qapi-schema.json | 19 ++++++++++++++++---
> qmp-commands.hx | 2 +-
> 4 files changed, 46 insertions(+), 5 deletions(-)
>
> diff --git a/hmp.c b/hmp.c
> index 2140605..ec038f1 100644
> --- a/hmp.c
> +++ b/hmp.c
> @@ -284,6 +284,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
> monitor_printf(mon, " %s: %" PRId64,
> MigrationParameter_lookup[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT],
> params->x_cpu_throttle_increment);
> + monitor_printf(mon, " %s: %" PRId64,
> + MigrationParameter_lookup[MIGRATION_PARAMETER_CHECKPOINT_DELAY],
> + params->checkpoint_delay);
> monitor_printf(mon, "\n");
> }
>
> @@ -1237,6 +1240,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
> bool has_decompress_threads = false;
> bool has_x_cpu_throttle_initial = false;
> bool has_x_cpu_throttle_increment = false;
> + bool has_checkpoint_delay = false;
> int i;
>
> for (i = 0; i < MIGRATION_PARAMETER_MAX; i++) {
> @@ -1256,6 +1260,8 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
> break;
> case MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT:
> has_x_cpu_throttle_increment = true;
> + case MIGRATION_PARAMETER_CHECKPOINT_DELAY:
> + has_checkpoint_delay = true;
> break;
> }
> qmp_migrate_set_parameters(has_compress_level, value,
> @@ -1263,6 +1269,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
> has_decompress_threads, value,
> has_x_cpu_throttle_initial, value,
> has_x_cpu_throttle_increment, value,
> + has_checkpoint_delay, value,
> &err);
> break;
> }
> diff --git a/migration/migration.c b/migration/migration.c
> index a4c690d..48545f3 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -56,6 +56,11 @@
> /* Migration XBZRLE default cache size */
> #define DEFAULT_MIGRATE_CACHE_SIZE (64 * 1024 * 1024)
>
> +/* The delay time (in ms) between two COLO checkpoints
> + * Note: Please change this default value to 10000 when we support hybrid mode.
> + */
> +#define DEFAULT_MIGRATE_CHECKPOINT_DELAY 200
> +
> static NotifierList migration_state_notifiers =
> NOTIFIER_LIST_INITIALIZER(migration_state_notifiers);
>
> @@ -91,6 +96,8 @@ MigrationState *migrate_get_current(void)
> DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL,
> .parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
> DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT,
> + .parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] =
> + DEFAULT_MIGRATE_CHECKPOINT_DELAY,
> };
>
> if (!once) {
> @@ -530,6 +537,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
> s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
> params->x_cpu_throttle_increment =
> s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
> + params->checkpoint_delay =
> + s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY];
>
> return params;
> }
> @@ -736,7 +745,10 @@ void qmp_migrate_set_parameters(bool has_compress_level,
> bool has_x_cpu_throttle_initial,
> int64_t x_cpu_throttle_initial,
> bool has_x_cpu_throttle_increment,
> - int64_t x_cpu_throttle_increment, Error **errp)
> + int64_t x_cpu_throttle_increment,
> + bool has_checkpoint_delay,
> + int64_t checkpoint_delay,
> + Error **errp)
> {
> MigrationState *s = migrate_get_current();
>
> @@ -771,6 +783,11 @@ void qmp_migrate_set_parameters(bool has_compress_level,
> "x_cpu_throttle_increment",
> "an integer in the range of 1 to 99");
> }
> + if (has_checkpoint_delay && (checkpoint_delay < 0)) {
> + error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
> + "checkpoint_delay",
> + "is invalid, it should be positive");
> + }
>
> if (has_compress_level) {
> s->parameters[MIGRATION_PARAMETER_COMPRESS_LEVEL] = compress_level;
> @@ -791,6 +808,10 @@ void qmp_migrate_set_parameters(bool has_compress_level,
> s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
> x_cpu_throttle_increment;
> }
> +
> + if (has_checkpoint_delay) {
> + s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] = checkpoint_delay;
> + }
> }
>
> void qmp_migrate_start_postcopy(Error **errp)
> diff --git a/qapi-schema.json b/qapi-schema.json
> index 5ee089d..b4b47ed 100644
> --- a/qapi-schema.json
> +++ b/qapi-schema.json
> @@ -625,11 +625,16 @@
> # @x-cpu-throttle-increment: throttle percentage increase each time
> # auto-converge detects that migration is not making
> # progress. The default value is 10. (Since 2.5)
> +#
> +# @checkpoint-delay: The delay time (in ms) between two COLO checkpoints in
> +# periodic mode. (Since 2.6)
> +#
> # Since: 2.4
> ##
> { 'enum': 'MigrationParameter',
> 'data': ['compress-level', 'compress-threads', 'decompress-threads',
> - 'x-cpu-throttle-initial', 'x-cpu-throttle-increment'] }
> + 'x-cpu-throttle-initial', 'x-cpu-throttle-increment',
> + 'checkpoint-delay' ] }
>
> #
> # @migrate-set-parameters
> @@ -649,6 +654,9 @@
> # @x-cpu-throttle-increment: throttle percentage increase each time
> # auto-converge detects that migration is not making
> # progress. The default value is 10. (Since 2.5)
> +#
> +# @checkpoint-delay: the delay time between two checkpoints. (Since 2.6)
> +#
> # Since: 2.4
> ##
> { 'command': 'migrate-set-parameters',
> @@ -656,7 +664,8 @@
> '*compress-threads': 'int',
> '*decompress-threads': 'int',
> '*x-cpu-throttle-initial': 'int',
> - '*x-cpu-throttle-increment': 'int'} }
> + '*x-cpu-throttle-increment': 'int',
> + '*checkpoint-delay': 'int' } }
>
> #
> # @MigrationParameters
> @@ -675,6 +684,8 @@
> # auto-converge detects that migration is not making
> # progress. The default value is 10. (Since 2.5)
> #
> +# @checkpoint-delay: the delay time between two COLO checkpoints
> +#
> # Since: 2.4
> ##
> { 'struct': 'MigrationParameters',
> @@ -682,7 +693,9 @@
> 'compress-threads': 'int',
> 'decompress-threads': 'int',
> 'x-cpu-throttle-initial': 'int',
> - 'x-cpu-throttle-increment': 'int'} }
> + 'x-cpu-throttle-increment': 'int',
> + 'checkpoint-delay': 'int'} }
> +
> ##
> # @query-migrate-parameters
> #
> diff --git a/qmp-commands.hx b/qmp-commands.hx
> index 92042d8..c39f2f0 100644
> --- a/qmp-commands.hx
> +++ b/qmp-commands.hx
> @@ -3664,7 +3664,7 @@ EQMP
> {
> .name = "migrate-set-parameters",
> .args_type =
> - "compress-level:i?,compress-threads:i?,decompress-threads:i?",
> + "compress-level:i?,compress-threads:i?,decompress-threads:i?,checkpoint-delay:i?",
> .mhandler.cmd_new = qmp_marshal_migrate_set_parameters,
> },
> SQMP
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 20/39] COLO: synchronize PVM's state to SVM periodically
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 20/39] COLO: synchronize PVM's state to SVM periodically zhanghailiang
@ 2015-12-09 18:53 ` Dr. David Alan Gilbert
0 siblings, 0 replies; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-09 18:53 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> Do checkpoint periodically, the default interval is 200ms.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> v11:
> - Fix wrong sleep time for checkpoint period. (Dave's review comment)
> ---
> migration/colo.c | 12 ++++++++++++
> 1 file changed, 12 insertions(+)
>
> diff --git a/migration/colo.c b/migration/colo.c
> index 4095d97..5b7ff87 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -11,6 +11,7 @@
> */
>
> #include <unistd.h>
> +#include "qemu/timer.h"
> #include "sysemu/sysemu.h"
> #include "migration/colo.h"
> #include "trace.h"
> @@ -183,6 +184,7 @@ out:
> static void colo_process_checkpoint(MigrationState *s)
> {
> QEMUSizedBuffer *buffer = NULL;
> + int64_t current_time, checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> int ret = 0;
> uint64_t value;
>
> @@ -216,11 +218,21 @@ static void colo_process_checkpoint(MigrationState *s)
> trace_colo_vm_state_change("stop", "run");
>
> while (s->state == MIGRATION_STATUS_COLO) {
> + current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> + if (current_time - checkpoint_time <
> + s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) {
> + int64_t delay_ms;
> +
> + delay_ms = s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] -
> + (current_time - checkpoint_time);
> + g_usleep(delay_ms * 1000);
> + }
> /* start a colo checkpoint */
> ret = colo_do_checkpoint_transaction(s, buffer);
> if (ret < 0) {
> goto out;
> }
> + checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> }
>
> out:
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 08/39] migration: Rename the'file' member of MigrationState
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 08/39] migration: Rename the'file' member of MigrationState zhanghailiang
2015-11-24 18:26 ` Dr. David Alan Gilbert
@ 2015-12-10 6:41 ` Wen Congyang
2015-12-11 3:40 ` Hailiang Zhang
1 sibling, 1 reply; 95+ messages in thread
From: Wen Congyang @ 2015-12-10 6:41 UTC (permalink / raw)
To: zhanghailiang, qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 11/24/2015 05:25 PM, zhanghailiang wrote:
> Rename the 'file' member of MigrationState to 'to_dst_file'.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
> ---
> v11:
> - Only rename 'file' member of MigrationState
You forgot to update migration/rdma.c.
Thanks
Wen Congyang
> ---
> include/migration/migration.h | 2 +-
> migration/exec.c | 4 +--
> migration/fd.c | 4 +--
> migration/migration.c | 72 ++++++++++++++++++++++---------------------
> migration/postcopy-ram.c | 6 ++--
> migration/savevm.c | 2 +-
> migration/tcp.c | 4 +--
> migration/unix.c | 4 +--
> 8 files changed, 51 insertions(+), 47 deletions(-)
>
> diff --git a/include/migration/migration.h b/include/migration/migration.h
> index a57a734..ba5bcec 100644
> --- a/include/migration/migration.h
> +++ b/include/migration/migration.h
> @@ -140,7 +140,7 @@ struct MigrationState
> size_t xfer_limit;
> QemuThread thread;
> QEMUBH *cleanup_bh;
> - QEMUFile *file;
> + QEMUFile *to_dst_file;
> int parameters[MIGRATION_PARAMETER_MAX];
>
> int state;
> diff --git a/migration/exec.c b/migration/exec.c
> index 8406d2b..9037109 100644
> --- a/migration/exec.c
> +++ b/migration/exec.c
> @@ -36,8 +36,8 @@
>
> void exec_start_outgoing_migration(MigrationState *s, const char *command, Error **errp)
> {
> - s->file = qemu_popen_cmd(command, "w");
> - if (s->file == NULL) {
> + s->to_dst_file = qemu_popen_cmd(command, "w");
> + if (s->to_dst_file == NULL) {
> error_setg_errno(errp, errno, "failed to popen the migration target");
> return;
> }
> diff --git a/migration/fd.c b/migration/fd.c
> index 3e4bed0..9a9d6c5 100644
> --- a/migration/fd.c
> +++ b/migration/fd.c
> @@ -50,9 +50,9 @@ void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **
> }
>
> if (fd_is_socket(fd)) {
> - s->file = qemu_fopen_socket(fd, "wb");
> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
> } else {
> - s->file = qemu_fdopen(fd, "wb");
> + s->to_dst_file = qemu_fdopen(fd, "wb");
> }
>
> migrate_fd_connect(s);
> diff --git a/migration/migration.c b/migration/migration.c
> index 41eac0d..a4c690d 100644
> --- a/migration/migration.c
> +++ b/migration/migration.c
> @@ -834,7 +834,7 @@ static void migrate_fd_cleanup(void *opaque)
>
> flush_page_queue(s);
>
> - if (s->file) {
> + if (s->to_dst_file) {
> trace_migrate_fd_cleanup();
> qemu_mutex_unlock_iothread();
> if (s->migration_thread_running) {
> @@ -844,8 +844,8 @@ static void migrate_fd_cleanup(void *opaque)
> qemu_mutex_lock_iothread();
>
> migrate_compress_threads_join();
> - qemu_fclose(s->file);
> - s->file = NULL;
> + qemu_fclose(s->to_dst_file);
> + s->to_dst_file = NULL;
> }
>
> assert((s->state != MIGRATION_STATUS_ACTIVE) &&
> @@ -862,7 +862,7 @@ static void migrate_fd_cleanup(void *opaque)
> void migrate_fd_error(MigrationState *s)
> {
> trace_migrate_fd_error();
> - assert(s->file == NULL);
> + assert(s->to_dst_file == NULL);
> migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
> MIGRATION_STATUS_FAILED);
> notifier_list_notify(&migration_state_notifiers, s);
> @@ -871,7 +871,7 @@ void migrate_fd_error(MigrationState *s)
> static void migrate_fd_cancel(MigrationState *s)
> {
> int old_state ;
> - QEMUFile *f = migrate_get_current()->file;
> + QEMUFile *f = migrate_get_current()->to_dst_file;
> trace_migrate_fd_cancel();
>
> if (s->rp_state.from_dst_file) {
> @@ -942,7 +942,7 @@ MigrationState *migrate_init(const MigrationParams *params)
> s->bytes_xfer = 0;
> s->xfer_limit = 0;
> s->cleanup_bh = 0;
> - s->file = NULL;
> + s->to_dst_file = NULL;
> s->state = MIGRATION_STATUS_NONE;
> s->params = *params;
> s->rp_state.from_dst_file = NULL;
> @@ -1122,8 +1122,9 @@ void qmp_migrate_set_speed(int64_t value, Error **errp)
>
> s = migrate_get_current();
> s->bandwidth_limit = value;
> - if (s->file) {
> - qemu_file_set_rate_limit(s->file, s->bandwidth_limit / XFER_LIMIT_RATIO);
> + if (s->to_dst_file) {
> + qemu_file_set_rate_limit(s->to_dst_file,
> + s->bandwidth_limit / XFER_LIMIT_RATIO);
> }
> }
>
> @@ -1393,7 +1394,7 @@ out:
> static int open_return_path_on_source(MigrationState *ms)
> {
>
> - ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->file);
> + ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->to_dst_file);
> if (!ms->rp_state.from_dst_file) {
> return -1;
> }
> @@ -1415,7 +1416,7 @@ static int await_return_path_close_on_source(MigrationState *ms)
> * rp_thread will exit, however if there's an error we need to cause
> * it to exit.
> */
> - if (qemu_file_get_error(ms->file) && ms->rp_state.from_dst_file) {
> + if (qemu_file_get_error(ms->to_dst_file) && ms->rp_state.from_dst_file) {
> /*
> * shutdown(2), if we have it, will cause it to unblock if it's stuck
> * waiting for the destination.
> @@ -1458,7 +1459,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> * Cause any non-postcopiable, but iterative devices to
> * send out their final data.
> */
> - qemu_savevm_state_complete_precopy(ms->file, true);
> + qemu_savevm_state_complete_precopy(ms->to_dst_file, true);
>
> /*
> * in Finish migrate and with the io-lock held everything should
> @@ -1476,9 +1477,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> * will notice we're in POSTCOPY_ACTIVE and not actually
> * wrap their state up here
> */
> - qemu_file_set_rate_limit(ms->file, INT64_MAX);
> + qemu_file_set_rate_limit(ms->to_dst_file, INT64_MAX);
> /* Ping just for debugging, helps line traces up */
> - qemu_savevm_send_ping(ms->file, 2);
> + qemu_savevm_send_ping(ms->to_dst_file, 2);
>
> /*
> * While loading the device state we may trigger page transfer
> @@ -1512,7 +1513,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> qsb = qemu_buf_get(fb);
>
> /* Now send that blob */
> - if (qemu_savevm_send_packaged(ms->file, qsb)) {
> + if (qemu_savevm_send_packaged(ms->to_dst_file, qsb)) {
> goto fail_closefb;
> }
> qemu_fclose(fb);
> @@ -1524,9 +1525,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
> * Although this ping is just for debug, it could potentially be
> * used for getting a better measurement of downtime at the source.
> */
> - qemu_savevm_send_ping(ms->file, 4);
> + qemu_savevm_send_ping(ms->to_dst_file, 4);
>
> - ret = qemu_file_get_error(ms->file);
> + ret = qemu_file_get_error(ms->to_dst_file);
> if (ret) {
> error_report("postcopy_start: Migration stream errored");
> migrate_set_state(&ms->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
> @@ -1569,8 +1570,8 @@ static void migration_completion(MigrationState *s, int current_active_state,
> if (!ret) {
> ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
> if (ret >= 0) {
> - qemu_file_set_rate_limit(s->file, INT64_MAX);
> - qemu_savevm_state_complete_precopy(s->file, false);
> + qemu_file_set_rate_limit(s->to_dst_file, INT64_MAX);
> + qemu_savevm_state_complete_precopy(s->to_dst_file, false);
> }
> }
> qemu_mutex_unlock_iothread();
> @@ -1581,7 +1582,7 @@ static void migration_completion(MigrationState *s, int current_active_state,
> } else if (s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) {
> trace_migration_completion_postcopy_end();
>
> - qemu_savevm_state_complete_postcopy(s->file);
> + qemu_savevm_state_complete_postcopy(s->to_dst_file);
> trace_migration_completion_postcopy_end_after_complete();
> }
>
> @@ -1602,7 +1603,7 @@ static void migration_completion(MigrationState *s, int current_active_state,
> }
> }
>
> - if (qemu_file_get_error(s->file)) {
> + if (qemu_file_get_error(s->to_dst_file)) {
> trace_migration_completion_file_err();
> goto fail;
> }
> @@ -1647,24 +1648,24 @@ static void *migration_thread(void *opaque)
>
> rcu_register_thread();
>
> - qemu_savevm_state_header(s->file);
> + qemu_savevm_state_header(s->to_dst_file);
>
> if (migrate_postcopy_ram()) {
> /* Now tell the dest that it should open its end so it can reply */
> - qemu_savevm_send_open_return_path(s->file);
> + qemu_savevm_send_open_return_path(s->to_dst_file);
>
> /* And do a ping that will make stuff easier to debug */
> - qemu_savevm_send_ping(s->file, 1);
> + qemu_savevm_send_ping(s->to_dst_file, 1);
>
> /*
> * Tell the destination that we *might* want to do postcopy later;
> * if the other end can't do postcopy it should fail now, nice and
> * early.
> */
> - qemu_savevm_send_postcopy_advise(s->file);
> + qemu_savevm_send_postcopy_advise(s->to_dst_file);
> }
>
> - qemu_savevm_state_begin(s->file, &s->params);
> + qemu_savevm_state_begin(s->to_dst_file, &s->params);
>
> s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
> current_active_state = MIGRATION_STATUS_ACTIVE;
> @@ -1678,10 +1679,10 @@ static void *migration_thread(void *opaque)
> int64_t current_time;
> uint64_t pending_size;
>
> - if (!qemu_file_rate_limit(s->file)) {
> + if (!qemu_file_rate_limit(s->to_dst_file)) {
> uint64_t pend_post, pend_nonpost;
>
> - qemu_savevm_state_pending(s->file, max_size, &pend_nonpost,
> + qemu_savevm_state_pending(s->to_dst_file, max_size, &pend_nonpost,
> &pend_post);
> pending_size = pend_nonpost + pend_post;
> trace_migrate_pending(pending_size, max_size,
> @@ -1702,7 +1703,7 @@ static void *migration_thread(void *opaque)
> continue;
> }
> /* Just another iteration step */
> - qemu_savevm_state_iterate(s->file, entered_postcopy);
> + qemu_savevm_state_iterate(s->to_dst_file, entered_postcopy);
> } else {
> trace_migration_thread_low_pending(pending_size);
> migration_completion(s, current_active_state,
> @@ -1711,7 +1712,7 @@ static void *migration_thread(void *opaque)
> }
> }
>
> - if (qemu_file_get_error(s->file)) {
> + if (qemu_file_get_error(s->to_dst_file)) {
> migrate_set_state(&s->state, current_active_state,
> MIGRATION_STATUS_FAILED);
> trace_migration_thread_file_err();
> @@ -1719,7 +1720,8 @@ static void *migration_thread(void *opaque)
> }
> current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
> if (current_time >= initial_time + BUFFER_DELAY) {
> - uint64_t transferred_bytes = qemu_ftell(s->file) - initial_bytes;
> + uint64_t transferred_bytes = qemu_ftell(s->to_dst_file) -
> + initial_bytes;
> uint64_t time_spent = current_time - initial_time;
> double bandwidth = transferred_bytes / time_spent;
> max_size = bandwidth * migrate_max_downtime() / 1000000;
> @@ -1735,11 +1737,11 @@ static void *migration_thread(void *opaque)
> s->expected_downtime = s->dirty_bytes_rate / bandwidth;
> }
>
> - qemu_file_reset_rate_limit(s->file);
> + qemu_file_reset_rate_limit(s->to_dst_file);
> initial_time = current_time;
> - initial_bytes = qemu_ftell(s->file);
> + initial_bytes = qemu_ftell(s->to_dst_file);
> }
> - if (qemu_file_rate_limit(s->file)) {
> + if (qemu_file_rate_limit(s->to_dst_file)) {
> /* usleep expects microseconds */
> g_usleep((initial_time + BUFFER_DELAY - current_time)*1000);
> }
> @@ -1757,7 +1759,7 @@ static void *migration_thread(void *opaque)
> qemu_savevm_state_cleanup();
> }
> if (s->state == MIGRATION_STATUS_COMPLETED) {
> - uint64_t transferred_bytes = qemu_ftell(s->file);
> + uint64_t transferred_bytes = qemu_ftell(s->to_dst_file);
> s->total_time = end_time - s->total_time;
> if (!entered_postcopy) {
> s->downtime = end_time - start_time;
> @@ -1794,7 +1796,7 @@ void migrate_fd_connect(MigrationState *s)
> s->expected_downtime = max_downtime/1000000;
> s->cleanup_bh = qemu_bh_new(migrate_fd_cleanup, s);
>
> - qemu_file_set_rate_limit(s->file,
> + qemu_file_set_rate_limit(s->to_dst_file,
> s->bandwidth_limit / XFER_LIMIT_RATIO);
>
> /* Notify before starting migration thread */
> diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
> index 22d6b18..f74016d 100644
> --- a/migration/postcopy-ram.c
> +++ b/migration/postcopy-ram.c
> @@ -733,7 +733,8 @@ void postcopy_discard_send_range(MigrationState *ms, PostcopyDiscardState *pds,
>
> if (pds->cur_entry == MAX_DISCARDS_PER_COMMAND) {
> /* Full set, ship it! */
> - qemu_savevm_send_postcopy_ram_discard(ms->file, pds->ramblock_name,
> + qemu_savevm_send_postcopy_ram_discard(ms->to_dst_file,
> + pds->ramblock_name,
> pds->cur_entry,
> pds->start_list,
> pds->length_list);
> @@ -753,7 +754,8 @@ void postcopy_discard_send_finish(MigrationState *ms, PostcopyDiscardState *pds)
> {
> /* Anything unsent? */
> if (pds->cur_entry) {
> - qemu_savevm_send_postcopy_ram_discard(ms->file, pds->ramblock_name,
> + qemu_savevm_send_postcopy_ram_discard(ms->to_dst_file,
> + pds->ramblock_name,
> pds->cur_entry,
> pds->start_list,
> pds->length_list);
> diff --git a/migration/savevm.c b/migration/savevm.c
> index 0ad1b93..f102870 100644
> --- a/migration/savevm.c
> +++ b/migration/savevm.c
> @@ -1163,7 +1163,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
> .shared = 0
> };
> MigrationState *ms = migrate_init(¶ms);
> - ms->file = f;
> + ms->to_dst_file = f;
>
> if (qemu_savevm_state_blocked(errp)) {
> return -EINVAL;
> diff --git a/migration/tcp.c b/migration/tcp.c
> index ae89172..e083d68 100644
> --- a/migration/tcp.c
> +++ b/migration/tcp.c
> @@ -39,11 +39,11 @@ static void tcp_wait_for_connect(int fd, Error *err, void *opaque)
>
> if (fd < 0) {
> DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
> - s->file = NULL;
> + s->to_dst_file = NULL;
> migrate_fd_error(s);
> } else {
> DPRINTF("migrate connect success\n");
> - s->file = qemu_fopen_socket(fd, "wb");
> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
> migrate_fd_connect(s);
> }
> }
> diff --git a/migration/unix.c b/migration/unix.c
> index b591813..5492dd6 100644
> --- a/migration/unix.c
> +++ b/migration/unix.c
> @@ -39,11 +39,11 @@ static void unix_wait_for_connect(int fd, Error *err, void *opaque)
>
> if (fd < 0) {
> DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
> - s->file = NULL;
> + s->to_dst_file = NULL;
> migrate_fd_error(s);
> } else {
> DPRINTF("migrate connect success\n");
> - s->file = qemu_fopen_socket(fd, "wb");
> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
> migrate_fd_connect(s);
> }
> }
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 23/39] COLO: Implement failover work for Primary VM
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 23/39] COLO: Implement failover work for Primary VM zhanghailiang
@ 2015-12-10 18:34 ` Dr. David Alan Gilbert
2015-12-11 7:54 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-10 18:34 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> For PVM, if there is failover request from users.
> The colo thread will exit the loop while the failover BH does the
> cleanup work and resumes VM.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> ---
> v11:
> - Don't call migration_end() in primary_vm_do_failover(),
> The cleanup work will be done in migration_thread().
> - Remove vm_start() in primary_vm_do_failover() which also been done
> in migraiton_thread()
> v10:
> - Call migration_end() in primary_vm_do_failover()
> ---
> include/migration/colo.h | 3 +++
> include/migration/failover.h | 1 +
> migration/colo-failover.c | 7 +++++-
> migration/colo.c | 56 ++++++++++++++++++++++++++++++++++++++++++--
> 4 files changed, 64 insertions(+), 3 deletions(-)
>
> diff --git a/include/migration/colo.h b/include/migration/colo.h
> index ba27719..0b02e95 100644
> --- a/include/migration/colo.h
> +++ b/include/migration/colo.h
> @@ -32,4 +32,7 @@ void *colo_process_incoming_thread(void *opaque);
> bool migration_incoming_in_colo_state(void);
>
> COLOMode get_colo_mode(void);
> +
> +/* failover */
> +void colo_do_failover(MigrationState *s);
> #endif
> diff --git a/include/migration/failover.h b/include/migration/failover.h
> index 882c625..fba3931 100644
> --- a/include/migration/failover.h
> +++ b/include/migration/failover.h
> @@ -26,5 +26,6 @@ void failover_init_state(void);
> int failover_set_state(int old_state, int new_state);
> int failover_get_state(void);
> void failover_request_active(Error **errp);
> +bool failover_request_is_active(void);
>
> #endif
> diff --git a/migration/colo-failover.c b/migration/colo-failover.c
> index 1b1be24..0c525da 100644
> --- a/migration/colo-failover.c
> +++ b/migration/colo-failover.c
> @@ -32,7 +32,7 @@ static void colo_failover_bh(void *opaque)
> error_report("Unkown error for failover, old_state=%d", old_state);
> return;
> }
> - /*TODO: Do failover work */
> + colo_do_failover(NULL);
> }
>
> void failover_request_active(Error **errp)
> @@ -67,6 +67,11 @@ int failover_get_state(void)
> return atomic_read(&failover_state);
> }
>
> +bool failover_request_is_active(void)
> +{
> + return ((failover_get_state() != FAILOVER_STATUS_NONE));
> +}
> +
> void qmp_x_colo_lost_heartbeat(Error **errp)
> {
> if (get_colo_mode() == COLO_MODE_UNKNOWN) {
> diff --git a/migration/colo.c b/migration/colo.c
> index cedfc63..7a42fc6 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -41,6 +41,42 @@ bool migration_incoming_in_colo_state(void)
> return mis && (mis->state == MIGRATION_STATUS_COLO);
> }
>
> +static bool colo_runstate_is_stopped(void)
> +{
> + return runstate_check(RUN_STATE_COLO) || !runstate_is_running();
> +}
> +
> +static void primary_vm_do_failover(void)
> +{
> + MigrationState *s = migrate_get_current();
> + int old_state;
> +
> + if (s->state != MIGRATION_STATUS_FAILED) {
> + migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
> + MIGRATION_STATUS_COMPLETED);
> + }
That's a little odd; it will only move to completed if the current
state is MIGRATION_STATUS_COLO, but you only do it if the state isn't
FAILED. You could remove the if and just call migrate_set_state
like that and it would be safe as long as you really only expect
to have to deal with MIGRATION_STATUS_COLO.
> + old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
> + FAILOVER_STATUS_COMPLETED);
> + if (old_state != FAILOVER_STATUS_HANDLING) {
> + error_report("Serious error while do failover for Primary VM,"
> + "old_state: %d", old_state);
It's generally better to state the reason rather than say it's 'serious';
so something like:
'Incorrect state (%d) while doing failover for Primary VM'
tells you more, and looks less scary!
> + return;
> + }
> +}
> +
> +void colo_do_failover(MigrationState *s)
> +{
> + /* Make sure vm stopped while failover */
> + if (!colo_runstate_is_stopped()) {
> + vm_stop_force_state(RUN_STATE_COLO);
> + }
That feels like a race? couldn't we be just at the end
of taking a checkpoint and about to restart when you do
the if, so it reads that it's currently stopped but
then it restarts it by the time you have a chance to
do anything?
I see in patch 13 (Save PVM....) you have:
qemu_mutex_lock_iothread();
vm_start();
qemu_mutex_unlock_iothread();
So maybe if that code is executed just before the
failover, then it would stop at the _lock_, we would
run here but then as soon as we finish wouldn't it vm_start
there?
> + if (get_colo_mode() == COLO_MODE_PRIMARY) {
> + primary_vm_do_failover();
> + }
> +}
> +
> /* colo checkpoint control helper */
> static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value)
> {
> @@ -122,9 +158,22 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
> }
>
> qemu_mutex_lock_iothread();
> + if (failover_request_is_active()) {
> + qemu_mutex_unlock_iothread();
> + ret = -1;
> + goto out;
> + }
> vm_stop_force_state(RUN_STATE_COLO);
> qemu_mutex_unlock_iothread();
> trace_colo_vm_state_change("run", "stop");
> + /*
> + * failover request bh could be called after
> + * vm_stop_force_state so we check failover_request_is_active() again.
> + */
> + if (failover_request_is_active()) {
> + ret = -1;
> + goto out;
> + }
I'm confused about why the check is needed specifically here;
for example can't that happen at any point where we're ousite of the
iothread lock? e.g. couldn't we set failover just a couple of
lines lower, lets say just after the s->params.blk= 0 ?
> /* Disable block migration */
> s->params.blk = 0;
> @@ -221,6 +270,11 @@ static void colo_process_checkpoint(MigrationState *s)
> trace_colo_vm_state_change("stop", "run");
>
> while (s->state == MIGRATION_STATUS_COLO) {
> + if (failover_request_is_active()) {
> + error_report("failover request");
> + goto out;
> + }
> +
> current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> if (current_time - checkpoint_time <
> s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) {
> @@ -242,8 +296,6 @@ out:
> if (ret < 0) {
> error_report("%s: %s", __func__, strerror(-ret));
> }
> - migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
> - MIGRATION_STATUS_COMPLETED);
I had to think about this; but yes, I guess the only way out is via the failover
which does the completed above.
Dave
> qsb_free(buffer);
> buffer = NULL;
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 24/39] COLO: Implement failover work for Secondary VM
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 24/39] COLO: Implement failover work for Secondary VM zhanghailiang
@ 2015-12-10 18:50 ` Dr. David Alan Gilbert
2015-12-11 8:27 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-10 18:50 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> If users require SVM to takeover work, colo incoming thread should
> exit from loop while failover BH helps backing to migration incoming
> coroutine.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> ---
> migration/colo.c | 42 +++++++++++++++++++++++++++++++++++++++---
> 1 file changed, 39 insertions(+), 3 deletions(-)
>
> diff --git a/migration/colo.c b/migration/colo.c
> index 7a42fc6..f31e957 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -46,6 +46,33 @@ static bool colo_runstate_is_stopped(void)
> return runstate_check(RUN_STATE_COLO) || !runstate_is_running();
> }
>
> +static void secondary_vm_do_failover(void)
> +{
> + int old_state;
> + MigrationIncomingState *mis = migration_incoming_get_current();
> +
> + migrate_set_state(&mis->state, MIGRATION_STATUS_COLO,
> + MIGRATION_STATUS_COMPLETED);
> +
> + if (!autostart) {
> + error_report("\"-S\" qemu option will be ignored in secondary side");
> + /* recover runstate to normal migration finish state */
> + autostart = true;
> + }
You might find libvirt will need something different for it to be
involved during the failover; but for now OK.
> + old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
> + FAILOVER_STATUS_COMPLETED);
> + if (old_state != FAILOVER_STATUS_HANDLING) {
> + error_report("Serious error while do failover for secondary VM,"
> + "old_state: %d", old_state);
Same suggestion as previous patch just to improve the error message.
> + return;
> + }
> + /* For Secondary VM, jump to incoming co */
> + if (mis->migration_incoming_co) {
> + qemu_coroutine_enter(mis->migration_incoming_co, NULL);
> + }
> +}
> +
> static void primary_vm_do_failover(void)
> {
> MigrationState *s = migrate_get_current();
> @@ -74,6 +101,8 @@ void colo_do_failover(MigrationState *s)
>
> if (get_colo_mode() == COLO_MODE_PRIMARY) {
> primary_vm_do_failover();
> + } else {
> + secondary_vm_do_failover();
> }
> }
>
> @@ -404,6 +433,12 @@ void *colo_process_incoming_thread(void *opaque)
> continue;
> }
> }
> +
> + if (failover_request_is_active()) {
> + error_report("failover request");
> + goto out;
> + }
> +
> /* FIXME: This is unnecessary for periodic checkpoint mode */
> ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_REPLY, 0);
> if (ret < 0) {
> @@ -473,10 +508,11 @@ out:
> qemu_fclose(fb);
> }
> qsb_free(buffer);
> -
> - qemu_mutex_lock_iothread();
> + /* Here, we can ensure BH is hold the global lock, and will join colo
> + * incoming thread, so here it is not necessary to lock here again,
> + * or there will be a deadlock error.
> + */
> colo_release_ram_cache();
> - qemu_mutex_unlock_iothread();
OK, I think I understand that - becuase we know there is a failover request
active, then it must be holding the lock?
Other than the error message improvement:
Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
Dave
>
> if (mis->to_src_file) {
> qemu_fclose(mis->to_src_file);
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 25/39] COLO: implement default failover treatment
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 25/39] COLO: implement default failover treatment zhanghailiang
@ 2015-12-10 19:01 ` Dr. David Alan Gilbert
2015-12-11 9:48 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-10 19:01 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> If we detect some error in colo, we will wait for some time,
> hoping users also detect it. If users don't issue failover command.
> We will go into default failover procedure, which the PVM will takeover
> work while SVM is exit in default.
I'm not sure this is needed; especially on the SVM. I don't see any harm
in the SVM waiting forever to be told what to do - it could be told to
failover or quit; I don't see any benefit to it automatically exiting.
In the primary, I can see if you didn't have some automated error
detection system then I can understand it (but I think it's rare);
but you really would want to make that failover delay configurable
so that you could turn it off in a system that did have failure detection;
because automatically restarting the primary after it had caused a failover
to the secondary would be very bad.
Dave
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> ---
> migration/colo.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
> 1 file changed, 46 insertions(+)
>
> diff --git a/migration/colo.c b/migration/colo.c
> index f31e957..1e6d3dd 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -19,6 +19,14 @@
> #include "qemu/sockets.h"
> #include "migration/failover.h"
>
> +/*
> + * The delay time before qemu begin the procedure of default failover treatment.
> + * Unit: ms
> + * Fix me: This value should be able to change by command
> + * 'migrate-set-parameters'
> + */
> +#define DEFAULT_FAILOVER_DELAY 2000
> +
> /* colo buffer */
> #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
>
> @@ -264,6 +272,7 @@ static void colo_process_checkpoint(MigrationState *s)
> {
> QEMUSizedBuffer *buffer = NULL;
> int64_t current_time, checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> + int64_t error_time;
> int ret = 0;
> uint64_t value;
>
> @@ -322,8 +331,25 @@ static void colo_process_checkpoint(MigrationState *s)
> }
>
> out:
> + current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> if (ret < 0) {
> error_report("%s: %s", __func__, strerror(-ret));
> + /* Give users time to get involved in this verdict */
> + while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
> + if (failover_request_is_active()) {
> + error_report("Primary VM will take over work");
> + break;
> + }
> + usleep(100 * 1000);
> + current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> + }
> +
> + qemu_mutex_lock_iothread();
> + if (!failover_request_is_active()) {
> + error_report("Primary VM will take over work in default");
> + failover_request_active(NULL);
> + }
> + qemu_mutex_unlock_iothread();
> }
>
> qsb_free(buffer);
> @@ -384,6 +410,7 @@ void *colo_process_incoming_thread(void *opaque)
> QEMUFile *fb = NULL;
> QEMUSizedBuffer *buffer = NULL; /* Cache incoming device state */
> uint64_t total_size;
> + int64_t error_time, current_time;
> int ret = 0;
> uint64_t value;
>
> @@ -499,9 +526,28 @@ void *colo_process_incoming_thread(void *opaque)
> }
>
> out:
> + current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> if (ret < 0) {
> error_report("colo incoming thread will exit, detect error: %s",
> strerror(-ret));
> + /* Give users time to get involved in this verdict */
> + while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
> + if (failover_request_is_active()) {
> + error_report("Secondary VM will take over work");
> + break;
> + }
> + usleep(100 * 1000);
> + current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> + }
> + /* check flag again*/
> + if (!failover_request_is_active()) {
> + /*
> + * We assume that Primary VM is still alive according to
> + * heartbeat, just kill Secondary VM
> + */
> + error_report("SVM is going to exit in default!");
> + exit(1);
> + }
> }
>
> if (fb) {
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 27/39] COLO failover: Shutdown related socket fd when do failover
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 27/39] COLO failover: Shutdown related socket fd when do failover zhanghailiang
@ 2015-12-10 20:03 ` Dr. David Alan Gilbert
2015-12-11 8:57 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-10 20:03 UTC (permalink / raw)
To: zhanghailiang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> If the net connection between COLO's two sides is broken while colo/colo incoming
> thread is blocked in 'read'/'write' socket fd. It will not detect this error until
> connect timeout. It will be a long time.
>
> Here we shutdown all the related socket file descriptors to wake up the blocking
> operation in failover BH. Besides, we should close the corresponding file descriptors
> after failvoer BH shutdown them, or there will be an error.
>
> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> ---
> v11:
> - Only shutdown fd for once
> ---
> migration/colo.c | 31 +++++++++++++++++++++++++++++--
> 1 file changed, 29 insertions(+), 2 deletions(-)
>
> diff --git a/migration/colo.c b/migration/colo.c
> index 4cd7b00..994b80d 100644
> --- a/migration/colo.c
> +++ b/migration/colo.c
> @@ -68,6 +68,14 @@ static void secondary_vm_do_failover(void)
> /* recover runstate to normal migration finish state */
> autostart = true;
> }
> + /*
> + * Make sure colo incoming thread not block in recv,
> + * mis->from_src_file and mis->to_src_file use the same fd,
> + * so here we only need to shutdown it for once.
> + */
That assumption is only valid for the current socket transport;
for example I have a (partially-working) RDMA transport where the
forward and reverse paths are quite separate.
> + if (mis->from_src_file) {
> + qemu_file_shutdown(mis->from_src_file);
> + }
>
> old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
> FAILOVER_STATUS_COMPLETED);
> @@ -92,6 +100,15 @@ static void primary_vm_do_failover(void)
> MIGRATION_STATUS_COMPLETED);
> }
>
> + /*
> + * Make sure colo thread no block in recv,
> + * Besides, s->rp_state.from_dst_file and s->to_dst_file use the
> + * same fd, so here we only need to shutdown it for once.
> + */
> + if (s->to_dst_file) {
> + qemu_file_shutdown(s->to_dst_file);
> + }
> +
> old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
> FAILOVER_STATUS_COMPLETED);
> if (old_state != FAILOVER_STATUS_HANDLING) {
> @@ -333,7 +350,7 @@ static void colo_process_checkpoint(MigrationState *s)
>
> out:
> current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> - if (ret < 0) {
> + if (ret < 0 || (!ret && !failover_request_is_active())) {
Why is this needed - when would you get a 0 return that is actually an error?
> error_report("%s: %s", __func__, strerror(-ret));
> qapi_event_send_colo_exit(COLO_MODE_PRIMARY, COLO_EXIT_REASON_ERROR,
> true, strerror(-ret), NULL);
> @@ -362,6 +379,11 @@ out:
> qsb_free(buffer);
> buffer = NULL;
>
> + /* Hope this not to be too long to loop here */
> + while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
> + ;
> + }
> + /* Must be called after failover BH is completed */
Is this the right patch for this?
Dave
> if (s->rp_state.from_dst_file) {
> qemu_fclose(s->rp_state.from_dst_file);
> }
> @@ -534,7 +556,7 @@ void *colo_process_incoming_thread(void *opaque)
>
> out:
> current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> - if (ret < 0) {
> + if (ret < 0 || (!ret && !failover_request_is_active())) {
> error_report("colo incoming thread will exit, detect error: %s",
> strerror(-ret));
> qapi_event_send_colo_exit(COLO_MODE_SECONDARY, COLO_EXIT_REASON_ERROR,
> @@ -573,6 +595,11 @@ out:
> */
> colo_release_ram_cache();
>
> + /* Hope this not to be too long to loop here */
> + while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
> + ;
> + }
> + /* Must be called after failover BH is completed */
> if (mis->to_src_file) {
> qemu_fclose(mis->to_src_file);
> }
> --
> 1.8.3.1
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 19/39] COLO: Add checkpoint-delay parameter for migrate-set-parameters
2015-12-09 18:50 ` Dr. David Alan Gilbert
@ 2015-12-11 3:20 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-11 3:20 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, Markus Armbruster, yunhong.jiang,
eddie.dong, peter.huangpeng, qemu-devel, arei.gonglei, stefanha,
amit.shah, Luiz Capitulino, hongyang.yang
On 2015/12/10 2:50, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> Add checkpoint-delay parameter for migrate-set-parameters, so that
>> we can control the checkpoint frequency when COLO is in periodic mode.
>>
>> Cc: Luiz Capitulino <lcapitulino@redhat.com>
>> Cc: Eric Blake <eblake@redhat.com>
>> Cc: Markus Armbruster <armbru@redhat.com>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>
> It's probably a good idea to make this x- but other than that:
>
OK, i will fix it, thanks.
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
>
>> ---
>> v11:
>> - Move this patch ahead of the patch where uses 'checkpoint_delay'
>> (Dave's suggestion)
>> v10:
>> - Fix related qmp command
>> ---
>> hmp.c | 7 +++++++
>> migration/migration.c | 23 ++++++++++++++++++++++-
>> qapi-schema.json | 19 ++++++++++++++++---
>> qmp-commands.hx | 2 +-
>> 4 files changed, 46 insertions(+), 5 deletions(-)
>>
>> diff --git a/hmp.c b/hmp.c
>> index 2140605..ec038f1 100644
>> --- a/hmp.c
>> +++ b/hmp.c
>> @@ -284,6 +284,9 @@ void hmp_info_migrate_parameters(Monitor *mon, const QDict *qdict)
>> monitor_printf(mon, " %s: %" PRId64,
>> MigrationParameter_lookup[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT],
>> params->x_cpu_throttle_increment);
>> + monitor_printf(mon, " %s: %" PRId64,
>> + MigrationParameter_lookup[MIGRATION_PARAMETER_CHECKPOINT_DELAY],
>> + params->checkpoint_delay);
>> monitor_printf(mon, "\n");
>> }
>>
>> @@ -1237,6 +1240,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
>> bool has_decompress_threads = false;
>> bool has_x_cpu_throttle_initial = false;
>> bool has_x_cpu_throttle_increment = false;
>> + bool has_checkpoint_delay = false;
>> int i;
>>
>> for (i = 0; i < MIGRATION_PARAMETER_MAX; i++) {
>> @@ -1256,6 +1260,8 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
>> break;
>> case MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT:
>> has_x_cpu_throttle_increment = true;
>> + case MIGRATION_PARAMETER_CHECKPOINT_DELAY:
>> + has_checkpoint_delay = true;
>> break;
>> }
>> qmp_migrate_set_parameters(has_compress_level, value,
>> @@ -1263,6 +1269,7 @@ void hmp_migrate_set_parameter(Monitor *mon, const QDict *qdict)
>> has_decompress_threads, value,
>> has_x_cpu_throttle_initial, value,
>> has_x_cpu_throttle_increment, value,
>> + has_checkpoint_delay, value,
>> &err);
>> break;
>> }
>> diff --git a/migration/migration.c b/migration/migration.c
>> index a4c690d..48545f3 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -56,6 +56,11 @@
>> /* Migration XBZRLE default cache size */
>> #define DEFAULT_MIGRATE_CACHE_SIZE (64 * 1024 * 1024)
>>
>> +/* The delay time (in ms) between two COLO checkpoints
>> + * Note: Please change this default value to 10000 when we support hybrid mode.
>> + */
>> +#define DEFAULT_MIGRATE_CHECKPOINT_DELAY 200
>> +
>> static NotifierList migration_state_notifiers =
>> NOTIFIER_LIST_INITIALIZER(migration_state_notifiers);
>>
>> @@ -91,6 +96,8 @@ MigrationState *migrate_get_current(void)
>> DEFAULT_MIGRATE_X_CPU_THROTTLE_INITIAL,
>> .parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
>> DEFAULT_MIGRATE_X_CPU_THROTTLE_INCREMENT,
>> + .parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] =
>> + DEFAULT_MIGRATE_CHECKPOINT_DELAY,
>> };
>>
>> if (!once) {
>> @@ -530,6 +537,8 @@ MigrationParameters *qmp_query_migrate_parameters(Error **errp)
>> s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INITIAL];
>> params->x_cpu_throttle_increment =
>> s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT];
>> + params->checkpoint_delay =
>> + s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY];
>>
>> return params;
>> }
>> @@ -736,7 +745,10 @@ void qmp_migrate_set_parameters(bool has_compress_level,
>> bool has_x_cpu_throttle_initial,
>> int64_t x_cpu_throttle_initial,
>> bool has_x_cpu_throttle_increment,
>> - int64_t x_cpu_throttle_increment, Error **errp)
>> + int64_t x_cpu_throttle_increment,
>> + bool has_checkpoint_delay,
>> + int64_t checkpoint_delay,
>> + Error **errp)
>> {
>> MigrationState *s = migrate_get_current();
>>
>> @@ -771,6 +783,11 @@ void qmp_migrate_set_parameters(bool has_compress_level,
>> "x_cpu_throttle_increment",
>> "an integer in the range of 1 to 99");
>> }
>> + if (has_checkpoint_delay && (checkpoint_delay < 0)) {
>> + error_setg(errp, QERR_INVALID_PARAMETER_VALUE,
>> + "checkpoint_delay",
>> + "is invalid, it should be positive");
>> + }
>>
>> if (has_compress_level) {
>> s->parameters[MIGRATION_PARAMETER_COMPRESS_LEVEL] = compress_level;
>> @@ -791,6 +808,10 @@ void qmp_migrate_set_parameters(bool has_compress_level,
>> s->parameters[MIGRATION_PARAMETER_X_CPU_THROTTLE_INCREMENT] =
>> x_cpu_throttle_increment;
>> }
>> +
>> + if (has_checkpoint_delay) {
>> + s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY] = checkpoint_delay;
>> + }
>> }
>>
>> void qmp_migrate_start_postcopy(Error **errp)
>> diff --git a/qapi-schema.json b/qapi-schema.json
>> index 5ee089d..b4b47ed 100644
>> --- a/qapi-schema.json
>> +++ b/qapi-schema.json
>> @@ -625,11 +625,16 @@
>> # @x-cpu-throttle-increment: throttle percentage increase each time
>> # auto-converge detects that migration is not making
>> # progress. The default value is 10. (Since 2.5)
>> +#
>> +# @checkpoint-delay: The delay time (in ms) between two COLO checkpoints in
>> +# periodic mode. (Since 2.6)
>> +#
>> # Since: 2.4
>> ##
>> { 'enum': 'MigrationParameter',
>> 'data': ['compress-level', 'compress-threads', 'decompress-threads',
>> - 'x-cpu-throttle-initial', 'x-cpu-throttle-increment'] }
>> + 'x-cpu-throttle-initial', 'x-cpu-throttle-increment',
>> + 'checkpoint-delay' ] }
>>
>> #
>> # @migrate-set-parameters
>> @@ -649,6 +654,9 @@
>> # @x-cpu-throttle-increment: throttle percentage increase each time
>> # auto-converge detects that migration is not making
>> # progress. The default value is 10. (Since 2.5)
>> +#
>> +# @checkpoint-delay: the delay time between two checkpoints. (Since 2.6)
>> +#
>> # Since: 2.4
>> ##
>> { 'command': 'migrate-set-parameters',
>> @@ -656,7 +664,8 @@
>> '*compress-threads': 'int',
>> '*decompress-threads': 'int',
>> '*x-cpu-throttle-initial': 'int',
>> - '*x-cpu-throttle-increment': 'int'} }
>> + '*x-cpu-throttle-increment': 'int',
>> + '*checkpoint-delay': 'int' } }
>>
>> #
>> # @MigrationParameters
>> @@ -675,6 +684,8 @@
>> # auto-converge detects that migration is not making
>> # progress. The default value is 10. (Since 2.5)
>> #
>> +# @checkpoint-delay: the delay time between two COLO checkpoints
>> +#
>> # Since: 2.4
>> ##
>> { 'struct': 'MigrationParameters',
>> @@ -682,7 +693,9 @@
>> 'compress-threads': 'int',
>> 'decompress-threads': 'int',
>> 'x-cpu-throttle-initial': 'int',
>> - 'x-cpu-throttle-increment': 'int'} }
>> + 'x-cpu-throttle-increment': 'int',
>> + 'checkpoint-delay': 'int'} }
>> +
>> ##
>> # @query-migrate-parameters
>> #
>> diff --git a/qmp-commands.hx b/qmp-commands.hx
>> index 92042d8..c39f2f0 100644
>> --- a/qmp-commands.hx
>> +++ b/qmp-commands.hx
>> @@ -3664,7 +3664,7 @@ EQMP
>> {
>> .name = "migrate-set-parameters",
>> .args_type =
>> - "compress-level:i?,compress-threads:i?,decompress-threads:i?",
>> + "compress-level:i?,compress-threads:i?,decompress-threads:i?,checkpoint-delay:i?",
>> .mhandler.cmd_new = qmp_marshal_migrate_set_parameters,
>> },
>> SQMP
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 08/39] migration: Rename the'file' member of MigrationState
2015-12-10 6:41 ` Wen Congyang
@ 2015-12-11 3:40 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-11 3:40 UTC (permalink / raw)
To: Wen Congyang, qemu-devel
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
dgilbert, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/10 14:41, Wen Congyang wrote:
> On 11/24/2015 05:25 PM, zhanghailiang wrote:
>> Rename the 'file' member of MigrationState to 'to_dst_file'.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Cc: Dr. David Alan Gilbert <dgilbert@redhat.com>
>> ---
>> v11:
>> - Only rename 'file' member of MigrationState
>
> You forgot to update migration/rdma.c.
>
Good catch, i will fix it, thanks.
> Thanks
> Wen Congyang
>
>> ---
>> include/migration/migration.h | 2 +-
>> migration/exec.c | 4 +--
>> migration/fd.c | 4 +--
>> migration/migration.c | 72 ++++++++++++++++++++++---------------------
>> migration/postcopy-ram.c | 6 ++--
>> migration/savevm.c | 2 +-
>> migration/tcp.c | 4 +--
>> migration/unix.c | 4 +--
>> 8 files changed, 51 insertions(+), 47 deletions(-)
>>
>> diff --git a/include/migration/migration.h b/include/migration/migration.h
>> index a57a734..ba5bcec 100644
>> --- a/include/migration/migration.h
>> +++ b/include/migration/migration.h
>> @@ -140,7 +140,7 @@ struct MigrationState
>> size_t xfer_limit;
>> QemuThread thread;
>> QEMUBH *cleanup_bh;
>> - QEMUFile *file;
>> + QEMUFile *to_dst_file;
>> int parameters[MIGRATION_PARAMETER_MAX];
>>
>> int state;
>> diff --git a/migration/exec.c b/migration/exec.c
>> index 8406d2b..9037109 100644
>> --- a/migration/exec.c
>> +++ b/migration/exec.c
>> @@ -36,8 +36,8 @@
>>
>> void exec_start_outgoing_migration(MigrationState *s, const char *command, Error **errp)
>> {
>> - s->file = qemu_popen_cmd(command, "w");
>> - if (s->file == NULL) {
>> + s->to_dst_file = qemu_popen_cmd(command, "w");
>> + if (s->to_dst_file == NULL) {
>> error_setg_errno(errp, errno, "failed to popen the migration target");
>> return;
>> }
>> diff --git a/migration/fd.c b/migration/fd.c
>> index 3e4bed0..9a9d6c5 100644
>> --- a/migration/fd.c
>> +++ b/migration/fd.c
>> @@ -50,9 +50,9 @@ void fd_start_outgoing_migration(MigrationState *s, const char *fdname, Error **
>> }
>>
>> if (fd_is_socket(fd)) {
>> - s->file = qemu_fopen_socket(fd, "wb");
>> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
>> } else {
>> - s->file = qemu_fdopen(fd, "wb");
>> + s->to_dst_file = qemu_fdopen(fd, "wb");
>> }
>>
>> migrate_fd_connect(s);
>> diff --git a/migration/migration.c b/migration/migration.c
>> index 41eac0d..a4c690d 100644
>> --- a/migration/migration.c
>> +++ b/migration/migration.c
>> @@ -834,7 +834,7 @@ static void migrate_fd_cleanup(void *opaque)
>>
>> flush_page_queue(s);
>>
>> - if (s->file) {
>> + if (s->to_dst_file) {
>> trace_migrate_fd_cleanup();
>> qemu_mutex_unlock_iothread();
>> if (s->migration_thread_running) {
>> @@ -844,8 +844,8 @@ static void migrate_fd_cleanup(void *opaque)
>> qemu_mutex_lock_iothread();
>>
>> migrate_compress_threads_join();
>> - qemu_fclose(s->file);
>> - s->file = NULL;
>> + qemu_fclose(s->to_dst_file);
>> + s->to_dst_file = NULL;
>> }
>>
>> assert((s->state != MIGRATION_STATUS_ACTIVE) &&
>> @@ -862,7 +862,7 @@ static void migrate_fd_cleanup(void *opaque)
>> void migrate_fd_error(MigrationState *s)
>> {
>> trace_migrate_fd_error();
>> - assert(s->file == NULL);
>> + assert(s->to_dst_file == NULL);
>> migrate_set_state(&s->state, MIGRATION_STATUS_SETUP,
>> MIGRATION_STATUS_FAILED);
>> notifier_list_notify(&migration_state_notifiers, s);
>> @@ -871,7 +871,7 @@ void migrate_fd_error(MigrationState *s)
>> static void migrate_fd_cancel(MigrationState *s)
>> {
>> int old_state ;
>> - QEMUFile *f = migrate_get_current()->file;
>> + QEMUFile *f = migrate_get_current()->to_dst_file;
>> trace_migrate_fd_cancel();
>>
>> if (s->rp_state.from_dst_file) {
>> @@ -942,7 +942,7 @@ MigrationState *migrate_init(const MigrationParams *params)
>> s->bytes_xfer = 0;
>> s->xfer_limit = 0;
>> s->cleanup_bh = 0;
>> - s->file = NULL;
>> + s->to_dst_file = NULL;
>> s->state = MIGRATION_STATUS_NONE;
>> s->params = *params;
>> s->rp_state.from_dst_file = NULL;
>> @@ -1122,8 +1122,9 @@ void qmp_migrate_set_speed(int64_t value, Error **errp)
>>
>> s = migrate_get_current();
>> s->bandwidth_limit = value;
>> - if (s->file) {
>> - qemu_file_set_rate_limit(s->file, s->bandwidth_limit / XFER_LIMIT_RATIO);
>> + if (s->to_dst_file) {
>> + qemu_file_set_rate_limit(s->to_dst_file,
>> + s->bandwidth_limit / XFER_LIMIT_RATIO);
>> }
>> }
>>
>> @@ -1393,7 +1394,7 @@ out:
>> static int open_return_path_on_source(MigrationState *ms)
>> {
>>
>> - ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->file);
>> + ms->rp_state.from_dst_file = qemu_file_get_return_path(ms->to_dst_file);
>> if (!ms->rp_state.from_dst_file) {
>> return -1;
>> }
>> @@ -1415,7 +1416,7 @@ static int await_return_path_close_on_source(MigrationState *ms)
>> * rp_thread will exit, however if there's an error we need to cause
>> * it to exit.
>> */
>> - if (qemu_file_get_error(ms->file) && ms->rp_state.from_dst_file) {
>> + if (qemu_file_get_error(ms->to_dst_file) && ms->rp_state.from_dst_file) {
>> /*
>> * shutdown(2), if we have it, will cause it to unblock if it's stuck
>> * waiting for the destination.
>> @@ -1458,7 +1459,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
>> * Cause any non-postcopiable, but iterative devices to
>> * send out their final data.
>> */
>> - qemu_savevm_state_complete_precopy(ms->file, true);
>> + qemu_savevm_state_complete_precopy(ms->to_dst_file, true);
>>
>> /*
>> * in Finish migrate and with the io-lock held everything should
>> @@ -1476,9 +1477,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
>> * will notice we're in POSTCOPY_ACTIVE and not actually
>> * wrap their state up here
>> */
>> - qemu_file_set_rate_limit(ms->file, INT64_MAX);
>> + qemu_file_set_rate_limit(ms->to_dst_file, INT64_MAX);
>> /* Ping just for debugging, helps line traces up */
>> - qemu_savevm_send_ping(ms->file, 2);
>> + qemu_savevm_send_ping(ms->to_dst_file, 2);
>>
>> /*
>> * While loading the device state we may trigger page transfer
>> @@ -1512,7 +1513,7 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
>> qsb = qemu_buf_get(fb);
>>
>> /* Now send that blob */
>> - if (qemu_savevm_send_packaged(ms->file, qsb)) {
>> + if (qemu_savevm_send_packaged(ms->to_dst_file, qsb)) {
>> goto fail_closefb;
>> }
>> qemu_fclose(fb);
>> @@ -1524,9 +1525,9 @@ static int postcopy_start(MigrationState *ms, bool *old_vm_running)
>> * Although this ping is just for debug, it could potentially be
>> * used for getting a better measurement of downtime at the source.
>> */
>> - qemu_savevm_send_ping(ms->file, 4);
>> + qemu_savevm_send_ping(ms->to_dst_file, 4);
>>
>> - ret = qemu_file_get_error(ms->file);
>> + ret = qemu_file_get_error(ms->to_dst_file);
>> if (ret) {
>> error_report("postcopy_start: Migration stream errored");
>> migrate_set_state(&ms->state, MIGRATION_STATUS_POSTCOPY_ACTIVE,
>> @@ -1569,8 +1570,8 @@ static void migration_completion(MigrationState *s, int current_active_state,
>> if (!ret) {
>> ret = vm_stop_force_state(RUN_STATE_FINISH_MIGRATE);
>> if (ret >= 0) {
>> - qemu_file_set_rate_limit(s->file, INT64_MAX);
>> - qemu_savevm_state_complete_precopy(s->file, false);
>> + qemu_file_set_rate_limit(s->to_dst_file, INT64_MAX);
>> + qemu_savevm_state_complete_precopy(s->to_dst_file, false);
>> }
>> }
>> qemu_mutex_unlock_iothread();
>> @@ -1581,7 +1582,7 @@ static void migration_completion(MigrationState *s, int current_active_state,
>> } else if (s->state == MIGRATION_STATUS_POSTCOPY_ACTIVE) {
>> trace_migration_completion_postcopy_end();
>>
>> - qemu_savevm_state_complete_postcopy(s->file);
>> + qemu_savevm_state_complete_postcopy(s->to_dst_file);
>> trace_migration_completion_postcopy_end_after_complete();
>> }
>>
>> @@ -1602,7 +1603,7 @@ static void migration_completion(MigrationState *s, int current_active_state,
>> }
>> }
>>
>> - if (qemu_file_get_error(s->file)) {
>> + if (qemu_file_get_error(s->to_dst_file)) {
>> trace_migration_completion_file_err();
>> goto fail;
>> }
>> @@ -1647,24 +1648,24 @@ static void *migration_thread(void *opaque)
>>
>> rcu_register_thread();
>>
>> - qemu_savevm_state_header(s->file);
>> + qemu_savevm_state_header(s->to_dst_file);
>>
>> if (migrate_postcopy_ram()) {
>> /* Now tell the dest that it should open its end so it can reply */
>> - qemu_savevm_send_open_return_path(s->file);
>> + qemu_savevm_send_open_return_path(s->to_dst_file);
>>
>> /* And do a ping that will make stuff easier to debug */
>> - qemu_savevm_send_ping(s->file, 1);
>> + qemu_savevm_send_ping(s->to_dst_file, 1);
>>
>> /*
>> * Tell the destination that we *might* want to do postcopy later;
>> * if the other end can't do postcopy it should fail now, nice and
>> * early.
>> */
>> - qemu_savevm_send_postcopy_advise(s->file);
>> + qemu_savevm_send_postcopy_advise(s->to_dst_file);
>> }
>>
>> - qemu_savevm_state_begin(s->file, &s->params);
>> + qemu_savevm_state_begin(s->to_dst_file, &s->params);
>>
>> s->setup_time = qemu_clock_get_ms(QEMU_CLOCK_HOST) - setup_start;
>> current_active_state = MIGRATION_STATUS_ACTIVE;
>> @@ -1678,10 +1679,10 @@ static void *migration_thread(void *opaque)
>> int64_t current_time;
>> uint64_t pending_size;
>>
>> - if (!qemu_file_rate_limit(s->file)) {
>> + if (!qemu_file_rate_limit(s->to_dst_file)) {
>> uint64_t pend_post, pend_nonpost;
>>
>> - qemu_savevm_state_pending(s->file, max_size, &pend_nonpost,
>> + qemu_savevm_state_pending(s->to_dst_file, max_size, &pend_nonpost,
>> &pend_post);
>> pending_size = pend_nonpost + pend_post;
>> trace_migrate_pending(pending_size, max_size,
>> @@ -1702,7 +1703,7 @@ static void *migration_thread(void *opaque)
>> continue;
>> }
>> /* Just another iteration step */
>> - qemu_savevm_state_iterate(s->file, entered_postcopy);
>> + qemu_savevm_state_iterate(s->to_dst_file, entered_postcopy);
>> } else {
>> trace_migration_thread_low_pending(pending_size);
>> migration_completion(s, current_active_state,
>> @@ -1711,7 +1712,7 @@ static void *migration_thread(void *opaque)
>> }
>> }
>>
>> - if (qemu_file_get_error(s->file)) {
>> + if (qemu_file_get_error(s->to_dst_file)) {
>> migrate_set_state(&s->state, current_active_state,
>> MIGRATION_STATUS_FAILED);
>> trace_migration_thread_file_err();
>> @@ -1719,7 +1720,8 @@ static void *migration_thread(void *opaque)
>> }
>> current_time = qemu_clock_get_ms(QEMU_CLOCK_REALTIME);
>> if (current_time >= initial_time + BUFFER_DELAY) {
>> - uint64_t transferred_bytes = qemu_ftell(s->file) - initial_bytes;
>> + uint64_t transferred_bytes = qemu_ftell(s->to_dst_file) -
>> + initial_bytes;
>> uint64_t time_spent = current_time - initial_time;
>> double bandwidth = transferred_bytes / time_spent;
>> max_size = bandwidth * migrate_max_downtime() / 1000000;
>> @@ -1735,11 +1737,11 @@ static void *migration_thread(void *opaque)
>> s->expected_downtime = s->dirty_bytes_rate / bandwidth;
>> }
>>
>> - qemu_file_reset_rate_limit(s->file);
>> + qemu_file_reset_rate_limit(s->to_dst_file);
>> initial_time = current_time;
>> - initial_bytes = qemu_ftell(s->file);
>> + initial_bytes = qemu_ftell(s->to_dst_file);
>> }
>> - if (qemu_file_rate_limit(s->file)) {
>> + if (qemu_file_rate_limit(s->to_dst_file)) {
>> /* usleep expects microseconds */
>> g_usleep((initial_time + BUFFER_DELAY - current_time)*1000);
>> }
>> @@ -1757,7 +1759,7 @@ static void *migration_thread(void *opaque)
>> qemu_savevm_state_cleanup();
>> }
>> if (s->state == MIGRATION_STATUS_COMPLETED) {
>> - uint64_t transferred_bytes = qemu_ftell(s->file);
>> + uint64_t transferred_bytes = qemu_ftell(s->to_dst_file);
>> s->total_time = end_time - s->total_time;
>> if (!entered_postcopy) {
>> s->downtime = end_time - start_time;
>> @@ -1794,7 +1796,7 @@ void migrate_fd_connect(MigrationState *s)
>> s->expected_downtime = max_downtime/1000000;
>> s->cleanup_bh = qemu_bh_new(migrate_fd_cleanup, s);
>>
>> - qemu_file_set_rate_limit(s->file,
>> + qemu_file_set_rate_limit(s->to_dst_file,
>> s->bandwidth_limit / XFER_LIMIT_RATIO);
>>
>> /* Notify before starting migration thread */
>> diff --git a/migration/postcopy-ram.c b/migration/postcopy-ram.c
>> index 22d6b18..f74016d 100644
>> --- a/migration/postcopy-ram.c
>> +++ b/migration/postcopy-ram.c
>> @@ -733,7 +733,8 @@ void postcopy_discard_send_range(MigrationState *ms, PostcopyDiscardState *pds,
>>
>> if (pds->cur_entry == MAX_DISCARDS_PER_COMMAND) {
>> /* Full set, ship it! */
>> - qemu_savevm_send_postcopy_ram_discard(ms->file, pds->ramblock_name,
>> + qemu_savevm_send_postcopy_ram_discard(ms->to_dst_file,
>> + pds->ramblock_name,
>> pds->cur_entry,
>> pds->start_list,
>> pds->length_list);
>> @@ -753,7 +754,8 @@ void postcopy_discard_send_finish(MigrationState *ms, PostcopyDiscardState *pds)
>> {
>> /* Anything unsent? */
>> if (pds->cur_entry) {
>> - qemu_savevm_send_postcopy_ram_discard(ms->file, pds->ramblock_name,
>> + qemu_savevm_send_postcopy_ram_discard(ms->to_dst_file,
>> + pds->ramblock_name,
>> pds->cur_entry,
>> pds->start_list,
>> pds->length_list);
>> diff --git a/migration/savevm.c b/migration/savevm.c
>> index 0ad1b93..f102870 100644
>> --- a/migration/savevm.c
>> +++ b/migration/savevm.c
>> @@ -1163,7 +1163,7 @@ static int qemu_savevm_state(QEMUFile *f, Error **errp)
>> .shared = 0
>> };
>> MigrationState *ms = migrate_init(¶ms);
>> - ms->file = f;
>> + ms->to_dst_file = f;
>>
>> if (qemu_savevm_state_blocked(errp)) {
>> return -EINVAL;
>> diff --git a/migration/tcp.c b/migration/tcp.c
>> index ae89172..e083d68 100644
>> --- a/migration/tcp.c
>> +++ b/migration/tcp.c
>> @@ -39,11 +39,11 @@ static void tcp_wait_for_connect(int fd, Error *err, void *opaque)
>>
>> if (fd < 0) {
>> DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
>> - s->file = NULL;
>> + s->to_dst_file = NULL;
>> migrate_fd_error(s);
>> } else {
>> DPRINTF("migrate connect success\n");
>> - s->file = qemu_fopen_socket(fd, "wb");
>> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
>> migrate_fd_connect(s);
>> }
>> }
>> diff --git a/migration/unix.c b/migration/unix.c
>> index b591813..5492dd6 100644
>> --- a/migration/unix.c
>> +++ b/migration/unix.c
>> @@ -39,11 +39,11 @@ static void unix_wait_for_connect(int fd, Error *err, void *opaque)
>>
>> if (fd < 0) {
>> DPRINTF("migrate connect error: %s\n", error_get_pretty(err));
>> - s->file = NULL;
>> + s->to_dst_file = NULL;
>> migrate_fd_error(s);
>> } else {
>> DPRINTF("migrate connect success\n");
>> - s->file = qemu_fopen_socket(fd, "wb");
>> + s->to_dst_file = qemu_fopen_socket(fd, "wb");
>> migrate_fd_connect(s);
>> }
>> }
>>
>
>
>
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 23/39] COLO: Implement failover work for Primary VM
2015-12-10 18:34 ` Dr. David Alan Gilbert
@ 2015-12-11 7:54 ` Hailiang Zhang
2015-12-11 9:22 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-11 7:54 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/11 2:34, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> For PVM, if there is failover request from users.
>> The colo thread will exit the loop while the failover BH does the
>> cleanup work and resumes VM.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> ---
>> v11:
>> - Don't call migration_end() in primary_vm_do_failover(),
>> The cleanup work will be done in migration_thread().
>> - Remove vm_start() in primary_vm_do_failover() which also been done
>> in migraiton_thread()
>> v10:
>> - Call migration_end() in primary_vm_do_failover()
>> ---
>> include/migration/colo.h | 3 +++
>> include/migration/failover.h | 1 +
>> migration/colo-failover.c | 7 +++++-
>> migration/colo.c | 56 ++++++++++++++++++++++++++++++++++++++++++--
>> 4 files changed, 64 insertions(+), 3 deletions(-)
>>
>> diff --git a/include/migration/colo.h b/include/migration/colo.h
>> index ba27719..0b02e95 100644
>> --- a/include/migration/colo.h
>> +++ b/include/migration/colo.h
>> @@ -32,4 +32,7 @@ void *colo_process_incoming_thread(void *opaque);
>> bool migration_incoming_in_colo_state(void);
>>
>> COLOMode get_colo_mode(void);
>> +
>> +/* failover */
>> +void colo_do_failover(MigrationState *s);
>> #endif
>> diff --git a/include/migration/failover.h b/include/migration/failover.h
>> index 882c625..fba3931 100644
>> --- a/include/migration/failover.h
>> +++ b/include/migration/failover.h
>> @@ -26,5 +26,6 @@ void failover_init_state(void);
>> int failover_set_state(int old_state, int new_state);
>> int failover_get_state(void);
>> void failover_request_active(Error **errp);
>> +bool failover_request_is_active(void);
>>
>> #endif
>> diff --git a/migration/colo-failover.c b/migration/colo-failover.c
>> index 1b1be24..0c525da 100644
>> --- a/migration/colo-failover.c
>> +++ b/migration/colo-failover.c
>> @@ -32,7 +32,7 @@ static void colo_failover_bh(void *opaque)
>> error_report("Unkown error for failover, old_state=%d", old_state);
>> return;
>> }
>> - /*TODO: Do failover work */
>> + colo_do_failover(NULL);
>> }
>>
>> void failover_request_active(Error **errp)
>> @@ -67,6 +67,11 @@ int failover_get_state(void)
>> return atomic_read(&failover_state);
>> }
>>
>> +bool failover_request_is_active(void)
>> +{
>> + return ((failover_get_state() != FAILOVER_STATUS_NONE));
>> +}
>> +
>> void qmp_x_colo_lost_heartbeat(Error **errp)
>> {
>> if (get_colo_mode() == COLO_MODE_UNKNOWN) {
>> diff --git a/migration/colo.c b/migration/colo.c
>> index cedfc63..7a42fc6 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -41,6 +41,42 @@ bool migration_incoming_in_colo_state(void)
>> return mis && (mis->state == MIGRATION_STATUS_COLO);
>> }
>>
>> +static bool colo_runstate_is_stopped(void)
>> +{
>> + return runstate_check(RUN_STATE_COLO) || !runstate_is_running();
>> +}
>> +
>> +static void primary_vm_do_failover(void)
>> +{
>> + MigrationState *s = migrate_get_current();
>> + int old_state;
>> +
>> + if (s->state != MIGRATION_STATUS_FAILED) {
>> + migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
>> + MIGRATION_STATUS_COMPLETED);
>> + }
>
> That's a little odd; it will only move to completed if the current
> state is MIGRATION_STATUS_COLO, but you only do it if the state isn't
> FAILED. You could remove the if and just call migrate_set_state
> like that and it would be safe as long as you really only expect
> to have to deal with MIGRATION_STATUS_COLO.
>
Yes, you are right, i will remove the judgement, and set the state directly.
>> + old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
>> + FAILOVER_STATUS_COMPLETED);
>> + if (old_state != FAILOVER_STATUS_HANDLING) {
>> + error_report("Serious error while do failover for Primary VM,"
>> + "old_state: %d", old_state);
>
> It's generally better to state the reason rather than say it's 'serious';
> so something like:
> 'Incorrect state (%d) while doing failover for Primary VM'
> tells you more, and looks less scary!
>
Ha, OK, i will fix it.
>> + return;
>> + }
>> +}
>> +
>> +void colo_do_failover(MigrationState *s)
>> +{
>> + /* Make sure vm stopped while failover */
>> + if (!colo_runstate_is_stopped()) {
>> + vm_stop_force_state(RUN_STATE_COLO);
>> + }
>
> That feels like a race? couldn't we be just at the end
> of taking a checkpoint and about to restart when you do
> the if, so it reads that it's currently stopped but
> then it restarts it by the time you have a chance to
> do anything?
Do you mean, after we stopped VM in failover() but before done the failover work,
the migration (checkpoint) thread may starts VM just in the middle time ?
The colo_do_failover() is in the context of BH, it holds
the __lock__, so checkpoint thread has no chance to execute vm_start().
> I see in patch 13 (Save PVM....) you have:
> qemu_mutex_lock_iothread();
> vm_start();
> qemu_mutex_unlock_iothread();
>
> So maybe if that code is executed just before the
> failover, then it would stop at the _lock_, we would
> run here but then as soon as we finish wouldn't it vm_start
> there?
>
But it seems that, we don't need to stop VM to do the failover work.
I'm not so sure, i will investigate if we can do it without stopping VM.
>> + if (get_colo_mode() == COLO_MODE_PRIMARY) {
>> + primary_vm_do_failover();
>> + }
>> +}
>> +
>> /* colo checkpoint control helper */
>> static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value)
>> {
>> @@ -122,9 +158,22 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
>> }
>>
>> qemu_mutex_lock_iothread();
>> + if (failover_request_is_active()) {
>> + qemu_mutex_unlock_iothread();
>> + ret = -1;
>> + goto out;
>> + }
>> vm_stop_force_state(RUN_STATE_COLO);
>> qemu_mutex_unlock_iothread();
>> trace_colo_vm_state_change("run", "stop");
>> + /*
>> + * failover request bh could be called after
>> + * vm_stop_force_state so we check failover_request_is_active() again.
>> + */
>> + if (failover_request_is_active()) {
>> + ret = -1;
>> + goto out;
>> + }
>
> I'm confused about why the check is needed specifically here;
> for example can't that happen at any point where we're ousite of the
> iothread lock? e.g. couldn't we set failover just a couple of
> lines lower, lets say just after the s->params.blk= 0 ?
>
Yes, you are right, failover could happen at any place where we are not
holding iothread lock. We should checkpoint the failover status after every
important steps. I'll add more check in next version ...
>> /* Disable block migration */
>> s->params.blk = 0;
>> @@ -221,6 +270,11 @@ static void colo_process_checkpoint(MigrationState *s)
>> trace_colo_vm_state_change("stop", "run");
>>
>> while (s->state == MIGRATION_STATUS_COLO) {
>> + if (failover_request_is_active()) {
>> + error_report("failover request");
>> + goto out;
>> + }
>> +
>> current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>> if (current_time - checkpoint_time <
>> s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) {
>> @@ -242,8 +296,6 @@ out:
>> if (ret < 0) {
>> error_report("%s: %s", __func__, strerror(-ret));
>> }
>> - migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
>> - MIGRATION_STATUS_COMPLETED);
>
> I had to think about this; but yes, I guess the only way out is via the failover
> which does the completed above.
>
Yes, thanks.
Hailiang
> Dave
>
>> qsb_free(buffer);
>> buffer = NULL;
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 24/39] COLO: Implement failover work for Secondary VM
2015-12-10 18:50 ` Dr. David Alan Gilbert
@ 2015-12-11 8:27 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-11 8:27 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/11 2:50, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> If users require SVM to takeover work, colo incoming thread should
>> exit from loop while failover BH helps backing to migration incoming
>> coroutine.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> ---
>> migration/colo.c | 42 +++++++++++++++++++++++++++++++++++++++---
>> 1 file changed, 39 insertions(+), 3 deletions(-)
>>
>> diff --git a/migration/colo.c b/migration/colo.c
>> index 7a42fc6..f31e957 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -46,6 +46,33 @@ static bool colo_runstate_is_stopped(void)
>> return runstate_check(RUN_STATE_COLO) || !runstate_is_running();
>> }
>>
>> +static void secondary_vm_do_failover(void)
>> +{
>> + int old_state;
>> + MigrationIncomingState *mis = migration_incoming_get_current();
>> +
>> + migrate_set_state(&mis->state, MIGRATION_STATUS_COLO,
>> + MIGRATION_STATUS_COMPLETED);
>> +
>> + if (!autostart) {
>> + error_report("\"-S\" qemu option will be ignored in secondary side");
>> + /* recover runstate to normal migration finish state */
>> + autostart = true;
>> + }
>
> You might find libvirt will need something different for it to be
> involved during the failover; but for now OK.
>
>> + old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
>> + FAILOVER_STATUS_COMPLETED);
>> + if (old_state != FAILOVER_STATUS_HANDLING) {
>> + error_report("Serious error while do failover for secondary VM,"
>> + "old_state: %d", old_state);
>
> Same suggestion as previous patch just to improve the error message.
>
OK, will fix it in next version.
>> + return;
>> + }
>> + /* For Secondary VM, jump to incoming co */
>> + if (mis->migration_incoming_co) {
>> + qemu_coroutine_enter(mis->migration_incoming_co, NULL);
>> + }
>> +}
>> +
>> static void primary_vm_do_failover(void)
>> {
>> MigrationState *s = migrate_get_current();
>> @@ -74,6 +101,8 @@ void colo_do_failover(MigrationState *s)
>>
>> if (get_colo_mode() == COLO_MODE_PRIMARY) {
>> primary_vm_do_failover();
>> + } else {
>> + secondary_vm_do_failover();
>> }
>> }
>>
>> @@ -404,6 +433,12 @@ void *colo_process_incoming_thread(void *opaque)
>> continue;
>> }
>> }
>> +
>> + if (failover_request_is_active()) {
>> + error_report("failover request");
>> + goto out;
>> + }
>> +
>> /* FIXME: This is unnecessary for periodic checkpoint mode */
>> ret = colo_ctl_put(mis->to_src_file, COLO_COMMAND_CHECKPOINT_REPLY, 0);
>> if (ret < 0) {
>> @@ -473,10 +508,11 @@ out:
>> qemu_fclose(fb);
>> }
>> qsb_free(buffer);
>> -
>> - qemu_mutex_lock_iothread();
>> + /* Here, we can ensure BH is hold the global lock, and will join colo
>> + * incoming thread, so here it is not necessary to lock here again,
>> + * or there will be a deadlock error.
>> + */
>> colo_release_ram_cache();
>> - qemu_mutex_unlock_iothread();
>
> OK, I think I understand that - becuase we know there is a failover request
> active, then it must be holding the lock?
>
Yes, we come here only when failover happened, and since Secondary VM
does failover in BH with holding iothread lock, and it will enter migration_incoming_co
at the end. The migration_incoming_co() will wait for colo incoming thread to finish.
So it can't try to get iothread lock, or there will be an deadlock error.
> Other than the error message improvement:
>
> Reviewed-by: Dr. David Alan Gilbert <dgilbert@redhat.com>
>
> Dave
>
Thanks,
Hailiang
>>
>> if (mis->to_src_file) {
>> qemu_fclose(mis->to_src_file);
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 27/39] COLO failover: Shutdown related socket fd when do failover
2015-12-10 20:03 ` Dr. David Alan Gilbert
@ 2015-12-11 8:57 ` Hailiang Zhang
2015-12-11 9:18 ` Dr. David Alan Gilbert
0 siblings, 1 reply; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-11 8:57 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/11 4:03, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> If the net connection between COLO's two sides is broken while colo/colo incoming
>> thread is blocked in 'read'/'write' socket fd. It will not detect this error until
>> connect timeout. It will be a long time.
>>
>> Here we shutdown all the related socket file descriptors to wake up the blocking
>> operation in failover BH. Besides, we should close the corresponding file descriptors
>> after failvoer BH shutdown them, or there will be an error.
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> ---
>> v11:
>> - Only shutdown fd for once
>> ---
>> migration/colo.c | 31 +++++++++++++++++++++++++++++--
>> 1 file changed, 29 insertions(+), 2 deletions(-)
>>
>> diff --git a/migration/colo.c b/migration/colo.c
>> index 4cd7b00..994b80d 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -68,6 +68,14 @@ static void secondary_vm_do_failover(void)
>> /* recover runstate to normal migration finish state */
>> autostart = true;
>> }
>> + /*
>> + * Make sure colo incoming thread not block in recv,
>> + * mis->from_src_file and mis->to_src_file use the same fd,
>> + * so here we only need to shutdown it for once.
>> + */
>
> That assumption is only valid for the current socket transport;
> for example I have a (partially-working) RDMA transport where the
> forward and reverse paths are quite separate.
>
Yes, you are right, but we really catch the bad stuck case in test for many times,
and it is easy to reproduce. So i think it's necessary to keep this codes.
>> + if (mis->from_src_file) {
>> + qemu_file_shutdown(mis->from_src_file);
>> + }
>>
>> old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
>> FAILOVER_STATUS_COMPLETED);
>> @@ -92,6 +100,15 @@ static void primary_vm_do_failover(void)
>> MIGRATION_STATUS_COMPLETED);
>> }
>>
>> + /*
>> + * Make sure colo thread no block in recv,
>> + * Besides, s->rp_state.from_dst_file and s->to_dst_file use the
>> + * same fd, so here we only need to shutdown it for once.
>> + */
>> + if (s->to_dst_file) {
>> + qemu_file_shutdown(s->to_dst_file);
>> + }
>> +
>> old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
>> FAILOVER_STATUS_COMPLETED);
>> if (old_state != FAILOVER_STATUS_HANDLING) {
>> @@ -333,7 +350,7 @@ static void colo_process_checkpoint(MigrationState *s)
>>
>> out:
>> current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>> - if (ret < 0) {
>> + if (ret < 0 || (!ret && !failover_request_is_active())) {
>
> Why is this needed - when would you get a 0 return that is actually an error?
>
If we shutdown the fd, it will return 0, and the ret will be 0, this only happen
when we do failover actively.
>> error_report("%s: %s", __func__, strerror(-ret));
>> qapi_event_send_colo_exit(COLO_MODE_PRIMARY, COLO_EXIT_REASON_ERROR,
>> true, strerror(-ret), NULL);
>> @@ -362,6 +379,11 @@ out:
>> qsb_free(buffer);
>> buffer = NULL;
>>
>> + /* Hope this not to be too long to loop here */
>> + while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
>> + ;
>> + }
>> + /* Must be called after failover BH is completed */
>
> Is this the right patch for this?
>
Yes, we must ensure the close() called after the shutdown() in failover,
or there maybe an shutdown wrong 'fd' error, the 'fd' maybe just re-used by
other thread after we close here, but before shutdown() in failover. This
is a race case, which is rarely happening. I'll add more comments here.
Thanks,
Hailiang
>> if (s->rp_state.from_dst_file) {
>> qemu_fclose(s->rp_state.from_dst_file);
>> }
>> @@ -534,7 +556,7 @@ void *colo_process_incoming_thread(void *opaque)
>>
>> out:
>> current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>> - if (ret < 0) {
>> + if (ret < 0 || (!ret && !failover_request_is_active())) {
>> error_report("colo incoming thread will exit, detect error: %s",
>> strerror(-ret));
>> qapi_event_send_colo_exit(COLO_MODE_SECONDARY, COLO_EXIT_REASON_ERROR,
>> @@ -573,6 +595,11 @@ out:
>> */
>> colo_release_ram_cache();
>>
>> + /* Hope this not to be too long to loop here */
>> + while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
>> + ;
>> + }
>> + /* Must be called after failover BH is completed */
>> if (mis->to_src_file) {
>> qemu_fclose(mis->to_src_file);
>> }
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 27/39] COLO failover: Shutdown related socket fd when do failover
2015-12-11 8:57 ` Hailiang Zhang
@ 2015-12-11 9:18 ` Dr. David Alan Gilbert
2015-12-11 9:29 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-11 9:18 UTC (permalink / raw)
To: Hailiang Zhang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* Hailiang Zhang (zhang.zhanghailiang@huawei.com) wrote:
> On 2015/12/11 4:03, Dr. David Alan Gilbert wrote:
> >* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> >>If the net connection between COLO's two sides is broken while colo/colo incoming
> >>thread is blocked in 'read'/'write' socket fd. It will not detect this error until
> >>connect timeout. It will be a long time.
> >>
> >>Here we shutdown all the related socket file descriptors to wake up the blocking
> >>operation in failover BH. Besides, we should close the corresponding file descriptors
> >>after failvoer BH shutdown them, or there will be an error.
> >>
> >>Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> >>Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> >>---
> >>v11:
> >>- Only shutdown fd for once
> >>---
> >> migration/colo.c | 31 +++++++++++++++++++++++++++++--
> >> 1 file changed, 29 insertions(+), 2 deletions(-)
> >>
> >>diff --git a/migration/colo.c b/migration/colo.c
> >>index 4cd7b00..994b80d 100644
> >>--- a/migration/colo.c
> >>+++ b/migration/colo.c
> >>@@ -68,6 +68,14 @@ static void secondary_vm_do_failover(void)
> >> /* recover runstate to normal migration finish state */
> >> autostart = true;
> >> }
> >>+ /*
> >>+ * Make sure colo incoming thread not block in recv,
> >>+ * mis->from_src_file and mis->to_src_file use the same fd,
> >>+ * so here we only need to shutdown it for once.
> >>+ */
> >
> >That assumption is only valid for the current socket transport;
> >for example I have a (partially-working) RDMA transport where the
> >forward and reverse paths are quite separate.
> >
>
> Yes, you are right, but we really catch the bad stuck case in test for many times,
> and it is easy to reproduce. So i think it's necessary to keep this codes.
Yes, I agree you must shutdown from_src_file, but I mean do you also need to
shutdown to_src_file ? On sockets they share an fd, but that might not always be
true, so you might have to shutdown both QEMUFile's - but only if you can
get blocked on them.
>
> >>+ if (mis->from_src_file) {
> >>+ qemu_file_shutdown(mis->from_src_file);
> >>+ }
> >>
> >> old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
> >> FAILOVER_STATUS_COMPLETED);
> >>@@ -92,6 +100,15 @@ static void primary_vm_do_failover(void)
> >> MIGRATION_STATUS_COMPLETED);
> >> }
> >>
> >>+ /*
> >>+ * Make sure colo thread no block in recv,
> >>+ * Besides, s->rp_state.from_dst_file and s->to_dst_file use the
> >>+ * same fd, so here we only need to shutdown it for once.
> >>+ */
> >>+ if (s->to_dst_file) {
> >>+ qemu_file_shutdown(s->to_dst_file);
> >>+ }
> >>+
> >> old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
> >> FAILOVER_STATUS_COMPLETED);
> >> if (old_state != FAILOVER_STATUS_HANDLING) {
> >>@@ -333,7 +350,7 @@ static void colo_process_checkpoint(MigrationState *s)
> >>
> >> out:
> >> current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> >>- if (ret < 0) {
> >>+ if (ret < 0 || (!ret && !failover_request_is_active())) {
> >
> >Why is this needed - when would you get a 0 return that is actually an error?
> >
>
> If we shutdown the fd, it will return 0, and the ret will be 0, this only happen
> when we do failover actively.
OK
> >> error_report("%s: %s", __func__, strerror(-ret));
> >> qapi_event_send_colo_exit(COLO_MODE_PRIMARY, COLO_EXIT_REASON_ERROR,
> >> true, strerror(-ret), NULL);
> >>@@ -362,6 +379,11 @@ out:
> >> qsb_free(buffer);
> >> buffer = NULL;
> >>
> >>+ /* Hope this not to be too long to loop here */
> >>+ while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
> >>+ ;
> >>+ }
> >>+ /* Must be called after failover BH is completed */
> >
> >Is this the right patch for this?
> >
>
> Yes, we must ensure the close() called after the shutdown() in failover,
> or there maybe an shutdown wrong 'fd' error, the 'fd' maybe just re-used by
> other thread after we close here, but before shutdown() in failover. This
> is a race case, which is rarely happening. I'll add more comments here.
Thanks, that makes sense; closing the wrong fd can be really hard to debug :-)
Dave
>
> Thanks,
> Hailiang
>
>
> >> if (s->rp_state.from_dst_file) {
> >> qemu_fclose(s->rp_state.from_dst_file);
> >> }
> >>@@ -534,7 +556,7 @@ void *colo_process_incoming_thread(void *opaque)
> >>
> >> out:
> >> current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> >>- if (ret < 0) {
> >>+ if (ret < 0 || (!ret && !failover_request_is_active())) {
> >> error_report("colo incoming thread will exit, detect error: %s",
> >> strerror(-ret));
> >> qapi_event_send_colo_exit(COLO_MODE_SECONDARY, COLO_EXIT_REASON_ERROR,
> >>@@ -573,6 +595,11 @@ out:
> >> */
> >> colo_release_ram_cache();
> >>
> >>+ /* Hope this not to be too long to loop here */
> >>+ while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
> >>+ ;
> >>+ }
> >>+ /* Must be called after failover BH is completed */
> >> if (mis->to_src_file) {
> >> qemu_fclose(mis->to_src_file);
> >> }
> >>--
> >>1.8.3.1
> >>
> >>
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> >.
> >
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 23/39] COLO: Implement failover work for Primary VM
2015-12-11 7:54 ` Hailiang Zhang
@ 2015-12-11 9:22 ` Dr. David Alan Gilbert
2015-12-11 9:38 ` Hailiang Zhang
0 siblings, 1 reply; 95+ messages in thread
From: Dr. David Alan Gilbert @ 2015-12-11 9:22 UTC (permalink / raw)
To: Hailiang Zhang
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
* Hailiang Zhang (zhang.zhanghailiang@huawei.com) wrote:
> On 2015/12/11 2:34, Dr. David Alan Gilbert wrote:
> >* zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
> >>For PVM, if there is failover request from users.
> >>The colo thread will exit the loop while the failover BH does the
> >>cleanup work and resumes VM.
> >>
> >>Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
> >>Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
> >>---
> >>v11:
> >>- Don't call migration_end() in primary_vm_do_failover(),
> >> The cleanup work will be done in migration_thread().
> >>- Remove vm_start() in primary_vm_do_failover() which also been done
> >> in migraiton_thread()
> >>v10:
> >>- Call migration_end() in primary_vm_do_failover()
> >>---
> >> include/migration/colo.h | 3 +++
> >> include/migration/failover.h | 1 +
> >> migration/colo-failover.c | 7 +++++-
> >> migration/colo.c | 56 ++++++++++++++++++++++++++++++++++++++++++--
> >> 4 files changed, 64 insertions(+), 3 deletions(-)
> >>
> >>diff --git a/include/migration/colo.h b/include/migration/colo.h
> >>index ba27719..0b02e95 100644
> >>--- a/include/migration/colo.h
> >>+++ b/include/migration/colo.h
> >>@@ -32,4 +32,7 @@ void *colo_process_incoming_thread(void *opaque);
> >> bool migration_incoming_in_colo_state(void);
> >>
> >> COLOMode get_colo_mode(void);
> >>+
> >>+/* failover */
> >>+void colo_do_failover(MigrationState *s);
> >> #endif
> >>diff --git a/include/migration/failover.h b/include/migration/failover.h
> >>index 882c625..fba3931 100644
> >>--- a/include/migration/failover.h
> >>+++ b/include/migration/failover.h
> >>@@ -26,5 +26,6 @@ void failover_init_state(void);
> >> int failover_set_state(int old_state, int new_state);
> >> int failover_get_state(void);
> >> void failover_request_active(Error **errp);
> >>+bool failover_request_is_active(void);
> >>
> >> #endif
> >>diff --git a/migration/colo-failover.c b/migration/colo-failover.c
> >>index 1b1be24..0c525da 100644
> >>--- a/migration/colo-failover.c
> >>+++ b/migration/colo-failover.c
> >>@@ -32,7 +32,7 @@ static void colo_failover_bh(void *opaque)
> >> error_report("Unkown error for failover, old_state=%d", old_state);
> >> return;
> >> }
> >>- /*TODO: Do failover work */
> >>+ colo_do_failover(NULL);
> >> }
> >>
> >> void failover_request_active(Error **errp)
> >>@@ -67,6 +67,11 @@ int failover_get_state(void)
> >> return atomic_read(&failover_state);
> >> }
> >>
> >>+bool failover_request_is_active(void)
> >>+{
> >>+ return ((failover_get_state() != FAILOVER_STATUS_NONE));
> >>+}
> >>+
> >> void qmp_x_colo_lost_heartbeat(Error **errp)
> >> {
> >> if (get_colo_mode() == COLO_MODE_UNKNOWN) {
> >>diff --git a/migration/colo.c b/migration/colo.c
> >>index cedfc63..7a42fc6 100644
> >>--- a/migration/colo.c
> >>+++ b/migration/colo.c
> >>@@ -41,6 +41,42 @@ bool migration_incoming_in_colo_state(void)
> >> return mis && (mis->state == MIGRATION_STATUS_COLO);
> >> }
> >>
> >>+static bool colo_runstate_is_stopped(void)
> >>+{
> >>+ return runstate_check(RUN_STATE_COLO) || !runstate_is_running();
> >>+}
> >>+
> >>+static void primary_vm_do_failover(void)
> >>+{
> >>+ MigrationState *s = migrate_get_current();
> >>+ int old_state;
> >>+
> >>+ if (s->state != MIGRATION_STATUS_FAILED) {
> >>+ migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
> >>+ MIGRATION_STATUS_COMPLETED);
> >>+ }
> >
> >That's a little odd; it will only move to completed if the current
> >state is MIGRATION_STATUS_COLO, but you only do it if the state isn't
> >FAILED. You could remove the if and just call migrate_set_state
> >like that and it would be safe as long as you really only expect
> >to have to deal with MIGRATION_STATUS_COLO.
> >
>
> Yes, you are right, i will remove the judgement, and set the state directly.
>
> >>+ old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
> >>+ FAILOVER_STATUS_COMPLETED);
> >>+ if (old_state != FAILOVER_STATUS_HANDLING) {
> >>+ error_report("Serious error while do failover for Primary VM,"
> >>+ "old_state: %d", old_state);
> >
> >It's generally better to state the reason rather than say it's 'serious';
> >so something like:
> > 'Incorrect state (%d) while doing failover for Primary VM'
> >tells you more, and looks less scary!
> >
>
> Ha, OK, i will fix it.
>
> >>+ return;
> >>+ }
> >>+}
> >>+
> >>+void colo_do_failover(MigrationState *s)
> >>+{
> >>+ /* Make sure vm stopped while failover */
> >>+ if (!colo_runstate_is_stopped()) {
> >>+ vm_stop_force_state(RUN_STATE_COLO);
> >>+ }
> >
> >That feels like a race? couldn't we be just at the end
> >of taking a checkpoint and about to restart when you do
> >the if, so it reads that it's currently stopped but
> >then it restarts it by the time you have a chance to
> >do anything?
>
> Do you mean, after we stopped VM in failover() but before done the failover work,
> the migration (checkpoint) thread may starts VM just in the middle time ?
> The colo_do_failover() is in the context of BH, it holds
> the __lock__, so checkpoint thread has no chance to execute vm_start().
What happens if the failover code was executed just before the checkpoint thread
executed the _lock_, when the failover code finishes what happens?
(I'm not sure this is a problem - I just thought I should check).
> >I see in patch 13 (Save PVM....) you have:
> > qemu_mutex_lock_iothread();
> > vm_start();
> > qemu_mutex_unlock_iothread();
> >
> >So maybe if that code is executed just before the
> >failover, then it would stop at the _lock_, we would
> >run here but then as soon as we finish wouldn't it vm_start
> >there?
> >
>
> But it seems that, we don't need to stop VM to do the failover work.
> I'm not so sure, i will investigate if we can do it without stopping VM.
Stopping it seems safer.
>
> >>+ if (get_colo_mode() == COLO_MODE_PRIMARY) {
> >>+ primary_vm_do_failover();
> >>+ }
> >>+}
> >>+
> >> /* colo checkpoint control helper */
> >> static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value)
> >> {
> >>@@ -122,9 +158,22 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
> >> }
> >>
> >> qemu_mutex_lock_iothread();
> >>+ if (failover_request_is_active()) {
> >>+ qemu_mutex_unlock_iothread();
> >>+ ret = -1;
> >>+ goto out;
> >>+ }
> >> vm_stop_force_state(RUN_STATE_COLO);
> >> qemu_mutex_unlock_iothread();
> >> trace_colo_vm_state_change("run", "stop");
> >>+ /*
> >>+ * failover request bh could be called after
> >>+ * vm_stop_force_state so we check failover_request_is_active() again.
> >>+ */
> >>+ if (failover_request_is_active()) {
> >>+ ret = -1;
> >>+ goto out;
> >>+ }
> >
> >I'm confused about why the check is needed specifically here;
> >for example can't that happen at any point where we're ousite of the
> >iothread lock? e.g. couldn't we set failover just a couple of
> >lines lower, lets say just after the s->params.blk= 0 ?
> >
>
> Yes, you are right, failover could happen at any place where we are not
> holding iothread lock. We should checkpoint the failover status after every
> important steps. I'll add more check in next version ...
>
>
> >> /* Disable block migration */
> >> s->params.blk = 0;
> >>@@ -221,6 +270,11 @@ static void colo_process_checkpoint(MigrationState *s)
> >> trace_colo_vm_state_change("stop", "run");
> >>
> >> while (s->state == MIGRATION_STATUS_COLO) {
> >>+ if (failover_request_is_active()) {
> >>+ error_report("failover request");
> >>+ goto out;
> >>+ }
> >>+
> >> current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
> >> if (current_time - checkpoint_time <
> >> s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) {
> >>@@ -242,8 +296,6 @@ out:
> >> if (ret < 0) {
> >> error_report("%s: %s", __func__, strerror(-ret));
> >> }
> >>- migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
> >>- MIGRATION_STATUS_COMPLETED);
> >
> >I had to think about this; but yes, I guess the only way out is via the failover
> >which does the completed above.
> >
>
> Yes, thanks.
>
> Hailiang
>
> >Dave
> >
> >> qsb_free(buffer);
> >> buffer = NULL;
> >>--
> >>1.8.3.1
> >>
> >>
> >--
> >Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
> >
> >.
> >
>
>
--
Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 27/39] COLO failover: Shutdown related socket fd when do failover
2015-12-11 9:18 ` Dr. David Alan Gilbert
@ 2015-12-11 9:29 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-11 9:29 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/11 17:18, Dr. David Alan Gilbert wrote:
> * Hailiang Zhang (zhang.zhanghailiang@huawei.com) wrote:
>> On 2015/12/11 4:03, Dr. David Alan Gilbert wrote:
>>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>>>> If the net connection between COLO's two sides is broken while colo/colo incoming
>>>> thread is blocked in 'read'/'write' socket fd. It will not detect this error until
>>>> connect timeout. It will be a long time.
>>>>
>>>> Here we shutdown all the related socket file descriptors to wake up the blocking
>>>> operation in failover BH. Besides, we should close the corresponding file descriptors
>>>> after failvoer BH shutdown them, or there will be an error.
>>>>
>>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>>>> ---
>>>> v11:
>>>> - Only shutdown fd for once
>>>> ---
>>>> migration/colo.c | 31 +++++++++++++++++++++++++++++--
>>>> 1 file changed, 29 insertions(+), 2 deletions(-)
>>>>
>>>> diff --git a/migration/colo.c b/migration/colo.c
>>>> index 4cd7b00..994b80d 100644
>>>> --- a/migration/colo.c
>>>> +++ b/migration/colo.c
>>>> @@ -68,6 +68,14 @@ static void secondary_vm_do_failover(void)
>>>> /* recover runstate to normal migration finish state */
>>>> autostart = true;
>>>> }
>>>> + /*
>>>> + * Make sure colo incoming thread not block in recv,
>>>> + * mis->from_src_file and mis->to_src_file use the same fd,
>>>> + * so here we only need to shutdown it for once.
>>>> + */
>>>
>>> That assumption is only valid for the current socket transport;
>>> for example I have a (partially-working) RDMA transport where the
>>> forward and reverse paths are quite separate.
>>>
>>
>> Yes, you are right, but we really catch the bad stuck case in test for many times,
>> and it is easy to reproduce. So i think it's necessary to keep this codes.
>
> Yes, I agree you must shutdown from_src_file, but I mean do you also need to
> shutdown to_src_file ? On sockets they share an fd, but that might not always be
Yes, in that case, we need to shutdown it, i will add it. Thanks.
> true, so you might have to shutdown both QEMUFile's - but only if you can
> get blocked on them.
>
>>
>>>> + if (mis->from_src_file) {
>>>> + qemu_file_shutdown(mis->from_src_file);
>>>> + }
>>>>
>>>> old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
>>>> FAILOVER_STATUS_COMPLETED);
>>>> @@ -92,6 +100,15 @@ static void primary_vm_do_failover(void)
>>>> MIGRATION_STATUS_COMPLETED);
>>>> }
>>>>
>>>> + /*
>>>> + * Make sure colo thread no block in recv,
>>>> + * Besides, s->rp_state.from_dst_file and s->to_dst_file use the
>>>> + * same fd, so here we only need to shutdown it for once.
>>>> + */
>>>> + if (s->to_dst_file) {
>>>> + qemu_file_shutdown(s->to_dst_file);
>>>> + }
>>>> +
>>>> old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
>>>> FAILOVER_STATUS_COMPLETED);
>>>> if (old_state != FAILOVER_STATUS_HANDLING) {
>>>> @@ -333,7 +350,7 @@ static void colo_process_checkpoint(MigrationState *s)
>>>>
>>>> out:
>>>> current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>>>> - if (ret < 0) {
>>>> + if (ret < 0 || (!ret && !failover_request_is_active())) {
>>>
>>> Why is this needed - when would you get a 0 return that is actually an error?
>>>
>>
>> If we shutdown the fd, it will return 0, and the ret will be 0, this only happen
>> when we do failover actively.
>
> OK
>
>>>> error_report("%s: %s", __func__, strerror(-ret));
>>>> qapi_event_send_colo_exit(COLO_MODE_PRIMARY, COLO_EXIT_REASON_ERROR,
>>>> true, strerror(-ret), NULL);
>>>> @@ -362,6 +379,11 @@ out:
>>>> qsb_free(buffer);
>>>> buffer = NULL;
>>>>
>>>> + /* Hope this not to be too long to loop here */
>>>> + while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
>>>> + ;
>>>> + }
>>>> + /* Must be called after failover BH is completed */
>>>
>>> Is this the right patch for this?
>>>
>>
>> Yes, we must ensure the close() called after the shutdown() in failover,
>> or there maybe an shutdown wrong 'fd' error, the 'fd' maybe just re-used by
>> other thread after we close here, but before shutdown() in failover. This
>> is a race case, which is rarely happening. I'll add more comments here.
>
> Thanks, that makes sense; closing the wrong fd can be really hard to debug :-)
>
> Dave
>
>>
>> Thanks,
>> Hailiang
>>
>>
>>>> if (s->rp_state.from_dst_file) {
>>>> qemu_fclose(s->rp_state.from_dst_file);
>>>> }
>>>> @@ -534,7 +556,7 @@ void *colo_process_incoming_thread(void *opaque)
>>>>
>>>> out:
>>>> current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>>>> - if (ret < 0) {
>>>> + if (ret < 0 || (!ret && !failover_request_is_active())) {
>>>> error_report("colo incoming thread will exit, detect error: %s",
>>>> strerror(-ret));
>>>> qapi_event_send_colo_exit(COLO_MODE_SECONDARY, COLO_EXIT_REASON_ERROR,
>>>> @@ -573,6 +595,11 @@ out:
>>>> */
>>>> colo_release_ram_cache();
>>>>
>>>> + /* Hope this not to be too long to loop here */
>>>> + while (failover_get_state() != FAILOVER_STATUS_COMPLETED) {
>>>> + ;
>>>> + }
>>>> + /* Must be called after failover BH is completed */
>>>> if (mis->to_src_file) {
>>>> qemu_fclose(mis->to_src_file);
>>>> }
>>>> --
>>>> 1.8.3.1
>>>>
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>>> .
>>>
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 23/39] COLO: Implement failover work for Primary VM
2015-12-11 9:22 ` Dr. David Alan Gilbert
@ 2015-12-11 9:38 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-11 9:38 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/11 17:22, Dr. David Alan Gilbert wrote:
> * Hailiang Zhang (zhang.zhanghailiang@huawei.com) wrote:
>> On 2015/12/11 2:34, Dr. David Alan Gilbert wrote:
>>> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>>>> For PVM, if there is failover request from users.
>>>> The colo thread will exit the loop while the failover BH does the
>>>> cleanup work and resumes VM.
>>>>
>>>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>>>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>>>> ---
>>>> v11:
>>>> - Don't call migration_end() in primary_vm_do_failover(),
>>>> The cleanup work will be done in migration_thread().
>>>> - Remove vm_start() in primary_vm_do_failover() which also been done
>>>> in migraiton_thread()
>>>> v10:
>>>> - Call migration_end() in primary_vm_do_failover()
>>>> ---
>>>> include/migration/colo.h | 3 +++
>>>> include/migration/failover.h | 1 +
>>>> migration/colo-failover.c | 7 +++++-
>>>> migration/colo.c | 56 ++++++++++++++++++++++++++++++++++++++++++--
>>>> 4 files changed, 64 insertions(+), 3 deletions(-)
>>>>
>>>> diff --git a/include/migration/colo.h b/include/migration/colo.h
>>>> index ba27719..0b02e95 100644
>>>> --- a/include/migration/colo.h
>>>> +++ b/include/migration/colo.h
>>>> @@ -32,4 +32,7 @@ void *colo_process_incoming_thread(void *opaque);
>>>> bool migration_incoming_in_colo_state(void);
>>>>
>>>> COLOMode get_colo_mode(void);
>>>> +
>>>> +/* failover */
>>>> +void colo_do_failover(MigrationState *s);
>>>> #endif
>>>> diff --git a/include/migration/failover.h b/include/migration/failover.h
>>>> index 882c625..fba3931 100644
>>>> --- a/include/migration/failover.h
>>>> +++ b/include/migration/failover.h
>>>> @@ -26,5 +26,6 @@ void failover_init_state(void);
>>>> int failover_set_state(int old_state, int new_state);
>>>> int failover_get_state(void);
>>>> void failover_request_active(Error **errp);
>>>> +bool failover_request_is_active(void);
>>>>
>>>> #endif
>>>> diff --git a/migration/colo-failover.c b/migration/colo-failover.c
>>>> index 1b1be24..0c525da 100644
>>>> --- a/migration/colo-failover.c
>>>> +++ b/migration/colo-failover.c
>>>> @@ -32,7 +32,7 @@ static void colo_failover_bh(void *opaque)
>>>> error_report("Unkown error for failover, old_state=%d", old_state);
>>>> return;
>>>> }
>>>> - /*TODO: Do failover work */
>>>> + colo_do_failover(NULL);
>>>> }
>>>>
>>>> void failover_request_active(Error **errp)
>>>> @@ -67,6 +67,11 @@ int failover_get_state(void)
>>>> return atomic_read(&failover_state);
>>>> }
>>>>
>>>> +bool failover_request_is_active(void)
>>>> +{
>>>> + return ((failover_get_state() != FAILOVER_STATUS_NONE));
>>>> +}
>>>> +
>>>> void qmp_x_colo_lost_heartbeat(Error **errp)
>>>> {
>>>> if (get_colo_mode() == COLO_MODE_UNKNOWN) {
>>>> diff --git a/migration/colo.c b/migration/colo.c
>>>> index cedfc63..7a42fc6 100644
>>>> --- a/migration/colo.c
>>>> +++ b/migration/colo.c
>>>> @@ -41,6 +41,42 @@ bool migration_incoming_in_colo_state(void)
>>>> return mis && (mis->state == MIGRATION_STATUS_COLO);
>>>> }
>>>>
>>>> +static bool colo_runstate_is_stopped(void)
>>>> +{
>>>> + return runstate_check(RUN_STATE_COLO) || !runstate_is_running();
>>>> +}
>>>> +
>>>> +static void primary_vm_do_failover(void)
>>>> +{
>>>> + MigrationState *s = migrate_get_current();
>>>> + int old_state;
>>>> +
>>>> + if (s->state != MIGRATION_STATUS_FAILED) {
>>>> + migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
>>>> + MIGRATION_STATUS_COMPLETED);
>>>> + }
>>>
>>> That's a little odd; it will only move to completed if the current
>>> state is MIGRATION_STATUS_COLO, but you only do it if the state isn't
>>> FAILED. You could remove the if and just call migrate_set_state
>>> like that and it would be safe as long as you really only expect
>>> to have to deal with MIGRATION_STATUS_COLO.
>>>
>>
>> Yes, you are right, i will remove the judgement, and set the state directly.
>>
>>>> + old_state = failover_set_state(FAILOVER_STATUS_HANDLING,
>>>> + FAILOVER_STATUS_COMPLETED);
>>>> + if (old_state != FAILOVER_STATUS_HANDLING) {
>>>> + error_report("Serious error while do failover for Primary VM,"
>>>> + "old_state: %d", old_state);
>>>
>>> It's generally better to state the reason rather than say it's 'serious';
>>> so something like:
>>> 'Incorrect state (%d) while doing failover for Primary VM'
>>> tells you more, and looks less scary!
>>>
>>
>> Ha, OK, i will fix it.
>>
>>>> + return;
>>>> + }
>>>> +}
>>>> +
>>>> +void colo_do_failover(MigrationState *s)
>>>> +{
>>>> + /* Make sure vm stopped while failover */
>>>> + if (!colo_runstate_is_stopped()) {
>>>> + vm_stop_force_state(RUN_STATE_COLO);
>>>> + }
>>>
>>> That feels like a race? couldn't we be just at the end
>>> of taking a checkpoint and about to restart when you do
>>> the if, so it reads that it's currently stopped but
>>> then it restarts it by the time you have a chance to
>>> do anything?
>>
>> Do you mean, after we stopped VM in failover() but before done the failover work,
>> the migration (checkpoint) thread may starts VM just in the middle time ?
>> The colo_do_failover() is in the context of BH, it holds
>> the __lock__, so checkpoint thread has no chance to execute vm_start().
>
> What happens if the failover code was executed just before the checkpoint thread
> executed the _lock_, when the failover code finishes what happens?
> (I'm not sure this is a problem - I just thought I should check).
>
Er, the colo loop will exit, and the failover flag will be detected in colo process thread finally.
It's OK in this case. But, yes. we should add more check for the failover status.
Thanks
Hailiang
>
>>> I see in patch 13 (Save PVM....) you have:
>>> qemu_mutex_lock_iothread();
>>> vm_start();
>>> qemu_mutex_unlock_iothread();
>>>
>>> So maybe if that code is executed just before the
>>> failover, then it would stop at the _lock_, we would
>>> run here but then as soon as we finish wouldn't it vm_start
>>> there?
>>>
>>
>> But it seems that, we don't need to stop VM to do the failover work.
>> I'm not so sure, i will investigate if we can do it without stopping VM.
>
> Stopping it seems safer.
>
OK.
>>
>>>> + if (get_colo_mode() == COLO_MODE_PRIMARY) {
>>>> + primary_vm_do_failover();
>>>> + }
>>>> +}
>>>> +
>>>> /* colo checkpoint control helper */
>>>> static int colo_ctl_put(QEMUFile *f, uint32_t cmd, uint64_t value)
>>>> {
>>>> @@ -122,9 +158,22 @@ static int colo_do_checkpoint_transaction(MigrationState *s,
>>>> }
>>>>
>>>> qemu_mutex_lock_iothread();
>>>> + if (failover_request_is_active()) {
>>>> + qemu_mutex_unlock_iothread();
>>>> + ret = -1;
>>>> + goto out;
>>>> + }
>>>> vm_stop_force_state(RUN_STATE_COLO);
>>>> qemu_mutex_unlock_iothread();
>>>> trace_colo_vm_state_change("run", "stop");
>>>> + /*
>>>> + * failover request bh could be called after
>>>> + * vm_stop_force_state so we check failover_request_is_active() again.
>>>> + */
>>>> + if (failover_request_is_active()) {
>>>> + ret = -1;
>>>> + goto out;
>>>> + }
>>>
>>> I'm confused about why the check is needed specifically here;
>>> for example can't that happen at any point where we're ousite of the
>>> iothread lock? e.g. couldn't we set failover just a couple of
>>> lines lower, lets say just after the s->params.blk= 0 ?
>>>
>>
>> Yes, you are right, failover could happen at any place where we are not
>> holding iothread lock. We should checkpoint the failover status after every
>> important steps. I'll add more check in next version ...
>>
>>
>>>> /* Disable block migration */
>>>> s->params.blk = 0;
>>>> @@ -221,6 +270,11 @@ static void colo_process_checkpoint(MigrationState *s)
>>>> trace_colo_vm_state_change("stop", "run");
>>>>
>>>> while (s->state == MIGRATION_STATUS_COLO) {
>>>> + if (failover_request_is_active()) {
>>>> + error_report("failover request");
>>>> + goto out;
>>>> + }
>>>> +
>>>> current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>>>> if (current_time - checkpoint_time <
>>>> s->parameters[MIGRATION_PARAMETER_CHECKPOINT_DELAY]) {
>>>> @@ -242,8 +296,6 @@ out:
>>>> if (ret < 0) {
>>>> error_report("%s: %s", __func__, strerror(-ret));
>>>> }
>>>> - migrate_set_state(&s->state, MIGRATION_STATUS_COLO,
>>>> - MIGRATION_STATUS_COMPLETED);
>>>
>>> I had to think about this; but yes, I guess the only way out is via the failover
>>> which does the completed above.
>>>
>>
>> Yes, thanks.
>>
>> Hailiang
>>
>>> Dave
>>>
>>>> qsb_free(buffer);
>>>> buffer = NULL;
>>>> --
>>>> 1.8.3.1
>>>>
>>>>
>>> --
>>> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>>>
>>> .
>>>
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
* Re: [Qemu-devel] [PATCH COLO-Frame v11 25/39] COLO: implement default failover treatment
2015-12-10 19:01 ` Dr. David Alan Gilbert
@ 2015-12-11 9:48 ` Hailiang Zhang
0 siblings, 0 replies; 95+ messages in thread
From: Hailiang Zhang @ 2015-12-11 9:48 UTC (permalink / raw)
To: Dr. David Alan Gilbert
Cc: lizhijian, quintela, yunhong.jiang, eddie.dong, peter.huangpeng,
qemu-devel, arei.gonglei, stefanha, amit.shah, hongyang.yang
On 2015/12/11 3:01, Dr. David Alan Gilbert wrote:
> * zhanghailiang (zhang.zhanghailiang@huawei.com) wrote:
>> If we detect some error in colo, we will wait for some time,
>> hoping users also detect it. If users don't issue failover command.
>> We will go into default failover procedure, which the PVM will takeover
>> work while SVM is exit in default.
>
> I'm not sure this is needed; especially on the SVM. I don't see any harm
> in the SVM waiting forever to be told what to do - it could be told to
> failover or quit; I don't see any benefit to it automatically exiting.
>
> In the primary, I can see if you didn't have some automated error
> detection system then I can understand it (but I think it's rare);
> but you really would want to make that failover delay configurable
> so that you could turn it off in a system that did have failure detection;
> because automatically restarting the primary after it had caused a failover
> to the secondary would be very bad.
Yes, automatically restarting the PVM may cause split-brain. I'll drop
this patch temporarily.
Thanks.
Hailiang
>>
>> Signed-off-by: zhanghailiang <zhang.zhanghailiang@huawei.com>
>> Signed-off-by: Li Zhijian <lizhijian@cn.fujitsu.com>
>> ---
>> migration/colo.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++
>> 1 file changed, 46 insertions(+)
>>
>> diff --git a/migration/colo.c b/migration/colo.c
>> index f31e957..1e6d3dd 100644
>> --- a/migration/colo.c
>> +++ b/migration/colo.c
>> @@ -19,6 +19,14 @@
>> #include "qemu/sockets.h"
>> #include "migration/failover.h"
>>
>> +/*
>> + * The delay time before qemu begin the procedure of default failover treatment.
>> + * Unit: ms
>> + * Fix me: This value should be able to change by command
>> + * 'migrate-set-parameters'
>> + */
>> +#define DEFAULT_FAILOVER_DELAY 2000
>> +
>> /* colo buffer */
>> #define COLO_BUFFER_BASE_SIZE (4 * 1024 * 1024)
>>
>> @@ -264,6 +272,7 @@ static void colo_process_checkpoint(MigrationState *s)
>> {
>> QEMUSizedBuffer *buffer = NULL;
>> int64_t current_time, checkpoint_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>> + int64_t error_time;
>> int ret = 0;
>> uint64_t value;
>>
>> @@ -322,8 +331,25 @@ static void colo_process_checkpoint(MigrationState *s)
>> }
>>
>> out:
>> + current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>> if (ret < 0) {
>> error_report("%s: %s", __func__, strerror(-ret));
>> + /* Give users time to get involved in this verdict */
>> + while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
>> + if (failover_request_is_active()) {
>> + error_report("Primary VM will take over work");
>> + break;
>> + }
>> + usleep(100 * 1000);
>> + current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>> + }
>> +
>> + qemu_mutex_lock_iothread();
>> + if (!failover_request_is_active()) {
>> + error_report("Primary VM will take over work in default");
>> + failover_request_active(NULL);
>> + }
>> + qemu_mutex_unlock_iothread();
>> }
>>
>> qsb_free(buffer);
>> @@ -384,6 +410,7 @@ void *colo_process_incoming_thread(void *opaque)
>> QEMUFile *fb = NULL;
>> QEMUSizedBuffer *buffer = NULL; /* Cache incoming device state */
>> uint64_t total_size;
>> + int64_t error_time, current_time;
>> int ret = 0;
>> uint64_t value;
>>
>> @@ -499,9 +526,28 @@ void *colo_process_incoming_thread(void *opaque)
>> }
>>
>> out:
>> + current_time = error_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>> if (ret < 0) {
>> error_report("colo incoming thread will exit, detect error: %s",
>> strerror(-ret));
>> + /* Give users time to get involved in this verdict */
>> + while (current_time - error_time <= DEFAULT_FAILOVER_DELAY) {
>> + if (failover_request_is_active()) {
>> + error_report("Secondary VM will take over work");
>> + break;
>> + }
>> + usleep(100 * 1000);
>> + current_time = qemu_clock_get_ms(QEMU_CLOCK_HOST);
>> + }
>> + /* check flag again*/
>> + if (!failover_request_is_active()) {
>> + /*
>> + * We assume that Primary VM is still alive according to
>> + * heartbeat, just kill Secondary VM
>> + */
>> + error_report("SVM is going to exit in default!");
>> + exit(1);
>> + }
>> }
>>
>> if (fb) {
>> --
>> 1.8.3.1
>>
>>
> --
> Dr. David Alan Gilbert / dgilbert@redhat.com / Manchester, UK
>
> .
>
^ permalink raw reply [flat|nested] 95+ messages in thread
end of thread, other threads:[~2015-12-11 9:49 UTC | newest]
Thread overview: 95+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2015-11-24 9:25 [Qemu-devel] [PATCH COLO-Frame v11 00/39] COarse-grain LOck-stepping(COLO) Virtual Machines for Non-stop Service (FT) zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 01/39] configure: Add parameter for configure to enable/disable COLO support zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 02/39] migration: Introduce capability 'x-colo' to migration zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 03/39] COLO: migrate colo related info to secondary node zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 04/39] migration: Export migrate_set_state() zhanghailiang
2015-11-24 17:31 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 05/39] migration: Add state records for migration incoming zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 06/39] migration: Integrate COLO checkpoint process into migration zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 07/39] migration: Integrate COLO checkpoint process into loadvm zhanghailiang
2015-11-24 18:14 ` Dr. David Alan Gilbert
2015-11-25 6:39 ` zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 08/39] migration: Rename the'file' member of MigrationState zhanghailiang
2015-11-24 18:26 ` Dr. David Alan Gilbert
2015-11-25 6:48 ` zhanghailiang
2015-12-10 6:41 ` Wen Congyang
2015-12-11 3:40 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 09/39] COLO/migration: Create a new communication path from destination to source zhanghailiang
2015-11-24 18:40 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 10/39] COLO: Implement colo checkpoint protocol zhanghailiang
2015-11-24 19:00 ` Dr. David Alan Gilbert
2015-11-25 14:01 ` Eric Blake
2015-11-26 6:52 ` Hailiang Zhang
2015-11-26 7:12 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 11/39] COLO: Add a new RunState RUN_STATE_COLO zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 12/39] QEMUSizedBuffer: Introduce two help functions for qsb zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 13/39] COLO: Save PVM state to secondary side when do checkpoint zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 14/39] ram: Split host_from_stream_offset() into two helper functions zhanghailiang
2015-12-01 18:19 ` Dr. David Alan Gilbert
2015-12-03 7:19 ` Hailiang Zhang
2015-12-03 7:29 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 15/39] COLO: Load PVM's dirty pages into SVM's RAM cache temporarily zhanghailiang
2015-12-01 19:02 ` Dr. David Alan Gilbert
2015-12-03 8:25 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 16/39] ram/COLO: Record the dirty pages that SVM received zhanghailiang
2015-12-01 19:36 ` Dr. David Alan Gilbert
2015-12-03 8:29 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 17/39] COLO: Load VMState into qsb before restore it zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 18/39] COLO: Flush PVM's cached RAM into SVM's memory zhanghailiang
2015-11-27 5:29 ` Li Zhijian
2015-12-01 12:02 ` Hailiang Zhang
2015-12-01 20:06 ` Dr. David Alan Gilbert
2015-12-03 8:50 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 19/39] COLO: Add checkpoint-delay parameter for migrate-set-parameters zhanghailiang
2015-12-09 18:50 ` Dr. David Alan Gilbert
2015-12-11 3:20 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 20/39] COLO: synchronize PVM's state to SVM periodically zhanghailiang
2015-12-09 18:53 ` Dr. David Alan Gilbert
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 21/39] COLO failover: Introduce a new command to trigger a failover zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 22/39] COLO failover: Introduce state to record failover process zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 23/39] COLO: Implement failover work for Primary VM zhanghailiang
2015-12-10 18:34 ` Dr. David Alan Gilbert
2015-12-11 7:54 ` Hailiang Zhang
2015-12-11 9:22 ` Dr. David Alan Gilbert
2015-12-11 9:38 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 24/39] COLO: Implement failover work for Secondary VM zhanghailiang
2015-12-10 18:50 ` Dr. David Alan Gilbert
2015-12-11 8:27 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 25/39] COLO: implement default failover treatment zhanghailiang
2015-12-10 19:01 ` Dr. David Alan Gilbert
2015-12-11 9:48 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 26/39] qmp event: Add event notification for COLO error zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 27/39] COLO failover: Shutdown related socket fd when do failover zhanghailiang
2015-12-10 20:03 ` Dr. David Alan Gilbert
2015-12-11 8:57 ` Hailiang Zhang
2015-12-11 9:18 ` Dr. David Alan Gilbert
2015-12-11 9:29 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 28/39] COLO failover: Don't do failover during loading VM's state zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 29/39] COLO: Process shutdown command for VM in COLO state zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 30/39] COLO: Update the global runstate after going into colo state zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 31/39] savevm: Split load vm state function qemu_loadvm_state zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 32/39] COLO: Separate the process of saving/loading ram and device state zhanghailiang
2015-11-27 5:10 ` Li Zhijian
2015-12-01 12:07 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 33/39] COLO: Split qemu_savevm_state_begin out of checkpoint process zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 34/39] net/filter-buffer: Add default filter-buffer for each netdev zhanghailiang
2015-11-27 11:39 ` Yang Hongyang
2015-11-28 5:55 ` Hailiang Zhang
2015-11-30 1:19 ` Li Zhijian
2015-12-01 8:56 ` Hailiang Zhang
2015-12-03 1:17 ` Wen Congyang
2015-12-03 3:53 ` Hailiang Zhang
2015-12-03 6:25 ` Wen Congyang
2015-12-03 6:48 ` Hailiang Zhang
2015-12-03 7:21 ` Yang Hongyang
2015-12-03 8:37 ` Hailiang Zhang
2015-12-07 7:38 ` Hailiang Zhang
2015-12-08 1:49 ` Yang Hongyang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 35/39] filter-buffer: Accept zero interval zhanghailiang
2015-11-27 11:42 ` Yang Hongyang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 36/39] filter-buffer: Introduce a helper function to enable/disable default filter zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 37/39] filter-buffer: Introduce a helper function to release packets zhanghailiang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 38/39] colo: Use default buffer-filter to buffer and " zhanghailiang
2015-11-27 12:51 ` Yang Hongyang
2015-11-28 6:15 ` Hailiang Zhang
2015-11-24 9:25 ` [Qemu-devel] [PATCH COLO-Frame v11 39/39] COLO: Add block replication into colo process zhanghailiang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.