From mboxrd@z Thu Jan 1 00:00:00 1970 From: Wen Congyang Subject: [RFC Patch v3 00/22] COarse-grain LOck-stepping Virtual Machines for Non-stop Service Date: Fri, 5 Sep 2014 17:25:35 +0800 Message-ID: <1409909158-19243-1-git-send-email-wency@cn.fujitsu.com> Mime-Version: 1.0 Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: 7bit Return-path: List-Unsubscribe: , List-Post: List-Help: List-Subscribe: , Sender: xen-devel-bounces@lists.xen.org Errors-To: xen-devel-bounces@lists.xen.org To: xen devel Cc: Ian Campbell , Wen Congyang , Ian Jackson , Jiang Yunhong , Dong Eddie , Yang Hongyang , Lai Jiangshan List-Id: xen-devel@lists.xenproject.org Virtual machine (VM) replication is a well known technique for providing application-agnostic software-implemented hardware fault tolerance - "non-stop service". Currently, remus provides this function, but it buffers all output packets, and the latency is unacceptable. In xen summit 2012, We introduce a new VM replication solution: colo (COarse-grain LOck-stepping virtual machine). The presentation is in the following URL: http://www.slideshare.net/xen_com_mgr/colo-coarsegrain-lockstepping-virtual-machines-for-nonstop-service Here is the summary of the solution: >>From the client's point of view, as long as the client observes identical responses from the primary and secondary VMs, according to the service semantics, then the secondary vm is a valid replica of the primary vm, and can successfully take over when a hardware failure of the primary vm is detected. This patchset is RFC, and implements the framework and disk replication of COLO: 1. Both primary vm and secondary vm are running 2. do checkoint 3. disk replication(use blktap2) 3. nic replication(use colo-agent) This patchset is based on remus-v19, bugfix, and colo-prepare patchset, and use migration v1. Only supports hvm guest now. The codes are also hosted on github: https://github.com/wencongyang/xen/tree/colo-v3 TODO list: 1. Use migration v2 to implement COLO 2. support pvm Known bugs: 1. qemu may segment fault 2. secondary vm is running, but don't response. Usage: 1. update the vm's configfile: disk: disk = [ 'format=colo,devtype=disk,access=w,vdev=hda,backendtype=tap,target=192.168.3.1:9000|aio:/root/images/hvm/hvm_nopv/hvm.img' ] nic: vif = [ 'mac=00:16:4f:00:00:11, bridge=br0, model=e1000, forwarddev=eth0' ] Note: the forwarddev of primary and secondary host should be connected directly, and no other app uses it. If you don't have such nic, you can use vlan to make it. 2. build colo-agent: You can get colo-agent from github: https://github.com/wencongyang/colo-agent 3. run: xl remus -c Patch 1-4 : update remus to reuse remus device codes Patch 5-12 : COLO framework related codes Patch 13-16: move some block-remus's codes to block-replication.c. These codes will be reused by COLO. Patch 17 : implement block-colo Patch 18 : update libxl to support blktap2 Patch 19 : implement disk replication Patch 20-22: implement nic replication Patch 23 : A patch for qemu-xen Changlog from v2 to v3: 1. rebase to newest remus 2. add nic replication support Changlog from v1 to v2: 1. rebase to newest remus 2. add disk replication support Wen Congyang (22): move remus related codes to libxl_remus.c rename remus device to checkpoint device adjust the indentation don't touch remus in checkpoint_device Update libxl_save_msgs_gen.pl to support return data from xl to xc Allow slave sends data to master secondary vm suspend/resume/checkpoint code primary vm suspend/get_dirty_pfn/resume/checkpoint code xc_domain_save: flush cache before calling callbacks->postcopy() in colo mode COLO: xc related codes send store mfn and console mfn to xl before resuming secondary vm implement the cmdline for COLO blktap2: connect to backup asynchronously switch to unprotected mode before closing blktap2: move async connect related codes to block-replication.c blktap2: move ramdisk related codes to block-replication.c block-colo: implement colo disk replication support blktap COLO in xl: libxl/colo: setup and control disk replication for blktap2 backends setup and control colo-agent for primary vm setup and control colo-agent for secondary vm colo: cmdline switches and config vars to control colo-agent docs/man/xl.conf.pod.5 | 6 + docs/man/xl.pod.1 | 11 +- tools/blktap2/drivers/Makefile | 5 +- tools/blktap2/drivers/block-colo.c | 1151 +++++++++++++++++++ tools/blktap2/drivers/block-remus.c | 1174 +++++--------------- tools/blktap2/drivers/block-replication.c | 1116 +++++++++++++++++++ tools/blktap2/drivers/block-replication.h | 217 ++++ tools/blktap2/drivers/tapdisk-control.c | 6 + tools/blktap2/drivers/tapdisk-disktype.c | 9 + tools/blktap2/drivers/tapdisk-disktype.h | 3 +- tools/blktap2/drivers/tapdisk-interface.c | 18 + tools/blktap2/drivers/tapdisk-interface.h | 1 + tools/blktap2/drivers/tapdisk-vbd.c | 9 + tools/blktap2/drivers/tapdisk-vbd.h | 1 + tools/blktap2/drivers/tapdisk.h | 1 + tools/hotplug/Linux/Makefile | 2 + tools/hotplug/Linux/colo-agent-setup | 198 ++++ tools/hotplug/Linux/remus-netbuf-setup | 45 +- tools/hotplug/Linux/xen-network-ft.sh | 102 ++ tools/libxc/xc_domain_restore.c | 44 +- tools/libxc/xc_domain_save.c | 66 +- tools/libxc/xenguest.h | 40 + tools/libxl/Makefile | 11 +- tools/libxl/colo-tc.c | 589 ++++++++++ tools/libxl/libxl.c | 78 +- tools/libxl/libxl.h | 3 +- tools/libxl/libxl_blktap2.c | 6 +- ...xl_remus_device.c => libxl_checkpoint_device.c} | 229 ++-- tools/libxl/libxl_colo.h | 48 + tools/libxl/libxl_colo_nic.c | 309 ++++++ tools/libxl/libxl_colo_restore.c | 990 +++++++++++++++++ tools/libxl/libxl_colo_save.c | 809 ++++++++++++++ tools/libxl/libxl_colo_save_disk_blktap2.c | 214 ++++ tools/libxl/libxl_create.c | 151 ++- tools/libxl/libxl_device.c | 4 +- tools/libxl/libxl_dm.c | 3 +- tools/libxl/libxl_dom.c | 234 +--- tools/libxl/libxl_internal.h | 218 ++-- tools/libxl/libxl_netbuffer.c | 127 ++- tools/libxl/libxl_noblktap2.c | 29 + tools/libxl/libxl_nonetbuffer.c | 14 +- tools/libxl/libxl_remus.c | 377 +++++++ tools/libxl/libxl_remus.h | 27 + tools/libxl/libxl_remus_disk_drbd.c | 67 +- tools/libxl/libxl_save_callout.c | 37 +- tools/libxl/libxl_save_helper.c | 17 + tools/libxl/libxl_save_msgs_gen.pl | 74 +- tools/libxl/libxl_types.idl | 15 +- tools/libxl/libxl_types_internal.idl | 4 +- tools/libxl/libxl_utils.c | 2 + tools/libxl/libxlu_disk_l.l | 1 + tools/libxl/xl.c | 3 + tools/libxl/xl.h | 1 + tools/libxl/xl_cmdimpl.c | 95 +- tools/libxl/xl_cmdtable.c | 3 +- 55 files changed, 7433 insertions(+), 1581 deletions(-) create mode 100644 tools/blktap2/drivers/block-colo.c create mode 100644 tools/blktap2/drivers/block-replication.c create mode 100644 tools/blktap2/drivers/block-replication.h create mode 100755 tools/hotplug/Linux/colo-agent-setup create mode 100644 tools/hotplug/Linux/xen-network-ft.sh create mode 100644 tools/libxl/colo-tc.c rename tools/libxl/{libxl_remus_device.c => libxl_checkpoint_device.c} (40%) create mode 100644 tools/libxl/libxl_colo.h create mode 100644 tools/libxl/libxl_colo_nic.c create mode 100644 tools/libxl/libxl_colo_restore.c create mode 100644 tools/libxl/libxl_colo_save.c create mode 100644 tools/libxl/libxl_colo_save_disk_blktap2.c create mode 100644 tools/libxl/libxl_remus.c create mode 100644 tools/libxl/libxl_remus.h -- 1.9.3