From mboxrd@z Thu Jan 1 00:00:00 1970 Received: from eggs.gnu.org ([2001:4830:134:3::10]:37552) by lists.gnu.org with esmtp (Exim 4.71) (envelope-from ) id 1c0nem-0005mw-Cq for qemu-devel@nongnu.org; Sun, 30 Oct 2016 06:48:07 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from ) id 1c0nek-0001Kd-PH for qemu-devel@nongnu.org; Sun, 30 Oct 2016 06:48:04 -0400 Received: from mx1.redhat.com ([209.132.183.28]:43282) by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32) (Exim 4.71) (envelope-from ) id 1c0nek-0001KW-Hv for qemu-devel@nongnu.org; Sun, 30 Oct 2016 06:48:02 -0400 From: Amit Shah Date: Sun, 30 Oct 2016 16:17:08 +0530 Message-Id: <1477824430-1460-17-git-send-email-amit.shah@redhat.com> In-Reply-To: <1477824430-1460-1-git-send-email-amit.shah@redhat.com> References: <1477824430-1460-1-git-send-email-amit.shah@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable Subject: [Qemu-devel] [PULL 16/18] docs: Add documentation for COLO feature List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , To: Peter Maydell Cc: Juan Quintela , "Dr. David Alan Gilbert" , qemu list , zhang.zhanghailiang@huawei.com From: zhanghailiang Introduce the design of COLO, and how to test it. Signed-off-by: zhanghailiang Reviewed-by: Amit Shah Signed-off-by: Amit Shah --- docs/COLO-FT.txt | 189 +++++++++++++++++++++++++++++++++++++++++++++++++= ++++++ 1 file changed, 189 insertions(+) create mode 100644 docs/COLO-FT.txt diff --git a/docs/COLO-FT.txt b/docs/COLO-FT.txt new file mode 100644 index 0000000..6282938 --- /dev/null +++ b/docs/COLO-FT.txt @@ -0,0 +1,189 @@ +COarse-grained LOck-stepping Virtual Machines for Non-stop Service +---------------------------------------- +Copyright (c) 2016 Intel Corporation +Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. +Copyright (c) 2016 Fujitsu, Corp. + +This work is licensed under the terms of the GNU GPL, version 2 or later= . +See the COPYING file in the top-level directory. + +This document gives an overview of COLO's design and how to use it. + +=3D=3D Background =3D=3D +Virtual machine (VM) replication is a well known technique for providing +application-agnostic software-implemented hardware fault tolerance, +also known as "non-stop service". + +COLO (COarse-grained LOck-stepping) is a high availability solution. +Both primary VM (PVM) and secondary VM (SVM) run in parallel. They recei= ve the +same request from client, and generate response in parallel too. +If the response packets from PVM and SVM are identical, they are release= d +immediately. Otherwise, a VM checkpoint (on demand) is conducted. + +=3D=3D Architecture =3D=3D + +The architecture of COLO is shown in the diagram below. +It consists of a pair of networked physical nodes: +The primary node running the PVM, and the secondary node running the SVM +to maintain a valid replica of the PVM. +PVM and SVM execute in parallel and generate output of response packets = for +client requests according to the application semantics. + +The incoming packets from the client or external network are received by= the +primary node, and then forwarded to the secondary node, so that both the= PVM +and the SVM are stimulated with the same requests. + +COLO receives the outbound packets from both the PVM and SVM and compare= s them +before allowing the output to be sent to clients. + +The SVM is qualified as a valid replica of the PVM, as long as it genera= tes +identical responses to all client requests. Once the differences in the = outputs +are detected between the PVM and SVM, COLO withholds transmission of the +outbound packets until it has successfully synchronized the PVM state to= the SVM. + + Primary Node = Secondary Node + +------------+ +-----------------------+ +----------------------= --+ +------------+ + | | | HeartBeat |<----->| HeartBeat = | | | + | Primary VM | +-----------|-----------+ +-----------|----------= --+ |Secondary VM| + | | | | = | | + | | +-----------|-----------+ +-----------|----------= --+ | | + | | |QEMU +---v----+ | |QEMU +----v---+ = | | | + | | | |Failover| | | |Failover| = | | | + | | | +--------+ | | +--------+ = | | | + | | | +---------------+ | | +---------------+ = | | | + | | | | VM Checkpoint |-------------->| VM Checkpoint | = | | | + | | | +---------------+ | | +---------------+ = | | | + | | | | | = | | | + |Requests<---------------------------^---------------------------------= --------->Requests| + |Responses----------------------\ /--|--------------\ /---------------= ---------Responses| + | | | | | | | | | | = | | | + | | | +-----------+ | | | | | | | +------------= + | | | + | | | | COLO disk | | | | | | | | | COLO disk = | | | | + | | | | Manager |-|-|--|--------------|--|->| Manager = | | | | + | | | +|----------+ | | | | | | | +-----------|= + | | | + | | | | | | | | | | | |= | | | + +------------+ +--|------------|-|--|--+ +---|--|--------------|= --+ +------------+ + | | | | | | | + +-------------+ | +----------v-v--|--+ +---|--v-----------+ |= +-------------+ + | VM Monitor | | | COLO Proxy | | COLO Proxy | |= | VM Monitor | + | | | |(compare packet) | | (adjust sequence)| |= | | + +-------------+ | +----------|----^--+ +------------------+ |= +-------------+ + | | | | + +------------------|------------|----|--+ +---------------------|= ------------------+ + | Kernel | | | | | Kernel |= | + +------------------|------------|----|--+ +---------------------|= ------------------+ + | | | | + +--------------v+ +--------v----|--+ +------------------+ +v= -------------+ + | Storage | |External Network| | External Network | | = Storage | + +---------------+ +----------------+ +------------------+ +-= -------------+ + +=3D=3D Components introduction =3D=3D + +You can see there are several components in COLO's diagram of architectu= re. +Their functions are described below. + +HeartBeat: +Runs on both the primary and secondary nodes, to periodically check plat= form +availability. When the primary node suffers a hardware fail-stop failure= , +the heartbeat stops responding, the secondary node will trigger a failov= er +as soon as it determines the absence. + +COLO disk Manager: +When primary VM writes data into image, the colo disk manger captures th= is data +and sends it to secondary VM's which makes sure the context of secondary= VM's +image is consistent with the context of primary VM 's image. +For more details, please refer to docs/block-replication.txt. + +Checkpoint/Failover Controller: +Modifications of save/restore flow to realize continuous migration, +to make sure the state of VM in Secondary side is always consistent with= VM in +Primary side. + +COLO Proxy: +Delivers packets to Primary and Seconday, and then compare the responses= from +both side. Then decide whether to start a checkpoint according to some r= ules. +Please refer to docs/colo-proxy.txt for more informations. + +Note: +HeartBeat has not been implemented yet, so you need to trigger failover = process +by using 'x-colo-lost-heartbeat' command. + +=3D=3D Test procedure =3D=3D +1. Startup qemu +Primary: +# qemu-kvm -enable-kvm -m 2048 -smp 2 -qmp stdio -vnc :7 -name primary \ + -device piix3-usb-uhci \ + -device usb-tablet -netdev tap,id=3Dhn0,vhost=3Doff \ + -device virtio-net-pci,id=3Dnet-pci0,netdev=3Dhn0 \ + -drive if=3Dvirtio,id=3Dprimary-disk0,driver=3Dquorum,read-pattern=3Df= ifo,vote-threshold=3D1,\ + children.0.file.filename=3D1.raw,\ + children.0.driver=3Draw -S +Secondary: +# qemu-kvm -enable-kvm -m 2048 -smp 2 -qmp stdio -vnc :7 -name secondary= \ + -device piix3-usb-uhci \ + -device usb-tablet -netdev tap,id=3Dhn0,vhost=3Doff \ + -device virtio-net-pci,id=3Dnet-pci0,netdev=3Dhn0 \ + -drive if=3Dnone,id=3Dsecondary-disk0,file.filename=3D1.raw,driver=3Dr= aw,node-name=3Dnode0 \ + -drive if=3Dvirtio,id=3Dactive-disk0,driver=3Dreplication,mode=3Dsecon= dary,\ + file.driver=3Dqcow2,top-id=3Dactive-disk0,\ + file.file.filename=3D/mnt/ramfs/active_disk.img,\ + file.backing.driver=3Dqcow2,\ + file.backing.file.filename=3D/mnt/ramfs/hidden_disk.img,\ + file.backing.backing=3Dsecondary-disk0 \ + -incoming tcp:0:8888 + +2. On Secondary VM's QEMU monitor, issue command +{'execute':'qmp_capabilities'} +{ 'execute': 'nbd-server-start', + 'arguments': {'addr': {'type': 'inet', 'data': {'host': 'xx.xx.xx.xx',= 'port': '8889'} } } +} +{'execute': 'nbd-server-add', 'arguments': {'device': 'secondeary-disk0'= , 'writable': true } } + +Note: + a. The qmp command nbd-server-start and nbd-server-add must be run + before running the qmp command migrate on primary QEMU + b. Active disk, hidden disk and nbd target's length should be the + same. + c. It is better to put active disk and hidden disk in ramdisk. + +3. On Primary VM's QEMU monitor, issue command: +{'execute':'qmp_capabilities'} +{ 'execute': 'human-monitor-command', + 'arguments': {'command-line': 'drive_add -n buddy driver=3Dreplication= ,mode=3Dprimary,file.driver=3Dnbd,file.host=3Dxx.xx.xx.xx,file.port=3D888= 9,file.export=3Dsecondary-disk0,node-name=3Dnbd_client0'}} +{ 'execute':'x-blockdev-change', 'arguments':{'parent': 'primary-disk0',= 'node': 'nbd_client0' } } +{ 'execute': 'migrate-set-capabilities', + 'arguments': {'capabilities': [ {'capability': 'x-colo', 'state': = true } ] } } +{ 'execute': 'migrate', 'arguments': {'uri': 'tcp:xx.xx.xx.xx:8888' } } + + Note: + a. There should be only one NBD Client for each primary disk. + b. xx.xx.xx.xx is the secondary physical machine's hostname or IP + c. The qmp command line must be run after running qmp command line in + secondary qemu. + +4. After the above steps, you will see, whenever you make changes to PVM= , SVM will be synced. +You can issue command '{ "execute": "migrate-set-parameters" , "argument= s":{ "x-checkpoint-delay": 2000 } }' +to change the checkpoint period time + +5. Failover test +You can kill Primary VM and run 'x_colo_lost_heartbeat' in Secondary VM'= s +monitor at the same time, then SVM will failover and client will not det= ect this +change. + +Before issuing '{ "execute": "x-colo-lost-heartbeat" }' command, we have= to +issue block related command to stop block replication. +Primary: + Remove the nbd child from the quorum: + { 'execute': 'x-blockdev-change', 'arguments': {'parent': 'colo-disk0'= , 'child': 'children.1'}} + { 'execute': 'human-monitor-command','arguments': {'command-line': 'dr= ive_del blk-buddy0'}} + Note: there is no qmp command to remove the blockdev now + +Secondary: + The primary host is down, so we should do the following thing: + { 'execute': 'nbd-server-stop' } + +=3D=3D TODO =3D=3D +1. Support continuous VM replication. +2. Support shared storage. +3. Develop the heartbeat part. +4. Reduce checkpoint VM=E2=80=99s downtime while doing checkpoint. --=20 2.7.4