* [PATCH 0/5] colo: Introduce resource agent and test suite/CI
@ 2020-05-11 12:26 Lukas Straub
2020-05-11 12:26 ` [PATCH 1/5] block/quorum.c: stable children names Lukas Straub
` (5 more replies)
0 siblings, 6 replies; 14+ messages in thread
From: Lukas Straub @ 2020-05-11 12:26 UTC (permalink / raw)
To: qemu-devel; +Cc: Zhang Chen, Alberto Garcia, Dr. David Alan Gilbert
[-- Attachment #1: Type: text/plain, Size: 2246 bytes --]
Hello Everyone,
These patches introduce a resource agent for fully automatic management of colo
and a test suite building upon the resource agent to extensively test colo.
Test suite features:
-Tests failover with peer crashing and hanging and failover during checkpoint
-Tests network using ssh and iperf3
-Quick test requires no special configuration
-Network test for testing colo-compare
-Stress test: failover all the time with network load
Resource agent features:
-Fully automatic management of colo
-Handles many failures: hanging/crashing qemu, replication error, disk error, ...
-Recovers from hanging qemu by using the "yank" oob command
-Tracks which node has up-to-date data
-Works well in clusters with more than 2 nodes
Run times on my laptop:
Quick test: 200s
Network test: 800s (tagged as slow)
Stress test: 1300s (tagged as slow)
The test suite needs access to a network bridge to properly test the network,
so some parameters need to be given to the test run. See
tests/acceptance/colo.py for more information.
I wonder how this integrates in existing CI infrastructure. Is there a common
CI for qemu where this can run or does every subsystem have to run their own
CI?
Regards,
Lukas Straub
Lukas Straub (5):
block/quorum.c: stable children names
colo: Introduce resource agent
colo: Introduce high-level test suite
configure,Makefile: Install colo resource-agent
MAINTAINERS: Add myself as maintainer for COLO resource agent
MAINTAINERS | 6 +
Makefile | 5 +
block/quorum.c | 20 +-
configure | 10 +
scripts/colo-resource-agent/colo | 1429 ++++++++++++++++++++++
scripts/colo-resource-agent/crm_master | 44 +
scripts/colo-resource-agent/crm_resource | 12 +
tests/acceptance/colo.py | 689 +++++++++++
8 files changed, 2209 insertions(+), 6 deletions(-)
create mode 100755 scripts/colo-resource-agent/colo
create mode 100755 scripts/colo-resource-agent/crm_master
create mode 100755 scripts/colo-resource-agent/crm_resource
create mode 100644 tests/acceptance/colo.py
--
2.20.1
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* [PATCH 1/5] block/quorum.c: stable children names
2020-05-11 12:26 [PATCH 0/5] colo: Introduce resource agent and test suite/CI Lukas Straub
@ 2020-05-11 12:26 ` Lukas Straub
2020-06-02 1:01 ` Zhang, Chen
2020-06-02 11:07 ` Alberto Garcia
2020-05-11 12:26 ` [PATCH 2/5] colo: Introduce resource agent Lukas Straub
` (4 subsequent siblings)
5 siblings, 2 replies; 14+ messages in thread
From: Lukas Straub @ 2020-05-11 12:26 UTC (permalink / raw)
To: qemu-devel; +Cc: Zhang Chen, Alberto Garcia, Dr. David Alan Gilbert
[-- Attachment #1: Type: text/plain, Size: 2784 bytes --]
If we remove the child with the highest index from the quorum,
decrement s->next_child_index. This way we get stable children
names as long as we only remove the last child.
Signed-off-by: Lukas Straub <lukasstraub2@web.de>
---
block/quorum.c | 20 ++++++++++++++------
1 file changed, 14 insertions(+), 6 deletions(-)
diff --git a/block/quorum.c b/block/quorum.c
index 6d7a56bd93..acfa09c2cc 100644
--- a/block/quorum.c
+++ b/block/quorum.c
@@ -29,6 +29,8 @@
#define HASH_LENGTH 32
+#define INDEXSTR_LEN 32
+
#define QUORUM_OPT_VOTE_THRESHOLD "vote-threshold"
#define QUORUM_OPT_BLKVERIFY "blkverify"
#define QUORUM_OPT_REWRITE "rewrite-corrupted"
@@ -972,9 +974,9 @@ static int quorum_open(BlockDriverState *bs, QDict *options, int flags,
opened = g_new0(bool, s->num_children);
for (i = 0; i < s->num_children; i++) {
- char indexstr[32];
- ret = snprintf(indexstr, 32, "children.%d", i);
- assert(ret < 32);
+ char indexstr[INDEXSTR_LEN];
+ ret = snprintf(indexstr, INDEXSTR_LEN, "children.%d", i);
+ assert(ret < INDEXSTR_LEN);
s->children[i] = bdrv_open_child(NULL, options, indexstr, bs,
&child_format, false, &local_err);
@@ -1026,7 +1028,7 @@ static void quorum_add_child(BlockDriverState *bs, BlockDriverState *child_bs,
{
BDRVQuorumState *s = bs->opaque;
BdrvChild *child;
- char indexstr[32];
+ char indexstr[INDEXSTR_LEN];
int ret;
if (s->is_blkverify) {
@@ -1041,8 +1043,8 @@ static void quorum_add_child(BlockDriverState *bs, BlockDriverState *child_bs,
return;
}
- ret = snprintf(indexstr, 32, "children.%u", s->next_child_index);
- if (ret < 0 || ret >= 32) {
+ ret = snprintf(indexstr, INDEXSTR_LEN, "children.%u", s->next_child_index);
+ if (ret < 0 || ret >= INDEXSTR_LEN) {
error_setg(errp, "cannot generate child name");
return;
}
@@ -1069,6 +1071,7 @@ static void quorum_del_child(BlockDriverState *bs, BdrvChild *child,
Error **errp)
{
BDRVQuorumState *s = bs->opaque;
+ char indexstr[INDEXSTR_LEN];
int i;
for (i = 0; i < s->num_children; i++) {
@@ -1090,6 +1093,11 @@ static void quorum_del_child(BlockDriverState *bs, BdrvChild *child,
/* We know now that num_children > threshold, so blkverify must be false */
assert(!s->is_blkverify);
+ snprintf(indexstr, INDEXSTR_LEN, "children.%u", s->next_child_index - 1);
+ if (!strncmp(child->name, indexstr, INDEXSTR_LEN)) {
+ s->next_child_index--;
+ }
+
bdrv_drained_begin(bs);
/* We can safely remove this child now */
--
2.20.1
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 2/5] colo: Introduce resource agent
2020-05-11 12:26 [PATCH 0/5] colo: Introduce resource agent and test suite/CI Lukas Straub
2020-05-11 12:26 ` [PATCH 1/5] block/quorum.c: stable children names Lukas Straub
@ 2020-05-11 12:26 ` Lukas Straub
2020-05-11 12:27 ` [PATCH 3/5] colo: Introduce high-level test suite Lukas Straub
` (3 subsequent siblings)
5 siblings, 0 replies; 14+ messages in thread
From: Lukas Straub @ 2020-05-11 12:26 UTC (permalink / raw)
To: qemu-devel; +Cc: Zhang Chen, Alberto Garcia, Dr. David Alan Gilbert
[-- Attachment #1: Type: text/plain, Size: 59893 bytes --]
Introduce a resource agent which can be used to manage qemu COLO
in a pacemaker cluster.
Signed-off-by: Lukas Straub <lukasstraub2@web.de>
---
scripts/colo-resource-agent/colo | 1429 ++++++++++++++++++++++++++++++
1 file changed, 1429 insertions(+)
create mode 100755 scripts/colo-resource-agent/colo
diff --git a/scripts/colo-resource-agent/colo b/scripts/colo-resource-agent/colo
new file mode 100755
index 0000000000..fbc5dc2c13
--- /dev/null
+++ b/scripts/colo-resource-agent/colo
@@ -0,0 +1,1429 @@
+#!/usr/bin/env python3
+
+# Resource agent for qemu COLO for use with Pacemaker CRM
+#
+# Copyright (c) Lukas Straub <lukasstraub2@web.de>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+from __future__ import print_function
+import subprocess
+import sys
+import os
+import os.path
+import signal
+import socket
+import select
+import json
+import re
+import time
+import logging
+import logging.handlers
+
+# Constants
+OCF_SUCCESS = 0
+OCF_ERR_GENERIC = 1
+OCF_ERR_ARGS = 2
+OCF_ERR_UNIMPLEMENTED = 3
+OCF_ERR_PERM = 4
+OCF_ERR_INSTALLED = 5
+OCF_ERR_CONFIGURED = 6
+OCF_NOT_RUNNING = 7
+OCF_RUNNING_MASTER = 8
+OCF_FAILED_MASTER = 9
+
+# Get environment variables
+OCF_RESKEY_CRM_meta_notify_type \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify_type")
+OCF_RESKEY_CRM_meta_notify_operation \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify_operation")
+OCF_RESKEY_CRM_meta_notify_key_operation \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify_key_operation")
+OCF_RESKEY_CRM_meta_notify_start_uname \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify_start_uname", "")
+OCF_RESKEY_CRM_meta_notify_stop_uname \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify_stop_uname", "")
+OCF_RESKEY_CRM_meta_notify_active_uname \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify_active_uname", "")
+OCF_RESKEY_CRM_meta_notify_promote_uname \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify_promote_uname", "")
+OCF_RESKEY_CRM_meta_notify_demote_uname \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify_demote_uname", "")
+OCF_RESKEY_CRM_meta_notify_master_uname \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify_master_uname", "")
+OCF_RESKEY_CRM_meta_notify_slave_uname \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify_slave_uname", "")
+
+HA_RSCTMP = os.getenv("HA_RSCTMP", "/run/resource-agents")
+HA_LOGFACILITY = os.getenv("HA_LOGFACILITY")
+HA_LOGFILE = os.getenv("HA_LOGFILE")
+HA_DEBUG = os.getenv("HA_debug", "0")
+HA_DEBUGLOG = os.getenv("HA_DEBUGLOG")
+OCF_RESOURCE_INSTANCE = os.getenv("OCF_RESOURCE_INSTANCE", "default-instance")
+OCF_RESKEY_CRM_meta_timeout \
+ = os.getenv("OCF_RESKEY_CRM_meta_timeout", "60000")
+OCF_RESKEY_CRM_meta_interval \
+ = int(os.getenv("OCF_RESKEY_CRM_meta_interval", "1"))
+OCF_RESKEY_CRM_meta_clone_max \
+ = int(os.getenv("OCF_RESKEY_CRM_meta_clone_max", "1"))
+OCF_RESKEY_CRM_meta_clone_node_max \
+ = int(os.getenv("OCF_RESKEY_CRM_meta_clone_node_max", "1"))
+OCF_RESKEY_CRM_meta_master_max \
+ = int(os.getenv("OCF_RESKEY_CRM_meta_master_max", "1"))
+OCF_RESKEY_CRM_meta_master_node_max \
+ = int(os.getenv("OCF_RESKEY_CRM_meta_master_node_max", "1"))
+OCF_RESKEY_CRM_meta_notify \
+ = os.getenv("OCF_RESKEY_CRM_meta_notify")
+OCF_RESKEY_CRM_meta_globally_unique \
+ = os.getenv("OCF_RESKEY_CRM_meta_globally_unique")
+
+HOSTNAME = os.getenv("OCF_RESKEY_CRM_meta_on_node", socket.gethostname())
+
+OCF_ACTION = os.getenv("__OCF_ACTION")
+if not OCF_ACTION and len(sys.argv) == 2:
+ OCF_ACTION = sys.argv[1]
+
+# Resource parameters
+OCF_RESKEY_qemu_binary_default = "qemu-system-x86_64"
+OCF_RESKEY_qemu_img_binary_default = "qemu-img"
+OCF_RESKEY_log_dir_default = HA_RSCTMP
+OCF_RESKEY_options_default = ""
+OCF_RESKEY_disk_size_default = ""
+OCF_RESKEY_active_hidden_dir_default = ""
+OCF_RESKEY_listen_address_default = "0.0.0.0"
+OCF_RESKEY_base_port_default = "9000"
+OCF_RESKEY_checkpoint_interval_default = "20000"
+OCF_RESKEY_compare_timeout_default = "3000"
+OCF_RESKEY_expired_scan_cycle_default = "3000"
+OCF_RESKEY_use_filter_rewriter_default = "true"
+OCF_RESKEY_vnet_hdr_default = "false"
+OCF_RESKEY_max_disk_errors_default = "1"
+OCF_RESKEY_monitor_timeout_default = "20000"
+OCF_RESKEY_yank_timeout_default = "10000"
+OCF_RESKEY_fail_fast_timeout_default = "5000"
+OCF_RESKEY_debug_default = "0"
+
+OCF_RESKEY_qemu_binary \
+ = os.getenv("OCF_RESKEY_qemu_binary", OCF_RESKEY_qemu_binary_default)
+OCF_RESKEY_qemu_img_binary \
+ = os.getenv("OCF_RESKEY_qemu_img_binary", OCF_RESKEY_qemu_img_binary_default)
+OCF_RESKEY_log_dir \
+ = os.getenv("OCF_RESKEY_log_dir", OCF_RESKEY_log_dir_default)
+OCF_RESKEY_options \
+ = os.getenv("OCF_RESKEY_options", OCF_RESKEY_options_default)
+OCF_RESKEY_disk_size \
+ = os.getenv("OCF_RESKEY_disk_size", OCF_RESKEY_disk_size_default)
+OCF_RESKEY_active_hidden_dir \
+ = os.getenv("OCF_RESKEY_active_hidden_dir", OCF_RESKEY_active_hidden_dir_default)
+OCF_RESKEY_listen_address \
+ = os.getenv("OCF_RESKEY_listen_address", OCF_RESKEY_listen_address_default)
+OCF_RESKEY_base_port \
+ = os.getenv("OCF_RESKEY_base_port", OCF_RESKEY_base_port_default)
+OCF_RESKEY_checkpoint_interval \
+ = os.getenv("OCF_RESKEY_checkpoint_interval", OCF_RESKEY_checkpoint_interval_default)
+OCF_RESKEY_compare_timeout \
+ = os.getenv("OCF_RESKEY_compare_timeout", OCF_RESKEY_compare_timeout_default)
+OCF_RESKEY_expired_scan_cycle \
+ = os.getenv("OCF_RESKEY_expired_scan_cycle", OCF_RESKEY_expired_scan_cycle_default)
+OCF_RESKEY_use_filter_rewriter \
+ = os.getenv("OCF_RESKEY_use_filter_rewriter", OCF_RESKEY_use_filter_rewriter_default)
+OCF_RESKEY_vnet_hdr \
+ = os.getenv("OCF_RESKEY_vnet_hdr", OCF_RESKEY_vnet_hdr_default)
+OCF_RESKEY_max_disk_errors \
+ = os.getenv("OCF_RESKEY_max_disk_errors", OCF_RESKEY_max_disk_errors_default)
+OCF_RESKEY_monitor_timeout \
+ = os.getenv("OCF_RESKEY_monitor_timeout", OCF_RESKEY_monitor_timeout_default)
+OCF_RESKEY_yank_timeout \
+ = os.getenv("OCF_RESKEY_yank_timeout", OCF_RESKEY_yank_timeout_default)
+OCF_RESKEY_fail_fast_timeout \
+ = os.getenv("OCF_RESKEY_fail_fast_timeout", OCF_RESKEY_fail_fast_timeout_default)
+OCF_RESKEY_debug \
+ = os.getenv("OCF_RESKEY_debug", OCF_RESKEY_debug_default)
+
+ACTIVE_IMAGE = os.path.join(OCF_RESKEY_active_hidden_dir, \
+ OCF_RESOURCE_INSTANCE + "-active.qcow2")
+HIDDEN_IMAGE = os.path.join(OCF_RESKEY_active_hidden_dir, \
+ OCF_RESOURCE_INSTANCE + "-hidden.qcow2")
+
+QMP_SOCK = os.path.join(HA_RSCTMP, OCF_RESOURCE_INSTANCE + "-qmp.sock")
+HELPER_SOCK = os.path.join(HA_RSCTMP, OCF_RESOURCE_INSTANCE + "-helper.sock")
+COMP_SOCK = os.path.join(HA_RSCTMP, OCF_RESOURCE_INSTANCE + "-compare.sock")
+COMP_OUT_SOCK = os.path.join(HA_RSCTMP, OCF_RESOURCE_INSTANCE \
+ + "-comp_out.sock")
+
+PID_FILE = os.path.join(HA_RSCTMP, OCF_RESOURCE_INSTANCE + "-qemu.pid")
+
+QMP_LOG = os.path.join(OCF_RESKEY_log_dir, OCF_RESOURCE_INSTANCE + "-qmp.log")
+QEMU_LOG = os.path.join(OCF_RESKEY_log_dir, OCF_RESOURCE_INSTANCE + "-qemu.log")
+HELPER_LOG = os.path.join(OCF_RESKEY_log_dir, OCF_RESOURCE_INSTANCE \
+ + "-helper.log")
+
+START_TIME = time.time()
+did_yank = False
+
+# Exception only raised by ourself
+class Error(Exception):
+ pass
+
+def setup_constants():
+ # This function is called after the parameters where validated
+ global OCF_RESKEY_CRM_meta_timeout
+ if OCF_ACTION == "monitor":
+ OCF_RESKEY_CRM_meta_timeout = OCF_RESKEY_monitor_timeout
+
+ global MIGRATE_PORT, MIRROR_PORT, COMPARE_IN_PORT, NBD_PORT
+ MIGRATE_PORT = int(OCF_RESKEY_base_port)
+ MIRROR_PORT = int(OCF_RESKEY_base_port) + 1
+ COMPARE_IN_PORT = int(OCF_RESKEY_base_port) + 2
+ NBD_PORT = int(OCF_RESKEY_base_port) + 3
+
+ global QEMU_PRIMARY_CMDLINE
+ QEMU_PRIMARY_CMDLINE = \
+ ("'%(OCF_RESKEY_qemu_binary)s' %(OCF_RESKEY_options)s"
+ " -drive if=none,node-name=colo-disk0,driver=quorum,read-pattern=fifo,"
+ "vote-threshold=1,children.0=parent0"
+ " -qmp unix:'%(QMP_SOCK)s',server,nowait"
+ " -daemonize -D '%(QEMU_LOG)s' -pidfile '%(PID_FILE)s'") % globals()
+
+ global QEMU_SECONDARY_CMDLINE
+ QEMU_SECONDARY_CMDLINE = \
+ ("'%(OCF_RESKEY_qemu_binary)s' %(OCF_RESKEY_options)s"
+ " -chardev socket,id=red0,host='%(OCF_RESKEY_listen_address)s',"
+ "port=%(MIRROR_PORT)s,server,nowait,nodelay,yank"
+ " -chardev socket,id=red1,host='%(OCF_RESKEY_listen_address)s',"
+ "port=%(COMPARE_IN_PORT)s,server,nowait,nodelay,yank"
+ " -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0"
+ " -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1") \
+ % globals()
+
+ if is_true(OCF_RESKEY_use_filter_rewriter):
+ QEMU_SECONDARY_CMDLINE += \
+ " -object filter-rewriter,id=rew0,netdev=hn0,queue=all"
+
+ QEMU_SECONDARY_CMDLINE += \
+ (" -drive if=none,node-name=childs0,top-id=colo-disk0,"
+ "driver=replication,mode=secondary,file.driver=qcow2,"
+ "file.file.filename='%(ACTIVE_IMAGE)s',file.backing.driver=qcow2,"
+ "file.backing.file.filename='%(HIDDEN_IMAGE)s',"
+ "file.backing.backing=parent0"
+ " -drive if=none,node-name=colo-disk0,driver=quorum,read-pattern=fifo,"
+ "vote-threshold=1,children.0=childs0"
+ " -incoming tcp:'%(OCF_RESKEY_listen_address)s':%(MIGRATE_PORT)s"
+ " -global migration.yank=true"
+ " -qmp unix:'%(QMP_SOCK)s',server,nowait"
+ " -daemonize -D '%(QEMU_LOG)s' -pidfile '%(PID_FILE)s'") % globals()
+
+def qemu_colo_meta_data():
+ print("""\
+<?xml version="1.0"?>
+<!DOCTYPE resource-agent SYSTEM "ra-api-1.dtd">
+<resource-agent name="colo">
+
+ <version>1.0</version>
+ <longdesc lang="en">
+Resource agent for qemu COLO. (https://wiki.qemu.org/Features/COLO)
+
+After defining the master/slave instance, the master score has to be
+manually set to show which node has up-to-date data. So you copy your
+image to one host (and create empty images the other host(s)) and then
+run "crm_master -r name_of_your_primitive -v 10" on that host.
+Also, you have to set 'notify=true' in the metadata attributes when
+defining the master/slave instance.
+
+Note:
+-If the instance is stopped cluster-wide, the resource agent will do a
+clean shutdown. Set the demote timeout to the time it takes for your
+guest to shutdown.
+-Colo replication is started from the monitor action. Set the monitor
+timeout to at least the time it takes for replication to start. You can
+set the monitor_timeout parameter for a soft timeout, which the resource
+agent tries to satisfy.
+-The resource agent may notify pacemaker about peer failure,
+these failures will show up with exitreason="Simulated failure".
+ </longdesc>
+ <shortdesc lang="en">Qemu COLO</shortdesc>
+
+ <parameters>
+
+ <parameter name="qemu_binary" unique="0" required="0">
+ <longdesc lang="en">qemu binary to use</longdesc>
+ <shortdesc lang="en">qemu binary</shortdesc>
+ <content type="string" default=\"""" \
+ + OCF_RESKEY_qemu_binary_default + """\"/>
+ </parameter>
+
+ <parameter name="qemu_img_binary" unique="0" required="0">
+ <longdesc lang="en">qemu-img binary to use</longdesc>
+ <shortdesc lang="en">qemu-img binary</shortdesc>
+ <content type="string" default=\"""" \
+ + OCF_RESKEY_qemu_img_binary_default + """\"/>
+ </parameter>
+
+ <parameter name="log_dir" unique="0" required="0">
+ <longdesc lang="en">Directory to place logs in</longdesc>
+ <shortdesc lang="en">Log directory</shortdesc>
+ <content type="string" default=\"""" \
+ + OCF_RESKEY_log_dir_default + """\"/>
+ </parameter>
+
+ <parameter name="options" unique="0" required="1">
+ <longdesc lang="en">
+Options to pass to qemu. These will be passed alongside COLO specific
+options, so you need to follow these conventions: The netdev should have
+id=hn0 and the disk controller drive=colo-disk0. The image node should
+have id/node-name=parent0, but should not be connected to the guest.
+Example:
+-vnc :0 -enable-kvm -cpu qemu64,+kvmclock -m 512 -netdev bridge,id=hn0
+-device e1000,netdev=hn0 -device virtio-blk,drive=colo-disk0
+-drive if=none,id=parent0,format=qcow2,file=/mnt/vms/vm01.qcow2
+ </longdesc>
+ <shortdesc lang="en">Options to pass to qemu.</shortdesc>
+ </parameter>
+
+ <parameter name="disk_size" unique="0" required="1">
+ <longdesc lang="en">Disk size of the image</longdesc>
+ <shortdesc lang="en">Disk size of the image</shortdesc>
+ <content type="string" default=\"""" \
+ + OCF_RESKEY_disk_size_default + """\"/>
+ </parameter>
+
+ <parameter name="active_hidden_dir" unique="0" required="1">
+ <longdesc lang="en">
+Directory where the active and hidden images will be stored. It is
+recommended to put this on a ramdisk.
+ </longdesc>
+ <shortdesc lang="en">Path to active and hidden images</shortdesc>
+ <content type="string" default=\"""" \
+ + OCF_RESKEY_active_hidden_dir_default + """\"/>
+ </parameter>
+
+ <parameter name="listen_address" unique="0" required="0">
+ <longdesc lang="en">Address to listen on.</longdesc>
+ <shortdesc lang="en">Listen address</shortdesc>
+ <content type="string" default=\"""" \
+ + OCF_RESKEY_listen_address_default + """\"/>
+ </parameter>
+
+ <parameter name="base_port" unique="1" required="0">
+ <longdesc lang="en">
+4 tcp ports that are unique for each instance. (base_port to base_port + 3)
+ </longdesc>
+ <shortdesc lang="en">Ports to use</shortdesc>
+ <content type="integer" default=\"""" \
+ + OCF_RESKEY_base_port_default + """\"/>
+ </parameter>
+
+ <parameter name="checkpoint_interval" unique="0" required="0">
+ <longdesc lang="en">
+Interval for regular checkpoints in milliseconds.
+ </longdesc>
+ <shortdesc lang="en">Interval for regular checkpoints</shortdesc>
+ <content type="integer" default=\"""" \
+ + OCF_RESKEY_checkpoint_interval_default + """\"/>
+ </parameter>
+
+ <parameter name="compare_timeout" unique="0" required="0">
+ <longdesc lang="en">
+Maximum time to hold a primary packet if secondary hasn't sent it yet,
+in milliseconds.
+You should also adjust "expired_scan_cycle" accordingly.
+ </longdesc>
+ <shortdesc lang="en">Compare timeout</shortdesc>
+ <content type="integer" default=\"""" \
+ + OCF_RESKEY_compare_timeout_default + """\"/>
+ </parameter>
+
+ <parameter name="expired_scan_cycle" unique="0" required="0">
+ <longdesc lang="en">
+Interval for checking for expired primary packets in milliseconds.
+ </longdesc>
+ <shortdesc lang="en">Expired packet check interval</shortdesc>
+ <content type="integer" default=\"""" \
+ + OCF_RESKEY_expired_scan_cycle_default + """\"/>
+ </parameter>
+
+ <parameter name="use_filter_rewriter" unique="0" required="0">
+ <longdesc lang="en">
+Use filter-rewriter to increase similarity between the VMs.
+ </longdesc>
+ <shortdesc lang="en">Use filter-rewriter</shortdesc>
+ <content type="string" default=\"""" \
+ + OCF_RESKEY_use_filter_rewriter_default + """\"/>
+ </parameter>
+
+ <parameter name="vnet_hdr" unique="0" required="0">
+ <longdesc lang="en">
+Set this to true if your system supports vnet_hdr and you enabled
+it on the tap netdev.
+ </longdesc>
+ <shortdesc lang="en">vnet_hdr support</shortdesc>
+ <content type="string" default=\"""" \
+ + OCF_RESKEY_vnet_hdr_default + """\"/>
+ </parameter>
+
+ <parameter name="max_disk_errors" unique="0" required="0">
+ <longdesc lang="en">
+Maximum disk read errors per monitor interval before marking the resource
+as failed. A write error is always fatal except if the value is 0.
+A value of 0 will disable disk error handling.
+Primary disk errors are only handled if there is a healthy secondary.
+ </longdesc>
+ <shortdesc lang="en">Maximum disk errors</shortdesc>
+ <content type="integer" default=\"""" \
+ + OCF_RESKEY_max_disk_errors_default + """\"/>
+ </parameter>
+
+ <parameter name="monitor_timeout" unique="0" required="0">
+ <longdesc lang="en">
+Soft timeout for monitor, in milliseconds.
+Must be lower than the monitor action timeout.
+ </longdesc>
+ <shortdesc lang="en">Monitor timeout</shortdesc>
+ <content type="integer" default=\"""" \
+ + OCF_RESKEY_monitor_timeout_default + """\"/>
+ </parameter>
+
+ <parameter name="yank_timeout" unique="0" required="0">
+ <longdesc lang="en">
+Timeout for QMP commands after which to execute the "yank" command,
+in milliseconds.
+Must be lower than any of the action timeouts.
+ </longdesc>
+ <shortdesc lang="en">Yank timeout</shortdesc>
+ <content type="integer" default=\"""" \
+ + OCF_RESKEY_yank_timeout_default + """\"/>
+ </parameter>
+
+ <parameter name="fail_fast_timeout" unique="0" required="0">
+ <longdesc lang="en">
+Timeout for QMP commands used in the stop and demote actions to speed
+up recovery from a hanging qemu, in milliseconds.
+Must be lower than any of the action timeouts.
+ </longdesc>
+ <shortdesc lang="en">Timeout for fast paths</shortdesc>
+ <content type="integer" default=\"""" \
+ + OCF_RESKEY_fail_fast_timeout_default + """\"/>
+ </parameter>
+
+ <parameter name="debug" unique="0" required="0">
+ <longdesc lang="en">
+Control debugging:
+0: disable debugging
+1: log debug messages and qmp commands
+2: + dump core of hanging qemu
+ </longdesc>
+ <shortdesc lang="en">Control debugging</shortdesc>
+ <content type="integer" default=\"""" \
+ + OCF_RESKEY_debug_default + """\"/>
+ </parameter>
+
+ </parameters>
+
+ <actions>
+ <action name="start" timeout="30s" />
+ <action name="stop" timeout="10s" />
+ <action name="monitor" timeout="30s" \
+ interval="1000ms" depth="0" role="Slave" />
+ <action name="monitor" timeout="30s" \
+ interval="1001ms" depth="0" role="Master" />
+ <action name="notify" timeout="30s" />
+ <action name="promote" timeout="30s" />
+ <action name="demote" timeout="120s" />
+ <action name="meta-data" timeout="5s" />
+ <action name="validate-all" timeout="20s" />
+ </actions>
+
+</resource-agent>
+""")
+
+def logs_open():
+ global log
+ log = logging.getLogger(OCF_RESOURCE_INSTANCE)
+ if int(OCF_RESKEY_debug) >= 1 or HA_DEBUG != "0":
+ log.setLevel(logging.DEBUG)
+ else:
+ log.setLevel(logging.INFO)
+
+ formater = logging.Formatter("(%(name)s) %(levelname)s: %(message)s")
+
+ if sys.stdout.isatty():
+ handler = logging.StreamHandler(stream=sys.stderr)
+ handler.setFormatter(formater)
+ log.addHandler(handler)
+
+ if HA_LOGFACILITY:
+ handler = logging.handlers.SysLogHandler("/dev/log")
+ handler.setFormatter(formater)
+ log.addHandler(handler)
+
+ if HA_LOGFILE:
+ handler = logging.FileHandler(HA_LOGFILE)
+ handler.setFormatter(formater)
+ log.addHandler(handler)
+
+ if HA_DEBUGLOG and HA_DEBUGLOG != HA_LOGFILE:
+ handler = logging.FileHandler(HA_DEBUGLOG)
+ handler.setFormatter(formater)
+ log.addHandler(handler)
+
+ global qmp_log
+ qmp_log = logging.getLogger("qmp_log")
+ qmp_log.setLevel(logging.DEBUG)
+ formater = logging.Formatter("%(message)s")
+
+ if int(OCF_RESKEY_debug) >= 1:
+ handler = logging.handlers.WatchedFileHandler(QMP_LOG)
+ handler.setFormatter(formater)
+ qmp_log.addHandler(handler)
+ else:
+ handler = logging.NullHandler()
+ qmp_log.addHandler(handler)
+
+def rotate_logfile(logfile, numlogs):
+ numlogs -= 1
+ for n in range(numlogs, -1, -1):
+ file = logfile
+ if n != 0:
+ file = "%s.%s" % (file, n)
+ if os.path.exists(file):
+ if n == numlogs:
+ os.remove(file)
+ else:
+ newname = "%s.%s" % (logfile, n + 1)
+ os.rename(file, newname)
+
+def is_writable(file):
+ return os.access(file, os.W_OK)
+
+def is_executable_file(file):
+ return os.path.isfile(file) and os.access(file, os.X_OK)
+
+def is_true(var):
+ return re.match("yes|true|1|YES|TRUE|True|ja|on|ON", str(var)) != None
+
+# Check if the binary exists and is executable
+def check_binary(binary):
+ if is_executable_file(binary):
+ return True
+ PATH = os.getenv("PATH", os.defpath)
+ for dir in PATH.split(os.pathsep):
+ if is_executable_file(os.path.join(dir, binary)):
+ return True
+ log.error("binary \"%s\" doesn't exist or not executable" % binary)
+ return False
+
+def run_command(commandline):
+ proc = subprocess.Popen(commandline, shell=True, stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT, universal_newlines=True)
+ stdout, stderr = proc.communicate()
+ if proc.returncode != 0:
+ log.error("command \"%s\" failed with code %s:\n%s" \
+ % (commandline, proc.returncode, stdout))
+ raise Error()
+
+# Functions for setting and getting the master score to tell Pacemaker which
+# host has the most recent data
+def set_master_score(score):
+ if score == 0:
+ run_command("crm_master -q -l forever -D")
+ else:
+ run_command("crm_master -q -l forever -v %s" % score)
+
+def set_remote_master_score(remote, score):
+ if score == 0:
+ run_command("crm_master -q -l forever -N '%s' -D" % remote)
+ else:
+ run_command("crm_master -q -l forever -N '%s' -v %s" % (remote, score))
+
+def get_master_score():
+ proc = subprocess.Popen("crm_master -q -G", shell=True,
+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
+ universal_newlines=True)
+ stdout, stderr = proc.communicate()
+ if proc.returncode != 0:
+ return 0
+ else:
+ return int(str.strip(stdout))
+
+def get_remote_master_score(remote):
+ proc = subprocess.Popen("crm_master -q -N '%s' -G" % remote, shell=True,
+ stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
+ universal_newlines=True)
+ stdout, stderr = proc.communicate()
+ if proc.returncode != 0:
+ return 0
+ else:
+ return int(str.strip(stdout))
+
+# Tell Pacemaker that the remote resource failed
+def report_remote_failure(remote):
+ run_command("crm_resource --resource '%s' --fail --node '%s'"
+ % (OCF_RESOURCE_INSTANCE, remote))
+
+def recv_line(fd):
+ line = ""
+ while True:
+ tmp = fd.recv(1).decode()
+ line += tmp
+ if tmp == "\n" or len(tmp) == 0:
+ break
+ return line
+
+# Filter out events
+def read_answer(fd):
+ while True:
+ line = recv_line(fd)
+ qmp_log.debug(str.strip(line))
+
+ if len(line) == 0:
+ log.error("qmp connection closed")
+ raise Error()
+
+ answer = json.loads(line)
+ # Ignore everything else
+ if "return" in answer or "error" in answer:
+ break
+ return answer
+
+# Execute one or more qmp commands
+def qmp_execute(fd, commands, ignore_error = False, do_yank = True):
+ for command in commands:
+ if not command:
+ continue
+
+ try:
+ to_send = json.dumps(command)
+ fd.sendall(str.encode(to_send + "\n"))
+ qmp_log.debug(to_send)
+
+ answer = read_answer(fd)
+ except Exception as e:
+ if isinstance(e, socket.timeout) and do_yank:
+ log.warning("Command timed out, trying to unfreeze qemu")
+ new_timeout = max(2, (int(OCF_RESKEY_CRM_meta_timeout)/1000) \
+ - (time.time() - START_TIME) - 2)
+ fd.settimeout(new_timeout)
+ answer = yank(fd)
+ # Read answer of timed-out command
+ try:
+ if "id" in answer:
+ answer = read_answer(fd)
+ else:
+ read_answer(fd)
+ except socket.error as e:
+ log.error("while reading answer of timed out command: "
+ "%s\n%s" % (json.dumps(command), e))
+ raise Error()
+ elif isinstance(e, socket.timeout) or isinstance(e, socket.error):
+ log.error("while executing qmp command: %s\n%s" \
+ % (json.dumps(command), e))
+ raise Error()
+ else:
+ raise
+
+ if not ignore_error and ("error" in answer):
+ log.error("qmp command returned error:\n%s\n%s" \
+ % (json.dumps(command), json.dumps(answer)))
+ raise Error()
+
+ return answer
+
+# Open qemu qmp connection
+def qmp_open(fail_fast = False):
+ if fail_fast:
+ timeout = int(OCF_RESKEY_fail_fast_timeout)/1000
+ else:
+ timeout = int(OCF_RESKEY_yank_timeout)/1000
+
+ try:
+ fd = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ fd.settimeout(timeout)
+ fd.connect(HELPER_SOCK)
+ except socket.error as e:
+ log.error("while connecting to helper socket: %s" % e)
+ raise Error()
+
+ return fd
+
+def yank(fd):
+ global did_yank
+ did_yank = True
+ answer = qmp_execute(fd, [{"exec-oob": "yank", "id": "yank0"}], \
+ do_yank = False, ignore_error = True)
+ return answer
+
+def oob_helper_exec(client, cmd, events):
+ if cmd["exec-helper"] == "get-events":
+ event = cmd["arguments"]["event"]
+ if (event in events):
+ to_send = json.dumps({"return": events[event]})
+ client.sendall(str.encode(to_send + "\n"))
+ else:
+ client.sendall(str.encode("{\"return\": []}\n"))
+ elif cmd["exec-helper"] == "clear-events":
+ events.clear()
+ client.sendall(str.encode("{\"return\": {}}\n"))
+ else:
+ client.sendall(str.encode("{\"error\": \"Unknown helper command\"}\n"))
+
+def oob_helper(qmp, server):
+ max_events = max(100, int(OCF_RESKEY_max_disk_errors))
+ events = {}
+ try:
+ os.close(0)
+ os.close(1)
+ os.close(2)
+ logging.shutdown()
+
+ client = None
+ while True:
+ if client:
+ watch = [client, qmp]
+ else:
+ watch = [server, qmp]
+ sel = select.select(watch, [], [])
+ try:
+ if client in sel[0]:
+ cmd = recv_line(client)
+ if len(cmd) == 0:
+ # client socket was closed: wait for new client
+ client.close()
+ client = None
+ continue
+ else:
+ parsed = json.loads(cmd)
+ if ("exec-helper" in parsed):
+ oob_helper_exec(client, parsed, events)
+ else:
+ qmp.sendall(str.encode(cmd))
+ if qmp in sel[0]:
+ answer = recv_line(qmp)
+ if len(answer) == 0:
+ # qmp socket was closed: qemu died, exit
+ os._exit(0)
+ else:
+ parsed = json.loads(answer)
+ if ("event" in parsed):
+ event = parsed["event"]
+ if (event not in events):
+ events[event] = []
+ if len(events[event]) < max_events:
+ events[event].append(parsed)
+ elif client:
+ client.sendall(str.encode(answer))
+ if server in sel[0]:
+ client, client_addr = server.accept()
+ except socket.error as e:
+ pass
+ except Exception as e:
+ with open(HELPER_LOG, 'a') as f:
+ f.write(str(e) + "\n")
+ os._exit(0)
+
+# Fork off helper to keep the oob qmp connection open and to catch events
+def oob_helper_open():
+ try:
+ qmp = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ qmp.connect(QMP_SOCK)
+ qmp_execute(qmp, [{"execute": "qmp_capabilities", "arguments": {"enable": ["oob"]}}])
+ except socket.error as e:
+ log.error("while connecting to qmp socket: %s" % e)
+ raise Error()
+
+ try:
+ if os.path.exists(HELPER_SOCK):
+ os.unlink(HELPER_SOCK)
+ server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
+ server.bind(HELPER_SOCK)
+ server.listen(1)
+ except socket.error as e:
+ log.error("while opening helper socket: %s" % e)
+ raise Error()
+
+ qmp.set_inheritable(True)
+ server.set_inheritable(True)
+
+ try:
+ pid = os.fork()
+ except OSError as e:
+ log.error("while forking off oob helper: %s" % e)
+ raise Error()
+
+ if pid == 0:
+ # child 1: Exits after forking off child 2, so pid 1 will become
+ # responsible for the child
+ os.setsid()
+
+ pid = os.fork()
+
+ if pid == 0:
+ # child 2: here the actual work is being done
+ oob_helper(qmp, server)
+ else:
+ os._exit(0)
+
+ qmp.close()
+ server.close()
+
+# Get the host of the nbd node
+def qmp_get_nbd_remote(fd):
+ block_nodes = qmp_execute(fd, [{"execute": "query-named-block-nodes", "arguments": {"flat": True}}])
+ for node in block_nodes["return"]:
+ if node["node-name"] == "nbd0":
+ url = str(node["image"]["filename"])
+ return str.split(url, "//")[1].split("/")[0].split(":")[0]
+ return None
+
+# Check if we are currently resyncing
+def qmp_check_resync(fd):
+ answer = qmp_execute(fd, [{"execute": "query-block-jobs"}])
+ for job in answer["return"]:
+ if job["device"] == "resync":
+ return job
+ return None
+
+def qmp_start_resync(fd, remote):
+ answer = qmp_execute(fd, [{"execute": "blockdev-add", "arguments": {"driver": "nbd", "node-name": "nbd0", "server": {"type": "inet", "host": str(remote), "port": str(NBD_PORT)}, "export": "parent0", "detect-zeroes": "on", "yank": True}}], ignore_error = True)
+ if "error" in answer:
+ log.warning("Failed to add nbd node: %s" % json.dumps(answer))
+ log.warning("Assuming peer failure")
+ report_remote_failure(remote)
+ else:
+ qmp_execute(fd, [{"execute": "blockdev-mirror", "arguments": {"device": "colo-disk0", "job-id": "resync", "target": "nbd0", "sync": "full", "on-target-error": "report", "on-source-error": "ignore", "auto-dismiss": False}}])
+
+def qmp_cancel_resync(fd):
+ timeout = START_TIME + (int(OCF_RESKEY_yank_timeout)/1000)
+
+ if qmp_check_resync(fd)["status"] != "concluded":
+ qmp_execute(fd, [{"execute": "block-job-cancel", "arguments": {"device": "resync", "force": True}}], ignore_error = True)
+ # Wait for the block-job to finish
+ while time.time() < timeout:
+ if qmp_check_resync(fd)["status"] == "concluded":
+ break
+ log.debug("Waiting for block-job to finish in qmp_cancel_resync()")
+ time.sleep(1)
+ else:
+ log.warning("Timed out, trying to unfreeze qemu")
+ yank(fd)
+ while qmp_check_resync(fd)["status"] != "concluded":
+ log.debug("Waiting for block-job to finish")
+ time.sleep(1)
+
+ qmp_execute(fd, [
+ {"execute": "block-job-dismiss", "arguments": {"id": "resync"}},
+ {"execute": "blockdev-del", "arguments": {"node-name": "nbd0"}}
+ ])
+
+def qmp_start_colo(fd, remote):
+ # Check if we have a filter-rewriter
+ answer = qmp_execute(fd, [{"execute": "qom-list", "arguments": {"path": "/objects/rew0"}}], ignore_error = True)
+ if "error" in answer:
+ if answer["error"]["class"] == "DeviceNotFound":
+ have_filter_rewriter = False
+ else:
+ log.error("while checking for filter-rewriter:\n%s" \
+ % json.dumps(answer))
+ raise Error()
+ else:
+ have_filter_rewriter = True
+
+ # Pause VM and cancel resync
+ qmp_execute(fd, [
+ {"execute": "stop"},
+ {"execute": "block-job-cancel", "arguments": {"device": "resync"}}
+ ])
+
+ # Wait for the block-job to finish
+ while qmp_check_resync(fd)["status"] != "concluded":
+ log.debug("Waiting for block-job to finish in qmp_start_colo()")
+ time.sleep(1)
+
+ # Add nbd to the quorum node
+ qmp_execute(fd, [
+ {"execute": "block-job-dismiss", "arguments": {"id": "resync"}},
+ {"execute": "x-blockdev-change", "arguments": {"parent": "colo-disk0", "node": "nbd0"}}
+ ])
+
+ # Connect mirror and compare_in to secondary
+ qmp_execute(fd, [
+ {"execute": "chardev-add", "arguments": {"id": "comp_pri_in0<", "backend": {"type": "socket", "data": {"addr": {"type": "unix", "data": {"path": str(COMP_SOCK)}}, "server": True}}}},
+ {"execute": "chardev-add", "arguments": {"id": "comp_pri_in0>", "backend": {"type": "socket", "data": {"addr": {"type": "unix", "data": {"path": str(COMP_SOCK)}}, "server": False}}}},
+ {"execute": "chardev-add", "arguments": {"id": "comp_out0<", "backend": {"type": "socket", "data": {"addr": {"type": "unix", "data": {"path": str(COMP_OUT_SOCK)}}, "server": True}}}},
+ {"execute": "chardev-add", "arguments": {"id": "comp_out0>", "backend": {"type": "socket", "data": {"addr": {"type": "unix", "data": {"path": str(COMP_OUT_SOCK)}}, "server": False}}}},
+ {"execute": "chardev-add", "arguments": {"id": "mirror0", "backend": {"type": "socket", "data": {"addr": {"type": "inet", "data": {"host": str(remote), "port": str(MIRROR_PORT)}}, "server": False, "nodelay": True, "yank": True}}}},
+ {"execute": "chardev-add", "arguments": {"id": "comp_sec_in0", "backend": {"type": "socket", "data": {"addr": {"type": "inet", "data": {"host": str(remote), "port": str(COMPARE_IN_PORT)}}, "server": False, "nodelay": True, "yank": True}}}}
+ ])
+
+ # Add the COLO filters
+ vnet_hdr_support = is_true(OCF_RESKEY_vnet_hdr)
+ if have_filter_rewriter:
+ qmp_execute(fd, [
+ {"execute": "object-add", "arguments": {"qom-type": "filter-mirror", "id": "m0", "props": {"insert": "before", "position": "id=rew0", "netdev": "hn0", "queue": "tx", "outdev": "mirror0", "vnet_hdr_support": vnet_hdr_support}}},
+ {"execute": "object-add", "arguments": {"qom-type": "filter-redirector", "id": "redire0", "props": {"insert": "before", "position": "id=rew0", "netdev": "hn0", "queue": "rx", "indev": "comp_out0<", "vnet_hdr_support": vnet_hdr_support}}},
+ {"execute": "object-add", "arguments": {"qom-type": "filter-redirector", "id": "redire1", "props": {"insert": "before", "position": "id=rew0", "netdev": "hn0", "queue": "rx", "outdev": "comp_pri_in0<", "vnet_hdr_support": vnet_hdr_support}}},
+ {"execute": "object-add", "arguments": {"qom-type": "iothread", "id": "iothread1"}},
+ {"execute": "object-add", "arguments": {"qom-type": "colo-compare", "id": "comp0", "props": {"primary_in": "comp_pri_in0>", "secondary_in": "comp_sec_in0", "outdev": "comp_out0>", "iothread": "iothread1", "compare_timeout": int(OCF_RESKEY_compare_timeout), "expired_scan_cycle": int(OCF_RESKEY_expired_scan_cycle), "vnet_hdr_support": vnet_hdr_support}}}
+ ])
+ else:
+ qmp_execute(fd, [
+ {"execute": "object-add", "arguments": {"qom-type": "filter-mirror", "id": "m0", "props": {"netdev": "hn0", "queue": "tx", "outdev": "mirror0", "vnet_hdr_support": vnet_hdr_support}}},
+ {"execute": "object-add", "arguments": {"qom-type": "filter-redirector", "id": "redire0", "props": {"netdev": "hn0", "queue": "rx", "indev": "comp_out0<", "vnet_hdr_support": vnet_hdr_support}}},
+ {"execute": "object-add", "arguments": {"qom-type": "filter-redirector", "id": "redire1", "props": {"netdev": "hn0", "queue": "rx", "outdev": "comp_pri_in0<", "vnet_hdr_support": vnet_hdr_support}}},
+ {"execute": "object-add", "arguments": {"qom-type": "iothread", "id": "iothread1"}},
+ {"execute": "object-add", "arguments": {"qom-type": "colo-compare", "id": "comp0", "props": {"primary_in": "comp_pri_in0>", "secondary_in": "comp_sec_in0", "outdev": "comp_out0>", "iothread": "iothread1", "compare_timeout": int(OCF_RESKEY_compare_timeout), "expired_scan_cycle": int(OCF_RESKEY_expired_scan_cycle), "vnet_hdr_support": vnet_hdr_support}}}
+ ])
+
+ # Start COLO
+ qmp_execute(fd, [
+ {"execute": "migrate-set-capabilities", "arguments": {"capabilities": [{"capability": "x-colo", "state": True }] }},
+ {"execute": "migrate-set-parameters" , "arguments": {"x-checkpoint-delay": int(OCF_RESKEY_checkpoint_interval), "yank": True}},
+ {"execute": "migrate", "arguments": {"uri": "tcp:%s:%s" % (remote, MIGRATE_PORT)}}
+ ])
+
+ # Wait for COLO to start
+ while qmp_execute(fd, [{"execute": "query-status"}])["return"]["status"] \
+ == "paused" \
+ or qmp_execute(fd, [{"execute": "query-colo-status"}])["return"]["mode"] \
+ != "primary" :
+ log.debug("Waiting for colo replication to start")
+ time.sleep(1)
+
+def qmp_primary_failover(fd):
+ qmp_execute(fd, [
+ {"execute": "object-del", "arguments": {"id": "m0"}},
+ {"execute": "object-del", "arguments": {"id": "redire0"}},
+ {"execute": "object-del", "arguments": {"id": "redire1"}},
+ {"execute": "x-colo-lost-heartbeat"},
+ {"execute": "object-del", "arguments": {"id": "comp0"}},
+ {"execute": "object-del", "arguments": {"id": "iothread1"}},
+ {"execute": "x-blockdev-change", "arguments": {"parent": "colo-disk0", "child": "children.1"}},
+ {"execute": "blockdev-del", "arguments": {"node-name": "nbd0"}},
+ {"execute": "chardev-remove", "arguments": {"id": "mirror0"}},
+ {"execute": "chardev-remove", "arguments": {"id": "comp_sec_in0"}},
+ {"execute": "chardev-remove", "arguments": {"id": "comp_pri_in0>"}},
+ {"execute": "chardev-remove", "arguments": {"id": "comp_pri_in0<"}},
+ {"execute": "chardev-remove", "arguments": {"id": "comp_out0>"}},
+ {"execute": "chardev-remove", "arguments": {"id": "comp_out0<"}}
+ ])
+
+def qmp_secondary_failover(fd):
+ qmp_execute(fd, [
+ {"execute": "nbd-server-stop"},
+ {"execute": "object-del", "arguments": {"id": "f2"}},
+ {"execute": "object-del", "arguments": {"id": "f1"}},
+ {"execute": "x-colo-lost-heartbeat"},
+ {"execute": "chardev-remove", "arguments": {"id": "red1"}},
+ {"execute": "chardev-remove", "arguments": {"id": "red0"}},
+ ])
+
+# Check qemu health and colo role
+def qmp_check_state(fd, fail_fast = False):
+ answer = qmp_execute(fd, [{"execute": "query-status"}], \
+ do_yank = not fail_fast)
+ vm_status = answer["return"]
+
+ answer = qmp_execute(fd, [{"execute": "query-colo-status"}], \
+ do_yank = not fail_fast)
+ colo_status = answer["return"]
+
+ if vm_status["status"] == "inmigrate":
+ role = OCF_SUCCESS
+ replication = OCF_NOT_RUNNING
+
+ elif (vm_status["status"] == "running" \
+ or vm_status["status"] == "colo" \
+ or vm_status["status"] == "finish-migrate") \
+ and colo_status["mode"] == "none" \
+ and (colo_status["reason"] == "request" \
+ or colo_status["reason"] == "none"):
+ role = OCF_RUNNING_MASTER
+ replication = OCF_NOT_RUNNING
+
+ elif (vm_status["status"] == "running" \
+ or vm_status["status"] == "colo" \
+ or vm_status["status"] == "finish-migrate") \
+ and colo_status["mode"] == "secondary":
+ role = OCF_SUCCESS
+ replication = OCF_SUCCESS
+
+ elif (vm_status["status"] == "running" \
+ or vm_status["status"] == "colo" \
+ or vm_status["status"] == "finish-migrate") \
+ and colo_status["mode"] == "primary":
+ role = OCF_RUNNING_MASTER
+ replication = OCF_SUCCESS
+
+ else:
+ log.error("Invalid qemu status:\nvm status: %s\ncolo status: %s" \
+ % (vm_status, colo_status))
+ role = OCF_ERR_GENERIC
+ replication = OCF_ERR_GENERIC
+
+ return role, replication
+
+# Sanity checks: check parameters, files, binaries, etc.
+def qemu_colo_validate_all():
+ # Check resource parameters
+ if not str.isdigit(OCF_RESKEY_base_port):
+ log.error("base_port needs to be a number")
+ return OCF_ERR_CONFIGURED
+
+ if not str.isdigit(OCF_RESKEY_checkpoint_interval):
+ log.error("checkpoint_interval needs to be a number")
+ return OCF_ERR_CONFIGURED
+
+ if not str.isdigit(OCF_RESKEY_compare_timeout):
+ log.error("compare_timeout needs to be a number")
+ return OCF_ERR_CONFIGURED
+
+ if not str.isdigit(OCF_RESKEY_expired_scan_cycle):
+ log.error("expired_scan_cycle needs to be a number")
+ return OCF_ERR_CONFIGURED
+
+ if not str.isdigit(OCF_RESKEY_max_disk_errors):
+ log.error("max_disk_errors needs to be a number")
+ return OCF_ERR_CONFIGURED
+
+ if not str.isdigit(OCF_RESKEY_monitor_timeout):
+ log.error("monitor_timeout needs to be a number")
+ return OCF_ERR_CONFIGURED
+
+ if not str.isdigit(OCF_RESKEY_yank_timeout):
+ log.error("yank_timeout needs to be a number")
+ return OCF_ERR_CONFIGURED
+
+ if not str.isdigit(OCF_RESKEY_fail_fast_timeout):
+ log.error("fail_fast_timeout needs to be a number")
+ return OCF_ERR_CONFIGURED
+
+ if not str.isdigit(OCF_RESKEY_debug):
+ log.error("debug needs to be a number")
+ return OCF_ERR_CONFIGURED
+
+ if not OCF_RESKEY_active_hidden_dir:
+ log.error("active_hidden_dir needs to be specified")
+ return OCF_ERR_CONFIGURED
+
+ if not OCF_RESKEY_disk_size:
+ log.error("disk_size needs to be specified")
+ return OCF_ERR_CONFIGURED
+
+ # Check resource meta configuration
+ if OCF_ACTION != "stop":
+ if OCF_RESKEY_CRM_meta_master_max != 1:
+ log.error("only one master allowed")
+ return OCF_ERR_CONFIGURED
+
+ if OCF_RESKEY_CRM_meta_clone_max > 2:
+ log.error("maximum 2 clones allowed")
+ return OCF_ERR_CONFIGURED
+
+ if OCF_RESKEY_CRM_meta_master_node_max != 1:
+ log.error("only one master per node allowed")
+ return OCF_ERR_CONFIGURED
+
+ if OCF_RESKEY_CRM_meta_clone_node_max != 1:
+ log.error("only one clone per node allowed")
+ return OCF_ERR_CONFIGURED
+
+ # Check if notify is enabled
+ if OCF_ACTION != "stop" and OCF_ACTION != "monitor":
+ if not is_true(OCF_RESKEY_CRM_meta_notify) \
+ and not OCF_RESKEY_CRM_meta_notify_start_uname:
+ log.error("notify needs to be enabled")
+ return OCF_ERR_CONFIGURED
+
+ # Check that globally-unique is disabled
+ if is_true(OCF_RESKEY_CRM_meta_globally_unique):
+ log.error("globally-unique needs to be disabled")
+ return OCF_ERR_CONFIGURED
+
+ # Check binaries
+ if not check_binary(OCF_RESKEY_qemu_binary):
+ return OCF_ERR_INSTALLED
+
+ if not check_binary(OCF_RESKEY_qemu_img_binary):
+ return OCF_ERR_INSTALLED
+
+ # Check paths and files
+ if not is_writable(OCF_RESKEY_active_hidden_dir) \
+ or not os.path.isdir(OCF_RESKEY_active_hidden_dir):
+ log.error("active and hidden image directory missing or not writable")
+ return OCF_ERR_PERM
+
+ return OCF_SUCCESS
+
+# Check if qemu is running
+def check_pid():
+ if not os.path.exists(PID_FILE):
+ return OCF_NOT_RUNNING, None
+
+ fd = open(PID_FILE, "r")
+ pid = int(str.strip(fd.readline()))
+ fd.close()
+ try:
+ os.kill(pid, 0)
+ except OSError:
+ log.info("qemu is not running")
+ return OCF_NOT_RUNNING, pid
+ else:
+ return OCF_SUCCESS, pid
+
+def qemu_colo_monitor(fail_fast = False):
+ status, pid = check_pid()
+ if status != OCF_SUCCESS:
+ return status, OCF_NOT_RUNNING
+
+ fd = qmp_open(fail_fast)
+
+ role, replication = qmp_check_state(fd, fail_fast)
+ if role != OCF_SUCCESS and role != OCF_RUNNING_MASTER:
+ return role, replication
+
+ colo_events = qmp_execute(fd, [{"exec-helper": "get-events", "arguments": {"event": "COLO_EXIT"}}], do_yank = False)
+ for event in colo_events["return"]:
+ if event["data"]["reason"] == "error":
+ if replication == OCF_SUCCESS:
+ replication = OCF_ERR_GENERIC
+
+ if did_yank and replication == OCF_SUCCESS:
+ replication = OCF_ERR_GENERIC
+
+ peer_disk_errors = 0
+ local_disk_errors = 0
+ quorum_events = qmp_execute(fd, [{"exec-helper": "get-events", "arguments": {"event": "QUORUM_REPORT_BAD"}}], do_yank = False)
+ for event in quorum_events["return"]:
+ if event["data"]["node-name"] == "nbd0":
+ if event["data"]["type"] == "read":
+ peer_disk_errors += 1
+ else:
+ peer_disk_errors += int(OCF_RESKEY_max_disk_errors)
+ else:
+ if event["data"]["type"] == "read":
+ local_disk_errors += 1
+ else:
+ local_disk_errors += int(OCF_RESKEY_max_disk_errors)
+
+ if int(OCF_RESKEY_max_disk_errors) != 0:
+ if peer_disk_errors >= int(OCF_RESKEY_max_disk_errors):
+ log.error("Peer disk error")
+ if replication == OCF_SUCCESS:
+ replication = OCF_ERR_GENERIC
+
+ if local_disk_errors >= int(OCF_RESKEY_max_disk_errors):
+ if replication == OCF_SUCCESS:
+ log.error("Local disk error")
+ role = OCF_ERR_GENERIC
+ else:
+ log.warning("Local disk error")
+
+ if not fail_fast and OCF_RESKEY_CRM_meta_interval != 0:
+ # This isn't a probe monitor
+ block_job = qmp_check_resync(fd)
+ if block_job:
+ if "error" in block_job:
+ log.error("resync error: %s" % block_job["error"])
+ peer = qmp_get_nbd_remote(fd)
+ qmp_cancel_resync(fd)
+ report_remote_failure(peer)
+ elif block_job["ready"] == True:
+ log.info("resync done, starting colo")
+ peer = qmp_get_nbd_remote(fd)
+ qmp_start_colo(fd, peer)
+ # COLO started, our secondary now can be promoted if the
+ # primary fails
+ set_remote_master_score(peer, 100)
+ else:
+ pct_done = (float(block_job["offset"]) \
+ / float(block_job["len"])) * 100
+ log.info("resync %.1f%% done" % pct_done)
+ else:
+ if replication == OCF_ERR_GENERIC:
+ if role == OCF_RUNNING_MASTER:
+ log.error("Replication error")
+ peer = qmp_get_nbd_remote(fd)
+ if peer:
+ report_remote_failure(peer)
+ else:
+ log.warning("Replication error")
+ qmp_execute(fd, [{"exec-helper": "clear-events"}], do_yank = False)
+
+ fd.close()
+
+ return role, replication
+
+def qemu_colo_start():
+ if check_pid()[0] == OCF_SUCCESS:
+ log.info("qemu is already running")
+ return OCF_SUCCESS
+
+ run_command("'%s' create -q -f qcow2 %s %s" \
+ % (OCF_RESKEY_qemu_img_binary, ACTIVE_IMAGE, OCF_RESKEY_disk_size))
+ run_command("'%s' create -q -f qcow2 %s %s" \
+ % (OCF_RESKEY_qemu_img_binary, HIDDEN_IMAGE, OCF_RESKEY_disk_size))
+
+ rotate_logfile(QMP_LOG, 8)
+ rotate_logfile(QEMU_LOG, 8)
+ run_command(QEMU_SECONDARY_CMDLINE)
+ oob_helper_open()
+
+ fd = qmp_open()
+ qmp_execute(fd, [
+ {"execute": "nbd-server-start", "arguments": {"addr": {"type": "inet", "data": {"host": str(OCF_RESKEY_listen_address), "port": str(NBD_PORT)}}}},
+ {"execute": "nbd-server-add", "arguments": {"device": "parent0", "writable": True}}
+ ])
+ fd.close()
+
+ return OCF_SUCCESS
+
+def env_do_shutdown_guest():
+ return OCF_RESKEY_CRM_meta_notify_active_uname \
+ and OCF_RESKEY_CRM_meta_notify_stop_uname \
+ and str.strip(OCF_RESKEY_CRM_meta_notify_active_uname) \
+ == str.strip(OCF_RESKEY_CRM_meta_notify_stop_uname)
+
+def env_find_secondary():
+ # slave(s) =
+ # OCF_RESKEY_CRM_meta_notify_slave_uname
+ # - OCF_RESKEY_CRM_meta_notify_stop_uname
+ # + OCF_RESKEY_CRM_meta_notify_start_uname
+ # Filter out hosts that are stopping and ourselves
+ for host in str.split(OCF_RESKEY_CRM_meta_notify_slave_uname, " "):
+ if host:
+ for stopping_host \
+ in str.split(OCF_RESKEY_CRM_meta_notify_stop_uname, " "):
+ if host == stopping_host:
+ break
+ else:
+ if host != HOSTNAME:
+ # we found a valid secondary
+ return host
+
+ for host in str.split(OCF_RESKEY_CRM_meta_notify_start_uname, " "):
+ if host != HOSTNAME:
+ # we found a valid secondary
+ return host
+
+ # we found no secondary
+ return None
+
+def _qemu_colo_stop(monstatus, shutdown_guest):
+ # stop action must do everything possible to stop the resource
+ try:
+ timeout = START_TIME + (int(OCF_RESKEY_CRM_meta_timeout)/1000) - 5
+ force_stop = False
+
+ if monstatus == OCF_NOT_RUNNING:
+ log.info("resource is already stopped")
+ return OCF_SUCCESS
+ elif monstatus == OCF_RUNNING_MASTER or monstatus == OCF_SUCCESS:
+ force_stop = False
+ else:
+ force_stop = True
+
+ if not force_stop:
+ fd = qmp_open(True)
+ if shutdown_guest:
+ if monstatus == OCF_RUNNING_MASTER:
+ qmp_execute(fd, [{"execute": "system_powerdown"}], \
+ do_yank = False)
+ else:
+ qmp_execute(fd, [{"execute": "quit"}], do_yank = False)
+ fd.close()
+
+ # wait for qemu to stop
+ while time.time() < timeout:
+ status, pid = check_pid()
+ if status == OCF_NOT_RUNNING:
+ # qemu stopped
+ return OCF_SUCCESS
+ elif status == OCF_SUCCESS:
+ # wait
+ log.debug("Waiting for qemu to stop")
+ time.sleep(1)
+ else:
+ # something went wrong, force stop instead
+ break
+
+ log.warning("clean stop timeout reached")
+ except Exception as e:
+ log.warning("error while stopping: %s" % e)
+
+ log.info("force stopping qemu")
+
+ status, pid = check_pid()
+ if status == OCF_NOT_RUNNING:
+ return OCF_SUCCESS
+ try:
+ if int(OCF_RESKEY_debug) >= 2:
+ os.kill(pid, signal.SIGSEGV)
+ else:
+ os.kill(pid, signal.SIGTERM)
+ time.sleep(2)
+ os.kill(pid, signal.SIGKILL)
+ except Exception:
+ pass
+
+ while check_pid()[0] != OCF_NOT_RUNNING:
+ time.sleep(1)
+
+ return OCF_SUCCESS
+
+def qemu_colo_stop():
+ shutdown_guest = env_do_shutdown_guest()
+ try:
+ role, replication = qemu_colo_monitor(True)
+ except Exception:
+ role, replication = OCF_ERR_GENERIC, OCF_ERR_GENERIC
+
+ status = _qemu_colo_stop(role, shutdown_guest)
+
+ if HOSTNAME == str.strip(OCF_RESKEY_CRM_meta_notify_master_uname):
+ if str.strip(OCF_RESKEY_CRM_meta_notify_promote_uname) != HOSTNAME:
+ # We where primary and the secondary is to be promoted.
+ # We are going to be out of date.
+ set_master_score(0)
+ else:
+ if role == OCF_RUNNING_MASTER:
+ # We where a healthy primary but had no healty secondary or it
+ # was stopped as well. So we have up-to-date data.
+ set_master_score(10)
+ else:
+ # We where a unhealthy primary but also had no healty secondary.
+ # So we still should have up-to-date data.
+ set_master_score(5)
+ else:
+ if get_master_score() > 10:
+ if role == OCF_SUCCESS:
+ if shutdown_guest:
+ # We where a healthy secondary and (probably) had a healthy
+ # primary and both where stopped. So we have up-to-date data
+ # too.
+ set_master_score(10)
+ else:
+ # We where a healthy secondary and (probably) had a healthy
+ # primary still running. So we are now out of date.
+ set_master_score(0)
+ else:
+ # We where a unhealthy secondary. So we are now out of date.
+ set_master_score(0)
+
+ return status
+
+def qemu_colo_notify():
+ action = "%s-%s" % (OCF_RESKEY_CRM_meta_notify_type, \
+ OCF_RESKEY_CRM_meta_notify_operation)
+
+ if action == "post-start":
+ if HOSTNAME == str.strip(OCF_RESKEY_CRM_meta_notify_master_uname):
+ peer = str.strip(OCF_RESKEY_CRM_meta_notify_start_uname)
+ fd = qmp_open()
+ qmp_start_resync(fd, peer)
+ # The secondary has inconsistent data until resync is finished
+ set_remote_master_score(peer, 0)
+ fd.close()
+
+ elif action == "pre-stop":
+ if not env_do_shutdown_guest() \
+ and HOSTNAME == str.strip(OCF_RESKEY_CRM_meta_notify_master_uname) \
+ and HOSTNAME != str.strip(OCF_RESKEY_CRM_meta_notify_stop_uname):
+ fd = qmp_open()
+ peer = qmp_get_nbd_remote(fd)
+ log.debug("our peer: %s" % peer)
+ if peer == str.strip(OCF_RESKEY_CRM_meta_notify_stop_uname):
+ if qmp_check_resync(fd):
+ qmp_cancel_resync(fd)
+ elif get_remote_master_score(peer) > 10:
+ qmp_primary_failover(fd)
+ qmp_execute(fd, [{"exec-helper": "clear-events"}],do_yank=False)
+ fd.close()
+
+ elif action == "post-stop" \
+ and OCF_RESKEY_CRM_meta_notify_key_operation == "stonith" \
+ and (HOSTNAME == str.strip(OCF_RESKEY_CRM_meta_notify_master_uname)
+ or str.strip(OCF_RESKEY_CRM_meta_notify_promote_uname)):
+ peer = str.strip(OCF_RESKEY_CRM_meta_notify_stop_uname)
+ set_remote_master_score(peer, 0)
+
+ return OCF_SUCCESS
+
+def qemu_colo_promote():
+ role, replication = qemu_colo_monitor()
+
+ if role == OCF_SUCCESS and replication == OCF_NOT_RUNNING:
+ status = _qemu_colo_stop(role, False)
+ if status != OCF_SUCCESS:
+ return status
+
+ rotate_logfile(QMP_LOG, 8)
+ rotate_logfile(QEMU_LOG, 8)
+ run_command(QEMU_PRIMARY_CMDLINE)
+ oob_helper_open()
+ set_master_score(101)
+
+ peer = env_find_secondary()
+ if peer:
+ fd = qmp_open()
+ qmp_start_resync(fd, peer)
+ # The secondary has inconsistent data until resync is finished
+ set_remote_master_score(peer, 0)
+ fd.close()
+ return OCF_SUCCESS
+ elif role == OCF_SUCCESS and replication != OCF_NOT_RUNNING:
+ fd = qmp_open()
+ qmp_secondary_failover(fd)
+ set_master_score(101)
+
+ peer = env_find_secondary()
+ if peer:
+ qmp_start_resync(fd, peer)
+ # The secondary has inconsistent data until resync is finished
+ set_remote_master_score(peer, 0)
+ qmp_execute(fd, [{"exec-helper": "clear-events"}], do_yank=False)
+ fd.close()
+ return OCF_SUCCESS
+ else:
+ return OCF_ERR_GENERIC
+
+def qemu_colo_demote():
+ status = qemu_colo_stop()
+ if status != OCF_SUCCESS:
+ return status
+ return qemu_colo_start()
+
+
+if OCF_ACTION == "meta-data":
+ qemu_colo_meta_data()
+ exit(OCF_SUCCESS)
+
+logs_open()
+
+status = qemu_colo_validate_all()
+# Exit here if our sanity checks fail, but try to continue if we need to stop
+if status != OCF_SUCCESS and OCF_ACTION != "stop":
+ exit(status)
+
+setup_constants()
+
+try:
+ if OCF_ACTION == "start":
+ status = qemu_colo_start()
+ elif OCF_ACTION == "stop":
+ status = qemu_colo_stop()
+ elif OCF_ACTION == "monitor":
+ status = qemu_colo_monitor()[0]
+ elif OCF_ACTION == "notify":
+ status = qemu_colo_notify()
+ elif OCF_ACTION == "promote":
+ status = qemu_colo_promote()
+ elif OCF_ACTION == "demote":
+ status = qemu_colo_demote()
+ elif OCF_ACTION == "validate-all":
+ status = qemu_colo_validate_all()
+ else:
+ status = OCF_ERR_UNIMPLEMENTED
+except Error:
+ exit(OCF_ERR_GENERIC)
+else:
+ exit(status)
--
2.20.1
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 3/5] colo: Introduce high-level test suite
2020-05-11 12:26 [PATCH 0/5] colo: Introduce resource agent and test suite/CI Lukas Straub
2020-05-11 12:26 ` [PATCH 1/5] block/quorum.c: stable children names Lukas Straub
2020-05-11 12:26 ` [PATCH 2/5] colo: Introduce resource agent Lukas Straub
@ 2020-05-11 12:27 ` Lukas Straub
2020-06-02 12:19 ` Philippe Mathieu-Daudé
2020-05-11 12:27 ` [PATCH 4/5] configure,Makefile: Install colo resource-agent Lukas Straub
` (2 subsequent siblings)
5 siblings, 1 reply; 14+ messages in thread
From: Lukas Straub @ 2020-05-11 12:27 UTC (permalink / raw)
To: qemu-devel; +Cc: Zhang Chen, Alberto Garcia, Dr. David Alan Gilbert
[-- Attachment #1: Type: text/plain, Size: 28057 bytes --]
Add high-level test relying on the colo resource-agent to test
all failover cases while checking guest network connectivity.
Signed-off-by: Lukas Straub <lukasstraub2@web.de>
---
scripts/colo-resource-agent/crm_master | 44 ++
scripts/colo-resource-agent/crm_resource | 12 +
tests/acceptance/colo.py | 689 +++++++++++++++++++++++
3 files changed, 745 insertions(+)
create mode 100755 scripts/colo-resource-agent/crm_master
create mode 100755 scripts/colo-resource-agent/crm_resource
create mode 100644 tests/acceptance/colo.py
diff --git a/scripts/colo-resource-agent/crm_master b/scripts/colo-resource-agent/crm_master
new file mode 100755
index 0000000000..886f523bda
--- /dev/null
+++ b/scripts/colo-resource-agent/crm_master
@@ -0,0 +1,44 @@
+#!/bin/bash
+
+# Fake crm_master for COLO testing
+#
+# Copyright (c) Lukas Straub <lukasstraub2@web.de>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+TMPDIR="$HA_RSCTMP"
+score=0
+query=0
+
+OPTIND=1
+while getopts 'Qql:Dv:N:G' opt; do
+ case "$opt" in
+ Q|q)
+ # Noop
+ ;;
+ "l")
+ # Noop
+ ;;
+ "D")
+ score=0
+ ;;
+ "v")
+ score=$OPTARG
+ ;;
+ "N")
+ TMPDIR="$COLO_TEST_REMOTE_TMP"
+ ;;
+ "G")
+ query=1
+ ;;
+ esac
+done
+
+if (( query )); then
+ cat "${TMPDIR}/master_score" || exit 1
+else
+ echo $score > "${TMPDIR}/master_score" || exit 1
+fi
+
+exit 0
diff --git a/scripts/colo-resource-agent/crm_resource b/scripts/colo-resource-agent/crm_resource
new file mode 100755
index 0000000000..ad69ff3c6b
--- /dev/null
+++ b/scripts/colo-resource-agent/crm_resource
@@ -0,0 +1,12 @@
+#!/bin/sh
+
+# Fake crm_resource for COLO testing
+#
+# Copyright (c) Lukas Straub <lukasstraub2@web.de>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+# Noop
+
+exit 0
diff --git a/tests/acceptance/colo.py b/tests/acceptance/colo.py
new file mode 100644
index 0000000000..465513fb6c
--- /dev/null
+++ b/tests/acceptance/colo.py
@@ -0,0 +1,689 @@
+# High-level test suite for qemu COLO testing all failover cases while checking
+# guest network connectivity
+#
+# Copyright (c) Lukas Straub <lukasstraub2@web.de>
+#
+# This work is licensed under the terms of the GNU GPL, version 2 or
+# later. See the COPYING file in the top-level directory.
+
+# HOWTO:
+#
+# This test has the following parameters:
+# bridge_name: name of the bridge interface to connect qemu to
+# host_address: ip address of the bridge interface
+# guest_address: ip address that the guest gets from the dhcp server
+# bridge_helper: path to the brige helper
+# (default: /usr/lib/qemu/qemu-bridge-helper)
+# install_cmd: command to run to install iperf3 and memtester in the guest
+# (default: "sudo -n dnf -q -y install iperf3 memtester")
+#
+# To run the network tests, you have to specify the parameters.
+#
+# Example for running the colo tests:
+# make check-acceptance FEDORA_31_ARCHES="x86_64" AVOCADO_TAGS="-t colo \
+# -p bridge_name=br0 -p host_address=192.168.220.1 \
+# -p guest_address=192.168.220.222"
+#
+# The colo tests currently only use x86_64 test vm images. With the
+# FEDORA_31_ARCHES make variable as in the example, only the x86_64 images will
+# be downloaded.
+#
+# If you're running the network tests as an unprivileged user, you need to set
+# the suid bit on the bridge helper (chmod +s <bridge-helper>).
+#
+# The dhcp server should assign a static ip to the guest, else the test may be
+# unreliable. The Mac address for the guest is always 52:54:00:12:34:56.
+
+
+import select
+import sys
+import subprocess
+import shutil
+import os
+import signal
+import os.path
+import time
+import tempfile
+
+from avocado import Test
+from avocado import skipUnless
+from avocado.utils import network
+from avocado.utils import vmimage
+from avocado.utils import cloudinit
+from avocado.utils import ssh
+from avocado.utils.path import find_command
+
+from avocado_qemu import pick_default_qemu_bin, BUILD_DIR, SOURCE_DIR
+from qemu.qmp import QEMUMonitorProtocol
+
+def iperf3_available():
+ try:
+ find_command("iperf3")
+ except CmdNotFoundError:
+ return False
+ return True
+
+class ColoTest(Test):
+
+ # Constants
+ OCF_SUCCESS = 0
+ OCF_ERR_GENERIC = 1
+ OCF_ERR_ARGS = 2
+ OCF_ERR_UNIMPLEMENTED = 3
+ OCF_ERR_PERM = 4
+ OCF_ERR_INSTALLED = 5
+ OCF_ERR_CONFIGURED = 6
+ OCF_NOT_RUNNING = 7
+ OCF_RUNNING_MASTER = 8
+ OCF_FAILED_MASTER = 9
+
+ HOSTA = 10
+ HOSTB = 11
+
+ QEMU_OPTIONS = (" -display none -vga none -enable-kvm"
+ " -smp 2 -cpu host -m 768"
+ " -device e1000,mac=52:54:00:12:34:56,netdev=hn0"
+ " -device virtio-blk,drive=colo-disk0")
+
+ FEDORA_VERSION = "31"
+ IMAGE_CHECKSUM = "e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0"
+ IMAGE_SIZE = "4294967296b"
+
+ hang_qemu = False
+ checkpoint_failover = False
+ traffic_procs = []
+
+ def get_image(self, temp_dir):
+ try:
+ return vmimage.get(
+ "fedora", arch="x86_64", version=self.FEDORA_VERSION,
+ checksum=self.IMAGE_CHECKSUM, algorithm="sha256",
+ cache_dir=self.cache_dirs[0],
+ snapshot_dir=temp_dir)
+ except:
+ self.cancel("Failed to download/prepare image")
+
+ @skipUnless(ssh.SSH_CLIENT_BINARY, "No SSH client available")
+ def setUp(self):
+ # Qemu and qemu-img binary
+ default_qemu_bin = pick_default_qemu_bin()
+ self.QEMU_BINARY = self.params.get("qemu_bin", default=default_qemu_bin)
+
+ # If qemu-img has been built, use it, otherwise the system wide one
+ # will be used. If none is available, the test will cancel.
+ qemu_img = os.path.join(BUILD_DIR, "qemu-img")
+ if not os.path.exists(qemu_img):
+ qemu_img = find_command("qemu-img", False)
+ if qemu_img is False:
+ self.cancel("Could not find \"qemu-img\", which is required to "
+ "create the bootable image")
+ self.QEMU_IMG_BINARY = qemu_img
+ vmimage.QEMU_IMG = qemu_img
+
+ self.RESOURCE_AGENT = os.path.join(SOURCE_DIR,
+ "scripts/colo-resource-agent/colo")
+ self.ADD_PATH = os.path.join(SOURCE_DIR, "scripts/colo-resource-agent")
+
+ # Logs
+ self.RA_LOG = os.path.join(self.outputdir, "resource-agent.log")
+ self.HOSTA_LOGDIR = os.path.join(self.outputdir, "hosta")
+ self.HOSTB_LOGDIR = os.path.join(self.outputdir, "hostb")
+ os.makedirs(self.HOSTA_LOGDIR)
+ os.makedirs(self.HOSTB_LOGDIR)
+
+ # Temporary directories
+ # We don't use self.workdir because of unix socket path length
+ # limitations
+ self.TMPDIR = tempfile.mkdtemp()
+ self.TMPA = os.path.join(self.TMPDIR, "hosta")
+ self.TMPB = os.path.join(self.TMPDIR, "hostb")
+ os.makedirs(self.TMPA)
+ os.makedirs(self.TMPB)
+
+ # Network
+ self.BRIDGE_NAME = self.params.get("bridge_name")
+ if self.BRIDGE_NAME:
+ self.HOST_ADDRESS = self.params.get("host_address")
+ self.GUEST_ADDRESS = self.params.get("guest_address")
+ self.BRIDGE_HELPER = self.params.get("bridge_helper",
+ default="/usr/lib/qemu/qemu-bridge-helper")
+ self.SSH_PORT = 22
+ else:
+ # QEMU's hard coded usermode router address
+ self.HOST_ADDRESS = "10.0.2.2"
+ self.GUEST_ADDRESS = "127.0.0.1"
+ self.BRIDGE_HOSTA_PORT = network.find_free_port(address="127.0.0.1")
+ self.BRIDGE_HOSTB_PORT = network.find_free_port(address="127.0.0.1")
+ self.SSH_PORT = network.find_free_port(address="127.0.0.1")
+
+ self.CLOUDINIT_HOME_PORT = network.find_free_port()
+
+ # Find free port range
+ base_port = 1024
+ while True:
+ base_port = network.find_free_port(start_port=base_port,
+ address="127.0.0.1")
+ if base_port == None:
+ self.cancel("Failed to find a free port")
+ for n in range(base_port, base_port +4):
+ if n > 65535:
+ break
+ if not network.is_port_free(n, "127.0.0.1"):
+ break
+ else:
+ # for loop above didn't break
+ break
+
+ self.BASE_PORT = base_port
+
+ # Disk images
+ self.log.info("Downloading/preparing boot image")
+ self.HOSTA_IMAGE = self.get_image(self.TMPA).path
+ self.HOSTB_IMAGE = self.get_image(self.TMPB).path
+ self.CLOUDINIT_ISO = os.path.join(self.TMPDIR, "cloudinit.iso")
+
+ self.log.info("Preparing cloudinit image")
+ try:
+ cloudinit.iso(self.CLOUDINIT_ISO, self.name,
+ username="test", password="password",
+ phone_home_host=self.HOST_ADDRESS,
+ phone_home_port=self.CLOUDINIT_HOME_PORT)
+ except Exception as e:
+ self.cancel("Failed to prepare cloudinit image")
+
+ self.QEMU_OPTIONS += " -cdrom %s" % self.CLOUDINIT_ISO
+
+ # Network bridge
+ if not self.BRIDGE_NAME:
+ self.BRIDGE_PIDFILE = os.path.join(self.TMPDIR, "bridge.pid")
+ self.run_command(("'%s' -pidfile '%s'"
+ " -M none -display none -daemonize"
+ " -netdev user,id=host,hostfwd=tcp:127.0.0.1:%s-:22"
+ " -netdev socket,id=hosta,listen=127.0.0.1:%s"
+ " -netdev socket,id=hostb,listen=127.0.0.1:%s"
+ " -netdev hubport,id=hostport,hubid=0,netdev=host"
+ " -netdev hubport,id=porta,hubid=0,netdev=hosta"
+ " -netdev hubport,id=portb,hubid=0,netdev=hostb")
+ % (self.QEMU_BINARY, self.BRIDGE_PIDFILE, self.SSH_PORT,
+ self.BRIDGE_HOSTA_PORT, self.BRIDGE_HOSTB_PORT), 0)
+
+ def tearDown(self):
+ try:
+ pid = self.read_pidfile(self.BRIDGE_PIDFILE)
+ if pid and self.check_pid(pid):
+ os.kill(pid, signal.SIGKILL)
+ except Exception as e:
+ pass
+
+ try:
+ self.ra_stop(self.HOSTA)
+ except Exception as e:
+ pass
+
+ try:
+ self.ra_stop(self.HOSTB)
+ except Exception as e:
+ pass
+
+ try:
+ self.ssh_close()
+ except Exception as e:
+ pass
+
+ for proc in self.traffic_procs:
+ try:
+ os.killpg(proc.pid, signal.SIGTERM)
+ except Exception as e:
+ pass
+
+ shutil.rmtree(self.TMPDIR)
+
+ def run_command(self, cmdline, expected_status, env=None):
+ proc = subprocess.Popen(cmdline, shell=True, stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ universal_newlines=True, env=env)
+ stdout, stderr = proc.communicate()
+ if proc.returncode != expected_status:
+ self.fail("command \"%s\" failed with code %s:\n%s"
+ % (cmdline, proc.returncode, stdout))
+
+ return proc.returncode
+
+ def cat_line(self, path):
+ line=""
+ try:
+ fd = open(path, "r")
+ line = str.strip(fd.readline())
+ fd.close()
+ except:
+ pass
+ return line
+
+ def read_pidfile(self, pidfile):
+ try:
+ pid = int(self.cat_line(pidfile))
+ except ValueError:
+ return None
+ else:
+ return pid
+
+ def check_pid(self, pid):
+ try:
+ os.kill(pid, 0)
+ except OSError:
+ return False
+ else:
+ return True
+
+ def ssh_open(self):
+ self.ssh_conn = ssh.Session(self.GUEST_ADDRESS, self.SSH_PORT,
+ user="test", password="password")
+ self.ssh_conn.DEFAULT_OPTIONS += (("UserKnownHostsFile", "/dev/null"),)
+ for i in range(10):
+ try:
+ if self.ssh_conn.connect():
+ return
+ except Exception as e:
+ pass
+ time.sleep(4)
+ self.fail("sshd timeout")
+
+ def ssh_ping(self):
+ if self.ssh_conn.cmd("echo ping").stdout != b"ping\n":
+ self.fail("ssh ping failed")
+
+ def ssh_close(self):
+ self.ssh_conn.quit()
+
+ def setup_base_env(self, host):
+ PATH = os.getenv("PATH", "")
+ env = { "PATH": "%s:%s" % (self.ADD_PATH, PATH),
+ "HA_LOGFILE": self.RA_LOG,
+ "OCF_RESOURCE_INSTANCE": "colo-test",
+ "OCF_RESKEY_CRM_meta_clone_max": "2",
+ "OCF_RESKEY_CRM_meta_notify": "true",
+ "OCF_RESKEY_CRM_meta_timeout": "30000",
+ "OCF_RESKEY_qemu_binary": self.QEMU_BINARY,
+ "OCF_RESKEY_qemu_img_binary": self.QEMU_IMG_BINARY,
+ "OCF_RESKEY_disk_size": str(self.IMAGE_SIZE),
+ "OCF_RESKEY_checkpoint_interval": "10000",
+ "OCF_RESKEY_base_port": str(self.BASE_PORT),
+ "OCF_RESKEY_debug": "2"}
+
+ if host == self.HOSTA:
+ env.update({"OCF_RESKEY_options":
+ ("%s -qmp unix:%s,server,nowait"
+ " -drive if=none,id=parent0,file='%s'")
+ % (self.QEMU_OPTIONS, self.get_qmp_sock(host),
+ self.HOSTA_IMAGE),
+ "OCF_RESKEY_active_hidden_dir": self.TMPA,
+ "OCF_RESKEY_listen_address": "127.0.0.1",
+ "OCF_RESKEY_log_dir": self.HOSTA_LOGDIR,
+ "OCF_RESKEY_CRM_meta_on_node": "127.0.0.1",
+ "HA_RSCTMP": self.TMPA,
+ "COLO_TEST_REMOTE_TMP": self.TMPB})
+
+ else:
+ env.update({"OCF_RESKEY_options":
+ ("%s -qmp unix:%s,server,nowait"
+ " -drive if=none,id=parent0,file='%s'")
+ % (self.QEMU_OPTIONS, self.get_qmp_sock(host),
+ self.HOSTB_IMAGE),
+ "OCF_RESKEY_active_hidden_dir": self.TMPB,
+ "OCF_RESKEY_listen_address": "127.0.0.2",
+ "OCF_RESKEY_log_dir": self.HOSTB_LOGDIR,
+ "OCF_RESKEY_CRM_meta_on_node": "127.0.0.2",
+ "HA_RSCTMP": self.TMPB,
+ "COLO_TEST_REMOTE_TMP": self.TMPA})
+
+ if self.BRIDGE_NAME:
+ env["OCF_RESKEY_options"] += \
+ " -netdev bridge,id=hn0,br=%s,helper='%s'" \
+ % (self.BRIDGE_NAME, self.BRIDGE_HELPER)
+ else:
+ if host == self.HOSTA:
+ env["OCF_RESKEY_options"] += \
+ " -netdev socket,id=hn0,connect=127.0.0.1:%s" \
+ % self.BRIDGE_HOSTA_PORT
+ else:
+ env["OCF_RESKEY_options"] += \
+ " -netdev socket,id=hn0,connect=127.0.0.1:%s" \
+ % self.BRIDGE_HOSTB_PORT
+ return env
+
+ def ra_start(self, host):
+ env = self.setup_base_env(host)
+ self.run_command(self.RESOURCE_AGENT + " start", self.OCF_SUCCESS, env)
+
+ def ra_stop(self, host):
+ env = self.setup_base_env(host)
+ self.run_command(self.RESOURCE_AGENT + " stop", self.OCF_SUCCESS, env)
+
+ def ra_monitor(self, host, expected_status):
+ env = self.setup_base_env(host)
+ self.run_command(self.RESOURCE_AGENT + " monitor", expected_status, env)
+
+ def ra_promote(self, host):
+ env = self.setup_base_env(host)
+ self.run_command(self.RESOURCE_AGENT + " promote", self.OCF_SUCCESS,env)
+
+ def ra_notify_start(self, host):
+ env = self.setup_base_env(host)
+
+ env.update({"OCF_RESKEY_CRM_meta_notify_type": "post",
+ "OCF_RESKEY_CRM_meta_notify_operation": "start"})
+
+ if host == self.HOSTA:
+ env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.1",
+ "OCF_RESKEY_CRM_meta_notify_start_uname": "127.0.0.2"})
+ else:
+ env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.2",
+ "OCF_RESKEY_CRM_meta_notify_start_uname": "127.0.0.1"})
+
+ self.run_command(self.RESOURCE_AGENT + " notify", self.OCF_SUCCESS, env)
+
+ def ra_notify_stop(self, host):
+ env = self.setup_base_env(host)
+
+ env.update({"OCF_RESKEY_CRM_meta_notify_type": "pre",
+ "OCF_RESKEY_CRM_meta_notify_operation": "stop"})
+
+ if host == self.HOSTA:
+ env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.1",
+ "OCF_RESKEY_CRM_meta_notify_stop_uname": "127.0.0.2"})
+ else:
+ env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.2",
+ "OCF_RESKEY_CRM_meta_notify_stop_uname": "127.0.0.1"})
+
+ self.run_command(self.RESOURCE_AGENT + " notify", self.OCF_SUCCESS, env)
+
+ def get_pid(self, host):
+ if host == self.HOSTA:
+ return self.read_pidfile(os.path.join(self.TMPA,
+ "colo-test-qemu.pid"))
+ else:
+ return self.read_pidfile(os.path.join(self.TMPB,
+ "colo-test-qemu.pid"))
+
+ def get_master_score(self, host):
+ if host == self.HOSTA:
+ return int(self.cat_line(os.path.join(self.TMPA, "master_score")))
+ else:
+ return int(self.cat_line(os.path.join(self.TMPB, "master_score")))
+
+ def get_qmp_sock(self, host):
+ if host == self.HOSTA:
+ return os.path.join(self.TMPA, "my-qmp.sock")
+ else:
+ return os.path.join(self.TMPB, "my-qmp.sock")
+
+
+ def kill_qemu_pre(self, host):
+ pid = self.get_pid(host)
+
+ qmp_sock = self.get_qmp_sock(host)
+
+ if self.checkpoint_failover:
+ qmp_conn = QEMUMonitorProtocol(qmp_sock)
+ qmp_conn.settimeout(10)
+ qmp_conn.connect()
+ while True:
+ event = qmp_conn.pull_event(wait=True)
+ if event["event"] == "STOP":
+ break
+ qmp_conn.close()
+
+
+ if pid and self.check_pid(pid):
+ if self.hang_qemu:
+ os.kill(pid, signal.SIGSTOP)
+ else:
+ os.kill(pid, signal.SIGKILL)
+ while self.check_pid(pid):
+ time.sleep(1)
+
+ def kill_qemu_post(self, host):
+ pid = self.get_pid(host)
+
+ if self.hang_qemu and pid and self.check_pid(pid):
+ os.kill(pid, signal.SIGKILL)
+ while self.check_pid(pid):
+ time.sleep(1)
+
+ def prepare_guest(self):
+ pass
+
+ def cycle_start(self, cycle):
+ pass
+
+ def active_section(self):
+ return False
+
+ def cycle_end(self, cycle):
+ pass
+
+ def check_connection(self):
+ self.ssh_ping()
+ for proc in self.traffic_procs:
+ if proc.poll() != None:
+ self.fail("Traffic process died")
+
+ def _test_colo(self, loop=1):
+ loop = max(loop, 1)
+ self.log.info("Will put logs in %s" % self.outputdir)
+
+ self.ra_stop(self.HOSTA)
+ self.ra_stop(self.HOSTB)
+
+ self.log.info("*** Startup ***")
+ self.ra_start(self.HOSTA)
+ self.ra_start(self.HOSTB)
+
+ self.ra_monitor(self.HOSTA, self.OCF_SUCCESS)
+ self.ra_monitor(self.HOSTB, self.OCF_SUCCESS)
+
+ self.log.info("*** Promoting ***")
+ self.ra_promote(self.HOSTA)
+ cloudinit.wait_for_phone_home(("0.0.0.0", self.CLOUDINIT_HOME_PORT),
+ self.name)
+ self.ssh_open()
+ self.prepare_guest()
+
+ self.ra_notify_start(self.HOSTA)
+
+ while self.get_master_score(self.HOSTB) != 100:
+ self.ra_monitor(self.HOSTA, self.OCF_RUNNING_MASTER)
+ self.ra_monitor(self.HOSTB, self.OCF_SUCCESS)
+ time.sleep(1)
+
+ self.log.info("*** Replication started ***")
+
+ self.check_connection()
+
+ primary = self.HOSTA
+ secondary = self.HOSTB
+
+ for n in range(loop):
+ self.cycle_start(n)
+ self.log.info("*** Secondary failover ***")
+ self.kill_qemu_pre(primary)
+ self.ra_notify_stop(secondary)
+ self.ra_monitor(secondary, self.OCF_SUCCESS)
+ self.ra_promote(secondary)
+ self.ra_monitor(secondary, self.OCF_RUNNING_MASTER)
+ self.kill_qemu_post(primary)
+
+ self.check_connection()
+
+ tmp = primary
+ primary = secondary
+ secondary = tmp
+
+ self.log.info("*** Secondary continue replication ***")
+ self.ra_start(secondary)
+ self.ra_notify_start(primary)
+
+ self.check_connection()
+
+ # Wait for resync
+ while self.get_master_score(secondary) != 100:
+ self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
+ self.ra_monitor(secondary, self.OCF_SUCCESS)
+ time.sleep(1)
+
+ self.log.info("*** Replication started ***")
+
+ self.check_connection()
+
+ if self.active_section():
+ break
+
+ self.log.info("*** Primary failover ***")
+ self.kill_qemu_pre(secondary)
+ self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
+ self.ra_notify_stop(primary)
+ self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
+ self.kill_qemu_post(secondary)
+
+ self.check_connection()
+
+ self.log.info("*** Primary continue replication ***")
+ self.ra_start(secondary)
+ self.ra_notify_start(primary)
+
+ self.check_connection()
+
+ # Wait for resync
+ while self.get_master_score(secondary) != 100:
+ self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
+ self.ra_monitor(secondary, self.OCF_SUCCESS)
+ time.sleep(1)
+
+ self.log.info("*** Replication started ***")
+
+ self.check_connection()
+
+ self.cycle_end(n)
+
+ self.ssh_close()
+
+ self.ra_stop(self.HOSTA)
+ self.ra_stop(self.HOSTB)
+
+ self.ra_monitor(self.HOSTA, self.OCF_NOT_RUNNING)
+ self.ra_monitor(self.HOSTB, self.OCF_NOT_RUNNING)
+ self.log.info("*** all ok ***")
+
+
+class ColoQuickTest(ColoTest):
+ """
+ :avocado: tags=colo
+ :avocado: tags=quick
+ :avocado: tags=arch:x86_64
+ """
+
+ timeout = 300
+
+ def cycle_end(self, cycle):
+ self.log.info("Testing with peer qemu hanging"
+ " and failover during checkpoint")
+ self.hang_qemu = True
+
+ def test_quick(self):
+ self.checkpoint_failover = True
+ self.log.info("Testing with peer qemu crashing"
+ " and failover during checkpoint")
+ self._test_colo(loop=2)
+
+
+class ColoNetworkTest(ColoTest):
+
+ def prepare_guest(self):
+ install_cmd = self.params.get("install_cmd", default=
+ "sudo -n dnf -q -y install iperf3 memtester")
+ self.ssh_conn.cmd(install_cmd)
+ # Use two instances to work around a bug in iperf3
+ self.ssh_conn.cmd("iperf3 -sD; iperf3 -sD -p 5202")
+
+ def _cycle_start(self, cycle):
+ pass
+
+ def cycle_start(self, cycle):
+ self._cycle_start(cycle)
+ tests = [("", "client-to-server tcp"), ("-R", "server-to-client tcp")]
+
+ self.log.info("Testing iperf %s" % tests[cycle % 2][1])
+ iperf_cmd = "iperf3 %s -t 60 -i 60 --connect-timeout 30000 -c %s" \
+ % (tests[cycle % 2][0], self.GUEST_ADDRESS)
+ proc = subprocess.Popen("while %s && %s; do sleep 1; done >>'%s' 2>&1"
+ % (iperf_cmd, iperf_cmd + " -p 5202",
+ os.path.join(self.outputdir, "iperf.log")),
+ shell=True, preexec_fn=os.setsid, stdin=subprocess.DEVNULL,
+ stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+ self.traffic_procs.append(proc)
+ time.sleep(5) # Wait for iperf to get up to speed
+
+ def cycle_end(self, cycle):
+ for proc in self.traffic_procs:
+ try:
+ os.killpg(proc.pid, signal.SIGTERM)
+ proc.wait()
+ except Exception as e:
+ pass
+ self.traffic_procs.clear()
+ time.sleep(20)
+
+class ColoRealNetworkTest(ColoNetworkTest):
+ """
+ :avocado: tags=colo
+ :avocado: tags=slow
+ :avocado: tags=network_test
+ :avocado: tags=arch:x86_64
+ """
+
+ timeout = 900
+
+ def active_section(self):
+ time.sleep(300)
+ return False
+
+ @skipUnless(iperf3_available(), "iperf3 not available")
+ def test_network(self):
+ if not self.BRIDGE_NAME:
+ self.cancel("bridge options not given, will skip network test")
+ self.log.info("Testing with peer qemu crashing and network load")
+ self._test_colo(loop=2)
+
+class ColoStressTest(ColoNetworkTest):
+ """
+ :avocado: tags=colo
+ :avocado: tags=slow
+ :avocado: tags=stress_test
+ :avocado: tags=arch:x86_64
+ """
+
+ timeout = 1800
+
+ def _cycle_start(self, cycle):
+ if cycle == 4:
+ self.log.info("Stresstest with peer qemu hanging, network load"
+ " and failover during checkpoint")
+ self.checkpoint_failover = True
+ self.hang_qemu = True
+ elif cycle == 8:
+ self.log.info("Stresstest with peer qemu crashing and network load")
+ self.checkpoint_failover = False
+ self.hang_qemu = False
+ elif cycle == 12:
+ self.log.info("Stresstest with peer qemu hanging and network load")
+ self.checkpoint_failover = False
+ self.hang_qemu = True
+
+ @skipUnless(iperf3_available(), "iperf3 not available")
+ def test_stress(self):
+ if not self.BRIDGE_NAME:
+ self.cancel("bridge options not given, will skip network test")
+ self.log.info("Stresstest with peer qemu crashing, network load"
+ " and failover during checkpoint")
+ self.checkpoint_failover = True
+ self._test_colo(loop=16)
--
2.20.1
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 4/5] configure,Makefile: Install colo resource-agent
2020-05-11 12:26 [PATCH 0/5] colo: Introduce resource agent and test suite/CI Lukas Straub
` (2 preceding siblings ...)
2020-05-11 12:27 ` [PATCH 3/5] colo: Introduce high-level test suite Lukas Straub
@ 2020-05-11 12:27 ` Lukas Straub
2020-05-11 12:27 ` [PATCH 5/5] MAINTAINERS: Add myself as maintainer for COLO resource agent Lukas Straub
2020-05-18 9:38 ` [PATCH 0/5] colo: Introduce resource agent and test suite/CI Zhang, Chen
5 siblings, 0 replies; 14+ messages in thread
From: Lukas Straub @ 2020-05-11 12:27 UTC (permalink / raw)
To: qemu-devel; +Cc: Zhang Chen, Alberto Garcia, Dr. David Alan Gilbert
[-- Attachment #1: Type: text/plain, Size: 2453 bytes --]
Optionally install the resouce-agent so it gets picked up by
pacemaker.
Signed-off-by: Lukas Straub <lukasstraub2@web.de>
---
Makefile | 5 +++++
configure | 10 ++++++++++
2 files changed, 15 insertions(+)
diff --git a/Makefile b/Makefile
index 8a9113e666..2ebffc4465 100644
--- a/Makefile
+++ b/Makefile
@@ -973,6 +973,11 @@ ifneq ($(DESCS),)
$(INSTALL_DATA) "$$tmpf" \
"$(DESTDIR)$(qemu_datadir)/firmware/$$x"; \
done
+endif
+ifdef INSTALL_COLO_RA
+ mkdir -p "$(DESTDIR)$(libdir)/ocf/resource.d/qemu"
+ $(INSTALL_PROG) "scripts/colo-resource-agent/colo" \
+ "$(DESTDIR)$(libdir)/ocf/resource.d/qemu/colo"
endif
for s in $(ICON_SIZES); do \
mkdir -p "$(DESTDIR)$(qemu_icondir)/hicolor/$${s}/apps"; \
diff --git a/configure b/configure
index 23b5e93752..c9252030cf 100755
--- a/configure
+++ b/configure
@@ -430,6 +430,7 @@ softmmu="yes"
linux_user="no"
bsd_user="no"
blobs="yes"
+colo_ra="no"
edk2_blobs="no"
pkgversion=""
pie=""
@@ -1309,6 +1310,10 @@ for opt do
;;
--disable-blobs) blobs="no"
;;
+ --disable-colo-ra) colo_ra="no"
+ ;;
+ --enable-colo-ra) colo_ra="yes"
+ ;;
--with-pkgversion=*) pkgversion="$optarg"
;;
--with-coroutine=*) coroutine="$optarg"
@@ -1776,6 +1781,7 @@ Advanced options (experts only):
--enable-gcov enable test coverage analysis with gcov
--gcov=GCOV use specified gcov [$gcov_tool]
--disable-blobs disable installing provided firmware blobs
+ --enable-colo-ra enable installing the COLO resource agent for pacemaker
--with-vss-sdk=SDK-path enable Windows VSS support in QEMU Guest Agent
--with-win-sdk=SDK-path path to Windows Platform SDK (to build VSS .tlb)
--tls-priority default TLS protocol/cipher priority string
@@ -6647,6 +6653,7 @@ echo "Linux AIO support $linux_aio"
echo "Linux io_uring support $linux_io_uring"
echo "ATTR/XATTR support $attr"
echo "Install blobs $blobs"
+echo "Install COLO resource agent $colo_ra"
echo "KVM support $kvm"
echo "HAX support $hax"
echo "HVF support $hvf"
@@ -7188,6 +7195,9 @@ fi
if test "$blobs" = "yes" ; then
echo "INSTALL_BLOBS=yes" >> $config_host_mak
fi
+if test "$colo_ra" = "yes" ; then
+ echo "INSTALL_COLO_RA=yes" >> $config_host_mak
+fi
if test "$iovec" = "yes" ; then
echo "CONFIG_IOVEC=y" >> $config_host_mak
fi
--
2.20.1
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 14+ messages in thread
* [PATCH 5/5] MAINTAINERS: Add myself as maintainer for COLO resource agent
2020-05-11 12:26 [PATCH 0/5] colo: Introduce resource agent and test suite/CI Lukas Straub
` (3 preceding siblings ...)
2020-05-11 12:27 ` [PATCH 4/5] configure,Makefile: Install colo resource-agent Lukas Straub
@ 2020-05-11 12:27 ` Lukas Straub
2020-05-18 9:38 ` [PATCH 0/5] colo: Introduce resource agent and test suite/CI Zhang, Chen
5 siblings, 0 replies; 14+ messages in thread
From: Lukas Straub @ 2020-05-11 12:27 UTC (permalink / raw)
To: qemu-devel; +Cc: Zhang Chen, Alberto Garcia, Dr. David Alan Gilbert
[-- Attachment #1: Type: text/plain, Size: 694 bytes --]
While I'm not going to have much time for this, I'll still
try to test and review patches.
Signed-off-by: Lukas Straub <lukasstraub2@web.de>
---
MAINTAINERS | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/MAINTAINERS b/MAINTAINERS
index 8cbc1fac2b..4c623a96e1 100644
--- a/MAINTAINERS
+++ b/MAINTAINERS
@@ -2466,6 +2466,12 @@ F: net/colo*
F: net/filter-rewriter.c
F: net/filter-mirror.c
+COLO resource agent and testing
+M: Lukas Straub <lukasstraub2@web.de>
+S: Odd fixes
+F: scripts/colo-resource-agent/*
+F: tests/acceptance/colo.py
+
Record/replay
M: Pavel Dovgalyuk <pavel.dovgaluk@ispras.ru>
R: Paolo Bonzini <pbonzini@redhat.com>
--
2.20.1
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply related [flat|nested] 14+ messages in thread
* RE: [PATCH 0/5] colo: Introduce resource agent and test suite/CI
2020-05-11 12:26 [PATCH 0/5] colo: Introduce resource agent and test suite/CI Lukas Straub
` (4 preceding siblings ...)
2020-05-11 12:27 ` [PATCH 5/5] MAINTAINERS: Add myself as maintainer for COLO resource agent Lukas Straub
@ 2020-05-18 9:38 ` Zhang, Chen
2020-06-06 18:59 ` Lukas Straub
5 siblings, 1 reply; 14+ messages in thread
From: Zhang, Chen @ 2020-05-18 9:38 UTC (permalink / raw)
To: Lukas Straub, qemu-devel
Cc: Jason Wang, Alberto Garcia, Dr. David Alan Gilbert
> -----Original Message-----
> From: Lukas Straub <lukasstraub2@web.de>
> Sent: Monday, May 11, 2020 8:27 PM
> To: qemu-devel <qemu-devel@nongnu.org>
> Cc: Alberto Garcia <berto@igalia.com>; Dr. David Alan Gilbert
> <dgilbert@redhat.com>; Zhang, Chen <chen.zhang@intel.com>
> Subject: [PATCH 0/5] colo: Introduce resource agent and test suite/CI
>
> Hello Everyone,
> These patches introduce a resource agent for fully automatic management of
> colo and a test suite building upon the resource agent to extensively test colo.
>
> Test suite features:
> -Tests failover with peer crashing and hanging and failover during checkpoint
> -Tests network using ssh and iperf3 -Quick test requires no special
> configuration -Network test for testing colo-compare -Stress test: failover all
> the time with network load
>
> Resource agent features:
> -Fully automatic management of colo
> -Handles many failures: hanging/crashing qemu, replication error, disk
> error, ...
> -Recovers from hanging qemu by using the "yank" oob command -Tracks
> which node has up-to-date data -Works well in clusters with more than 2
> nodes
>
> Run times on my laptop:
> Quick test: 200s
> Network test: 800s (tagged as slow)
> Stress test: 1300s (tagged as slow)
>
> The test suite needs access to a network bridge to properly test the network,
> so some parameters need to be given to the test run. See
> tests/acceptance/colo.py for more information.
>
> I wonder how this integrates in existing CI infrastructure. Is there a common
> CI for qemu where this can run or does every subsystem have to run their
> own CI?
Wow~ Very happy to see this series.
I have checked the "how to" in tests/acceptance/colo.py,
But it looks not enough for users, can you write an independent document for this series?
Include test Infrastructure ASC II diagram, test cases design , detailed how to and more information for
pacemaker cluster and resource agent..etc ?
Thanks
Zhang Chen
>
> Regards,
> Lukas Straub
>
>
> Lukas Straub (5):
> block/quorum.c: stable children names
> colo: Introduce resource agent
> colo: Introduce high-level test suite
> configure,Makefile: Install colo resource-agent
> MAINTAINERS: Add myself as maintainer for COLO resource agent
>
> MAINTAINERS | 6 +
> Makefile | 5 +
> block/quorum.c | 20 +-
> configure | 10 +
> scripts/colo-resource-agent/colo | 1429 ++++++++++++++++++++++
> scripts/colo-resource-agent/crm_master | 44 +
> scripts/colo-resource-agent/crm_resource | 12 +
> tests/acceptance/colo.py | 689 +++++++++++
> 8 files changed, 2209 insertions(+), 6 deletions(-) create mode 100755
> scripts/colo-resource-agent/colo create mode 100755 scripts/colo-resource-
> agent/crm_master
> create mode 100755 scripts/colo-resource-agent/crm_resource
> create mode 100644 tests/acceptance/colo.py
>
> --
> 2.20.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* RE: [PATCH 1/5] block/quorum.c: stable children names
2020-05-11 12:26 ` [PATCH 1/5] block/quorum.c: stable children names Lukas Straub
@ 2020-06-02 1:01 ` Zhang, Chen
2020-06-02 11:07 ` Alberto Garcia
1 sibling, 0 replies; 14+ messages in thread
From: Zhang, Chen @ 2020-06-02 1:01 UTC (permalink / raw)
To: Lukas Straub, qemu-devel; +Cc: Alberto Garcia, Dr. David Alan Gilbert
> -----Original Message-----
> From: Lukas Straub <lukasstraub2@web.de>
> Sent: Monday, May 11, 2020 8:27 PM
> To: qemu-devel <qemu-devel@nongnu.org>
> Cc: Alberto Garcia <berto@igalia.com>; Dr. David Alan Gilbert
> <dgilbert@redhat.com>; Zhang, Chen <chen.zhang@intel.com>
> Subject: [PATCH 1/5] block/quorum.c: stable children names
>
> If we remove the child with the highest index from the quorum, decrement
> s->next_child_index. This way we get stable children names as long as we
> only remove the last child.
>
Looks good for me, and it can solve this bug:
colo: Can not recover colo after svm failover twice
https://bugs.launchpad.net/bugs/1881231
Reviewed-by: Zhang Chen <chen.zhang@intel.com>
> Signed-off-by: Lukas Straub <lukasstraub2@web.de>
> ---
> block/quorum.c | 20 ++++++++++++++------
> 1 file changed, 14 insertions(+), 6 deletions(-)
>
> diff --git a/block/quorum.c b/block/quorum.c index 6d7a56bd93..acfa09c2cc
> 100644
> --- a/block/quorum.c
> +++ b/block/quorum.c
> @@ -29,6 +29,8 @@
>
> #define HASH_LENGTH 32
>
> +#define INDEXSTR_LEN 32
> +
> #define QUORUM_OPT_VOTE_THRESHOLD "vote-threshold"
> #define QUORUM_OPT_BLKVERIFY "blkverify"
> #define QUORUM_OPT_REWRITE "rewrite-corrupted"
> @@ -972,9 +974,9 @@ static int quorum_open(BlockDriverState *bs, QDict
> *options, int flags,
> opened = g_new0(bool, s->num_children);
>
> for (i = 0; i < s->num_children; i++) {
> - char indexstr[32];
> - ret = snprintf(indexstr, 32, "children.%d", i);
> - assert(ret < 32);
> + char indexstr[INDEXSTR_LEN];
> + ret = snprintf(indexstr, INDEXSTR_LEN, "children.%d", i);
> + assert(ret < INDEXSTR_LEN);
>
> s->children[i] = bdrv_open_child(NULL, options, indexstr, bs,
> &child_format, false, &local_err); @@ -1026,7 +1028,7
> @@ static void quorum_add_child(BlockDriverState *bs, BlockDriverState
> *child_bs, {
> BDRVQuorumState *s = bs->opaque;
> BdrvChild *child;
> - char indexstr[32];
> + char indexstr[INDEXSTR_LEN];
> int ret;
>
> if (s->is_blkverify) {
> @@ -1041,8 +1043,8 @@ static void quorum_add_child(BlockDriverState *bs,
> BlockDriverState *child_bs,
> return;
> }
>
> - ret = snprintf(indexstr, 32, "children.%u", s->next_child_index);
> - if (ret < 0 || ret >= 32) {
> + ret = snprintf(indexstr, INDEXSTR_LEN, "children.%u", s-
> >next_child_index);
> + if (ret < 0 || ret >= INDEXSTR_LEN) {
> error_setg(errp, "cannot generate child name");
> return;
> }
> @@ -1069,6 +1071,7 @@ static void quorum_del_child(BlockDriverState *bs,
> BdrvChild *child,
> Error **errp) {
> BDRVQuorumState *s = bs->opaque;
> + char indexstr[INDEXSTR_LEN];
> int i;
>
> for (i = 0; i < s->num_children; i++) { @@ -1090,6 +1093,11 @@ static void
> quorum_del_child(BlockDriverState *bs, BdrvChild *child,
> /* We know now that num_children > threshold, so blkverify must be
> false */
> assert(!s->is_blkverify);
>
> + snprintf(indexstr, INDEXSTR_LEN, "children.%u", s->next_child_index - 1);
> + if (!strncmp(child->name, indexstr, INDEXSTR_LEN)) {
> + s->next_child_index--;
> + }
> +
> bdrv_drained_begin(bs);
>
> /* We can safely remove this child now */
> --
> 2.20.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 1/5] block/quorum.c: stable children names
2020-05-11 12:26 ` [PATCH 1/5] block/quorum.c: stable children names Lukas Straub
2020-06-02 1:01 ` Zhang, Chen
@ 2020-06-02 11:07 ` Alberto Garcia
1 sibling, 0 replies; 14+ messages in thread
From: Alberto Garcia @ 2020-06-02 11:07 UTC (permalink / raw)
To: Lukas Straub, qemu-devel; +Cc: Zhang Chen, Dr. David Alan Gilbert
On Mon 11 May 2020 02:26:54 PM CEST, Lukas Straub wrote:
> If we remove the child with the highest index from the quorum,
> decrement s->next_child_index. This way we get stable children
> names as long as we only remove the last child.
>
> Signed-off-by: Lukas Straub <lukasstraub2@web.de>
Reviewed-by: Alberto Garcia <berto@igalia.com>
Berto
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/5] colo: Introduce high-level test suite
2020-05-11 12:27 ` [PATCH 3/5] colo: Introduce high-level test suite Lukas Straub
@ 2020-06-02 12:19 ` Philippe Mathieu-Daudé
2020-06-04 10:55 ` Lukas Straub
0 siblings, 1 reply; 14+ messages in thread
From: Philippe Mathieu-Daudé @ 2020-06-02 12:19 UTC (permalink / raw)
To: Lukas Straub, qemu-devel
Cc: Zhang Chen, Alberto Garcia, Dr. David Alan Gilbert,
Wainer dos Santos Moschetta, Cleber Rosa
+Cleber/Wainer
On 5/11/20 2:27 PM, Lukas Straub wrote:
> Add high-level test relying on the colo resource-agent to test
> all failover cases while checking guest network connectivity.
>
> Signed-off-by: Lukas Straub <lukasstraub2@web.de>
> ---
> scripts/colo-resource-agent/crm_master | 44 ++
> scripts/colo-resource-agent/crm_resource | 12 +
> tests/acceptance/colo.py | 689 +++++++++++++++++++++++
> 3 files changed, 745 insertions(+)
> create mode 100755 scripts/colo-resource-agent/crm_master
> create mode 100755 scripts/colo-resource-agent/crm_resource
> create mode 100644 tests/acceptance/colo.py
>
> diff --git a/scripts/colo-resource-agent/crm_master b/scripts/colo-resource-agent/crm_master
> new file mode 100755
> index 0000000000..886f523bda
> --- /dev/null
> +++ b/scripts/colo-resource-agent/crm_master
> @@ -0,0 +1,44 @@
> +#!/bin/bash
> +
> +# Fake crm_master for COLO testing
> +#
> +# Copyright (c) Lukas Straub <lukasstraub2@web.de>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later. See the COPYING file in the top-level directory.
> +
> +TMPDIR="$HA_RSCTMP"
> +score=0
> +query=0
> +
> +OPTIND=1
> +while getopts 'Qql:Dv:N:G' opt; do
> + case "$opt" in
> + Q|q)
> + # Noop
> + ;;
> + "l")
> + # Noop
> + ;;
> + "D")
> + score=0
> + ;;
> + "v")
> + score=$OPTARG
> + ;;
> + "N")
> + TMPDIR="$COLO_TEST_REMOTE_TMP"
> + ;;
> + "G")
> + query=1
> + ;;
> + esac
> +done
> +
> +if (( query )); then
> + cat "${TMPDIR}/master_score" || exit 1
> +else
> + echo $score > "${TMPDIR}/master_score" || exit 1
> +fi
> +
> +exit 0
> diff --git a/scripts/colo-resource-agent/crm_resource b/scripts/colo-resource-agent/crm_resource
> new file mode 100755
> index 0000000000..ad69ff3c6b
> --- /dev/null
> +++ b/scripts/colo-resource-agent/crm_resource
> @@ -0,0 +1,12 @@
> +#!/bin/sh
> +
> +# Fake crm_resource for COLO testing
> +#
> +# Copyright (c) Lukas Straub <lukasstraub2@web.de>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later. See the COPYING file in the top-level directory.
> +
> +# Noop
> +
> +exit 0
> diff --git a/tests/acceptance/colo.py b/tests/acceptance/colo.py
> new file mode 100644
> index 0000000000..465513fb6c
> --- /dev/null
> +++ b/tests/acceptance/colo.py
> @@ -0,0 +1,689 @@
> +# High-level test suite for qemu COLO testing all failover cases while checking
> +# guest network connectivity
> +#
> +# Copyright (c) Lukas Straub <lukasstraub2@web.de>
> +#
> +# This work is licensed under the terms of the GNU GPL, version 2 or
> +# later. See the COPYING file in the top-level directory.
> +
> +# HOWTO:
> +#
> +# This test has the following parameters:
> +# bridge_name: name of the bridge interface to connect qemu to
> +# host_address: ip address of the bridge interface
> +# guest_address: ip address that the guest gets from the dhcp server
> +# bridge_helper: path to the brige helper
> +# (default: /usr/lib/qemu/qemu-bridge-helper)
> +# install_cmd: command to run to install iperf3 and memtester in the guest
> +# (default: "sudo -n dnf -q -y install iperf3 memtester")
> +#
> +# To run the network tests, you have to specify the parameters.
> +#
> +# Example for running the colo tests:
> +# make check-acceptance FEDORA_31_ARCHES="x86_64" AVOCADO_TAGS="-t colo \
> +# -p bridge_name=br0 -p host_address=192.168.220.1 \
> +# -p guest_address=192.168.220.222"
> +#
> +# The colo tests currently only use x86_64 test vm images. With the
> +# FEDORA_31_ARCHES make variable as in the example, only the x86_64 images will
> +# be downloaded.
> +#
> +# If you're running the network tests as an unprivileged user, you need to set
> +# the suid bit on the bridge helper (chmod +s <bridge-helper>).
> +#
> +# The dhcp server should assign a static ip to the guest, else the test may be
> +# unreliable. The Mac address for the guest is always 52:54:00:12:34:56.
> +
> +
> +import select
> +import sys
> +import subprocess
> +import shutil
> +import os
> +import signal
> +import os.path
> +import time
> +import tempfile
> +
> +from avocado import Test
> +from avocado import skipUnless
> +from avocado.utils import network
> +from avocado.utils import vmimage
> +from avocado.utils import cloudinit
> +from avocado.utils import ssh
> +from avocado.utils.path import find_command
> +
> +from avocado_qemu import pick_default_qemu_bin, BUILD_DIR, SOURCE_DIR
> +from qemu.qmp import QEMUMonitorProtocol
> +
> +def iperf3_available():
> + try:
> + find_command("iperf3")
> + except CmdNotFoundError:
> + return False
> + return True
> +
> +class ColoTest(Test):
> +
> + # Constants
> + OCF_SUCCESS = 0
> + OCF_ERR_GENERIC = 1
> + OCF_ERR_ARGS = 2
> + OCF_ERR_UNIMPLEMENTED = 3
> + OCF_ERR_PERM = 4
> + OCF_ERR_INSTALLED = 5
> + OCF_ERR_CONFIGURED = 6
> + OCF_NOT_RUNNING = 7
> + OCF_RUNNING_MASTER = 8
> + OCF_FAILED_MASTER = 9
> +
> + HOSTA = 10
> + HOSTB = 11
> +
> + QEMU_OPTIONS = (" -display none -vga none -enable-kvm"
> + " -smp 2 -cpu host -m 768"
> + " -device e1000,mac=52:54:00:12:34:56,netdev=hn0"
> + " -device virtio-blk,drive=colo-disk0")
> +
> + FEDORA_VERSION = "31"
> + IMAGE_CHECKSUM = "e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0"
> + IMAGE_SIZE = "4294967296b"
> +
> + hang_qemu = False
> + checkpoint_failover = False
> + traffic_procs = []
> +
> + def get_image(self, temp_dir):
> + try:
> + return vmimage.get(
> + "fedora", arch="x86_64", version=self.FEDORA_VERSION,
> + checksum=self.IMAGE_CHECKSUM, algorithm="sha256",
> + cache_dir=self.cache_dirs[0],
> + snapshot_dir=temp_dir)
> + except:
> + self.cancel("Failed to download/prepare image")
> +
> + @skipUnless(ssh.SSH_CLIENT_BINARY, "No SSH client available")
> + def setUp(self):
> + # Qemu and qemu-img binary
> + default_qemu_bin = pick_default_qemu_bin()
> + self.QEMU_BINARY = self.params.get("qemu_bin", default=default_qemu_bin)
> +
> + # If qemu-img has been built, use it, otherwise the system wide one
> + # will be used. If none is available, the test will cancel.
> + qemu_img = os.path.join(BUILD_DIR, "qemu-img")
> + if not os.path.exists(qemu_img):
> + qemu_img = find_command("qemu-img", False)
> + if qemu_img is False:
> + self.cancel("Could not find \"qemu-img\", which is required to "
> + "create the bootable image")
> + self.QEMU_IMG_BINARY = qemu_img
Probably worth refactoring that as re-usable pick_qemuimg_bin() or
better named?
Similarly with BRIDGE_HELPER... We need a generic class to get binaries
from environment or build dir.
> + vmimage.QEMU_IMG = qemu_img
> +
> + self.RESOURCE_AGENT = os.path.join(SOURCE_DIR,
> + "scripts/colo-resource-agent/colo")
> + self.ADD_PATH = os.path.join(SOURCE_DIR, "scripts/colo-resource-agent")
> +
> + # Logs
> + self.RA_LOG = os.path.join(self.outputdir, "resource-agent.log")
> + self.HOSTA_LOGDIR = os.path.join(self.outputdir, "hosta")
> + self.HOSTB_LOGDIR = os.path.join(self.outputdir, "hostb")
> + os.makedirs(self.HOSTA_LOGDIR)
> + os.makedirs(self.HOSTB_LOGDIR)
> +
> + # Temporary directories
> + # We don't use self.workdir because of unix socket path length
> + # limitations
> + self.TMPDIR = tempfile.mkdtemp()
> + self.TMPA = os.path.join(self.TMPDIR, "hosta")
> + self.TMPB = os.path.join(self.TMPDIR, "hostb")
> + os.makedirs(self.TMPA)
> + os.makedirs(self.TMPB)
> +
> + # Network
> + self.BRIDGE_NAME = self.params.get("bridge_name")
> + if self.BRIDGE_NAME:
> + self.HOST_ADDRESS = self.params.get("host_address")
> + self.GUEST_ADDRESS = self.params.get("guest_address")
> + self.BRIDGE_HELPER = self.params.get("bridge_helper",
> + default="/usr/lib/qemu/qemu-bridge-helper")
> + self.SSH_PORT = 22
> + else:
> + # QEMU's hard coded usermode router address
> + self.HOST_ADDRESS = "10.0.2.2"
> + self.GUEST_ADDRESS = "127.0.0.1"
> + self.BRIDGE_HOSTA_PORT = network.find_free_port(address="127.0.0.1")
> + self.BRIDGE_HOSTB_PORT = network.find_free_port(address="127.0.0.1")
> + self.SSH_PORT = network.find_free_port(address="127.0.0.1")
> +
> + self.CLOUDINIT_HOME_PORT = network.find_free_port()
> +
> + # Find free port range
> + base_port = 1024
> + while True:
> + base_port = network.find_free_port(start_port=base_port,
> + address="127.0.0.1")
> + if base_port == None:
> + self.cancel("Failed to find a free port")
> + for n in range(base_port, base_port +4):
> + if n > 65535:
> + break
> + if not network.is_port_free(n, "127.0.0.1"):
> + break
> + else:
> + # for loop above didn't break
> + break
> +
> + self.BASE_PORT = base_port
> +
> + # Disk images
> + self.log.info("Downloading/preparing boot image")
> + self.HOSTA_IMAGE = self.get_image(self.TMPA).path
> + self.HOSTB_IMAGE = self.get_image(self.TMPB).path
> + self.CLOUDINIT_ISO = os.path.join(self.TMPDIR, "cloudinit.iso")
> +
> + self.log.info("Preparing cloudinit image")
> + try:
> + cloudinit.iso(self.CLOUDINIT_ISO, self.name,
> + username="test", password="password",
> + phone_home_host=self.HOST_ADDRESS,
> + phone_home_port=self.CLOUDINIT_HOME_PORT)
> + except Exception as e:
> + self.cancel("Failed to prepare cloudinit image")
> +
> + self.QEMU_OPTIONS += " -cdrom %s" % self.CLOUDINIT_ISO
> +
> + # Network bridge
> + if not self.BRIDGE_NAME:
> + self.BRIDGE_PIDFILE = os.path.join(self.TMPDIR, "bridge.pid")
> + self.run_command(("'%s' -pidfile '%s'"
> + " -M none -display none -daemonize"
> + " -netdev user,id=host,hostfwd=tcp:127.0.0.1:%s-:22"
> + " -netdev socket,id=hosta,listen=127.0.0.1:%s"
> + " -netdev socket,id=hostb,listen=127.0.0.1:%s"
> + " -netdev hubport,id=hostport,hubid=0,netdev=host"
> + " -netdev hubport,id=porta,hubid=0,netdev=hosta"
> + " -netdev hubport,id=portb,hubid=0,netdev=hostb")
> + % (self.QEMU_BINARY, self.BRIDGE_PIDFILE, self.SSH_PORT,
> + self.BRIDGE_HOSTA_PORT, self.BRIDGE_HOSTB_PORT), 0)
> +
> + def tearDown(self):
> + try:
> + pid = self.read_pidfile(self.BRIDGE_PIDFILE)
> + if pid and self.check_pid(pid):
> + os.kill(pid, signal.SIGKILL)
> + except Exception as e:
> + pass
> +
> + try:
> + self.ra_stop(self.HOSTA)
> + except Exception as e:
> + pass
> +
> + try:
> + self.ra_stop(self.HOSTB)
> + except Exception as e:
> + pass
> +
> + try:
> + self.ssh_close()
> + except Exception as e:
> + pass
> +
> + for proc in self.traffic_procs:
> + try:
> + os.killpg(proc.pid, signal.SIGTERM)
> + except Exception as e:
> + pass
> +
> + shutil.rmtree(self.TMPDIR)
> +
> + def run_command(self, cmdline, expected_status, env=None):
> + proc = subprocess.Popen(cmdline, shell=True, stdout=subprocess.PIPE,
> + stderr=subprocess.STDOUT,
> + universal_newlines=True, env=env)
> + stdout, stderr = proc.communicate()
> + if proc.returncode != expected_status:
> + self.fail("command \"%s\" failed with code %s:\n%s"
> + % (cmdline, proc.returncode, stdout))
> +
> + return proc.returncode
> +
> + def cat_line(self, path):
> + line=""
> + try:
> + fd = open(path, "r")
> + line = str.strip(fd.readline())
> + fd.close()
> + except:
> + pass
> + return line
> +
> + def read_pidfile(self, pidfile):
> + try:
> + pid = int(self.cat_line(pidfile))
> + except ValueError:
> + return None
> + else:
> + return pid
> +
> + def check_pid(self, pid):
> + try:
> + os.kill(pid, 0)
> + except OSError:
> + return False
> + else:
> + return True
> +
> + def ssh_open(self):
> + self.ssh_conn = ssh.Session(self.GUEST_ADDRESS, self.SSH_PORT,
> + user="test", password="password")
> + self.ssh_conn.DEFAULT_OPTIONS += (("UserKnownHostsFile", "/dev/null"),)
> + for i in range(10):
> + try:
> + if self.ssh_conn.connect():
> + return
> + except Exception as e:
> + pass
> + time.sleep(4)
> + self.fail("sshd timeout")
> +
> + def ssh_ping(self):
> + if self.ssh_conn.cmd("echo ping").stdout != b"ping\n":
> + self.fail("ssh ping failed")
> +
> + def ssh_close(self):
> + self.ssh_conn.quit()
> +
> + def setup_base_env(self, host):
> + PATH = os.getenv("PATH", "")
> + env = { "PATH": "%s:%s" % (self.ADD_PATH, PATH),
> + "HA_LOGFILE": self.RA_LOG,
> + "OCF_RESOURCE_INSTANCE": "colo-test",
> + "OCF_RESKEY_CRM_meta_clone_max": "2",
> + "OCF_RESKEY_CRM_meta_notify": "true",
> + "OCF_RESKEY_CRM_meta_timeout": "30000",
> + "OCF_RESKEY_qemu_binary": self.QEMU_BINARY,
> + "OCF_RESKEY_qemu_img_binary": self.QEMU_IMG_BINARY,
> + "OCF_RESKEY_disk_size": str(self.IMAGE_SIZE),
We can remove IMAGE_SIZE and use a runtime filesize(HOSTA_IMAGE) instead.
> + "OCF_RESKEY_checkpoint_interval": "10000",
> + "OCF_RESKEY_base_port": str(self.BASE_PORT),
> + "OCF_RESKEY_debug": "2"}
> +
> + if host == self.HOSTA:
> + env.update({"OCF_RESKEY_options":
> + ("%s -qmp unix:%s,server,nowait"
> + " -drive if=none,id=parent0,file='%s'")
> + % (self.QEMU_OPTIONS, self.get_qmp_sock(host),
> + self.HOSTA_IMAGE),
> + "OCF_RESKEY_active_hidden_dir": self.TMPA,
> + "OCF_RESKEY_listen_address": "127.0.0.1",
> + "OCF_RESKEY_log_dir": self.HOSTA_LOGDIR,
> + "OCF_RESKEY_CRM_meta_on_node": "127.0.0.1",
> + "HA_RSCTMP": self.TMPA,
> + "COLO_TEST_REMOTE_TMP": self.TMPB})
> +
> + else:
> + env.update({"OCF_RESKEY_options":
> + ("%s -qmp unix:%s,server,nowait"
> + " -drive if=none,id=parent0,file='%s'")
> + % (self.QEMU_OPTIONS, self.get_qmp_sock(host),
> + self.HOSTB_IMAGE),
> + "OCF_RESKEY_active_hidden_dir": self.TMPB,
> + "OCF_RESKEY_listen_address": "127.0.0.2",
> + "OCF_RESKEY_log_dir": self.HOSTB_LOGDIR,
> + "OCF_RESKEY_CRM_meta_on_node": "127.0.0.2",
> + "HA_RSCTMP": self.TMPB,
> + "COLO_TEST_REMOTE_TMP": self.TMPA})
> +
> + if self.BRIDGE_NAME:
> + env["OCF_RESKEY_options"] += \
> + " -netdev bridge,id=hn0,br=%s,helper='%s'" \
> + % (self.BRIDGE_NAME, self.BRIDGE_HELPER)
> + else:
> + if host == self.HOSTA:
> + env["OCF_RESKEY_options"] += \
> + " -netdev socket,id=hn0,connect=127.0.0.1:%s" \
> + % self.BRIDGE_HOSTA_PORT
> + else:
> + env["OCF_RESKEY_options"] += \
> + " -netdev socket,id=hn0,connect=127.0.0.1:%s" \
> + % self.BRIDGE_HOSTB_PORT
> + return env
> +
> + def ra_start(self, host):
> + env = self.setup_base_env(host)
> + self.run_command(self.RESOURCE_AGENT + " start", self.OCF_SUCCESS, env)
> +
> + def ra_stop(self, host):
> + env = self.setup_base_env(host)
> + self.run_command(self.RESOURCE_AGENT + " stop", self.OCF_SUCCESS, env)
> +
> + def ra_monitor(self, host, expected_status):
> + env = self.setup_base_env(host)
> + self.run_command(self.RESOURCE_AGENT + " monitor", expected_status, env)
> +
> + def ra_promote(self, host):
> + env = self.setup_base_env(host)
> + self.run_command(self.RESOURCE_AGENT + " promote", self.OCF_SUCCESS,env)
> +
> + def ra_notify_start(self, host):
> + env = self.setup_base_env(host)
> +
> + env.update({"OCF_RESKEY_CRM_meta_notify_type": "post",
> + "OCF_RESKEY_CRM_meta_notify_operation": "start"})
> +
> + if host == self.HOSTA:
> + env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.1",
> + "OCF_RESKEY_CRM_meta_notify_start_uname": "127.0.0.2"})
> + else:
> + env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.2",
> + "OCF_RESKEY_CRM_meta_notify_start_uname": "127.0.0.1"})
> +
> + self.run_command(self.RESOURCE_AGENT + " notify", self.OCF_SUCCESS, env)
> +
> + def ra_notify_stop(self, host):
> + env = self.setup_base_env(host)
> +
> + env.update({"OCF_RESKEY_CRM_meta_notify_type": "pre",
> + "OCF_RESKEY_CRM_meta_notify_operation": "stop"})
> +
> + if host == self.HOSTA:
> + env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.1",
> + "OCF_RESKEY_CRM_meta_notify_stop_uname": "127.0.0.2"})
> + else:
> + env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.2",
> + "OCF_RESKEY_CRM_meta_notify_stop_uname": "127.0.0.1"})
> +
> + self.run_command(self.RESOURCE_AGENT + " notify", self.OCF_SUCCESS, env)
> +
> + def get_pid(self, host):
> + if host == self.HOSTA:
> + return self.read_pidfile(os.path.join(self.TMPA,
> + "colo-test-qemu.pid"))
> + else:
> + return self.read_pidfile(os.path.join(self.TMPB,
> + "colo-test-qemu.pid"))
> +
> + def get_master_score(self, host):
> + if host == self.HOSTA:
> + return int(self.cat_line(os.path.join(self.TMPA, "master_score")))
> + else:
> + return int(self.cat_line(os.path.join(self.TMPB, "master_score")))
> +
> + def get_qmp_sock(self, host):
> + if host == self.HOSTA:
> + return os.path.join(self.TMPA, "my-qmp.sock")
> + else:
> + return os.path.join(self.TMPB, "my-qmp.sock")
> +
> +
> + def kill_qemu_pre(self, host):
> + pid = self.get_pid(host)
> +
> + qmp_sock = self.get_qmp_sock(host)
> +
> + if self.checkpoint_failover:
> + qmp_conn = QEMUMonitorProtocol(qmp_sock)
> + qmp_conn.settimeout(10)
> + qmp_conn.connect()
> + while True:
> + event = qmp_conn.pull_event(wait=True)
> + if event["event"] == "STOP":
> + break
> + qmp_conn.close()
> +
> +
> + if pid and self.check_pid(pid):
> + if self.hang_qemu:
> + os.kill(pid, signal.SIGSTOP)
> + else:
> + os.kill(pid, signal.SIGKILL)
> + while self.check_pid(pid):
> + time.sleep(1)
> +
> + def kill_qemu_post(self, host):
> + pid = self.get_pid(host)
> +
> + if self.hang_qemu and pid and self.check_pid(pid):
> + os.kill(pid, signal.SIGKILL)
> + while self.check_pid(pid):
> + time.sleep(1)
> +
> + def prepare_guest(self):
> + pass
> +
> + def cycle_start(self, cycle):
> + pass
> +
> + def active_section(self):
> + return False
> +
> + def cycle_end(self, cycle):
> + pass
> +
> + def check_connection(self):
> + self.ssh_ping()
> + for proc in self.traffic_procs:
> + if proc.poll() != None:
> + self.fail("Traffic process died")
> +
> + def _test_colo(self, loop=1):
> + loop = max(loop, 1)
> + self.log.info("Will put logs in %s" % self.outputdir)
> +
> + self.ra_stop(self.HOSTA)
> + self.ra_stop(self.HOSTB)
> +
> + self.log.info("*** Startup ***")
> + self.ra_start(self.HOSTA)
> + self.ra_start(self.HOSTB)
> +
> + self.ra_monitor(self.HOSTA, self.OCF_SUCCESS)
> + self.ra_monitor(self.HOSTB, self.OCF_SUCCESS)
> +
> + self.log.info("*** Promoting ***")
> + self.ra_promote(self.HOSTA)
> + cloudinit.wait_for_phone_home(("0.0.0.0", self.CLOUDINIT_HOME_PORT),
> + self.name)
> + self.ssh_open()
> + self.prepare_guest()
> +
> + self.ra_notify_start(self.HOSTA)
> +
> + while self.get_master_score(self.HOSTB) != 100:
> + self.ra_monitor(self.HOSTA, self.OCF_RUNNING_MASTER)
> + self.ra_monitor(self.HOSTB, self.OCF_SUCCESS)
> + time.sleep(1)
> +
> + self.log.info("*** Replication started ***")
> +
> + self.check_connection()
> +
> + primary = self.HOSTA
> + secondary = self.HOSTB
> +
> + for n in range(loop):
> + self.cycle_start(n)
> + self.log.info("*** Secondary failover ***")
> + self.kill_qemu_pre(primary)
> + self.ra_notify_stop(secondary)
> + self.ra_monitor(secondary, self.OCF_SUCCESS)
> + self.ra_promote(secondary)
> + self.ra_monitor(secondary, self.OCF_RUNNING_MASTER)
> + self.kill_qemu_post(primary)
> +
> + self.check_connection()
> +
> + tmp = primary
> + primary = secondary
> + secondary = tmp
> +
> + self.log.info("*** Secondary continue replication ***")
> + self.ra_start(secondary)
> + self.ra_notify_start(primary)
> +
> + self.check_connection()
> +
> + # Wait for resync
> + while self.get_master_score(secondary) != 100:
> + self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
> + self.ra_monitor(secondary, self.OCF_SUCCESS)
> + time.sleep(1)
> +
> + self.log.info("*** Replication started ***")
> +
> + self.check_connection()
> +
> + if self.active_section():
> + break
> +
> + self.log.info("*** Primary failover ***")
> + self.kill_qemu_pre(secondary)
> + self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
> + self.ra_notify_stop(primary)
> + self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
> + self.kill_qemu_post(secondary)
> +
> + self.check_connection()
> +
> + self.log.info("*** Primary continue replication ***")
> + self.ra_start(secondary)
> + self.ra_notify_start(primary)
> +
> + self.check_connection()
> +
> + # Wait for resync
> + while self.get_master_score(secondary) != 100:
> + self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
> + self.ra_monitor(secondary, self.OCF_SUCCESS)
> + time.sleep(1)
> +
> + self.log.info("*** Replication started ***")
> +
> + self.check_connection()
> +
> + self.cycle_end(n)
Interesting test :)
> +
> + self.ssh_close()
> +
> + self.ra_stop(self.HOSTA)
> + self.ra_stop(self.HOSTB)
> +
> + self.ra_monitor(self.HOSTA, self.OCF_NOT_RUNNING)
> + self.ra_monitor(self.HOSTB, self.OCF_NOT_RUNNING)
> + self.log.info("*** all ok ***")
> +
> +
> +class ColoQuickTest(ColoTest):
> + """
> + :avocado: tags=colo
> + :avocado: tags=quick
> + :avocado: tags=arch:x86_64
> + """
> +
> + timeout = 300
> +
> + def cycle_end(self, cycle):
> + self.log.info("Testing with peer qemu hanging"
> + " and failover during checkpoint")
> + self.hang_qemu = True
> +
> + def test_quick(self):
> + self.checkpoint_failover = True
> + self.log.info("Testing with peer qemu crashing"
> + " and failover during checkpoint")
> + self._test_colo(loop=2)
> +
> +
> +class ColoNetworkTest(ColoTest):
> +
> + def prepare_guest(self):
> + install_cmd = self.params.get("install_cmd", default=
> + "sudo -n dnf -q -y install iperf3 memtester")
> + self.ssh_conn.cmd(install_cmd)
> + # Use two instances to work around a bug in iperf3
> + self.ssh_conn.cmd("iperf3 -sD; iperf3 -sD -p 5202")
> +
> + def _cycle_start(self, cycle):
> + pass
> +
> + def cycle_start(self, cycle):
> + self._cycle_start(cycle)
> + tests = [("", "client-to-server tcp"), ("-R", "server-to-client tcp")]
> +
> + self.log.info("Testing iperf %s" % tests[cycle % 2][1])
> + iperf_cmd = "iperf3 %s -t 60 -i 60 --connect-timeout 30000 -c %s" \
> + % (tests[cycle % 2][0], self.GUEST_ADDRESS)
> + proc = subprocess.Popen("while %s && %s; do sleep 1; done >>'%s' 2>&1"
> + % (iperf_cmd, iperf_cmd + " -p 5202",
> + os.path.join(self.outputdir, "iperf.log")),
> + shell=True, preexec_fn=os.setsid, stdin=subprocess.DEVNULL,
> + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
So this run on the host, require the host to be Linux + iperf3
installed. Don't we need to be privileged to run it?
> + self.traffic_procs.append(proc)
> + time.sleep(5) # Wait for iperf to get up to speed
> +
> + def cycle_end(self, cycle):
> + for proc in self.traffic_procs:
> + try:
> + os.killpg(proc.pid, signal.SIGTERM)
> + proc.wait()
> + except Exception as e:
> + pass
> + self.traffic_procs.clear()
> + time.sleep(20)
> +
> +class ColoRealNetworkTest(ColoNetworkTest):
> + """
> + :avocado: tags=colo
> + :avocado: tags=slow
> + :avocado: tags=network_test
> + :avocado: tags=arch:x86_64
> + """
> +
> + timeout = 900
> +
> + def active_section(self):
> + time.sleep(300)
> + return False
> +
> + @skipUnless(iperf3_available(), "iperf3 not available")
> + def test_network(self):
> + if not self.BRIDGE_NAME:
> + self.cancel("bridge options not given, will skip network test")
> + self.log.info("Testing with peer qemu crashing and network load")
> + self._test_colo(loop=2)
> +
> +class ColoStressTest(ColoNetworkTest):
> + """
> + :avocado: tags=colo
> + :avocado: tags=slow
> + :avocado: tags=stress_test
> + :avocado: tags=arch:x86_64
> + """
> +
> + timeout = 1800
How long does this test take on your hw (what CPU, to compare)?
> +
> + def _cycle_start(self, cycle):
> + if cycle == 4:
> + self.log.info("Stresstest with peer qemu hanging, network load"
> + " and failover during checkpoint")
> + self.checkpoint_failover = True
> + self.hang_qemu = True
> + elif cycle == 8:
> + self.log.info("Stresstest with peer qemu crashing and network load")
> + self.checkpoint_failover = False
> + self.hang_qemu = False
> + elif cycle == 12:
> + self.log.info("Stresstest with peer qemu hanging and network load")
> + self.checkpoint_failover = False
> + self.hang_qemu = True
> +
> + @skipUnless(iperf3_available(), "iperf3 not available")
> + def test_stress(self):
> + if not self.BRIDGE_NAME:
> + self.cancel("bridge options not given, will skip network test")
> + self.log.info("Stresstest with peer qemu crashing, network load"
> + " and failover during checkpoint")
> + self.checkpoint_failover = True
> + self._test_colo(loop=16)
>
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 3/5] colo: Introduce high-level test suite
2020-06-02 12:19 ` Philippe Mathieu-Daudé
@ 2020-06-04 10:55 ` Lukas Straub
0 siblings, 0 replies; 14+ messages in thread
From: Lukas Straub @ 2020-06-04 10:55 UTC (permalink / raw)
To: Philippe Mathieu-Daudé
Cc: Alberto Garcia, qemu-devel, Wainer dos Santos Moschetta,
Dr. David Alan Gilbert, Zhang Chen, Cleber Rosa
[-- Attachment #1: Type: text/plain, Size: 32332 bytes --]
On Tue, 2 Jun 2020 14:19:08 +0200
Philippe Mathieu-Daudé <philmd@redhat.com> wrote:
> +Cleber/Wainer
>
> On 5/11/20 2:27 PM, Lukas Straub wrote:
> > Add high-level test relying on the colo resource-agent to test
> > all failover cases while checking guest network connectivity.
> >
> > Signed-off-by: Lukas Straub <lukasstraub2@web.de>
> > ---
> > scripts/colo-resource-agent/crm_master | 44 ++
> > scripts/colo-resource-agent/crm_resource | 12 +
> > tests/acceptance/colo.py | 689 +++++++++++++++++++++++
> > 3 files changed, 745 insertions(+)
> > create mode 100755 scripts/colo-resource-agent/crm_master
> > create mode 100755 scripts/colo-resource-agent/crm_resource
> > create mode 100644 tests/acceptance/colo.py
> >
> > diff --git a/scripts/colo-resource-agent/crm_master b/scripts/colo-resource-agent/crm_master
> > new file mode 100755
> > index 0000000000..886f523bda
> > --- /dev/null
> > +++ b/scripts/colo-resource-agent/crm_master
> > @@ -0,0 +1,44 @@
> > +#!/bin/bash
> > +
> > +# Fake crm_master for COLO testing
> > +#
> > +# Copyright (c) Lukas Straub <lukasstraub2@web.de>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2 or
> > +# later. See the COPYING file in the top-level directory.
> > +
> > +TMPDIR="$HA_RSCTMP"
> > +score=0
> > +query=0
> > +
> > +OPTIND=1
> > +while getopts 'Qql:Dv:N:G' opt; do
> > + case "$opt" in
> > + Q|q)
> > + # Noop
> > + ;;
> > + "l")
> > + # Noop
> > + ;;
> > + "D")
> > + score=0
> > + ;;
> > + "v")
> > + score=$OPTARG
> > + ;;
> > + "N")
> > + TMPDIR="$COLO_TEST_REMOTE_TMP"
> > + ;;
> > + "G")
> > + query=1
> > + ;;
> > + esac
> > +done
> > +
> > +if (( query )); then
> > + cat "${TMPDIR}/master_score" || exit 1
> > +else
> > + echo $score > "${TMPDIR}/master_score" || exit 1
> > +fi
> > +
> > +exit 0
> > diff --git a/scripts/colo-resource-agent/crm_resource b/scripts/colo-resource-agent/crm_resource
> > new file mode 100755
> > index 0000000000..ad69ff3c6b
> > --- /dev/null
> > +++ b/scripts/colo-resource-agent/crm_resource
> > @@ -0,0 +1,12 @@
> > +#!/bin/sh
> > +
> > +# Fake crm_resource for COLO testing
> > +#
> > +# Copyright (c) Lukas Straub <lukasstraub2@web.de>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2 or
> > +# later. See the COPYING file in the top-level directory.
> > +
> > +# Noop
> > +
> > +exit 0
> > diff --git a/tests/acceptance/colo.py b/tests/acceptance/colo.py
> > new file mode 100644
> > index 0000000000..465513fb6c
> > --- /dev/null
> > +++ b/tests/acceptance/colo.py
> > @@ -0,0 +1,689 @@
> > +# High-level test suite for qemu COLO testing all failover cases while checking
> > +# guest network connectivity
> > +#
> > +# Copyright (c) Lukas Straub <lukasstraub2@web.de>
> > +#
> > +# This work is licensed under the terms of the GNU GPL, version 2 or
> > +# later. See the COPYING file in the top-level directory.
> > +
> > +# HOWTO:
> > +#
> > +# This test has the following parameters:
> > +# bridge_name: name of the bridge interface to connect qemu to
> > +# host_address: ip address of the bridge interface
> > +# guest_address: ip address that the guest gets from the dhcp server
> > +# bridge_helper: path to the brige helper
> > +# (default: /usr/lib/qemu/qemu-bridge-helper)
> > +# install_cmd: command to run to install iperf3 and memtester in the guest
> > +# (default: "sudo -n dnf -q -y install iperf3 memtester")
> > +#
> > +# To run the network tests, you have to specify the parameters.
> > +#
> > +# Example for running the colo tests:
> > +# make check-acceptance FEDORA_31_ARCHES="x86_64" AVOCADO_TAGS="-t colo \
> > +# -p bridge_name=br0 -p host_address=192.168.220.1 \
> > +# -p guest_address=192.168.220.222"
> > +#
> > +# The colo tests currently only use x86_64 test vm images. With the
> > +# FEDORA_31_ARCHES make variable as in the example, only the x86_64 images will
> > +# be downloaded.
> > +#
> > +# If you're running the network tests as an unprivileged user, you need to set
> > +# the suid bit on the bridge helper (chmod +s <bridge-helper>).
> > +#
> > +# The dhcp server should assign a static ip to the guest, else the test may be
> > +# unreliable. The Mac address for the guest is always 52:54:00:12:34:56.
> > +
> > +
> > +import select
> > +import sys
> > +import subprocess
> > +import shutil
> > +import os
> > +import signal
> > +import os.path
> > +import time
> > +import tempfile
> > +
> > +from avocado import Test
> > +from avocado import skipUnless
> > +from avocado.utils import network
> > +from avocado.utils import vmimage
> > +from avocado.utils import cloudinit
> > +from avocado.utils import ssh
> > +from avocado.utils.path import find_command
> > +
> > +from avocado_qemu import pick_default_qemu_bin, BUILD_DIR, SOURCE_DIR
> > +from qemu.qmp import QEMUMonitorProtocol
> > +
> > +def iperf3_available():
> > + try:
> > + find_command("iperf3")
> > + except CmdNotFoundError:
> > + return False
> > + return True
> > +
> > +class ColoTest(Test):
> > +
> > + # Constants
> > + OCF_SUCCESS = 0
> > + OCF_ERR_GENERIC = 1
> > + OCF_ERR_ARGS = 2
> > + OCF_ERR_UNIMPLEMENTED = 3
> > + OCF_ERR_PERM = 4
> > + OCF_ERR_INSTALLED = 5
> > + OCF_ERR_CONFIGURED = 6
> > + OCF_NOT_RUNNING = 7
> > + OCF_RUNNING_MASTER = 8
> > + OCF_FAILED_MASTER = 9
> > +
> > + HOSTA = 10
> > + HOSTB = 11
> > +
> > + QEMU_OPTIONS = (" -display none -vga none -enable-kvm"
> > + " -smp 2 -cpu host -m 768"
> > + " -device e1000,mac=52:54:00:12:34:56,netdev=hn0"
> > + " -device virtio-blk,drive=colo-disk0")
> > +
> > + FEDORA_VERSION = "31"
> > + IMAGE_CHECKSUM = "e3c1b309d9203604922d6e255c2c5d098a309c2d46215d8fc026954f3c5c27a0"
> > + IMAGE_SIZE = "4294967296b"
> > +
> > + hang_qemu = False
> > + checkpoint_failover = False
> > + traffic_procs = []
> > +
> > + def get_image(self, temp_dir):
> > + try:
> > + return vmimage.get(
> > + "fedora", arch="x86_64", version=self.FEDORA_VERSION,
> > + checksum=self.IMAGE_CHECKSUM, algorithm="sha256",
> > + cache_dir=self.cache_dirs[0],
> > + snapshot_dir=temp_dir)
> > + except:
> > + self.cancel("Failed to download/prepare image")
> > +
> > + @skipUnless(ssh.SSH_CLIENT_BINARY, "No SSH client available")
> > + def setUp(self):
> > + # Qemu and qemu-img binary
> > + default_qemu_bin = pick_default_qemu_bin()
> > + self.QEMU_BINARY = self.params.get("qemu_bin", default=default_qemu_bin)
> > +
> > + # If qemu-img has been built, use it, otherwise the system wide one
> > + # will be used. If none is available, the test will cancel.
> > + qemu_img = os.path.join(BUILD_DIR, "qemu-img")
> > + if not os.path.exists(qemu_img):
> > + qemu_img = find_command("qemu-img", False)
> > + if qemu_img is False:
> > + self.cancel("Could not find \"qemu-img\", which is required to "
> > + "create the bootable image")
> > + self.QEMU_IMG_BINARY = qemu_img
>
> Probably worth refactoring that as re-usable pick_qemuimg_bin() or
> better named?
>
> Similarly with BRIDGE_HELPER... We need a generic class to get binaries
> from environment or build dir.
Agreed, I think a new function pick_qemu_util similar to this code in tests/acceptance/avocado_qemu/__init__.py should suffice.
> > + vmimage.QEMU_IMG = qemu_img
> > +
> > + self.RESOURCE_AGENT = os.path.join(SOURCE_DIR,
> > + "scripts/colo-resource-agent/colo")
> > + self.ADD_PATH = os.path.join(SOURCE_DIR, "scripts/colo-resource-agent")
> > +
> > + # Logs
> > + self.RA_LOG = os.path.join(self.outputdir, "resource-agent.log")
> > + self.HOSTA_LOGDIR = os.path.join(self.outputdir, "hosta")
> > + self.HOSTB_LOGDIR = os.path.join(self.outputdir, "hostb")
> > + os.makedirs(self.HOSTA_LOGDIR)
> > + os.makedirs(self.HOSTB_LOGDIR)
> > +
> > + # Temporary directories
> > + # We don't use self.workdir because of unix socket path length
> > + # limitations
> > + self.TMPDIR = tempfile.mkdtemp()
> > + self.TMPA = os.path.join(self.TMPDIR, "hosta")
> > + self.TMPB = os.path.join(self.TMPDIR, "hostb")
> > + os.makedirs(self.TMPA)
> > + os.makedirs(self.TMPB)
> > +
> > + # Network
> > + self.BRIDGE_NAME = self.params.get("bridge_name")
> > + if self.BRIDGE_NAME:
> > + self.HOST_ADDRESS = self.params.get("host_address")
> > + self.GUEST_ADDRESS = self.params.get("guest_address")
> > + self.BRIDGE_HELPER = self.params.get("bridge_helper",
> > + default="/usr/lib/qemu/qemu-bridge-helper")
> > + self.SSH_PORT = 22
> > + else:
> > + # QEMU's hard coded usermode router address
> > + self.HOST_ADDRESS = "10.0.2.2"
> > + self.GUEST_ADDRESS = "127.0.0.1"
> > + self.BRIDGE_HOSTA_PORT = network.find_free_port(address="127.0.0.1")
> > + self.BRIDGE_HOSTB_PORT = network.find_free_port(address="127.0.0.1")
> > + self.SSH_PORT = network.find_free_port(address="127.0.0.1")
> > +
> > + self.CLOUDINIT_HOME_PORT = network.find_free_port()
> > +
> > + # Find free port range
> > + base_port = 1024
> > + while True:
> > + base_port = network.find_free_port(start_port=base_port,
> > + address="127.0.0.1")
> > + if base_port == None:
> > + self.cancel("Failed to find a free port")
> > + for n in range(base_port, base_port +4):
> > + if n > 65535:
> > + break
> > + if not network.is_port_free(n, "127.0.0.1"):
> > + break
> > + else:
> > + # for loop above didn't break
> > + break
> > +
> > + self.BASE_PORT = base_port
> > +
> > + # Disk images
> > + self.log.info("Downloading/preparing boot image")
> > + self.HOSTA_IMAGE = self.get_image(self.TMPA).path
> > + self.HOSTB_IMAGE = self.get_image(self.TMPB).path
> > + self.CLOUDINIT_ISO = os.path.join(self.TMPDIR, "cloudinit.iso")
> > +
> > + self.log.info("Preparing cloudinit image")
> > + try:
> > + cloudinit.iso(self.CLOUDINIT_ISO, self.name,
> > + username="test", password="password",
> > + phone_home_host=self.HOST_ADDRESS,
> > + phone_home_port=self.CLOUDINIT_HOME_PORT)
> > + except Exception as e:
> > + self.cancel("Failed to prepare cloudinit image")
> > +
> > + self.QEMU_OPTIONS += " -cdrom %s" % self.CLOUDINIT_ISO
> > +
> > + # Network bridge
> > + if not self.BRIDGE_NAME:
> > + self.BRIDGE_PIDFILE = os.path.join(self.TMPDIR, "bridge.pid")
> > + self.run_command(("'%s' -pidfile '%s'"
> > + " -M none -display none -daemonize"
> > + " -netdev user,id=host,hostfwd=tcp:127.0.0.1:%s-:22"
> > + " -netdev socket,id=hosta,listen=127.0.0.1:%s"
> > + " -netdev socket,id=hostb,listen=127.0.0.1:%s"
> > + " -netdev hubport,id=hostport,hubid=0,netdev=host"
> > + " -netdev hubport,id=porta,hubid=0,netdev=hosta"
> > + " -netdev hubport,id=portb,hubid=0,netdev=hostb")
> > + % (self.QEMU_BINARY, self.BRIDGE_PIDFILE, self.SSH_PORT,
> > + self.BRIDGE_HOSTA_PORT, self.BRIDGE_HOSTB_PORT), 0)
> > +
> > + def tearDown(self):
> > + try:
> > + pid = self.read_pidfile(self.BRIDGE_PIDFILE)
> > + if pid and self.check_pid(pid):
> > + os.kill(pid, signal.SIGKILL)
> > + except Exception as e:
> > + pass
> > +
> > + try:
> > + self.ra_stop(self.HOSTA)
> > + except Exception as e:
> > + pass
> > +
> > + try:
> > + self.ra_stop(self.HOSTB)
> > + except Exception as e:
> > + pass
> > +
> > + try:
> > + self.ssh_close()
> > + except Exception as e:
> > + pass
> > +
> > + for proc in self.traffic_procs:
> > + try:
> > + os.killpg(proc.pid, signal.SIGTERM)
> > + except Exception as e:
> > + pass
> > +
> > + shutil.rmtree(self.TMPDIR)
> > +
> > + def run_command(self, cmdline, expected_status, env=None):
> > + proc = subprocess.Popen(cmdline, shell=True, stdout=subprocess.PIPE,
> > + stderr=subprocess.STDOUT,
> > + universal_newlines=True, env=env)
> > + stdout, stderr = proc.communicate()
> > + if proc.returncode != expected_status:
> > + self.fail("command \"%s\" failed with code %s:\n%s"
> > + % (cmdline, proc.returncode, stdout))
> > +
> > + return proc.returncode
> > +
> > + def cat_line(self, path):
> > + line=""
> > + try:
> > + fd = open(path, "r")
> > + line = str.strip(fd.readline())
> > + fd.close()
> > + except:
> > + pass
> > + return line
> > +
> > + def read_pidfile(self, pidfile):
> > + try:
> > + pid = int(self.cat_line(pidfile))
> > + except ValueError:
> > + return None
> > + else:
> > + return pid
> > +
> > + def check_pid(self, pid):
> > + try:
> > + os.kill(pid, 0)
> > + except OSError:
> > + return False
> > + else:
> > + return True
> > +
> > + def ssh_open(self):
> > + self.ssh_conn = ssh.Session(self.GUEST_ADDRESS, self.SSH_PORT,
> > + user="test", password="password")
> > + self.ssh_conn.DEFAULT_OPTIONS += (("UserKnownHostsFile", "/dev/null"),)
> > + for i in range(10):
> > + try:
> > + if self.ssh_conn.connect():
> > + return
> > + except Exception as e:
> > + pass
> > + time.sleep(4)
> > + self.fail("sshd timeout")
> > +
> > + def ssh_ping(self):
> > + if self.ssh_conn.cmd("echo ping").stdout != b"ping\n":
> > + self.fail("ssh ping failed")
> > +
> > + def ssh_close(self):
> > + self.ssh_conn.quit()
> > +
> > + def setup_base_env(self, host):
> > + PATH = os.getenv("PATH", "")
> > + env = { "PATH": "%s:%s" % (self.ADD_PATH, PATH),
> > + "HA_LOGFILE": self.RA_LOG,
> > + "OCF_RESOURCE_INSTANCE": "colo-test",
> > + "OCF_RESKEY_CRM_meta_clone_max": "2",
> > + "OCF_RESKEY_CRM_meta_notify": "true",
> > + "OCF_RESKEY_CRM_meta_timeout": "30000",
> > + "OCF_RESKEY_qemu_binary": self.QEMU_BINARY,
> > + "OCF_RESKEY_qemu_img_binary": self.QEMU_IMG_BINARY,
> > + "OCF_RESKEY_disk_size": str(self.IMAGE_SIZE),
>
> We can remove IMAGE_SIZE and use a runtime filesize(HOSTA_IMAGE) instead.
I will change the resource-agent so it doesn't need OCF_RESKEY_disk_size.
> > + "OCF_RESKEY_checkpoint_interval": "10000",
> > + "OCF_RESKEY_base_port": str(self.BASE_PORT),
> > + "OCF_RESKEY_debug": "2"}
> > +
> > + if host == self.HOSTA:
> > + env.update({"OCF_RESKEY_options":
> > + ("%s -qmp unix:%s,server,nowait"
> > + " -drive if=none,id=parent0,file='%s'")
> > + % (self.QEMU_OPTIONS, self.get_qmp_sock(host),
> > + self.HOSTA_IMAGE),
> > + "OCF_RESKEY_active_hidden_dir": self.TMPA,
> > + "OCF_RESKEY_listen_address": "127.0.0.1",
> > + "OCF_RESKEY_log_dir": self.HOSTA_LOGDIR,
> > + "OCF_RESKEY_CRM_meta_on_node": "127.0.0.1",
> > + "HA_RSCTMP": self.TMPA,
> > + "COLO_TEST_REMOTE_TMP": self.TMPB})
> > +
> > + else:
> > + env.update({"OCF_RESKEY_options":
> > + ("%s -qmp unix:%s,server,nowait"
> > + " -drive if=none,id=parent0,file='%s'")
> > + % (self.QEMU_OPTIONS, self.get_qmp_sock(host),
> > + self.HOSTB_IMAGE),
> > + "OCF_RESKEY_active_hidden_dir": self.TMPB,
> > + "OCF_RESKEY_listen_address": "127.0.0.2",
> > + "OCF_RESKEY_log_dir": self.HOSTB_LOGDIR,
> > + "OCF_RESKEY_CRM_meta_on_node": "127.0.0.2",
> > + "HA_RSCTMP": self.TMPB,
> > + "COLO_TEST_REMOTE_TMP": self.TMPA})
> > +
> > + if self.BRIDGE_NAME:
> > + env["OCF_RESKEY_options"] += \
> > + " -netdev bridge,id=hn0,br=%s,helper='%s'" \
> > + % (self.BRIDGE_NAME, self.BRIDGE_HELPER)
> > + else:
> > + if host == self.HOSTA:
> > + env["OCF_RESKEY_options"] += \
> > + " -netdev socket,id=hn0,connect=127.0.0.1:%s" \
> > + % self.BRIDGE_HOSTA_PORT
> > + else:
> > + env["OCF_RESKEY_options"] += \
> > + " -netdev socket,id=hn0,connect=127.0.0.1:%s" \
> > + % self.BRIDGE_HOSTB_PORT
> > + return env
> > +
> > + def ra_start(self, host):
> > + env = self.setup_base_env(host)
> > + self.run_command(self.RESOURCE_AGENT + " start", self.OCF_SUCCESS, env)
> > +
> > + def ra_stop(self, host):
> > + env = self.setup_base_env(host)
> > + self.run_command(self.RESOURCE_AGENT + " stop", self.OCF_SUCCESS, env)
> > +
> > + def ra_monitor(self, host, expected_status):
> > + env = self.setup_base_env(host)
> > + self.run_command(self.RESOURCE_AGENT + " monitor", expected_status, env)
> > +
> > + def ra_promote(self, host):
> > + env = self.setup_base_env(host)
> > + self.run_command(self.RESOURCE_AGENT + " promote", self.OCF_SUCCESS,env)
> > +
> > + def ra_notify_start(self, host):
> > + env = self.setup_base_env(host)
> > +
> > + env.update({"OCF_RESKEY_CRM_meta_notify_type": "post",
> > + "OCF_RESKEY_CRM_meta_notify_operation": "start"})
> > +
> > + if host == self.HOSTA:
> > + env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.1",
> > + "OCF_RESKEY_CRM_meta_notify_start_uname": "127.0.0.2"})
> > + else:
> > + env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.2",
> > + "OCF_RESKEY_CRM_meta_notify_start_uname": "127.0.0.1"})
> > +
> > + self.run_command(self.RESOURCE_AGENT + " notify", self.OCF_SUCCESS, env)
> > +
> > + def ra_notify_stop(self, host):
> > + env = self.setup_base_env(host)
> > +
> > + env.update({"OCF_RESKEY_CRM_meta_notify_type": "pre",
> > + "OCF_RESKEY_CRM_meta_notify_operation": "stop"})
> > +
> > + if host == self.HOSTA:
> > + env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.1",
> > + "OCF_RESKEY_CRM_meta_notify_stop_uname": "127.0.0.2"})
> > + else:
> > + env.update({"OCF_RESKEY_CRM_meta_notify_master_uname": "127.0.0.2",
> > + "OCF_RESKEY_CRM_meta_notify_stop_uname": "127.0.0.1"})
> > +
> > + self.run_command(self.RESOURCE_AGENT + " notify", self.OCF_SUCCESS, env)
> > +
> > + def get_pid(self, host):
> > + if host == self.HOSTA:
> > + return self.read_pidfile(os.path.join(self.TMPA,
> > + "colo-test-qemu.pid"))
> > + else:
> > + return self.read_pidfile(os.path.join(self.TMPB,
> > + "colo-test-qemu.pid"))
> > +
> > + def get_master_score(self, host):
> > + if host == self.HOSTA:
> > + return int(self.cat_line(os.path.join(self.TMPA, "master_score")))
> > + else:
> > + return int(self.cat_line(os.path.join(self.TMPB, "master_score")))
> > +
> > + def get_qmp_sock(self, host):
> > + if host == self.HOSTA:
> > + return os.path.join(self.TMPA, "my-qmp.sock")
> > + else:
> > + return os.path.join(self.TMPB, "my-qmp.sock")
> > +
> > +
> > + def kill_qemu_pre(self, host):
> > + pid = self.get_pid(host)
> > +
> > + qmp_sock = self.get_qmp_sock(host)
> > +
> > + if self.checkpoint_failover:
> > + qmp_conn = QEMUMonitorProtocol(qmp_sock)
> > + qmp_conn.settimeout(10)
> > + qmp_conn.connect()
> > + while True:
> > + event = qmp_conn.pull_event(wait=True)
> > + if event["event"] == "STOP":
> > + break
> > + qmp_conn.close()
> > +
> > +
> > + if pid and self.check_pid(pid):
> > + if self.hang_qemu:
> > + os.kill(pid, signal.SIGSTOP)
> > + else:
> > + os.kill(pid, signal.SIGKILL)
> > + while self.check_pid(pid):
> > + time.sleep(1)
> > +
> > + def kill_qemu_post(self, host):
> > + pid = self.get_pid(host)
> > +
> > + if self.hang_qemu and pid and self.check_pid(pid):
> > + os.kill(pid, signal.SIGKILL)
> > + while self.check_pid(pid):
> > + time.sleep(1)
> > +
> > + def prepare_guest(self):
> > + pass
> > +
> > + def cycle_start(self, cycle):
> > + pass
> > +
> > + def active_section(self):
> > + return False
> > +
> > + def cycle_end(self, cycle):
> > + pass
> > +
> > + def check_connection(self):
> > + self.ssh_ping()
> > + for proc in self.traffic_procs:
> > + if proc.poll() != None:
> > + self.fail("Traffic process died")
> > +
> > + def _test_colo(self, loop=1):
> > + loop = max(loop, 1)
> > + self.log.info("Will put logs in %s" % self.outputdir)
> > +
> > + self.ra_stop(self.HOSTA)
> > + self.ra_stop(self.HOSTB)
> > +
> > + self.log.info("*** Startup ***")
> > + self.ra_start(self.HOSTA)
> > + self.ra_start(self.HOSTB)
> > +
> > + self.ra_monitor(self.HOSTA, self.OCF_SUCCESS)
> > + self.ra_monitor(self.HOSTB, self.OCF_SUCCESS)
> > +
> > + self.log.info("*** Promoting ***")
> > + self.ra_promote(self.HOSTA)
> > + cloudinit.wait_for_phone_home(("0.0.0.0", self.CLOUDINIT_HOME_PORT),
> > + self.name)
> > + self.ssh_open()
> > + self.prepare_guest()
> > +
> > + self.ra_notify_start(self.HOSTA)
> > +
> > + while self.get_master_score(self.HOSTB) != 100:
> > + self.ra_monitor(self.HOSTA, self.OCF_RUNNING_MASTER)
> > + self.ra_monitor(self.HOSTB, self.OCF_SUCCESS)
> > + time.sleep(1)
> > +
> > + self.log.info("*** Replication started ***")
> > +
> > + self.check_connection()
> > +
> > + primary = self.HOSTA
> > + secondary = self.HOSTB
> > +
> > + for n in range(loop):
> > + self.cycle_start(n)
> > + self.log.info("*** Secondary failover ***")
> > + self.kill_qemu_pre(primary)
> > + self.ra_notify_stop(secondary)
> > + self.ra_monitor(secondary, self.OCF_SUCCESS)
> > + self.ra_promote(secondary)
> > + self.ra_monitor(secondary, self.OCF_RUNNING_MASTER)
> > + self.kill_qemu_post(primary)
> > +
> > + self.check_connection()
> > +
> > + tmp = primary
> > + primary = secondary
> > + secondary = tmp
> > +
> > + self.log.info("*** Secondary continue replication ***")
> > + self.ra_start(secondary)
> > + self.ra_notify_start(primary)
> > +
> > + self.check_connection()
> > +
> > + # Wait for resync
> > + while self.get_master_score(secondary) != 100:
> > + self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
> > + self.ra_monitor(secondary, self.OCF_SUCCESS)
> > + time.sleep(1)
> > +
> > + self.log.info("*** Replication started ***")
> > +
> > + self.check_connection()
> > +
> > + if self.active_section():
> > + break
> > +
> > + self.log.info("*** Primary failover ***")
> > + self.kill_qemu_pre(secondary)
> > + self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
> > + self.ra_notify_stop(primary)
> > + self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
> > + self.kill_qemu_post(secondary)
> > +
> > + self.check_connection()
> > +
> > + self.log.info("*** Primary continue replication ***")
> > + self.ra_start(secondary)
> > + self.ra_notify_start(primary)
> > +
> > + self.check_connection()
> > +
> > + # Wait for resync
> > + while self.get_master_score(secondary) != 100:
> > + self.ra_monitor(primary, self.OCF_RUNNING_MASTER)
> > + self.ra_monitor(secondary, self.OCF_SUCCESS)
> > + time.sleep(1)
> > +
> > + self.log.info("*** Replication started ***")
> > +
> > + self.check_connection()
> > +
> > + self.cycle_end(n)
>
> Interesting test :)
>
> > +
> > + self.ssh_close()
> > +
> > + self.ra_stop(self.HOSTA)
> > + self.ra_stop(self.HOSTB)
> > +
> > + self.ra_monitor(self.HOSTA, self.OCF_NOT_RUNNING)
> > + self.ra_monitor(self.HOSTB, self.OCF_NOT_RUNNING)
> > + self.log.info("*** all ok ***")
> > +
> > +
> > +class ColoQuickTest(ColoTest):
> > + """
> > + :avocado: tags=colo
> > + :avocado: tags=quick
> > + :avocado: tags=arch:x86_64
> > + """
> > +
> > + timeout = 300
> > +
> > + def cycle_end(self, cycle):
> > + self.log.info("Testing with peer qemu hanging"
> > + " and failover during checkpoint")
> > + self.hang_qemu = True
> > +
> > + def test_quick(self):
> > + self.checkpoint_failover = True
> > + self.log.info("Testing with peer qemu crashing"
> > + " and failover during checkpoint")
> > + self._test_colo(loop=2)
> > +
> > +
> > +class ColoNetworkTest(ColoTest):
> > +
> > + def prepare_guest(self):
> > + install_cmd = self.params.get("install_cmd", default=
> > + "sudo -n dnf -q -y install iperf3 memtester")
> > + self.ssh_conn.cmd(install_cmd)
> > + # Use two instances to work around a bug in iperf3
> > + self.ssh_conn.cmd("iperf3 -sD; iperf3 -sD -p 5202")
> > +
> > + def _cycle_start(self, cycle):
> > + pass
> > +
> > + def cycle_start(self, cycle):
> > + self._cycle_start(cycle)
> > + tests = [("", "client-to-server tcp"), ("-R", "server-to-client tcp")]
> > +
> > + self.log.info("Testing iperf %s" % tests[cycle % 2][1])
> > + iperf_cmd = "iperf3 %s -t 60 -i 60 --connect-timeout 30000 -c %s" \
> > + % (tests[cycle % 2][0], self.GUEST_ADDRESS)
> > + proc = subprocess.Popen("while %s && %s; do sleep 1; done >>'%s' 2>&1"
> > + % (iperf_cmd, iperf_cmd + " -p 5202",
> > + os.path.join(self.outputdir, "iperf.log")),
> > + shell=True, preexec_fn=os.setsid, stdin=subprocess.DEVNULL,
> > + stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
>
> So this run on the host, require the host to be Linux + iperf3
> installed. Don't we need to be privileged to run it?
Not for iperf3, but the bridge helper needs the setuid bit to be set if the test runs unprivileged.
> > + self.traffic_procs.append(proc)
> > + time.sleep(5) # Wait for iperf to get up to speed
> > +
> > + def cycle_end(self, cycle):
> > + for proc in self.traffic_procs:
> > + try:
> > + os.killpg(proc.pid, signal.SIGTERM)
> > + proc.wait()
> > + except Exception as e:
> > + pass
> > + self.traffic_procs.clear()
> > + time.sleep(20)
> > +
> > +class ColoRealNetworkTest(ColoNetworkTest):
> > + """
> > + :avocado: tags=colo
> > + :avocado: tags=slow
> > + :avocado: tags=network_test
> > + :avocado: tags=arch:x86_64
> > + """
> > +
> > + timeout = 900
> > +
> > + def active_section(self):
> > + time.sleep(300)
> > + return False
> > +
> > + @skipUnless(iperf3_available(), "iperf3 not available")
> > + def test_network(self):
> > + if not self.BRIDGE_NAME:
> > + self.cancel("bridge options not given, will skip network test")
> > + self.log.info("Testing with peer qemu crashing and network load")
> > + self._test_colo(loop=2)
> > +
> > +class ColoStressTest(ColoNetworkTest):
> > + """
> > + :avocado: tags=colo
> > + :avocado: tags=slow
> > + :avocado: tags=stress_test
> > + :avocado: tags=arch:x86_64
> > + """
> > +
> > + timeout = 1800
>
> How long does this test take on your hw (what CPU, to compare)?
The CPU is an Intel i7-5600U and M.2 SATA SSD (during resync the whole image is copied), here are the runtimes:
Quick test: ~200s
Network test: ~800s
Stress test: ~1300s
> > +
> > + def _cycle_start(self, cycle):
> > + if cycle == 4:
> > + self.log.info("Stresstest with peer qemu hanging, network load"
> > + " and failover during checkpoint")
> > + self.checkpoint_failover = True
> > + self.hang_qemu = True
> > + elif cycle == 8:
> > + self.log.info("Stresstest with peer qemu crashing and network load")
> > + self.checkpoint_failover = False
> > + self.hang_qemu = False
> > + elif cycle == 12:
> > + self.log.info("Stresstest with peer qemu hanging and network load")
> > + self.checkpoint_failover = False
> > + self.hang_qemu = True
> > +
> > + @skipUnless(iperf3_available(), "iperf3 not available")
> > + def test_stress(self):
> > + if not self.BRIDGE_NAME:
> > + self.cancel("bridge options not given, will skip network test")
> > + self.log.info("Stresstest with peer qemu crashing, network load"
> > + " and failover during checkpoint")
> > + self.checkpoint_failover = True
> > + self._test_colo(loop=16)
> >
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/5] colo: Introduce resource agent and test suite/CI
2020-05-18 9:38 ` [PATCH 0/5] colo: Introduce resource agent and test suite/CI Zhang, Chen
@ 2020-06-06 18:59 ` Lukas Straub
2020-06-16 1:42 ` Zhang, Chen
0 siblings, 1 reply; 14+ messages in thread
From: Lukas Straub @ 2020-06-06 18:59 UTC (permalink / raw)
To: Zhang, Chen
Cc: Jason Wang, Alberto Garcia, qemu-devel, Dr. David Alan Gilbert
[-- Attachment #1: Type: text/plain, Size: 3535 bytes --]
On Mon, 18 May 2020 09:38:24 +0000
"Zhang, Chen" <chen.zhang@intel.com> wrote:
> > -----Original Message-----
> > From: Lukas Straub <lukasstraub2@web.de>
> > Sent: Monday, May 11, 2020 8:27 PM
> > To: qemu-devel <qemu-devel@nongnu.org>
> > Cc: Alberto Garcia <berto@igalia.com>; Dr. David Alan Gilbert
> > <dgilbert@redhat.com>; Zhang, Chen <chen.zhang@intel.com>
> > Subject: [PATCH 0/5] colo: Introduce resource agent and test suite/CI
> >
> > Hello Everyone,
> > These patches introduce a resource agent for fully automatic management of
> > colo and a test suite building upon the resource agent to extensively test colo.
> >
> > Test suite features:
> > -Tests failover with peer crashing and hanging and failover during checkpoint
> > -Tests network using ssh and iperf3 -Quick test requires no special
> > configuration -Network test for testing colo-compare -Stress test: failover all
> > the time with network load
> >
> > Resource agent features:
> > -Fully automatic management of colo
> > -Handles many failures: hanging/crashing qemu, replication error, disk
> > error, ...
> > -Recovers from hanging qemu by using the "yank" oob command -Tracks
> > which node has up-to-date data -Works well in clusters with more than 2
> > nodes
> >
> > Run times on my laptop:
> > Quick test: 200s
> > Network test: 800s (tagged as slow)
> > Stress test: 1300s (tagged as slow)
> >
> > The test suite needs access to a network bridge to properly test the network,
> > so some parameters need to be given to the test run. See
> > tests/acceptance/colo.py for more information.
> >
> > I wonder how this integrates in existing CI infrastructure. Is there a common
> > CI for qemu where this can run or does every subsystem have to run their
> > own CI?
>
> Wow~ Very happy to see this series.
> I have checked the "how to" in tests/acceptance/colo.py,
> But it looks not enough for users, can you write an independent document for this series?
> Include test Infrastructure ASC II diagram, test cases design , detailed how to and more information for
> pacemaker cluster and resource agent..etc ?
Hi,
I quickly created a more complete howto for configuring a pacemaker cluster and using the resource agent, I hope it helps:
https://wiki.qemu.org/Features/COLO/Managed_HOWTO
Regards,
Lukas Straub
> Thanks
> Zhang Chen
>
>
> >
> > Regards,
> > Lukas Straub
> >
> >
> > Lukas Straub (5):
> > block/quorum.c: stable children names
> > colo: Introduce resource agent
> > colo: Introduce high-level test suite
> > configure,Makefile: Install colo resource-agent
> > MAINTAINERS: Add myself as maintainer for COLO resource agent
> >
> > MAINTAINERS | 6 +
> > Makefile | 5 +
> > block/quorum.c | 20 +-
> > configure | 10 +
> > scripts/colo-resource-agent/colo | 1429 ++++++++++++++++++++++
> > scripts/colo-resource-agent/crm_master | 44 +
> > scripts/colo-resource-agent/crm_resource | 12 +
> > tests/acceptance/colo.py | 689 +++++++++++
> > 8 files changed, 2209 insertions(+), 6 deletions(-) create mode 100755
> > scripts/colo-resource-agent/colo create mode 100755 scripts/colo-resource-
> > agent/crm_master
> > create mode 100755 scripts/colo-resource-agent/crm_resource
> > create mode 100644 tests/acceptance/colo.py
> >
> > --
> > 2.20.1
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
* RE: [PATCH 0/5] colo: Introduce resource agent and test suite/CI
2020-06-06 18:59 ` Lukas Straub
@ 2020-06-16 1:42 ` Zhang, Chen
2020-06-19 13:55 ` Lukas Straub
0 siblings, 1 reply; 14+ messages in thread
From: Zhang, Chen @ 2020-06-16 1:42 UTC (permalink / raw)
To: Lukas Straub
Cc: Zhanghailiang, Jason Wang, Alberto Garcia, qemu-devel,
Dr. David Alan Gilbert
> -----Original Message-----
> From: Lukas Straub <lukasstraub2@web.de>
> Sent: Sunday, June 7, 2020 3:00 AM
> To: Zhang, Chen <chen.zhang@intel.com>
> Cc: qemu-devel <qemu-devel@nongnu.org>; Alberto Garcia
> <berto@igalia.com>; Dr. David Alan Gilbert <dgilbert@redhat.com>; Jason
> Wang <jasowang@redhat.com>
> Subject: Re: [PATCH 0/5] colo: Introduce resource agent and test suite/CI
>
> On Mon, 18 May 2020 09:38:24 +0000
> "Zhang, Chen" <chen.zhang@intel.com> wrote:
>
> > > -----Original Message-----
> > > From: Lukas Straub <lukasstraub2@web.de>
> > > Sent: Monday, May 11, 2020 8:27 PM
> > > To: qemu-devel <qemu-devel@nongnu.org>
> > > Cc: Alberto Garcia <berto@igalia.com>; Dr. David Alan Gilbert
> > > <dgilbert@redhat.com>; Zhang, Chen <chen.zhang@intel.com>
> > > Subject: [PATCH 0/5] colo: Introduce resource agent and test
> > > suite/CI
> > >
> > > Hello Everyone,
> > > These patches introduce a resource agent for fully automatic
> > > management of colo and a test suite building upon the resource agent to
> extensively test colo.
> > >
> > > Test suite features:
> > > -Tests failover with peer crashing and hanging and failover during
> > > checkpoint -Tests network using ssh and iperf3 -Quick test requires
> > > no special configuration -Network test for testing colo-compare
> > > -Stress test: failover all the time with network load
> > >
> > > Resource agent features:
> > > -Fully automatic management of colo
> > > -Handles many failures: hanging/crashing qemu, replication error,
> > > disk error, ...
> > > -Recovers from hanging qemu by using the "yank" oob command -Tracks
> > > which node has up-to-date data -Works well in clusters with more
> > > than 2 nodes
> > >
> > > Run times on my laptop:
> > > Quick test: 200s
> > > Network test: 800s (tagged as slow)
> > > Stress test: 1300s (tagged as slow)
> > >
> > > The test suite needs access to a network bridge to properly test the
> > > network, so some parameters need to be given to the test run. See
> > > tests/acceptance/colo.py for more information.
> > >
> > > I wonder how this integrates in existing CI infrastructure. Is there
> > > a common CI for qemu where this can run or does every subsystem have
> > > to run their own CI?
> >
> > Wow~ Very happy to see this series.
> > I have checked the "how to" in tests/acceptance/colo.py, But it looks
> > not enough for users, can you write an independent document for this
> series?
> > Include test Infrastructure ASC II diagram, test cases design ,
> > detailed how to and more information for pacemaker cluster and resource
> agent..etc ?
>
> Hi,
> I quickly created a more complete howto for configuring a pacemaker cluster
> and using the resource agent, I hope it helps:
> https://wiki.qemu.org/Features/COLO/Managed_HOWTO
Hi Lukas,
I noticed you contribute some content in Qemu COLO WIKI.
For the Features/COLO/Manual HOWTO
https://wiki.qemu.org/Features/COLO/Manual_HOWTO
Why not keep the Secondary side start command same with the qemu/docs/COLO-FT.txt?
If I understand correctly, add the quorum related command in secondary will support resume replication.
Then, we can add primary/secondary resume step here.
Thanks
Zhang Chen
>
> Regards,
> Lukas Straub
>
> > Thanks
> > Zhang Chen
> >
> >
> > >
> > > Regards,
> > > Lukas Straub
> > >
> > >
> > > Lukas Straub (5):
> > > block/quorum.c: stable children names
> > > colo: Introduce resource agent
> > > colo: Introduce high-level test suite
> > > configure,Makefile: Install colo resource-agent
> > > MAINTAINERS: Add myself as maintainer for COLO resource agent
> > >
> > > MAINTAINERS | 6 +
> > > Makefile | 5 +
> > > block/quorum.c | 20 +-
> > > configure | 10 +
> > > scripts/colo-resource-agent/colo | 1429 ++++++++++++++++++++++
> > > scripts/colo-resource-agent/crm_master | 44 +
> > > scripts/colo-resource-agent/crm_resource | 12 +
> > > tests/acceptance/colo.py | 689 +++++++++++
> > > 8 files changed, 2209 insertions(+), 6 deletions(-) create mode
> > > 100755 scripts/colo-resource-agent/colo create mode 100755
> > > scripts/colo-resource- agent/crm_master create mode 100755
> > > scripts/colo-resource-agent/crm_resource
> > > create mode 100644 tests/acceptance/colo.py
> > >
> > > --
> > > 2.20.1
^ permalink raw reply [flat|nested] 14+ messages in thread
* Re: [PATCH 0/5] colo: Introduce resource agent and test suite/CI
2020-06-16 1:42 ` Zhang, Chen
@ 2020-06-19 13:55 ` Lukas Straub
0 siblings, 0 replies; 14+ messages in thread
From: Lukas Straub @ 2020-06-19 13:55 UTC (permalink / raw)
To: Zhang, Chen
Cc: Zhanghailiang, Jason Wang, Alberto Garcia, qemu-devel,
Dr. David Alan Gilbert
[-- Attachment #1: Type: text/plain, Size: 3657 bytes --]
On Tue, 16 Jun 2020 01:42:45 +0000
"Zhang, Chen" <chen.zhang@intel.com> wrote:
> > -----Original Message-----
> > From: Lukas Straub <lukasstraub2@web.de>
> > Sent: Sunday, June 7, 2020 3:00 AM
> > To: Zhang, Chen <chen.zhang@intel.com>
> > Cc: qemu-devel <qemu-devel@nongnu.org>; Alberto Garcia
> > <berto@igalia.com>; Dr. David Alan Gilbert <dgilbert@redhat.com>; Jason
> > Wang <jasowang@redhat.com>
> > Subject: Re: [PATCH 0/5] colo: Introduce resource agent and test suite/CI
> >
> > On Mon, 18 May 2020 09:38:24 +0000
> > "Zhang, Chen" <chen.zhang@intel.com> wrote:
> >
> > > > -----Original Message-----
> > > > From: Lukas Straub <lukasstraub2@web.de>
> > > > Sent: Monday, May 11, 2020 8:27 PM
> > > > To: qemu-devel <qemu-devel@nongnu.org>
> > > > Cc: Alberto Garcia <berto@igalia.com>; Dr. David Alan Gilbert
> > > > <dgilbert@redhat.com>; Zhang, Chen <chen.zhang@intel.com>
> > > > Subject: [PATCH 0/5] colo: Introduce resource agent and test
> > > > suite/CI
> > > >
> > > > Hello Everyone,
> > > > These patches introduce a resource agent for fully automatic
> > > > management of colo and a test suite building upon the resource agent to
> > extensively test colo.
> > > >
> > > > Test suite features:
> > > > -Tests failover with peer crashing and hanging and failover during
> > > > checkpoint -Tests network using ssh and iperf3 -Quick test requires
> > > > no special configuration -Network test for testing colo-compare
> > > > -Stress test: failover all the time with network load
> > > >
> > > > Resource agent features:
> > > > -Fully automatic management of colo
> > > > -Handles many failures: hanging/crashing qemu, replication error,
> > > > disk error, ...
> > > > -Recovers from hanging qemu by using the "yank" oob command -Tracks
> > > > which node has up-to-date data -Works well in clusters with more
> > > > than 2 nodes
> > > >
> > > > Run times on my laptop:
> > > > Quick test: 200s
> > > > Network test: 800s (tagged as slow)
> > > > Stress test: 1300s (tagged as slow)
> > > >
> > > > The test suite needs access to a network bridge to properly test the
> > > > network, so some parameters need to be given to the test run. See
> > > > tests/acceptance/colo.py for more information.
> > > >
> > > > I wonder how this integrates in existing CI infrastructure. Is there
> > > > a common CI for qemu where this can run or does every subsystem have
> > > > to run their own CI?
> > >
> > > Wow~ Very happy to see this series.
> > > I have checked the "how to" in tests/acceptance/colo.py, But it looks
> > > not enough for users, can you write an independent document for this
> > series?
> > > Include test Infrastructure ASC II diagram, test cases design ,
> > > detailed how to and more information for pacemaker cluster and resource
> > agent..etc ?
> >
> > Hi,
> > I quickly created a more complete howto for configuring a pacemaker cluster
> > and using the resource agent, I hope it helps:
> > https://wiki.qemu.org/Features/COLO/Managed_HOWTO
>
> Hi Lukas,
>
> I noticed you contribute some content in Qemu COLO WIKI.
> For the Features/COLO/Manual HOWTO
> https://wiki.qemu.org/Features/COLO/Manual_HOWTO
>
> Why not keep the Secondary side start command same with the qemu/docs/COLO-FT.txt?
> If I understand correctly, add the quorum related command in secondary will support resume replication.
> Then, we can add primary/secondary resume step here.
I haven't updated the wiki from qemu/docs/COLO-FT.txt yet, I just moved it there from the main page.
Regards,
Lukas Straub
> Thanks
> Zhang Chen
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]
^ permalink raw reply [flat|nested] 14+ messages in thread
end of thread, other threads:[~2020-06-19 14:00 UTC | newest]
Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-05-11 12:26 [PATCH 0/5] colo: Introduce resource agent and test suite/CI Lukas Straub
2020-05-11 12:26 ` [PATCH 1/5] block/quorum.c: stable children names Lukas Straub
2020-06-02 1:01 ` Zhang, Chen
2020-06-02 11:07 ` Alberto Garcia
2020-05-11 12:26 ` [PATCH 2/5] colo: Introduce resource agent Lukas Straub
2020-05-11 12:27 ` [PATCH 3/5] colo: Introduce high-level test suite Lukas Straub
2020-06-02 12:19 ` Philippe Mathieu-Daudé
2020-06-04 10:55 ` Lukas Straub
2020-05-11 12:27 ` [PATCH 4/5] configure,Makefile: Install colo resource-agent Lukas Straub
2020-05-11 12:27 ` [PATCH 5/5] MAINTAINERS: Add myself as maintainer for COLO resource agent Lukas Straub
2020-05-18 9:38 ` [PATCH 0/5] colo: Introduce resource agent and test suite/CI Zhang, Chen
2020-06-06 18:59 ` Lukas Straub
2020-06-16 1:42 ` Zhang, Chen
2020-06-19 13:55 ` Lukas Straub
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.